This paper addresses the challenging problem of in-hand, contact-rich, and long-horizon dexterous robot manipulation by proposing drumming as a comprehensive testbed. The DexDrummer framework employs a hierarchical object-centric bimanual policy that combines trajectory planning with residual reinforcement learning corrections, enabling effective sim-to-real transfer. The approach specifically targets dexterous manipulation skills including in-hand control for drumstick stabilization, contact-rich striking interactions, and long-horizon rhythmic coordination across multiple drums. By integrating these three challenging aspects into a single complex task, this work advances the field of robotic dexterity and manipulation planning.