This paper addresses the challenge of geometrically infeasible pose hypotheses in object pose estimation, particularly for dexterous manipulation scenarios. The authors propose a multi-modal approach combining differentiable physics simulation, differentiable rendering, and visuo-tactile sensing to refine object poses while ensuring physical consistency. The method significantly reduces intersection volume errors between objects and robotic hands by 73% under accurate initial estimates and over 87% under high uncertainty, outperforming ICP-based approaches. By integrating physical constraints into pose optimization, this work enables robots to achieve more reliable manipulation by ensuring estimated object poses respect both geometric accuracy and physical reality.