This paper presents KLDrive, a novel knowledge-graph-augmented large language model reasoning framework specifically designed for fine-grained question answering in autonomous driving scenarios. The framework addresses critical challenges in autonomous driving perception by consolidating multi-source evidence through an energy-based scene fact construction module that builds reliable scene knowledge graphs. A specialized LLM agent performs fact-grounded reasoning over constrained action spaces using explicit structural constraints, combining structured prompting with few-shot in-context exemplars to adapt to diverse driving reasoning tasks. This approach tackles issues of unreliable scene facts, hallucinations, and opaque reasoning found in existing perception pipelines and driving-oriented LLM methods.