European Conference on Computer Vision (ECCV) 2026

Disentangling Hallucinations

Orthogonal Semantic Projection for Robust Interpretability

Emirhan Bilgiç1,2,3   Baptiste Caramiaux2   Zhi Yan1   Gianni Franchi4

1U2IS, ENSTA, Institut Polytechnique de Paris, Palaiseau, France  ·  2ISIR, Sorbonne Université, Paris, France  ·  3Imperial College London, United Kingdom  ·  4AMIAD, Pôle Recherche, Palaiseau, France

Paper (arXiv) Code

Overview of OSP: a geometric intervention in the multimodal embedding space that removes hallucinations from attribution maps.

Abstract

As Vision-Language Models are increasingly deployed in safety-critical applications, the trustworthiness of their explanations becomes crucial. Explainable AI (XAI) methods for Vision-Language Models often suffer from explanation hallucination, where attribution maps highlight prominent image regions even when prompted with incorrect text descriptions (e.g., highlighting a dog when prompted “cat”). Although this problem is widespread, a formal mathematical analysis of XAI methods and CLIP embeddings is largely missing in the literature. We demonstrate that this phenomenon is not specific to a single architecture but is a fundamental consequence of Linear Semantic Leakage in high-dimensional embedding spaces. We propose a unified theoretical framework, Linear Semantic Attribution (LSA), which generalizes across discriminative methods. We introduce Orthogonal Semantic Projection (OSP), a geometric intervention that utilizes the residual property of Orthogonal Matching Pursuit (OMP) to disentangle unique semantic signals from shared concepts. We prove theoretically and demonstrate empirically that OSP minimizes hallucination by orthogonalizing the query vector against distractor concepts, rendering the attribution model blind to shared features while preserving fidelity for correct prompts.

Method

Attribution methods unified formulation
Most saliency methods reduce to one structure: a function of the image features and the text query, Vc(x) = f(aimg, atxtc). OSP purifies the text query before this function is applied.
Overall pipeline of OSP
Overall pipeline of OSP. Our method performs a geometric intervention in the multimodal embedding space to disentangle shared semantic components that cause hallucination, resulting in more faithful and grounded explanations.
Vanilla XAI vs OSP-grounded XAI
Standard XAI hallucinates absent concepts (top). OSP-grounded explanations stay faithful to the prompt: “correct for the right reasons” (bottom).

Highlights

Plug-and-play

A training-free module applied directly in the embedding space, before attribution.

Universal

Works across CLIP, SigLIP and Stable Diffusion, and 5 attribution methods.

Removes ghost signals

Orthogonality against distractors eliminates hallucinated saliency by construction.

Validated

Consistent AUROC gains on ImageNet-Segmentation, backed by a 200-participant user study.

BibTeX

Cite this work 
@article{bilgicc2026disentangling,
  title={Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretability},
  author={Bilgi{\c{c}}, Emirhan and Caramiaux, Baptiste and Yan, Zhi and Franchi, Gianni},
  journal={arXiv preprint arXiv:2606.14758},
  year={2026}
}