Highlights:
- New research compares text-based and image-based retrieval in multimodal RAG systems.
- Direct multimodal embedding achieves a 13% absolute improvement in mAP@5 and 11% in nDCG@5 over text-based summarization.
- Study introduces a unique financial benchmark combining charts, tables, and textual data.
- Eight co-authors from diverse computational research backgrounds designed the comparative framework.
TLDR:
A 2025 study by Elias Lumer and colleagues demonstrates that direct multimodal embedding methods significantly outperform text-only summarization in Retrieval-Augmented Generation (RAG) systems, improving retrieval accuracy and factual consistency by preserving visual context in datasets like financial documents.
In a groundbreaking 2025 publication on arXiv (arXiv:2511.16654), a team of researchers led by Elias Lumer, Alex Cardenas, Matt Melich, Myles Mason, Sara Dieter, Vamse Kumar Subbiah, Pradeep Honaganahalli Basavaraju, and Roberto Hernandez presents a detailed comparative analysis between text-based and image-based retrieval in multimodal Retrieval-Augmented Generation (RAG) Large Language Model (LLM) systems. Their paper, titled *Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems*, explores a critical challenge in modern AI: how to maintain visual fidelity when integrating textual and visual information for retrieval and question answering tasks.
Traditional multimodal RAG pipelines rely heavily on converting images—such as charts, tables, or diagrams—into textual summaries before embedding them into vector databases. However, this process introduces information loss, particularly contextual and spatial relationships that are essential for accurate downstream reasoning. The authors address this by evaluating two distinct retrieval strategies: (1) text-based chunk retrieval, where image data is preprocessed into text form, and (2) direct multimodal embedding retrieval, where both images and text are embedded together in a shared multimodal vector space without summarization.
Using six different LLM configurations and two multimodal embedding models, the team created a unique benchmark dataset built from financial earnings call documents containing both numerical tables and associated text. Across 40 question-answer pairs, direct multimodal embedding retrieval achieved an impressive 13% improvement in mean average precision (mAP@5) and 11% in normalized discounted cumulative gain (nDCG@5). These gains translate to relative performance boosts of 32% and 20%, respectively, underscoring the tangible advantages of retaining image representations. Furthermore, evaluations using LLMs as judges revealed that direct retrieval methods produced more accurate and factually consistent outputs. The study’s results highlight a key shift in RAG design philosophy—one that prioritizes native multimodal understanding over lossy text summarization, pushing the frontier of AI systems that can effectively interpret and reason across both language and visual data sources.
Technically, the research showcases how multimodal encoders—capable of embedding image and text pairs jointly—enable vector similarity searches that preserve context-rich features. This advancement is crucial for fields like finance, healthcare, and scientific publishing, where visualizations often carry unique insights unavailable from text alone. By showing strong empirical improvements and reliable factual consistency, Lumer and his co-authors provide a blueprint for the next generation of multimodal RAG systems that combine retrieval speed, accuracy, and interpretability.
Source:
Source:
arXiv:2511.16654 [cs.CL] – https://doi.org/10.48550/arXiv.2511.16654 (Submitted on 20 Nov 2025)