EVALUATING GROUNDING STRATEGIES FOR FACTUAL ACCURACY AND INFORMATION COVERAGE IN LLM-GENERATED REVIEW SUMMARIES
2026 (English) Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE credits
Student thesis
Abstract [en]
Large Language Models (LLMs) are increasingly used to summarize business data such as customer reviews. Although these models produce fluent outputs, they can generate statements that are not fully supported by the input data, a problem referred to as hallucination. In business analytics, where summaries may directly influence operational decisions, this creates a significant reliability risk. This thesis investigates how different grounding strategies affect the quality of LLM-generated restaurant review summaries. Grounding constrains generation to rely on explicitly provided external data rather than on knowledge learned during training. The research question is: How do different grounding strategies affect the quality and reliability of LLM-generated review summaries? Four strategies are evaluated in a single-factor controlled experiment: Plain Prompting, Retrieval-Augmented Generation (RAG), Aggregated-Statistics Grounding, and Template-Constrained Generation. Each is applied to eleven evaluation scenarios built from real restaurant review data and supplemented with synthetic cases covering text-rating conflicts, ambiguous opinions, and high information density. Generated summaries are assessed using atomic-fact verification, golden-insight coverage scoring, citation-quality metrics, and operational telemetry, with an LLM-as-a-Judge pipeline as the primary evaluator. The results reveal clear trade-offs. RAG is the most computationally efficient but underperforms in information coverage. Aggregated-Statistics Grounding and Template-Constrained Generation achieve the highest mean factual accuracy, but their advantage over the Plain Prompting long-context baseline is not statistically significant. Template-Constrained Generation shows the weakest attribution quality. No single strategy is superior across factual accuracy, coverage, attribution quality, and operational cost, so the findings provide practical guidance for selecting a strategy based on deployment requirements.
Place, publisher, year, edition, pages 2026. , p. 88
Keywords [en]
Large Language Models, Text Summarization, Grounding Strategies, Factual Accuracy, Information Coverage, Attribution Faithfulness, Hallucination, Retrieval-Augmented Generation (RAG), Hybrid Data, LLM-as-a-Judge
National Category
Computer and Information Sciences
Identifiers URN: urn:nbn:se:mdh:diva-78986 OAI: oai:DiVA.org:mdh-78986 DiVA, id: diva2:2100389
External cooperation
Zoined Oy
Subject / course Computer Science
Presentation
2026-09-02, U2-158, Universitetsplan 1, 722 20 Västerås, Västerås, 11:00 (English)
Supervisors
Examiners
2026-09-292026-09-102026-09-29 Bibliographically approved