ClaimLens

Introduction

ClaimLens is a production-oriented Retrieval-Augmented Generation (RAG) system designed for insurance policy analysis, where accuracy and traceability are critical.

Traditional RAG pipelines often rely on heuristic chunking and loosely grounded outputs, which can lead to inconsistent retrieval and hallucinations. In domains like insurance, where decisions depend on precise clauses, this becomes a major limitation.

This project focuses on treating retrieval and reasoning as structured, deterministic systems rather than black-box pipelines, ensuring that every output is grounded, traceable, and evaluable.

Problem & Motivation

Most RAG tutorials suggest a simple pipeline: chunk documents, embed them, and retrieve with an LLM. This works well for clean text, but breaks down in real-world documents like insurance policies.

Insurance PDFs are structurally complex, with inconsistent numbering, repeated headings, annexures, and noisy formatting. Naive token-based chunking ignores these structures, often splitting clauses incorrectly or missing important context entirely.

The core problem wasn't retrieval — it was structure.

To address this, I designed a deterministic clause parser that:

• • Detects multiple clause formats (numbered, roman, alphabetic, definitions)
• • Assigns canonical IDs to each clause for traceability
• • Enforces fail-fast behavior to avoid silent parsing errors

This ensures that each retrieval unit maps directly to a real legal clause, improving both retrieval accuracy and interpretability.

While not perfect due to challenges like multi-column layouts and inconsistent formatting, the system significantly outperforms naive chunking and provides a clear evaluation framework to iteratively improve performance.

Overview

ClaimLens is designed as a structured retrieval system rather than a naive RAG pipeline.

The system enforces:

• • Deterministic parsing for stable retrieval units
• • Canonical identifiers for traceability
• • Evaluation-driven design for measurable performance

The goal is to move from "LLM-generated answers" to reliable, reproducible decision support.

System Architecture

The architecture is designed to separate concerns across ingestion, retrieval, ranking, and reasoning, ensuring each component is independently optimizable and testable.

• • Ingestion → Page-level document loading
• • Clause Splitter → Deterministic clause extraction
• • Retriever → Dense retrieval (FAISS)
• • Reranker → Cross-Encoder ranking refinement
• • Reasoner → LLM with strict schema validation
• • Pipeline → End-to-end orchestration

Architecture

The two diagrams below translate the system description into a product view and an execution flow, making it easier to see how ClaimLens moves from a policy question to a grounded answer.

Design Constraints

• • High precision required for legal clause interpretation
• • Inconsistent document structures across insurers
• • Need for traceable and explainable outputs
• • Minimizing hallucinations in LLM reasoning

Key Engineering Decisions

Retrieval Pipeline

• • Dense Retrieval (FAISS + BGE embeddings)
• • Top-K = 40 candidate generation
• • Cross-Encoder reranking → Top 5
• • Eliminated manual hybrid weighting

Reasoning & Validation

• • Strict JSON schema enforcement (Pydantic)
• • Citation grounding constraints
• • Retry mechanism on validation failure

What Makes This Different

Evaluation

Evaluation was treated as a first-class component rather than an afterthought.

Metrics such as Recall@20 and MRR were used to measure retrieval effectiveness, ensuring that relevant clauses are consistently surfaced before reasoning.

This enabled iterative improvements in retrieval quality instead of relying on subjective output inspection.

Trade-offs

Challenges

• • Handling inconsistent clause structures across insurers
• • Reducing noise from dense retrieval
• • Enforcing strict schema validation on LLM outputs

Future Improvements

• • Adaptive retrieval based on query intent
• • Learning-to-rank for dynamic reranking optimization
• • Feedback loop for continuous evaluation improvement
• • Integration with LangGraph for agentic workflows

What I Learned

Key Insight