RAG Evaluation in Python: RAGAS vs DeepEval vs TruLens Compared (2026)

Updated: August 8, 2026

RAG evaluation in Python is the practice of measuring how well a retrieval-augmented generation pipeline pulls back relevant context and then produces faithful, on-topic answers. In 2026, the three open-source frameworks most Python teams reach for are RAGAS, DeepEval, and TruLens. RAGAS is the reference-metrics library that kicked off the movement. DeepEval treats evals like Pytest test cases. TruLens focuses on live observability with a feedback-function model. I've shipped RAG systems with all three, and this guide compares them with runnable code, shows which metrics each supports, and gives you a decision framework so you can pick without regretting it three months in.

• RAGAS (v0.2+) is the strongest choice for reference-free metrics on retrieval quality: faithfulness, answer relevancy, context precision, and context recall, with a mature integration for LangChain and LlamaIndex.
• DeepEval is the best fit when you want evaluation to feel like unit testing (Pytest-style assert_test), with 40+ built-in metrics covering hallucination, bias, toxicity, and G-Eval.
• TruLens shines for live production tracing. Its "feedback functions" attach to spans and let you monitor faithfulness and answer relevance on real user traffic, not just golden sets.
• All three now support arbitrary LLM judges via LiteLLM or a callable, so you're not locked into OpenAI, and all three integrate with observability stacks like Langfuse and Arize.
• For CI/CD gating pick DeepEval; for ad-hoc dataset evals pick RAGAS; for continuous production monitoring pick TruLens. Or combine RAGAS metrics inside DeepEval or TruLens (both wrap them).

What is RAG evaluation and why does it need its own tooling?

RAG evaluation is the process of scoring a retrieval-augmented generation system on two coupled axes: did the retriever pull back the right context, and did the generator use that context faithfully to answer the question. Traditional NLP metrics like BLEU or ROUGE score surface overlap between a generated answer and a reference, but they miss the two failure modes that RAG systems actually exhibit. Namely, the retriever missing the relevant chunk, and the generator hallucinating facts that were never in the retrieved context. That's why a separate class of tooling emerged.

The problem is compounded because ground-truth answers are painfully expensive to collect. If you spent a week hand-labeling 500 (question, answer) pairs, a model swap or a chunking-strategy change can invalidate the reference set overnight. I hit this exact wall on a doc-QA project last spring. Every framework in this comparison supports reference-free metrics that use an LLM judge to score faithfulness and relevance without a written-down "correct answer", plus reference-based metrics when you do have labels. In practice most teams start reference-free, layer in a small golden set, and only reach for full labels when a metric flags a regression they want to investigate.

You should also treat RAG evaluation as continuous, not one-off. Data drift, index updates, and prompt tweaks all shift scores, so the tool you pick has to fit into the loop you already run, whether that's Pytest in CI, a scheduled Airflow job, or live production traces. Our earlier guide to LLM observability in Python covers the tracing layer; this article focuses on the scoring layer that runs on top.

RAGAS vs DeepEval vs TruLens at a glance

Before we walk through code for each library, here's a side-by-side view of how they compare on the dimensions Python teams actually care about in 2026. All three are actively maintained and had major releases in the past year (RAGAS 0.2, DeepEval 2.x, and TruLens 1.x), so version pinning matters.

Dimension	RAGAS	DeepEval	TruLens
Primary use case	Dataset-level metric scoring	Pytest-style eval assertions	Live tracing + feedback functions
Built-in metrics	~12 (RAG-focused)	40+ (RAG, safety, custom)	~15 (RAG-focused)
Reference-free scoring	Yes (default)	Yes	Yes
Pytest integration	Manual	Native (assert_test)	Manual
Live production tracing	No	Optional (Confident AI)	Yes (built-in)
LLM judge providers	Any (LiteLLM, LangChain)	Any (LiteLLM native)	Any (LiteLLM, callable)
Cost per 1k evals	~$0.30 (gpt-4o-mini)	~$0.40 (default judge)	~$0.35 (per feedback fn)
Best-fit team	ML/data scientists iterating	Engineers with CI/CD discipline	Platform teams running prod

The core RAG metrics you actually need

Regardless of the framework, five metrics do 90% of the work in production RAG evaluation. Understanding them decoupled from any specific library helps you read the docs faster and swap frameworks without confusion, because the same idea often has three different names across the three tools.

Faithfulness (a.k.a. groundedness)

Faithfulness measures whether every factual claim in the generated answer is supported by the retrieved context. An LLM judge extracts atomic claims from the answer, then checks each one against the context; the score is supported_claims / total_claims. This is the single most useful metric for catching hallucinations, and all three frameworks compute it in essentially the same way.

Answer relevancy

Answer relevancy asks whether the generated answer actually addresses the user's question. Not whether it's correct, but whether it's on topic. RAGAS computes this by having the LLM generate hypothetical questions that the answer could plausibly answer, then measuring cosine similarity between those questions and the original one.

Context precision and context recall

Context precision measures the signal-to-noise ratio of your retriever: of the chunks you pulled back, how many were actually useful? Context recall measures the inverse. Of the information needed to answer, how much did the retriever find? These two together let you separate retriever failures from generator failures, which is honestly the most valuable diagnostic move in RAG debugging.

Retrieval hit rate and MRR

When you do have a golden dataset with labeled ground-truth chunks, classic IR metrics (hit rate at K, Mean Reciprocal Rank, NDCG) tell you whether your embedding model and reranker are pulling the right chunks into the top-K window. DeepEval and TruLens expose these natively; with RAGAS you compute them yourself and pass them through.

RAGAS in practice: reference-free retrieval scoring

RAGAS is the closest thing to a "just evaluate my RAG pipeline on a batch of queries" tool. You give it a list of question/answer/context triples and it returns a scored DataFrame. Install it in a fresh Python 3.11+ environment:

pip install "ragas>=0.2.10" "datasets>=3.0" "langchain-openai>=0.2"

A minimal evaluation script looks like this. Note that in RAGAS 0.2 the API moved to per-sample SingleTurnSample objects and an EvaluationDataset wrapper. Older tutorials using Dataset.from_dict still work, but the new API is clearer:

from ragas import EvaluationDataset, evaluate
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import (
    Faithfulness,
    ResponseRelevancy,
    LLMContextPrecisionWithoutReference,
)
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

# 1. Wrap your judge model. Any LiteLLM-compatible model works.
judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))

# 2. Build samples from your RAG pipeline output.
samples = [
    SingleTurnSample(
        user_input="What is the default chunk size in LlamaIndex?",
        response="LlamaIndex uses a default chunk size of 1024 tokens.",
        retrieved_contexts=[
            "The SentenceSplitter default chunk_size is 1024 tokens with 200 overlap.",
            "Nodes are the atomic units of LlamaIndex indexing.",
        ],
    ),
    # ... more samples from your test set
]

dataset = EvaluationDataset(samples=samples)

# 3. Evaluate. All three metrics are reference-free.
result = evaluate(
    dataset=dataset,
    metrics=[
        Faithfulness(llm=judge),
        ResponseRelevancy(llm=judge),
        LLMContextPrecisionWithoutReference(llm=judge),
    ],
)

# 4. Inspect. result is convertible to a pandas DataFrame.
df = result.to_pandas()
print(df[["user_input", "faithfulness", "answer_relevancy",
          "llm_context_precision_without_reference"]])

The output is a DataFrame you can persist to Parquet, load into a Jupyter notebook, and diff between experiments. That's why RAGAS wins for the experimenting data scientist persona: the ergonomics match how you already think about model comparison. For deeper integration into a broader analytics workflow, the resulting frames slot naturally into pipelines built with Polars and DuckDB.

DeepEval in practice: Pytest-style RAG assertions

DeepEval takes a different philosophy: your evaluations are tests. You write them in a tests/ directory, run them with Pytest, and get a pass/fail signal you can wire into GitHub Actions. This maps beautifully to the "unit tests for LLMs" mental model that engineering teams already have.

pip install "deepeval>=2.4" pytest

A minimal DeepEval test file:

# tests/test_rag_faithfulness.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualPrecisionMetric,
)

# Import your production RAG pipeline
from myapp.rag import answer_question

@pytest.mark.parametrize("question,expected_topic", [
    ("What is the default chunk size in LlamaIndex?", "chunk size"),
    ("How does Qdrant handle hybrid search?", "hybrid search"),
])
def test_rag_pipeline(question, expected_topic):
    result = answer_question(question)  # returns {answer, contexts}

    test_case = LLMTestCase(
        input=question,
        actual_output=result["answer"],
        retrieval_context=result["contexts"],
    )

    faithfulness = FaithfulnessMetric(threshold=0.8, model="gpt-4o-mini")
    relevancy = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini")
    precision = ContextualPrecisionMetric(threshold=0.7, model="gpt-4o-mini")

    assert_test(test_case, [faithfulness, relevancy, precision])

Run it with pytest tests/ -v. Each metric prints its score, reasoning trace, and pass/fail against the threshold. Because it's just Pytest, all the tooling you already have (fixtures, parametrize, xdist for parallelism, coverage reports) works out of the box.

DeepEval also ships G-Eval, a general-purpose LLM-judge metric where you write the rubric in natural language and the framework converts it into a scoring prompt. This is invaluable for domain-specific criteria that don't map to a named metric, such as "the answer should cite the source document number in square brackets when it uses a fact from context". If your RAG pipeline produces JSON outputs, the assertions compose nicely with the type-safe output layer we covered in structured LLM outputs with Instructor and Pydantic AI.

TruLens in practice: feedback functions on live traces

TruLens is the odd one out. It's designed to instrument your RAG pipeline in production and score every real user query, not to run batch evals on a dataset. You attach feedback functions to spans in the trace, and they compute scores asynchronously, storing everything in a local SQLite DB or a hosted backend.

pip install "trulens>=1.4" "trulens-providers-litellm"

A minimal TruLens integration with a LangChain-style RAG chain:

from trulens.core import TruSession, Feedback
from trulens.apps.langchain import TruChain
from trulens.providers.litellm import LiteLLM
import numpy as np

session = TruSession()
session.reset_database()  # dev only

# 1. Pick a judge provider (any LiteLLM model).
provider = LiteLLM(model_engine="openai/gpt-4o-mini")

# 2. Define feedback functions. These are just scored callables.
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons, name="groundedness")
    .on(context=TruChain.select_context())
    .on_output()
)
f_answer_rel = Feedback(provider.relevance, name="answer_relevance").on_input_output()
f_context_rel = (
    Feedback(provider.context_relevance, name="context_relevance")
    .on_input()
    .on(context=TruChain.select_context())
    .aggregate(np.mean)
)

# 3. Wrap your existing chain. Every invocation is now scored.
tru_chain = TruChain(
    rag_chain,  # your existing LangChain runnable
    app_name="prod-rag",
    app_version="v1.2",
    feedbacks=[f_groundedness, f_answer_rel, f_context_rel],
)

with tru_chain as recording:
    answer = rag_chain.invoke("What is the default chunk size in LlamaIndex?")

# 4. Launch the dashboard to inspect traces + scores.
from trulens.dashboard import run_dashboard
run_dashboard(session)

The dashboard shows every trace, the retrieved context for each, and the feedback scores side by side. In production, feedback functions run in a background thread pool so they don't block the user response. Honestly, this is the killer feature: you get real-world evaluation on real user traffic, not just the synthetic questions in your golden set. TruLens pairs particularly well with document-loader pipelines built on top of tools like the ones covered in our Docling PDF parsing guide, because you can trace and score end-to-end from ingestion to answer.

How does LLM-as-judge evaluation work?

Every reference-free metric in these three libraries is a variation of LLM-as-judge. You use a language model to score the output of another (or the same) language model. The judge is prompted with a rubric, such as "extract atomic claims from the answer, then for each claim decide whether the context supports it", and returns a structured score plus a reasoning trace.

The obvious concern is circularity. If the same GPT-family model wrote the answer and grades it, how do you trust the score? The 2025–2026 research consensus is that LLM-as-judge correlates well with human judgment (κ ≈ 0.6–0.8 on faithfulness) provided you use a stronger judge than the generator and you use temperature 0 for reproducibility. The "Judging LLM-as-a-Judge" paper (Zheng et al.) is still the reference here. When possible, use a different family for the judge: for example, generate with Claude Sonnet 4.5 and judge with GPT-4.1, or vice versa.

All three frameworks let you plug in a custom judge via LiteLLM, which means you can point them at a local Qwen3-32B served by vLLM for zero-cost evaluation on sensitive data. If you go local, keep the judge at ≥8B parameters and validate correlation with a hand-labeled subset first. Smaller open models have wide variance on nuanced faithfulness calls, so don't skip this step.

Building a golden dataset for RAG evaluation

Even with reference-free metrics, you eventually want a small, stable, hand-curated evaluation set to catch regressions the LLM judges miss. RAGAS ships a TestsetGenerator that reads your document corpus and synthesizes question/context pairs; DeepEval has a Synthesizer class with similar mechanics. Both are useful for bootstrapping, but the output always needs human review.

from ragas.testset import TestsetGenerator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.document_loaders import DirectoryLoader

# 1. Load your corpus (any LangChain loader works).
docs = DirectoryLoader("./corpus", glob="**/*.md").load()

# 2. Generate a synthetic testset.
generator = TestsetGenerator.from_langchain(
    llm=ChatOpenAI(model="gpt-4o", temperature=0),
    embedding_model=OpenAIEmbeddings(model="text-embedding-3-small"),
)
testset = generator.generate_with_langchain_docs(docs, testset_size=50)

# 3. Persist and hand-review before using in CI.
testset.to_pandas().to_parquet("./eval/golden_v1.parquet")

The pattern that's worked for me in practice: generate ~200 candidates, discard the 30–40% that a domain expert flags as ambiguous or wrongly-labeled, and commit the remaining 120–140 to your repo as golden_v1.parquet. Version this file the same way you version model weights. A new golden set means a new baseline, not a hotfix.

Wiring RAG evals into CI/CD

The end goal for most teams is a GitHub Actions workflow that fails a PR if faithfulness drops below a threshold. DeepEval is designed for exactly this. A pytest run returns a non-zero exit code and the job fails. Here's a minimal .github/workflows/rag-eval.yml:

name: RAG Evaluation
on:
  pull_request:
    paths:
      - "src/rag/**"
      - "prompts/**"
      - "tests/rag/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-eval.txt
      - name: Run RAG evaluations
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          pytest tests/rag/ -v --tb=short \
            --deepeval-cache-file .deepeval-cache.json
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-cache
          path: .deepeval-cache.json

Two production tips from actually running this. First, use the --deepeval-cache-file flag so repeated runs on unchanged (input, output) pairs don't re-call the judge. That drops CI cost by 5–10× on a stable golden set. Second, run the eval only on PRs that touch src/rag/** or prompts/**. There's no reason to spend $2 in judge tokens on a README typo.

Which framework should you pick?

The decision really comes down to where in the loop you want evaluation to live.

• Pick RAGAS if you're a data scientist iterating on retrievers, chunkers, or embedding models in Jupyter and you want a scored DataFrame you can slice and dice. Its metric set is the reference implementation the other two often wrap.
• Pick DeepEval if you have a mature CI/CD culture and you want RAG quality treated as a first-class test signal alongside your unit tests. The Pytest ergonomics are unmatched and the metric library is the broadest of the three.
• Pick TruLens if you have a RAG system in production and you want to score real user queries, not synthetic ones. The tracing model catches issues that no synthetic golden set can predict.

These aren't mutually exclusive. A common 2026 setup is RAGAS metrics called from inside DeepEval assertions during CI, with TruLens instrumenting production for the drift-detection loop that feeds back into next month's golden set. Both DeepEval and TruLens explicitly wrap RAGAS metrics as of their latest releases, so you rarely need to pick just one.

For deeper reading, the primary sources are the RAGAS documentation, the DeepEval GitHub repository, and the TruLens documentation. All three are actively maintained with substantive changelogs to track.

Frequently Asked Questions

Do I need ground-truth answers to evaluate a RAG pipeline?

No. RAGAS, DeepEval, and TruLens all default to reference-free metrics that use an LLM judge to score faithfulness, answer relevancy, and context precision without written-down correct answers. Ground-truth labels are only required for metrics like context recall and semantic similarity, and even a small hand-curated set of 50–150 examples is enough for regression detection.

What is the difference between faithfulness and answer relevancy?

Faithfulness asks whether every factual claim in the answer is supported by the retrieved context, which catches hallucinations. Answer relevancy asks whether the answer is on-topic for the question, which catches off-topic or evasive answers. A response can be faithful but irrelevant ("I don't know" is always faithful), or relevant but unfaithful (a hallucinated but on-topic answer), so you need both.

Can I use a local model as the LLM judge?

Yes. All three frameworks accept a LiteLLM-compatible model, which lets you point at an Ollama, vLLM, or TGI endpoint serving Qwen3, Llama 3.3, or any other open-weight model. Keep the judge at ≥8B parameters, run at temperature 0, and validate correlation with a hand-labeled subset before trusting the scores in CI.

How much does RAG evaluation cost per run?

With gpt-4o-mini as the judge and typical 2k-token contexts, expect roughly $0.30–$0.40 per 1,000 test cases across faithfulness, answer relevancy, and context precision. Enabling the DeepEval or RAGAS response cache drops repeat-run cost by 5–10×, and switching to a locally hosted judge model pushes marginal cost to zero if you already have the GPU.

Can I combine RAGAS, DeepEval, and TruLens?

Yes, and many teams do. DeepEval and TruLens both explicitly wrap RAGAS metrics, so a common pattern is: RAGAS as the metric library, DeepEval as the CI/CD harness that runs it, and TruLens instrumenting production for continuous scoring. They share the LLM-judge abstraction and can point at the same LiteLLM endpoint, so operational overhead stays low.