Stackvora.tech

Evaluating LLM Outputs

Evaluating Large Language Model (LLM) outputs presents unique challenges that traditional software testing methodologies often fail to address…

Evaluating Large Language Model (LLM) outputs presents unique challenges that traditional software testing methodologies often fail to address comprehensively. Unlike deterministic code, LLMs generate varied responses, making unit tests insufficient. A robust evaluation harness is critical for maintaining output quality, especially with frequent prompt adjustments, model updates, or data retraining. This article outlines a structured approach to building and operationalizing an LLM evaluation pipeline, focusing on practical methods and critical metrics.

The foundation of any effective LLM evaluation system is a meticulously curated dataset. Start by collecting 100-300 real-world inputs, ensuring coverage of common use cases, known failure modes, and a selection of adversarial examples. For each input, label the expected behavior. This doesn't necessarily mean providing an exact text match but defining the essential properties the answer must possess. For instance, an expected behavior could specify that the output must be a valid JSON object, contain specific keywords, or adhere to a particular conversational tone.

Building Your Golden Dataset

The quality of your evaluation hinges directly on the quality and diversity of your golden dataset. This dataset acts as your ground truth against which all subsequent LLM responses will be measured.

Dataset Collection Strategy

  • Common Cases: Include prompts representing typical user interactions or application queries. These should constitute the majority of your dataset (e.g., 60-70%).
  • Known Failure Modes: Deliberately include prompts that have historically caused the model to hallucinate, refuse inappropriately, or generate incorrect outputs. This helps in regression testing (e.g., 15-20%).
  • Edge Cases & Adversarial Examples: Craft prompts designed to stress-test the model's boundaries, including ambiguous queries, trick questions, or inputs designed to elicit harmful responses. These are crucial for robustness (e.g., 10-15%).
  • Diversity: Ensure the dataset covers various topics, linguistic styles, and lengths relevant to your application domain.

Labeling Expected Behavior

For each prompt, define clear, actionable expected outcomes. These labels can be multi-faceted:

  • Categorical Labels: e.g., "expected_intent": "summarization", "expected_sentiment": "positive".
  • Structural Constraints: e.g., "must_be_json": true, "json_schema": "{'type': 'object', 'properties': {'name': {'type': 'string'}}}".
  • Keyword Presence/Absence: e.g., "must_contain_keywords": ["Python", "Docker"], "must_not_contain_substrings": ["unhelpful", "irrelevant"].
  • Factual Grounding: For RAG systems, "retrieved_context_ids": ["doc_123", "doc_456"] and "all_claims_traceable": true.
  • Safety & Refusal: "expected_refusal": false or "expected_refusal_category": "PII_redaction".

Four Key Evaluation Methods

A comprehensive evaluation pipeline integrates several methods, each offering distinct advantages.

1. Deterministic Checks

These are the cheapest and most efficient checks, suitable for immediate feedback and catching obvious regressions.

  • JSON/XML Schema Validation: For outputs expected in structured formats. Tools like jsonschema in Python are excellent.
  • Required/Forbidden Fields/Strings: Verifying the presence of critical information or absence of undesirable terms. 
    
import re
import json

def check_json_schema(output_text, schema):
    try:
        data = json.loads(output_text)
        jsonschema.validate(instance=data, schema=schema)
        return True, "Valid JSON and schema"
    except (json.JSONDecodeError, jsonschema.ValidationError) as e:
        return False, f"JSON/Schema error: {e}"

def check_keyword_presence(output_text, keywords):
    missing = [kw for kw in keywords if kw not in output_text]
    return not bool(missing), f"Missing keywords: {', '.join(missing)}"

# Example Schema (example_schema.json)
# {
#     "type": "object",
#     "properties": {
#         "title": {"type": "string", "minLength": 5},
#         "authors": {"type": "array", "items": {"type": "string"}},
#         "year": {"type": "integer", "minimum": 1900}
#     },
#     "required": ["title", "authors", "year"]
# }
  • Citation Presence/Format: For RAG-augmented models, ensuring that claims are properly attributed to source documents (e.g., [1] or (Source: document_id)). Regular expressions are often used here.
  • Length Constraints: Enforcing minimum or maximum token/word counts.
  • Language Detection: Ensuring the output is in the expected language.

2. Reference Metrics

These methods compare the LLM output against a predefined ground truth or reference answer. They are most effective when a precise target output can be defined.

  • Exact Match (EM): Binary metric (1 if identical, 0 otherwise). Useful for factual questions with single, canonical answers (e.g., "What is the capital of France?").
  • F1 Score (Token-based): Measures overlap between generated and reference text, balancing precision and recall. Ideal for summarization or open-ended questions where exact phrasing varies but content overlap is crucial. Often used with frameworks like 🤗 Evaluate.
  • ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Family of metrics (ROUGE-N, ROUGE-L) commonly used for summarization tasks, measuring n-gram overlap between generated and reference summaries.
  • BLEU (Bilingual Evaluation Understudy): Originally for machine translation, measures n-gram precision against a set of reference translations. Can be adapted for text generation tasks where output fidelity to a reference is important.
  • Embedding Similarity: Calculates the cosine similarity between vector embeddings of the generated and reference texts. Useful when semantic similarity is more important than lexical overlap. Models like sentence-transformers/all-MiniLM-L6-v2 are good choices for generating embeddings.

Trade-off: Requires significant effort to create high-quality reference answers, especially for open-ended generative tasks. Small variations in phrasing can lead to low scores even if the meaning is correct.

3. LLM-as-Judge

Leverages a powerful LLM to evaluate the output of another (or the same) LLM. This method is highly flexible and can assess subjective qualities.

  • Mechanism: A prompt is constructed that includes the original user query, the LLM's response, and a detailed rubric. The judge LLM then scores the response against this rubric, often on a numeric scale (e.g., 1-5).
  • Rubric Design: Crucial for consistent evaluation. The rubric should define clear criteria and anchor points for each score. For example:
    Score Factual Accuracy Coherence
    1 Contains multiple factual errors. Completely illogical or disjointed.
    3 Minor factual inaccuracies or unsubstantiated claims. Generally coherent but with awkward transitions.
    5 All claims are accurate and well-supported. Fluent, well-structured, and easy to understand.
  • Calibration: Before trusting LLM-as-judge scores, calibrate them against human labels. Compare scores from the judge LLM with scores from human reviewers on a subset of the data. Adjust the judge's prompt or rubric if there are significant discrepancies.
  • Fixed Rubric: Always use a fixed, version-controlled rubric to ensure consistency across evaluation runs. Avoid ad-hoc instructions.

Benefits: Scales well, can evaluate complex criteria, and often correlates well with human judgment once calibrated. Challenges: Costly (API calls), susceptible to judge LLM bias, and still requires human oversight for calibration and outlier review.

4. Human Review

The ultimate arbiter of quality. While not scalable for all evaluations, human review is indispensable for qualitative assessment, identifying new failure modes, and validating automated methods.

  • Sampling Strategy: Review a small, carefully selected weekly or bi-weekly sample. Focus on:
    • Cases where automated evaluations disagree or yield ambiguous results.
    • New prompts or domains.
    • Outputs flagged by safety filters.
    • Adversarial examples.
  • Feedback Loop: Human feedback should inform improvements to the golden dataset, prompt engineering, and automated evaluation metrics.

What to Score

A comprehensive evaluation covers multiple dimensions of LLM performance.

  • Factual Grounding: Is every claim traceable to the provided context (for RAG systems) or to general world knowledge? Tools like RAGAS can help evaluate this for RAG applications.
  • Instruction Following: Does the model adhere to all explicit instructions in the prompt (e.g., "answer in 3 bullet points", "use a formal tone", "provide a JSON object")?
  • Format Compliance: Does the output conform to the expected structure (JSON, XML, markdown list, etc.)?
  • Safety & Refusal Behavior: Does the model refuse appropriately when asked unsafe, illegal, or unethical questions? Does it avoid generating harmful content? This often involves specialized classifiers or predefined refusal patterns.
  • Coherence & Fluency: Is the text well-written, easy to understand, and logically structured? (Often best assessed by LLM-as-judge or human review).
  • Relevance: Does the answer directly address the user's query without veering off-topic?
  • Latency and Cost Per Request: Non-functional but critical metrics. Track average and P95 latency for response generation, and token usage for cost analysis. These should be part of every evaluation run.

Running Evaluation Like CI/CD

Integrate your evaluation harness into your development workflow, treating it like a continuous integration (CI) pipeline.


# Example CI/CD stage (e.g., in .github/workflows/llm-eval.yml or GitLab CI)
name: LLM Evaluation
on:
  push:
    branches:
      - main
    paths:
      - 'prompts/**'
      - 'models/**'
      - 'eval_data/**' # Trigger on changes to prompts, models, or eval dataset
jobs:
  evaluate_llm:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Run LLM Evaluation
        run: |
          python scripts/run_evals.py \
            --model_version ${{ github.sha }} \
            --prompt_path prompts/current_prompt.txt \
            --dataset_path eval_data/golden_dataset.json \
            --output_dir eval_results/${{ github.sha }}
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - name: Check Evaluation Thresholds
        run: python scripts/check_thresholds.py eval_results/${{ github.sha }}/summary.json
        # This script should fail if metrics (e.g., F1, accuracy) drop below a predefined threshold
      - name: Upload Evaluation Results
        uses: actions/upload-artifact@v3
        with:
          name: llm-eval-results-${{ github.sha }}
          path: eval_results/${{ github.sha }}
  • Version Control Everything: Treat prompts, model configurations, and evaluation datasets as code. Store them in version control (Git).
  • Automated Runs: Trigger evaluation runs on every code commit, prompt change, or model update.
  • Fail the Build: Configure your CI pipeline to fail if key evaluation metrics (e.g., factual accuracy, instruction following) drop below predefined thresholds. This prevents regressions from reaching production.
    • Threshold Example: min_f1_score: 0.75, max_hallucination_rate: 0.05, min_safety_score: 0.90.
  • Store All Runs: Archive evaluation results (metrics, raw model outputs, judge LLM scores) for every run. This historical data is crucial for attributing quality changes to specific commits or model versions.
  • Track Production Traces: Periodically sample real production traffic and add these samples (with human labels if possible) to your evaluation dataset. This ensures your evaluation dataset remains relevant and representative of actual user interactions, preventing drift.

Common Pitfalls

  • Stale Evaluation Data: Not updating your golden dataset frequently enough to reflect new user behaviors or product features. This leads to evaluations that don't represent real-world performance.
  • Over-reliance on Single Metrics: Believing one metric (e.g., F1) perfectly captures output quality. LLM evaluation is multi-faceted; use a suite of metrics.
  • Ignoring Latency/Cost: Focusing solely on qualitative metrics and neglecting operational concerns. High latency or cost can make a perfectly accurate model unusable.
  • Uncalibrated LLM-as-Judge: Deploying an LLM judge without verifying its alignment with human judgment, leading to misleading scores.
  • Lack of Versioning: Not tracking changes to prompts, models, and eval datasets, making it impossible to reproduce results or diagnose regressions.
  • Biased Evaluation Data: A dataset that doesn't cover known failure modes or adversarial cases will give an artificially inflated sense of model robustness.

Back to the knowledge base · Ask the AI assistant