What is LLM evaluation?
LLM evaluation is the process of measuring how well a large language model performs on a task, by running it against test inputs and scoring its outputs against criteria such as correctness, relevance, faithfulness, and safety. Evaluation can be reference-based (compared to a known answer) or reference-free (judged by a rubric or a model).
Two things are usually being evaluated, and it helps to keep them separate. Model evaluation asks whether the underlying model is capable, using general tests. System or product evaluation asks whether your application, prompts, retrieval, and guardrails included, produces good outcomes on your real inputs. A model can top every public benchmark and still fail your system evaluation, which is the one that decides whether it is worth anything to you.
How do you evaluate an LLM?
You evaluate an LLM by assembling a dataset of representative inputs, choosing a scoring method (exact match, a metric, or a judge), running the model, and comparing results across models and versions. Evaluation happens offline before deployment and online in production, and the hardest part is defining what a good answer is for your task.
There are a few organizing distinctions. Evaluation is offline (on a fixed test set before shipping) or online (on live traffic in production). Scoring is reference-based (compared to a known-good answer) or reference-free (judged by a rubric or a model when there is no single right answer). And the judge is a deterministic check, a human rater, or another model. Most serious setups use all of these in combination, run continuously as a loop rather than as a one-time test.
A concrete example: to evaluate an LLM that drafts customer-support replies, you might assemble 200 real tickets with their known-good responses, score each draft two ways (exact match on the resolution code it picks, and an LLM judge on tone and completeness against a rubric), run both candidate models over the same set, and compare pass rates and cost per ticket. The setup, not the model, is where most of the work lives.
What are the main LLM evaluation metrics?
Common LLM evaluation metrics include exact match and accuracy for tasks with a known answer, BLEU and ROUGE for overlap with a reference text, BERTScore for semantic similarity, faithfulness and answer relevancy for retrieval systems, hallucination rate, task success for agents, and latency and cost for operations.
| Metric | What it measures | When to use |
|---|---|---|
| Exact match / accuracy | Output matches a known-correct answer | Tasks with one right answer (classification, extraction) |
| BLEU / ROUGE | Overlap with a reference text | Translation, summarization (reference-based) |
| BERTScore | Semantic similarity to a reference via embeddings | When wording varies but meaning should match |
| Faithfulness / groundedness | Whether the answer stays supported by the provided context | RAG and retrieval systems |
| Answer relevancy | Whether the answer addresses the question asked | RAG, question answering |
| Hallucination rate | Share of outputs with unsupported claims | Any generative use where accuracy matters |
| Task success / completion | Whether the model accomplished the end goal | Agents and multi-step workflows |
| LLM-as-judge score | A model's rubric-based rating of an output | Open-ended outputs with no single reference |
| Latency, cost | Operational efficiency | Every production system |
What is LLM-as-a-judge?
LLM-as-a-judge is a method where one language model scores a candidate model's outputs against a rubric or a reference, often used for open-ended tasks that have no single correct answer. It scales human-style judgment cheaply, but it carries known biases such as self-preference and position bias, so its scores are best treated as one signal rather than ground truth.
It comes in a few forms (pairwise, single-answer grading, and reference-guided scoring) and carries real biases (position, verbosity, and self-preference), so a judge is only trustworthy once it is validated against human labels. LLM-as-a-judge is its own discipline; see our full guide to LLM-as-a-judge for judge types, prompt and rubric design, the full bias catalog and mitigations, and how to validate a judge against human labels.
How do you evaluate a RAG system or an AI agent?
You evaluate a RAG system with retrieval and generation metrics such as context precision, context recall, faithfulness, and answer relevancy. You evaluate an AI agent on task success, tool-call correctness, and step efficiency across a full trajectory, not just a single response.
For retrieval-augmented generation, evaluate the two halves separately: retrieval (context precision and recall, whether the right documents were fetched) and generation (faithfulness and answer relevancy, whether the answer used them correctly). RAG evaluation is its own discipline; see our full guide to RAG evaluation for retrieval and generation metrics, the RAG triad, golden datasets, and the same private-ownership logic applied to RAG.
For agents, a single response is not enough: score the whole trajectory on task success, tool-call correctness, and efficiency. Agent evaluation is its own growing discipline too; see our full guide to AI agent evaluation for tool-call correctness, trajectory scoring, benchmarks, and the same logic applied to agents.
What tools are used for LLM evaluation?
Open-source and commercial frameworks handle the mechanics: harnesses like OpenAI's evals and EleutherAI's lm-evaluation-harness, and eval libraries such as DeepEval, plus commercial platforms. They are useful for developer-time model testing. The question is not which framework, but who runs the evaluation, on whose data, and whether your vendor can see it. We compare the categories in AI evaluation tools, compared.
Why enterprises run LLM evaluation privately
Everything above is the mechanics. The part that decides real money is different: a public benchmark cannot measure your business value, your vendor's own eval is conflicted, and your evaluation criteria are intellectual property that gets gamed or leaked when you expose them. That is why the evaluation an enterprise trusts is private and owned. We make the full argument in what private AI evaluation is, Goodhart's Law for AI, and why your eval criteria are IP.
How do you evaluate an LLM on your own data?
You evaluate an LLM on your own data by building golden datasets from your real inputs and outcomes, standing up an eval harness inside your perimeter, defining what good means for your business, and re-scoring on every model change, all kept invisible to the model vendor. That is private, owned LLM evaluation.
In practice that is four steps, all kept invisible to the model vendor:
- Build golden datasets from your real inputs and known-good outcomes.
- Stand up an eval harness inside your own perimeter, so criteria and data never leave.
- Define your scoring to encode what a good answer is for your business, not a vendor default.
- Re-score on every model change, re-running whenever a model is swapped or updated.
Getting your data evaluable in the first place is the groundwork. That capability is what PrivateEval builds and transfers to your team. We do not evaluate the LLM for you; we give you the private machinery to evaluate it yourself.