Project milestone H2: your group's proposal canvas is due Wednesday 14 Oct, 23:59.
Warm-up
Three questions on last session: LoRA, overfitting and the test set.
Warm-up · 10'Warm-up · Q1
With LoRA, a weight matrix becomes \(W = W_0 + BA\). During fine-tuning, what is trained?
\(B\) is \(d \times r\) and \(A\) is \(r \times k\) with a small rank \(r\), so each adapted
matrix trains \(r(d + k)\) parameters instead of \(d \cdot k\). The pretrained \(W_0\) is untouched, which is why
adapters are small files and several can share one base model.
Warm-up · Q2
Fine-tuning on 200 examples: training loss keeps falling, but validation loss rises after epoch 2. What is happening?
The model is memorising the 200 training examples. Early stopping on the validation
loss, fewer epochs, weight decay, LoRA dropout or more (and more varied) data all help. Today we make
"validation" precise: how to split, how many folds, and what to keep sealed.
Warm-up · Q3
In the S04 challenge, the AI-proposed TrainingArguments picked the best epoch by looking at the test set. Why is that a problem?
Every decision you take while looking at a dataset tunes the model to it. Choose with a
validation set (or cross-validation) and open the test set once, at the end. That rule is today's backbone.
Learning objectives
By the end of this session you can…
Explain why generated text is hard to evaluate and separate intrinsic from extrinsic evaluation RAE 2
Compute and interpret perplexity, BLEU, ROUGE, chrF and embedding similarity, and name what each misses RAE 2
Design a human or LLM-as-a-judge evaluation that controls known biases RAE 2
Choose classification metrics and a validation scheme without leakage RAE 2
Link an offline metric to a business KPI and a go/no-go threshold RAE 2
RAE 2: evaluate trained models with standard metrics and validation methods (course syllabus).
Part 1
Why is generation hard to evaluate?
Many right answers, many ways to be wrong, and a model's own measure of surprise.
Theory · 60'
The core difficulty
One input, many right answers
Input: a 6-minute call in which an Andes Bank customer reports a card charged twice at a supermarket and asks for a refund.
Three good summaries
Customer reports a duplicate supermarket charge and requests a refund.
Double card payment at a supermarket; refund requested.
Client was billed twice for one purchase and wants the money back.
One fluent, wrong summary
Customer reports a duplicate supermarket charge; the refund was already approved.
It shares most words with summary 1, yet it invents a commitment the bank never made.
No single reference: good outputs differ in words and order.
Many dimensions: fluency, faithfulness, coverage, tone, safety.
Humans disagree too, and their time is expensive.
A map of evaluation
Intrinsic vs extrinsic evaluation
Intrinsic: the output itself
Reference-free: perplexity, an LLM judge, human ratings
Lexical overlap with a reference: BLEU, ROUGE, chrF
Semantic similarity to a reference: embeddings, BERTScore
Task metrics for labels: precision, recall, F1
Extrinsic: the effect it causes
Did the agent resolve the case faster?
Did fewer customers call back?
Did fraud losses or review hours go down?
Measured offline (a simulation, a pilot) or online (A/B test, shadow mode)
Rule of thumb: intrinsic metrics are cheap and fast, so you use them to iterate;
extrinsic ones are what the business pays for, so you use them to decide.
Reference-free · perplexity
Perplexity: how surprised is the model?
For a text of \(N\) tokens, perplexity is the exponential of the average negative log-likelihood:
As uncertain as a fair choice among 4 tokens at every step.
What it is good for
Tracking training and validation loss (it is \(e^{\text{loss}}\))
Comparing models that share a tokenizer on the same text
Spotting out-of-domain text (tweets, another language)
Introduced for speech recognition by Jelinek et al. (1977); see Jurafsky & Martin, SLP3, ch. 3.
Reference-free · a trap
Perplexity depends on the tokenizer
GPT-2 on a short text
Tokens / word
PPL per token
PPL per word
News paragraph (English)
1.10
42.8
63.4
Same news, words shuffled
1.10
3,489.9
8,163.2
Informal tweet (English)
1.43
315.0
3,841.6
News paragraph (Spanish)
2.02
141.3
22,538.0
Computed in today's lab with openai-community/gpt2 on four short texts written for class. PPL per word = exp(total NLL ÷ number of words).
Not comparable across tokenizers: a model that cuts words into more, easier pieces gets a lower per-token PPL.
Compare fairly: same text, same tokenizer; or normalise per word, character or byte; or use a task metric.
Check · perplexity
On the same Spanish test set, model A reports perplexity 18 and model B reports 27. They use different tokenizers. Which one models Spanish better?
Per-token perplexity averages over different numbers of tokens for the same text. Convert to a
shared unit (e.g., bits per byte or per word) or, better for a business decision, compare both models on the
task you care about. This exact trap is one of the planted errors in today's challenge.
Buying on benchmarks
A leaderboard is not your test set
Andes Bank (fictional) must pick a model for a Spanish-language customer chat. Two vendors
send impressive scores: one a low perplexity, the other a top rank on a public English benchmark.
Neither number was measured on Andes Bank's customers, language or task.
Ask before you believe a score
Which data? Language, domain, date. Could the model have seen it during training (contamination)?
Which unit and settings? Tokenizer, prompt, decoding, number of examples.
Against what baseline? The current process, a simple rule, the cheaper model.
Build a small private evaluation set from real, anonymised cases and run every candidate on it.
Part 2
Comparing with a reference, and beyond
BLEU, ROUGE, chrF, embeddings, human raters and LLM judges.
Reference-based · translation
BLEU: clipped n-gram precision + brevity penalty
1 · Clipped precision \(p_n\): each candidate n-gram counts at most as many times as it appears in the reference.
2 · Brevity penalty for a candidate of length \(c\) and reference length \(r\):
$$ \text{BP} = \begin{cases} 1 & c > r \\ e^{\,1 - r/c} & c \le r \end{cases} $$
Reference: the cat is on the mat Candidate: the the the the the the the
Plain unigram precision: 7/7. Clipped: 2/7, because "the" appears only twice in the reference.
BLEU is precision-based; recall is only approximated by the brevity penalty. Corpus BLEU pools counts over all sentences.
Reference-based · summaries and morphology
ROUGE looks at recall, chrF at characters
ROUGE (Lin, 2004)
ROUGE-N recall: the share of the reference's n-grams that the candidate covers (ROUGE-1 unigrams, ROUGE-2 bigrams).
ROUGE-L: longest common subsequence (LCS) of \(m\) reference and \(n\) candidate words; order matters, gaps are allowed.
\(R = \tfrac{\text{LCS}}{m}\) · \(P = \tfrac{\text{LCS}}{n}\) · \(F_1 = \tfrac{2PR}{P + R}\) (as in the rouge_score package)
chrF (Popović, 2015)
An F-score over character n-grams (up to 6 characters). Partial credit for aprobó vs aprobado, which share most characters.
\(F_\beta\) with \(\beta = 2\) in sacrebleu: recall counts more than precision.
Useful for morphologically rich languages such as Spanish.
Shared blind spot: all three reward surface overlap with one reference. None of them checks that the facts are right.
Simulator · BLEU & ROUGE calculator
Score a candidate against a reference
Candidate words found in the reference (clipped):
n-gram
clipped matches / total
\(p_n\)
Length \(c\) = – · \(r\) = – · BP = –
BLEU (0–100): –
Precision
Recall
F1
ROUGE-1
–
–
–
ROUGE-L
–
–
–
Lower-cased words, punctuation ignored. * Smoothing: the \(k\)-th order with 0 matches uses \(p_n = 1/(2^k \cdot \text{total})\) (sacrebleu's "exp" method).
Check · BLEU
Reference: “El banco aprobó ayer la transferencia del cliente.” Which candidate gets the higher sentence BLEU with sacrebleu?
In the lab: the negated sentence gets BLEU 70.7 (chrF 94.8), the paraphrase BLEU 23.5 (chrF 70.0).
One inserted word keeps most n-grams; a reordering breaks them. Overlap is not meaning, so pair these metrics with
a meaning check: embeddings, a judge or a person.
Reference-based · meaning
Embedding similarity and BERTScore
Sentence embeddings (e.g., all-MiniLM-L6-v2; Reimers & Gurevych, 2019): one vector per text; cosine similarity near 1 means "about the same thing".
BERTScore (Zhang et al., 2020): match every token to its most similar token in the other text using contextual embeddings, then compute precision, recall and F1.
Both reward paraphrases that n-gram metrics miss.
Pair (English)
ROUGE-L F1
Cosine
Paraphrase
0.500
0.668
Negation
0.933
0.801
Unrelated
0.000
−0.035
From the lab: MiniLM cosine and ROUGE-L on three hand-written pairs. The negation still scores high on both.
Still not a fact-checker: "was approved" and "was not approved" sit close in embedding space.
Pick an embedding model that covers your language (MiniLM was trained mainly on English).
The reference standard
Human evaluation, designed properly
What to rate
Explicit criteria: fluency, faithfulness to the source, coverage, tone, safety. One question per criterion.
How to rate
Likert (1–5 per criterion) or pairwise (A vs B, which is better?). Pairwise is often easier for raters to answer consistently.
Who and how many
At least two raters on a shared sample; measure agreement (e.g., Cohen's κ); blind them to which system wrote what.
Guidelines with examples are the cheapest way to raise agreement: show a 1, a 3 and a 5 for each criterion before rating starts.
Scaling human judgment
LLM-as-a-judge: useful, but biased
A strong LLM grades one answer against a rubric, or compares two answers. Zheng et al. (2023) found that
strong judges such as GPT-4 agreed with human preferences more than 80% of the time on their benchmarks,
similar to the agreement between humans.
Known biases
Position: prefers the answer in a given slot
Verbosity: prefers longer answers
Self-enhancement: favours answers from its own model family
Limited reasoning when grading maths or logic
Mitigations
Swap A and B; accept a verdict only if both orders agree
Give a rubric and, when possible, a reference answer
Ask for a justification before the verdict
Use a judge from a different family than the system under test
Validate the judge on a sample with human ratings before trusting it
Self-check · name the problem
What went wrong in each evaluation?
The judge prefers whichever answer is shown first.
Position bias. Swap the order and require agreement.
The judge prefers the longer answer even when it adds nothing.
Verbosity bias. Rubric that rewards concision; control length.
A perfect paraphrase gets a low BLEU.
Lexical metric limit. Add chrF, embeddings or human review.
Two raters agree only about half of the time.
Unclear guidelines. Add examples per score; measure κ again.
The test sentences were also in the model's training data.
Contamination. The score is optimistic; use fresh, private data.
The summarizer wins on ROUGE but invents a refund that was never approved.
Faithfulness failure. Overlap ≠ truth; check facts with people or a rubric judge.
Beyond one number
Holistic evaluation (HELM)
Liang et al. (2022) evaluated many language models on many scenarios, and for each scenario reported seven kinds of metrics, not one:
Report a small set of metrics that expose trade-offs (quality vs cost vs risk).
Evaluate every candidate under the same conditions (data, prompts, decoding).
Say what you did not measure.
Final project minimum: at least two justified metrics, one technical and one tied to the business
or to risk, compared against a baseline on the same test set.
The metric your customer feels
Pick metrics that match the failure that hurts
NovaTel (fictional) machine-translates English support replies into Spanish. A wrong
number or a negation flipped in a reply about charges triggers a complaint; a slightly unusual word order does not.
chrF to track quality over time; BLEU only next to a baseline
Weekly human review by bilingual agents, focused on numbers and negations
KPI: complaints and repeat contacts about translated replies
≈ 6.7 h
of reviewer time per round to rate 200 replies with 2 raters at about 1 minute each
(200 × 2 × 1 min = 400 min; illustrative arithmetic).
Human evaluation has a budget: plan the sample size and how often you repeat it.
Part 3
When the output is a label
Precision, recall, thresholds, honest validation, error analysis and business KPIs.
40 synthetic transactions (12 fraud, 28 legit) with made-up model scores. Cost is in "false-alert units": a false alert costs 1.
Check · metrics
Andes Bank's fraud flagger: a missed fraud costs far more than a false alert, and the analyst team can review at most 300 alerts a day. What should drive the threshold?
Misses are expensive, so maximise recall, but alerts need people: the review capacity sets a
floor on precision (or a cap on alerts). Accuracy hides the rare class. Write the rule down before looking at test results.
Validation
Holdout, k-fold and the sealed test set
Stratified 5-fold cross-validation on the training data:
Each fold keeps the class proportions (stratification). Report the mean and the spread across folds.
Holdout: one train/validation/test split. Simple, but one split can be lucky or unlucky.
k-fold: every example is validated once; use it to choose models and hyperparameters when data are scarce.
Test set: touched once, at the end, to confirm. If you tune on cross-validation, the CV score is optimistic: keep the test set (or use nested CV).
Kohavi (1995) recommended stratified 10-fold cross-validation for model selection.
Validation · the silent killer
Data leakage: when the test is not really unseen
Leak
Example
Fix
Duplicates
The same template message appears in train and test
Deduplicate before splitting
Preprocessing
TF-IDF vocabulary or a scaler fitted on all the data
Put it inside a Pipeline
Groups
Messages from the same customer on both sides
Group-aware split (GroupKFold)
Time
Training on next month to predict last month
Split by date
Tuning
Choosing the epoch or threshold on the test set
Decide on validation; test once
Contamination
A public test set inside a pretrained model's training data
Private, fresh evaluation data
Leakage is a widespread cause of irreproducible results in applied ML (Kapoor & Narayanan, 2023).
Check · leakage
NovaTel trains an intent classifier on chat messages; many customers wrote ten or more. Messages were shuffled randomly into train and test. Test F1 is excellent, but in production it is much lower. Most likely cause?
The model partly memorised how each customer writes, and the test set rewarded that. Split by
customer (group-aware split) so that test customers are new, as they are in production.
From numbers to fixes
Error analysis: read the mistakes
1 · Sample
20+ errors from the validation set, or the top confused pairs.
How many errors per group? Fix the biggest group first.
4 · Hypothesise
One cause and one change per group.
5 · Re-evaluate
On validation (CV); confirm on test once.
In today's lab (Banking77, 10 intents)
Top confusions include pending_transfer → transfer_not_received_by_recipient: "Why hasn't my transfer gone through yet?" could be either. That is a label-definition problem more than a model problem.
Slices: also break the metric down by segment (channel, language, product, customer type). An average can hide a group where the model fails.
From offline to business
Link each model metric to a KPI
Product
Offline metric
Operational signal
Business KPI
Call summarizer
Human faithfulness score; ROUGE vs a baseline
Minutes of after-call work
Cost per call
FAQ bot
Correct-answer rate (human or validated judge)
Share of chats resolved without an agent
Cost to serve · satisfaction
Fraud flagger
Recall at the capacity-limited precision
Alerts reviewed per day
Fraud losses · analyst hours
Reply translation
chrF + human adequacy review
Replies edited by agents
Repeat contacts · complaints
Go / no-go: fix the thresholds before you look at the test set (e.g., "beats the baseline on the
offline metric and no critical errors in the human review sample"), then confirm online with a pilot or an A/B test.
Thresholds are business decisions
Put a price on each cell of the confusion matrix
In the threshold simulator, the cheapest threshold moved from 0.73 to 0.21 when a missed
fraud went from costing the same as a false alert to costing ten times more.
Same model, same data: only the cost assumption changed.
Questions for the business owner
What does one false positive cost (time, customer friction)?
What does one false negative cost (losses, regulatory risk)?
Which capacity limits the decision (reviewers, budget)?
Who signs off the threshold, and when is it reviewed?
Apply it
Choose the metrics for four products
Evaluation plan in one line each · 7 min
NovaTel call center: live EN→ES translation of chat replies.
Andes Bank: summarizer of internal committee meetings.
Banco Andino: FAQ bot on the public website.
Andes Bank: fraud flagger for card transactions.
For each: one automatic metric, one human or judge check, one business KPI and one go/no-go threshold.
Rooms of 3–4, each room takes all four. All companies are fictional.
Each room reports one product in 30 seconds.
Break
Stretch, refill your coffee, and open Kaggle for the lab.
Hands-on
Guided lab
Four small models, five metrics, one confusion matrix, all on a free CPU.
Lab · 60'
Lab bridge
What we will build
1
Perplexity of gpt2 on news, tweets and Spanish; same tokenizer vs different units
12'
2
EN→ES with opus-mt-en-es on opus_books: BLEU and chrF with sacrebleu
12'
3
Summaries with flan-t5-small on CNN/DailyMail: ROUGE, lead-3 baseline, MiniLM similarity
15'
4
Banking77 (10 intents): stratified 5-fold, confusion matrix, top confused pairs
15'
Open the notebook
Kaggle checklist: Internet on · Accelerator none (CPU is enough today).
Perplexity measures surprise under a model; compare it only with the same text and tokenizer.
BLEU is precision with a brevity penalty, ROUGE is recall-oriented, chrF works on characters: all measure overlap, not truth.
Embeddings, human raters and validated LLM judges add meaning, at a cost and with biases.
Choose the threshold with explicit costs; report k-fold mean ± spread; open the test set once.
Every metric should point to a KPI and a go/no-go rule written in advance.
Exit question
For your final project: which failure would hurt the user most, and which metric would see it?
Next session: basic AI applications with Gradio and Startti, today at 10:00 after a short break.
Its pre-work and your Startti licence were due before Saturday. Project: the group's H2 proposal canvas is due Wed 14 Oct, 23:59.
References
Sources for this session
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. ACL.
Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out (ACL workshop).
Popović, M. (2015). chrF: Character n-gram F-score for automatic MT evaluation. WMT.
Post, M. (2018). A call for clarity in reporting BLEU scores. WMT.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. ICLR.
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. EMNLP.
Zheng, L., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks.
Liang, P., et al. (2022). Holistic evaluation of language models. arXiv:2211.09110.
Jelinek, F., Mercer, R. L., Bahl, L. R., & Baker, J. K. (1977). Perplexity: A measure of the difficulty of speech recognition tasks. JASA, 62(S1), S63.
Jurafsky, D., & Martin, J. H. (2023). Speech and language processing (3rd ed. draft), ch. 3: N-gram language models.