Especialización en IA Generativa y Desarrollo de Negocios
Universidad del Rosario · Rosario GSB
Today
Agenda · 3 hours
Warm-up: three questions on ethics and sustainability (S10) 10'
Theory: the improvement loop, trustworthy experiments, robustness & efficiency, production 60'
Break 10'
Guided lab: one full turn of the loop on Banking77 60'
Session challenge (graded, individual) 40'
Project: your improvement iteration (H4) went in with the final deliverable on Friday 23 Oct. Today you review it; nothing project-related is due.
Warm-up
Three questions on last session: fairness metrics, carbon footprint and counterfactual tests.
Warm-up · 10'Warm-up · Q1
A credit model has the same true positive rate for men and women among applicants who repay, but approves women less often overall. Which criterion does it satisfy?
Equal opportunity asks for equal true positive rates among the people who deserve the positive
outcome (Hardt, Price & Srebro, 2016). Demographic parity asks for equal approval rates overall, which fails here.
When base rates differ between groups, the two criteria can conflict.
Warm-up · Q2
The same fine-tuning job (same GPU, hours and PUE) runs in two regions. Region A's grid emits 4× more CO₂ per kWh than region B's. Emissions in A are…
Emissions ≈ energy (kWh) × PUE × grid carbon intensity (kg CO₂e/kWh). Same energy and PUE,
4× the intensity → about 4× the emissions. That is why a hydro-heavy grid such as Colombia's changes the footprint
of the same job.
Warm-up · Q3
In S10 you swapped names (“María” ↔ “Juan”) in otherwise identical sentences and compared sentiment scores. In testing vocabulary, that is…
Counterfactual fairness tests are invariance (INV) tests in CheckList terms (Ribeiro et al., 2020).
Today we apply the same idea to typos, casing and irrelevant sentences, and report pass rates.
Learning objectives
By the end of this session you can…
Run an improvement loop: error analysis → hypothesis → targeted fix → decision, with an experiment log RAE 2
Find likely label errors and choose between data-centric and model-centric fixes RAE 2
Tune hyperparameters with Optuna without leaking the test set RAE 2
Design behavioural tests and recognise distribution shift RAE 2RAE 6
Trade accuracy for size and latency (quantization, distillation) and decide under constraints RAE 6
Plan monitoring and safe rollout for models and LLM applications RAE 6
Part 1
The improvement loop
Models are never done: from errors to hypotheses to targeted fixes.
Theory · 60'
Why models decay
The model is the small box
Configuration
Data collection
Data verification
Feature extraction
ML code
Serving infrastructure
Resource management
Analysis tools
Monitoring
“Only a small fraction of real-world ML systems is composed of the ML code” (Sculley et al., 2015).
Debt hides around it: CACE (changing anything changes everything), hidden feedback loops, undeclared consumers, unstable data dependencies.
The data stages and the feedback loops are what make ML different from classic software. Huyen (2022) calls ML systems iterative by nature.
Simulator · the improvement loop
Click a stage to see what good practice looks like
or click / Tab + Enter on a stage
Error analysis
From 100 errors to a ranked list of hypotheses
Category (Andes Bank, 100 misrouted messages)
Count
Example
Hypothesis
Confused pair: card arrival vs delivery estimate
34
“Is my card on its way?”
Too few distinguishing examples
Wrong label in the training data
22
“My top-up failed” → cancel_transfer
Vendor labelling errors
New product, no intent for it yet
18
“How do savings pockets work?”
Distribution shift: new intent
Typos and slang
15
“wheres my crad”
Word features miss misspellings
Two intents in one message
11
“Cancel it and block my card”
Task definition (one label only)
Illustrative counts. A category's share of the errors is the ceiling of its fix.
Method: sample errors from validation, tag and count them, estimate each fix's ceiling and cost, then attack the best ratio.
Check · prioritisation
One sprint left. Confused pair: 34% of errors, 2 days to write 200 targeted examples. Typos: 15%, 1 day to try character n-grams. Two-intent messages: 11%, 3 weeks for a new architecture. What goes first?
Rank fixes by expected gain (at most the category's share of errors) per unit of effort.
Random extra data mostly repeats what the model already knows; targeted data attacks the confusion directly.
Two ways to improve
Data-centric vs model-centric
🧠
Model-centric
Hold the data fixed, iterate on the model: features, architecture, hyperparameters, loss.
Easy to automate; limited by the quality of the labels.
🗂️
Data-centric
Hold the model fixed, iterate on the data: consistent labels and guidelines, targeted collection, removing duplicates and leakage, augmentation.
Often the bigger lever for small teams (Ng, data-centric AI).
Data cascades: 92% of the 53 AI practitioners interviewed by Sambasivan et al. (2021) reported at least one compounding data problem.
Where is your bottleneck?
In your organisation: the data or the model?
Label noise
Let the model point at suspicious labels
Get out-of-sample probabilities \(\hat{p}(x_i)\) with k-fold cross-validation: the model never scores an example it trained on.
Self-confidence of the received label \(\tilde{y}_i\): \(s_i = \hat{p}(\tilde{y}_i \mid x_i)\).
Per-class threshold \(t_j\) = mean self-confidence of the examples labelled \(j\).
Flag \(i\) when another class clears its threshold with more probability than the received one.
This simplifies confident learning (Northcutt, Jiang & Chuang, 2021); their open-source library is cleanlab.
Even benchmarks have wrong labels
Northcutt, Athalye & Mueller (2021) estimated an average of at least 3.3% label errors across the test sets of 10 popular datasets.
Review, don't auto-delete: a flagged example may be a rare but correct case.
Labelling is a supply chain
Put quality control in the vendor contract
1 in 10
wrong labels in the vendor batch of today's lab scenario (a simulation we control, not a market statistic).
Andes Bank (fictional) pays a vendor per label. Before accepting a batch:
Double-label a random sample and measure agreement.
Write guidelines with examples of the confusable intents.
Run a CL-style audit and send the flagged items back for review.
Keep a small gold set, labelled in-house, only for evaluation.
Part 2
Experiments you can trust
Tracking, honest splits and hyperparameter search.
Reproducibility
If it isn't logged, it didn't happen
Log every run
Code version (git commit) and data version (snapshot or hash)
MLflow (open source) and Weights & Biases (hosted) store runs, compare them and register models. A pandas DataFrame, as in today's lab, is the minimum.
Honest numbers
Decide on validation, report on test once
Train
The model learns here. It may be noisy.
Validation
Every decision: keep or discard a change, pick hyperparameters, choose a threshold.
Test
Touched once, at the end, to report all versions. Nothing is tuned on it.
Best of 20 seeds is not a result. Run several seeds and report mean ± standard deviation; fix the seed before you start.
Is the gain real? With ~300 validation messages, one message ≈ 0.3 points of accuracy. Use a paired bootstrap or a decision threshold written in advance.
Hyperparameter optimisation
Grid, random or Bayesian?
Grid search
9 trials, but only 3 distinct values of the parameter that matters.
Random search
Same budget, 9 distinct values: better when few parameters matter (Bergstra & Bengio, 2012).
Bayesian (TPE)
Uses past trials to model where good scores are and samples there next. Optuna's default sampler is TPE (Tree-structured Parzen Estimator).
Horizontal axis: the important hyperparameter; red ticks: the values actually tested.
Optuna · define-by-run
The search space is written as code
import optuna
def objective(trial):
analyzer = trial.suggest_categorical("analyzer", ["word", "char_wb"])
if analyzer == "word":
n = trial.suggest_int("word_ngram_max", 1, 3)
else:
n = trial.suggest_int("char_ngram_max", 3, 5)
C = trial.suggest_float("C", 0.1, 100, log=True)
model = make_model(C, n, analyzer).fit(X_train, y_train)
return macro_f1(model, X_valid, y_valid) # never the test set
sampler = optuna.samplers.TPESampler(seed=42)
study = optuna.create_study(direction="maximize", sampler=sampler)
study.optimize(objective, n_trials=20)
Define-by-run (Akiba et al., 2019): the space can branch (word vs character n-grams).
Seeded sampler: the search is reproducible.
Pruning is early stopping for trials: report intermediate scores with trial.report and stop when trial.should_prune() says the trial is hopeless.
Inside one training run, early stopping watches the validation loss with some patience.
Check · selection
After 200 Optuna trials, the best trial scores 0.912 macro-F1 on validation; the next ten are around 0.900. On test it scores 0.899. Most likely explanation?
Selecting the best of many trials on the same validation set rewards lucky noise. The honest
estimate is the untouched test set, or a nested cross-validation.
Experiments cost money
What is one point of accuracy worth?
≈ 100 h
of agent time per month saved by one point of routing accuracy, if 200,000 messages a month
are routed and each misroute costs 3 minutes (illustrative arithmetic: 2,000 × 3 min).
Before the sprint, Andes Bank (fictional) fixes:
the budget: number of trials × cost of one trial;
the stopping rule: when is "no gain" an answer?
the gain that justifies a release, in business units.
Part 3
Robust and efficient
Behavioural tests, distribution shift, and making models smaller and faster.
Behavioural testing · CheckList
Test capabilities, not only accuracy
MFT · minimum functionality
Simple cases with a known answer, like unit tests.
“The app is not good.” → NEGATIVE
INV · invariance
A label-preserving change must not change the prediction.
“Where is my card?” → “Where is my crad? Thanks!” → same intent
DIR · directional
A change must move the output in a known direction.
Adding “and nobody answered my calls” must not make a complaint less negative.
Pass rate = share of test cases that behave as expected. Ribeiro et al. (2020) found actionable failures this way even in extensively tested commercial models.
Self-check · name the test
MFT, INV or DIR?
Replace “Bogotá” with “Medellín” in a complaint: the queue must not change.
INV: a label-preserving change
“The transfer was not fast” must be classified as negative.
MFT: a simple case with a known answer (negation)
Adding “This is urgent!” must not lower the priority score.
DIR: a known direction of change
Swap two letters in one word: the intent must stay the same.
INV: robustness to typos
“How do I cancel a transfer?” must go to cancel_transfer.
MFT: a unit test for one intent
Adding “I was charged twice” must raise the probability of a billing complaint.
DIR: the output must move up
Distribution shift
The world moves after deployment
Covariate shift
\(p(x)\) changes, \(p(y \mid x)\) stays.
A new mobile app sends shorter messages with emojis.
Label shift
\(p(y)\) changes.
A mass card reissue triples “where is my card?” messages.
Concept drift
\(p(y \mid x)\) changes: the same input now needs a different answer.
A new policy moves pending transfers to the fraud team.
Detect: monitor input statistics and the distribution of predictions (fast, no labels needed), and performance on a labelled sample (slow, but the real signal). Huyen (2022, ch. 8); Gama et al. (2014).
Efficiency
Four ways to make a model cheaper to run
Quantization
Store weights (and sometimes activations) with fewer bits: int8, int4. Integer-arithmetic inference: Jacob et al. (2018); 8-bit inference for large LLMs: Dettmers et al. (2022).
Distillation
A small student learns to imitate a large teacher's softened outputs (Hinton, Vinyals & Dean, 2015). DistilBERT: 40% smaller and 60% faster than BERT, keeping 97% of its language understanding (Sanh et al., 2019).
Pruning
Remove weights, attention heads or layers that contribute little, then fine-tune to recover (e.g., Han, Mao & Dally, 2016).
Batching
Process several requests together: more throughput, but each request may wait longer. Live chat cares about latency; nightly jobs about throughput.
Quantization in one formula
Round weights to a few integer levels
Symmetric \(b\)-bit quantization of a weight tensor \(w\):
$$ s = \frac{\max_i |w_i|}{2^{b-1}-1}, \qquad q_i = \operatorname{round}\!\left(\frac{w_i}{s}\right), \qquad \hat{w}_i = s \, q_i $$
Memory of the weights
parameters × bits ÷ 8 bytes. A 7B model: 14 GB in 16-bit, 7 GB in int8, 3.5 GB in 4-bit, plus activations and cache.
When it hurts
One large outlier inflates \(s\), so small weights round to zero. LLM.int8() keeps outlier features in 16-bit (Dettmers et al., 2022); per-channel or per-group scales help too.
Dynamic quantization (today's lab) converts weights ahead of time and activations on the fly; static quantization calibrates activations; quantization-aware training learns with the rounding.
Simulator · quantization
Bits, memory and rounding error
Weights only
16-bit
int8
0.5B params
1.00 GB
–
7B params
14.0 GB
–
Levels: – · mean |error|: – (– of mean |w|) · rounded to zero: –
Latency is a contract
Measure on the hardware you will deploy on
Andes Bank's (fictional) live router must answer within 50 ms on shared CPUs. Before choosing a model, the team measures:
latency percentiles (p95, p99), not only the average;
size in memory: how many replicas fit per server;
accuracy and pass rates after compression, on the same evaluation set.
Why not trust the brochure?
int8 speed-ups depend on the CPU's integer kernels: on some ARM laptops dynamic int8 can even be slower.
Only some layers get quantized: size falls by less than 4×.
A smaller model can fail different behavioural tests.
Part 4
Improving in production
Monitoring, safe rollouts and LLMOps, then a review of your project's iteration.
Share of each intent, mean confidence, “other” rate
No
Outcomes
Messages re-routed by agents, escalations, satisfaction
Delayed
System
Latency p95, errors, cost per 1,000 messages
No
Feedback loops: if agents only label what the model routed to them, the model's mistakes shape its own future training data (Sculley et al., 2015).
One signal
Which single signal would you watch first for your project's model?
Safe rollouts
Shadow → canary → A/B test
1 · Shadow
The new model sees live traffic; its predictions are logged, not used. Compare with the old one at zero customer risk.
2 · Canary
A small share of traffic uses the new model; watch errors and latency, with a rollback ready.
3 · A/B test
Random split, one primary metric, guardrail metrics, duration and sample size fixed in advance (Kohavi, Tang & Xu, 2020).
Andes Bank (fictional) A/B test for the new router. Primary: share of messages resolved without re-routing.
Guardrails: satisfaction, latency p95, complaints. Version the model, data and config so a rollback takes minutes.
Check · rollout
The new router is 2 points better offline. What is the safest first step in production?
Shadow mode tests the new model on real inputs (and real latency) with no customer impact.
Then a canary, then an A/B test with guardrails.
LLMOps
Every prompt change is a release
Eval suite: a versioned golden set of inputs with expected behaviour: exact checks, rubrics, LLM-as-judge calibrated against humans.
Regression tests run on every change of prompt, model version, tool or retrieval index.
Safety tests from S10: prompt injection, PII leaks.
Track cost, tokens and latency per request; pin provider model versions.
Monitor traces and user feedback; failures become new test cases.
CASES = [
("How do I block my card?", "lost_or_stolen_card"),
("Cancel my last transfer", "cancel_transfer"),
# ... every past incident becomes a case
]
def test_router_does_not_regress():
hits = [route(q) == expected for q, expected in CASES]
assert sum(hits) / len(hits) >= 0.95 # last release's bar
A pytest-style regression test for an agent or a prompt-based router (S07).
Your project
Review your improvement iteration
Project groups · 8 min
The error category your iteration targeted, with a count from your baseline.
Your hypothesis and the change: was it one change, data-centric or model-centric?
Decision rule: written before the experiment? Decided on validation? Is the gain above the noise?
One behavioural test and one efficiency constraint for your demo.
The next iteration you would run, and one signal you would monitor after launch.
Nothing to resubmit: this becomes the “iteration” minute of your S12 pitch.
Rooms = project groups. Two groups share in 30 seconds.
Final project · milestone H4
Your improvement iteration, already submitted
What: §5 of the technical report: an experiment log with at least one iteration (hypothesis → change → result → decision) and a link to the code.
Rule: decide with validation; at the end, report every version on the same test set without tuning anything on it.
When: submitted with the final deliverable on Friday 23 Oct at 23:59, inside PF_G<NN>_informe.pdf.
Why it matters: a missing iteration subtracts 0.3 from the technical deliverable; in S12 you defend it with your numbers.
Improvement is a loop with a log: hypothesis → one change → validation → decision.
Look at the errors first; with noisy labels, data fixes often beat tuning.
Decide on validation, report on test once, and check that gains exceed the noise.
“Better” includes robustness, size, latency and a safe rollout.
Exit question
What is the next change you would try on your project model, and what result would make you discard it?
Next session: S12 starts at 10:00 today, after a short break: pitches (8'), demos (3') and questions (4').
No pre-work in between: your final deliverables went in on Friday 23 Oct, and the pitch and demo were rehearsed before today.
References
Sources for this session
Sculley, D., et al. (2015). Hidden technical debt in machine learning systems. NeurIPS.
Amershi, S., et al. (2019). Software engineering for machine learning: A case study. ICSE-SEIP.
Huyen, C. (2022). Designing machine learning systems. O'Reilly.
Ng, A. Data-centric AI resources. DeepLearning.AI: deeplearning.ai
Sambasivan, N., et al. (2021). “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. CHI.
Northcutt, C., Jiang, L., & Chuang, I. (2021). Confident learning: Estimating uncertainty in dataset labels. JAIR, 70.
Northcutt, C., Athalye, A., & Mueller, J. (2021). Pervasive label errors in test sets destabilize machine learning benchmarks. NeurIPS Datasets and Benchmarks.
Akiba, T., et al. (2019). Optuna: A next-generation hyperparameter optimization framework. KDD.
Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. JMLR, 13.
Casanueva, I., et al. (2020). Efficient intent detection with dual sentence encoders. NLP4ConvAI workshop, ACL (Banking77).
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. ACL.
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4).
Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531.
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT. arXiv:1910.01108.
Jacob, B., et al. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. CVPR.
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit matrix multiplication for transformers at scale. NeurIPS.
Han, S., Mao, H., & Dally, W. J. (2016). Deep compression. ICLR.
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments. Cambridge University Press.
Hardt, M., Price, E., & Srebro, N. (2016). Equality of opportunity in supervised learning. NeurIPS.
Wei, J., & Zou, K. (2019). EDA: Easy data augmentation techniques for boosting performance on text classification tasks. EMNLP-IJCNLP.