Especialización en IA Generativa y Desarrollo de Negocios
Universidad del Rosario · Rosario GSB
Today
Agenda · 3 hours
Warm-up: transfer learning recap + your S1–S3 balance 10'
Theory: full fine-tuning vs PEFT, instruction data, hyperparameters, when to fine-tune 60'
Break 10'
Guided lab: DistilBERT on Banking77 with Trainer · LoRA on SmolLM2 60'
Session challenge (graded, individual) 40'
Kaggle today: Settings → Accelerator → GPU T4 x2 · Internet on.
Warm-up
Three questions on Session 03: what changed in the model, and what did not?
Warm-up · 10'Warm-up · Q1
In the S03 lab you trained a logistic regression on MiniLM sentence embeddings (feature extraction). Which weights changed?
Feature extraction freezes the encoder and trains a small head on its outputs. Today we
go further: full fine-tuning updates every weight, and LoRA trains a small set of new
weights while the original ones stay frozen.
Warm-up · Q2
In S03 you classified news with Qwen2.5 using 4 labelled examples in the prompt (few-shot). What happened to the model's weights?
In-context learning changes the input, not the weights: remove the examples and the
behaviour disappears. Fine-tuning bakes behaviour into the weights, so it persists without long prompts.
Warm-up · Q3
The same customer message often needs more tokens in Spanish than in English. Why does that matter when you fine-tune?
Tokenizers trained mostly on English split Spanish words into more pieces. Compute and
memory grow with sequence length, and text beyond max_length is truncated, so check your token
lengths before you choose it (you will do that in the lab).
Formative check-in · sessions 4–7
Your balance after challenges S1–S3
📬 In e-Aulas before today's session
Your grades for S1–S3, with individual feedback for each rubric component.
Compare them with your own self-assessments: where did you over- or under-estimate?
Pick one habit to change from S04 on (e.g., cite your own numbers, fill the AI log as you go).
Need to talk? Book a short conversation in office hours between sessions 4 and 7.
Project: your group's proposal canvas (H2) is due Wednesday 14 Oct, 23:59; written feedback comes back in S07.
This is formative: it has no grade of its own. Details in the course guide (diálogo formativo).
Learning objectives
By the end of this session you can…
Contrast full fine-tuning with PEFT (adapters, prompt tuning, LoRA, QLoRA) and compute LoRA's trainable parameters RAE 1
Fine-tune an encoder with Trainer and report accuracy and macro-F1 on a held-out test set RAE 1RAE 2
Adapt a small chat model to a brand voice with LoRA and compare outputs before and after RAE 1
Diagnose underfitting, overfitting and divergence from learning curves RAE 2
Decide between prompting, RAG and fine-tuning for a business case RAE 1
RAE numbers refer to the learning outcomes in the course syllabus (guía de asignatura).
Part 1
How much of the model should change?
Full fine-tuning, parameter-efficient fine-tuning, and the maths of LoRA.
Theory · 60'
Fine-tuning in one line
Start from pretrained weights, keep training on your task
Pretraining gives weights \(\theta_0\). Fine-tuning minimises the loss on your labelled data, starting from \(\theta_0\) instead of random numbers:
$$ \theta^{*} = \arg\min_{\theta} \; \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}\big(f_\theta(x_i),\, y_i\big), \qquad \theta \leftarrow \theta_0 \text{ at the start} $$
1 · General LM
Pretrain a language model on a large general corpus.
2 · Target-domain LM
Continue language modelling on text from your domain.
3 · Task head
Fine-tune a classifier, unfreezing layers gradually, with layer-specific learning rates.
ULMFiT (Howard & Ruder, 2018) made this recipe standard in NLP: with 100 labelled examples it matched training from scratch on 100× more data.
Two strategies
Full fine-tuning vs parameter-efficient fine-tuning
Full fine-tuning
PEFT (adapters, LoRA, prompt tuning…)
What trains
Every weight of the model
A small set of new weights; the original weights stay frozen
Training memory
Weights + gradients + optimizer state for all parameters
Frozen weights + gradients and optimizer state for a tiny fraction
What you store per task
A full copy of the model
A small adapter file (often a few MB)
Forgetting risk
Higher: all weights can drift
Lower, and the base can be recovered by removing the adapter
Typical use
Small encoders (DistilBERT, BERT) and plenty of compute
LLMs, many tasks on one base model, limited GPUs
Today's lab does both: full fine-tuning of DistilBERT (67 M parameters) and LoRA on SmolLM2 (135 M).
Why full fine-tuning is expensive
About 16 bytes per parameter, before activations
Training with AdamW in fp32
Bytes / parameter
Weights
4
Gradients
4
Adam first moment \(m\)
4
Adam second moment \(v\)
4
Total
16
Mixed precision also totals about 16 bytes (Rajbhandari et al., 2020). Activations come on top and grow with batch size × sequence length.
≈ 108 GB
to fully fine-tune a 6.7 B-parameter model: 6.74 × 10⁹ × 16 bytes, before activations. A free Kaggle T4 has 16 GB.
DistilBERT: 67 M × 16 B ≈ 1.1 GB, which is why we can fully fine-tune it today.
The PEFT family
Four ways to train a few weights
🔌 Adapters
Small bottleneck layers inserted in every Transformer block; only they train. Within 0.4% of full fine-tuning on GLUE with 3.6% extra parameters per task.
Houlsby et al. (2019)
🧩 Prefix & prompt tuning
Learn a few continuous "virtual tokens" prepended to the input; the model is frozen. Close to full fine-tuning for very large models.
Li & Liang (2021) · Lester et al. (2021)
🪶 LoRA
Add a trainable low-rank update \(BA\) to chosen weight matrices. Can be merged after training: no extra inference latency.
Hu et al. (2021)
🗜️ QLoRA
LoRA on top of a base model quantized to 4 bits (NF4). Fine-tuned a 65 B model on a single 48 GB GPU.
Dettmers et al. (2023)
LoRA · the maths
Freeze \(W_0\), learn a low-rank update
$$ W = W_0 + \Delta W = W_0 + BA, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d, k) $$
$$ h = W_0 x + \frac{\alpha}{r}\, B A x $$
\(W_0\) (\(d \times k\)) is frozen; only \(A\) and \(B\) train.
\(A\) starts random, \(B\) starts at zero, so \(\Delta W = 0\) at step 0: training starts exactly from the pretrained model.
\(\alpha / r\) scales the update (in the lab: \(\alpha = 16,\ r = 8\) → ×2).
LoRA with r = 8 on the query and value projections of SmolLM2-135M. Roughly what share of the model's weights is trained?
30 layers × 8 × [(576 + 576) + (576 + 192)] = 460,800 trainable weights out of ≈134.5 M,
about 0.34%. The value projection is narrower (192) because SmolLM2 shares keys and values across heads (GQA).
One base model, many products
Adapters change the economics of customization
≈ 1.8 MB
per LoRA adapter (460,800 weights × 4 bytes) vs ≈ 538 MB for a full fp32 copy of SmolLM2-135M
(134.5 M × 4 bytes). Illustrative arithmetic; the lab prints the real file size.
Andes Bank (fictional) wants five assistants: collections, onboarding, card support, complaints and
internal HR. With full fine-tuning it maintains five full models; with LoRA, one base model and five small adapters.
Serve one base and swap adapters per request.
Version, audit and roll back each adapter separately.
But: five adapters still need five evaluation sets.
Part 2
Instruction tuning and your own data
How chat models are made, what the training data looks like, and how to build a dataset you can trust.
From base model to assistant
Pretraining → SFT → preference tuning
Pretraining
Next-token prediction on huge unlabelled text. Knows language, not how to follow requests.
→
Supervised fine-tuning
Instruction → ideal response pairs written or checked by people. Learns to follow instructions and a format.
→
Preference tuning
People rank several answers; the model is optimised towards preferred ones (RLHF).
InstructGPT (Ouyang et al., 2022): labelers preferred the outputs of a
1.3 B instruction-tuned model over those of the 175 B GPT-3, despite 100× fewer parameters.
SmolLM2-135M-Instruct already went through these steps. Today we add one more, small SFT step for a brand voice.
SFT data format
From a JSON line to training tokens
1 · One example per line (brand_voice.jsonl)
{"instruction": "How do I change my PIN?",
"response": "Easy! In the Andes app, go to Cards → Security → …"}
2 · As chat messages: system, user, assistant
3 · Through the model's chat template (SmolLM2 uses ChatML):
<|im_start|>system
You are the virtual assistant of Andes Bank…<|im_end|>
<|im_start|>user
How do I change my PIN?<|im_end|>
<|im_start|>assistant
Easy! In the Andes app, go to Cards → …<|im_end|>
Loss masking: labels = −100 for the system and user tokens, so the model
learns to write answers, not to predict questions.
Building a custom dataset
Five steps, quality over quantity
1 · Collect
Real inputs: tickets, chats, emails. Cover rare but important cases.
Write a one-line definition and an exclusion rule ("not when…") per intent.
Decide a rule for messages with two intents, and when to use other.
Label: (a) "My new card never came and now I see a purchase I didn't make." (b) "You charged me for a transfer that bounced." (c) "Close my account, your fees are ridiculous." (d) "Hi, my ID is 1020304050, card lost."
Add one privacy rule for annotators.
Andes Bank is fictional. One person per room reports one rule.
Rooms of 3 (use your project groups). Write in a shared doc.
Data is the real cost
Budget for labelling, not only for GPUs
≈ 33 h
of annotator time to double-label 2,000 messages at 30 seconds per label
(2,000 × 2 × 30 s = 120,000 s). Illustrative arithmetic: measure your own time per label on a pilot of 50.
A fine-tuning run on a small model costs minutes of GPU. The dataset behind it costs weeks of expert time, and it is
the asset that is hardest for competitors to copy.
Pilot: label 50–100 examples, measure agreement, fix the guide.
Who labels? Experts are slow and expensive; crowd workers need a better guide.
Anonymize before any data leaves the organisation.
Part 3
Training that works
Hyperparameters, learning curves, overfitting and catastrophic forgetting.
Hyperparameters
The knobs, and sensible starting points
Knob
What it controls
Full fine-tuning (BERT-size)
LoRA
Learning rate
Step size of each update
2e-5 – 5e-5
Higher: ≈ 1e-4 – 1e-3
Epochs
Passes over the training set
2 – 4
More on tiny sets; watch validation
Batch size
Examples per update; memory
16 or 32
As memory allows (or gradient accumulation)
Warmup
First steps with the learning rate rising from 0
≈ 5–10% of steps
Same
Weight decay
Pulls weights towards 0 (AdamW)
0 – 0.01
Often 0
Max length
Truncation; cost grows with length
Cover ≈ 95% of your examples' token lengths
r · α · dropout · targets
LoRA capacity, scale α/r, regularisation, which matrices
—
r 4–16 · α = 2r · 0.05 · q,v (or all linear)
BERT's fine-tuning grid: batch {16, 32}, learning rate {5e-5, 3e-5, 2e-5}, epochs {2, 3, 4} (Devlin et al., 2019). LoRA ranges are common starting points, not rules.
Reading learning curves
Four shapes you will meet
📉 Underfitting
Training and validation loss both still high and falling when training stops.
Fix: more epochs, higher learning rate, bigger model or LoRA rank.
✅ Good fit
Validation loss reaches its minimum and flattens; small gap to training loss.
Keep the checkpoint with the best validation score.
📈 Overfitting
Training loss keeps falling while validation loss turns up.
Fix: fewer epochs, early stopping, more data, weight decay, smaller r.
💥 Divergence
Loss jumps, oscillates or becomes NaN; accuracy near chance.
Fix: lower the learning rate, add warmup, clip gradients.
Simulator · the tuning game
Find a good fit in as few runs as possible
Runs: 0 · example-passes used: 0
Illustrative curves from a transparent rule (see notes), not real training runs.
Verdict: press Train · best epoch: –
Small data
Overfitting: memorising 200 examples is easy
A 67 M-parameter model can memorise a few hundred sentences perfectly. Training accuracy then says nothing; only held-out data does.
Evaluate every epoch on validation (eval_strategy="epoch").
Early stopping: keep the best validation checkpoint.
Regularise: weight decay, dropout, fewer epochs, PEFT with a small r.
More or better data beats clever tricks.
Tiny validation sets are noisy: with 50 examples, one error moves accuracy by 2 points.
Three datasets, three jobs
Train
fit the weights
Validation
choose epochs, learning rate, configuration
Test
report once, at the end; never used to choose
If you pick the configuration with the best test score, the test score is no longer an honest estimate.
Catastrophic forgetting
Learning task B can erase task A
Fine-tuning pushes weights towards the new data; skills that are not in that data can degrade. Elastic weight consolidation (Kirkpatrick et al., 2017) penalises changes to weights that mattered for the old task:
\(F_i\): Fisher information (importance of weight \(i\) for task A) · \(\lambda\): how much to protect A.
Practical defences
Small learning rate and few epochs.
PEFT: the base stays frozen; switch the adapter off to recover it.
Mix some general examples into the training data (replay).
Run a "general skills" test set before and after every fine-tune.
Check · learning curves
Training loss keeps falling; validation loss has risen for the last 3 epochs; test accuracy has not been computed. What do you do?
Rising validation loss with falling training loss is overfitting. Choose with validation, report
with test. Option b turns the test set into a second validation set and makes the reported score optimistic.
Side effects are a business risk
Re-test everything after every fine-tune
NovaTel (fictional) fine-tunes its support model on 500 friendly conversations. Tone improves, but
nobody re-runs the safety tests: the model now also answers requests it used to refuse.
Qi et al. (2024) showed that fine-tuning aligned models, even on benign data, can weaken their safety behaviour.
A release checklist for fine-tuned models
Task metrics on the test set (accuracy, F1, human rating).
General skills and safety prompts: same answers as the base model?
Prompts in Spanish and English, short and long.
Model card: data, hyperparameters, known limits (S10).
Part 4
Prompt, RAG or fine-tune?
A decision guide, and hosted fine-tuning versus open-weights models.
Decision guide
Climb the ladder only when the evidence says so
1 · Prompting
Clear instructions + a few examples + an evaluation set of real cases.
Always the baseline. Stop here if it passes your evaluation.
2 · RAG
Failures come from missing or changing knowledge: policies, prices, product manuals.
Retrieve the facts at question time; update documents, not weights.
3 · Fine-tuning
Failures come from behaviour: format, tone, a narrow classification, jargon, or cost and latency at scale.
Needs labelled examples, evaluation and maintenance.
They combine: fine-tune for how to answer, RAG for what is true today.
Fine-tuning is a poor way to teach facts that change every month.
Where to fine-tune
Hosted fine-tuning API vs open-weights model
Hosted API (e.g., OpenAI)
Open weights + PEFT (today's lab)
Data format
JSONL, one chat conversation (messages) per line
Any format you convert with the chat template
Infrastructure
None: upload data, create a job
Your GPUs (Kaggle, cloud or on-premise) and code
Weights
Stay with the provider
Yours: run them anywhere the licence allows
Data leaves you?
Yes, under the provider's terms
Not necessarily
Continuity
The provider decides which models and features remain
You decide when to update or retire
Real example: OpenAI's fine-tuning guide (consulted September 2026) says the platform is being wound down:
closed to new users, with existing fine-tuned models served until their base models are deprecated. The same guide recommends
starting with about 50 well-crafted examples (minimum 10).
Check · decide
Banco Andino's assistant must quote current fees and interest rates, which change every month. The answers are fluent but often outdated. What should the team try first?
The failure is missing, changing knowledge, so the fix is in the context, not the weights.
Fine-tuning could still help with tone or format, on top of RAG.
Cost at scale
Long prompts are paid on every request
150 M → 10 M
input tokens per month for 100,000 requests if a 1,500-token few-shot prompt can be replaced by a 100-token prompt
to a fine-tuned model (illustrative arithmetic; outputs not included).
Andes Bank classifies 100,000 messages a month. A fine-tuned DistilBERT runs on one small server with no per-token bill;
a large LLM with a long prompt pays for the examples every time.
Put on the other side of the scale: labelling, training, evaluation, hosting and a person who maintains it.
Break
Open the lab in Kaggle now and turn on the GPU: the session restarts when you change the accelerator.
Hands-on
Guided lab
Full fine-tuning of DistilBERT on Banking77, then a LoRA brand voice for SmolLM2.
Individual, graded, 40 minutes in class. Submit by 23:59 today.
Challenge de la sesión · evaluado
Challenge S04: afina, compara y decide
Ejecución: afina DistilBERT en tus 10 intents de Banking77 con tu configuración A; reporta accuracy y F1 macro.
Análisis: entrena la configuración B, compara las curvas, diagnostica si hay sub- o sobreajuste (o ninguno) y elige con validación; evalúa en test una sola vez.
Explica y decide: ¿afinamiento, RAG o prompting para tu dominio asignado?
Crítica a la IA: encuentra, explica y corrige 4 errores en unos TrainingArguments propuestos por un asistente.
Rúbrica: ejecución 40% · análisis 40% · reflexión y registro de IA 20%.
IA permitida con registro; micro-sustentaciones: si no explicas tu entrega, máximo 3.0.
Entrega: Archivo → Descargar notebook → e-Aulas como S04_<codigo>.ipynb, 23:59.
Full fine-tuning trains every weight (≈16 bytes per parameter); PEFT trains a tiny fraction.
LoRA: \(W = W_0 + BA\), rank \(r\) sets capacity; adapters are small, swappable and mergeable.
Choose with validation, report with test once; read the curves before trusting a number.
Prompt first, RAG for changing knowledge, fine-tune for behaviour, and re-test after every fine-tune.
Exit question
For your project: what data would you fine-tune on, how many examples could you label in a week, and what is your baseline?
Next session: Evaluating generative models (BLEU, ROUGE, chrF, human evaluation).
Saturday has S05 (7:00) and S06 (10:00): do the pre-work for both and activate your Startti licence before Saturday 7:00. Draft your group's proposal canvas (H2, due Wed 14 Oct).
References
Sources for this session
Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for text classification. ACL.
Houlsby, N., et al. (2019). Parameter-efficient transfer learning for NLP. ICML.
Li, X. L., & Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. ACL.
Lester, B., Al-Rfou, R., & Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. EMNLP.
Hu, E. J., et al. (2021). LoRA: Low-rank adaptation of large language models. arXiv:2106.09685 (ICLR 2022).
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. NeurIPS.
Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.
Zhou, C., et al. (2023). LIMA: Less is more for alignment. NeurIPS.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT. NAACL.
Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., & Vulić, I. (2020). Efficient intent detection with dual sentence encoders. NLP4ConvAI Workshop, ACL.
Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS, 114(13).
Gebru, T., et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12).
Qi, X., et al. (2024). Fine-tuning aligned language models compromises safety, even when users do not intend to! ICLR.
Rajbhandari, S., et al. (2020). ZeRO: Memory optimizations toward training trillion parameter models. SC20.
Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. ICLR.