Pretrained Models and Transfer Learning

Modelos preentrenados y transferencia de aprendizaje

Desarrollo y Evaluación de Modelos Propios

Especialización en IA Generativa y Desarrollo de Negocios

Today

Agenda · 3 hours

Warm-up: three questions on training generative models (S02) 10'
Theory: why pretraining works, Transformers, tokens, transfer strategies 60'
Break 10'
Guided lab: tokenizers, BERT, embeddings + logistic regression, zero/few-shot LLM 60'
Session challenge (graded, individual) + project groups (H1) 40'

Warm-up

Three questions to reactivate Session 02: the training loop, overfitting and VAEs.

Warm-up · 10'

Warm-up · Q1

Which order is correct for one training step in PyTorch?

PyTorch accumulates gradients by default, so each step clears them, computes the loss, back-propagates and only then updates the weights. Forgetting zero_grad() was one of the planted bugs in the S02 challenge.

Warm-up · Q2

Training loss keeps falling, but validation loss starts rising after epoch 6. What is the best reading?

A widening gap between training and validation loss is the signature of overfitting. Early stopping keeps the checkpoint with the lowest validation loss; dropout, weight decay and more data also help.

Warm-up · Q3

In a VAE, what usually happens when you increase β, the weight of the KL term?

The loss is reconstruction + β · KL. A larger β pushes the encoder's distribution towards $\mathcal{N}(0, I)$, which makes sampling from the prior work better at the cost of reconstruction detail.

Learning objectives

By the end of this session you can…

Explain why self-supervised pretraining produces knowledge you can reuse RAE 1
Describe self-attention and contrast BERT, GPT and T5 RAE 1
Measure how tokenization changes the cost of Spanish text RAE 3
Build a classifier on frozen embeddings and compare it with zero- and few-shot prompting RAE 1 RAE 3
Choose a transfer strategy and a model (size, licence, language) for a business case RAE 3

RAE numbers refer to the learning outcomes in the course syllabus (guía de asignatura).

Part 1

Why pretraining works

Labels from the data itself, at a scale no company could label by hand.

Theory · 60'

Self-supervised learning

The data labels itself

Next-token prediction (GPT)

The client paid the → ?

Every position of every text is a free training example: $\;p(x_1,\dots,x_T) = \prod_t p(x_t \mid x_{<t})$.

Masked-token prediction (BERT)

The client [MASK] the invoice → paid

Hide some tokens and recover them using the context on both sides.

BERT (Devlin et al., 2019)	BooksCorpus + English Wikipedia: about 3.3 billion words
GPT-3 (Brown et al., 2020)	175 billion parameters trained on 300 billion tokens

What did it learn?

Ask BERT to fill in the blank

Outputs of the fill-mask pipeline, rounded; you reproduce them in Part 2 of the lab. *Full sentence: “… would be back in five minutes.”
Input to bert-base-uncased	Top predictions (probability)	What it shows
The bank approved my [MASK] application.	loan 0.50 · bank 0.04 · initial 0.03	Domain knowledge
Paris is the [MASK] of France.	capital 0.997	World facts
The nurse said [MASK] would be back soon.*	she 0.77 · they 0.10 · he 0.08	Social bias from the data
The engineer said [MASK] would be back soon.*	he 0.62 · they 0.23 · she 0.06	Social bias from the data
El banco aprobó mi [MASK] de crédito.	##o 0.49 · casa 0.05	No Spanish: wrong language

Your corpus: what could a model learn from your organisation's unlabelled text (emails, tickets, contracts)? One line in the chat.

Transfer learning

Pretrain once, reuse many times

Pretraining

Huge unlabelled corpus, self-supervised. Weeks of compute, done once by a lab.

→

Pretrained model

General representations of language: a starting point published on the Hub.

→

Adaptation

Your task and your data: prompting, a small head, or fine-tuning.

Transfer learning: reuse what a model learned on one task (predicting tokens) to learn another task (classifying tickets) with far less labelled data.

Why it helps: with 16 labels per class you cannot learn language, but you can learn a boundary on top of a model that already knows it.

Who pays for what

Someone else paid for pretraining

Andes Bank (fictional) wants to route complaint emails. It will not pretrain a model: it downloads a pretrained encoder and adapts it.

What remains on the bank's bill: labelling examples, evaluation, hosting, monitoring and people.

But you also inherit…

the language of the corpus (an English model cannot read Spanish emails);
its biases (the nurse/engineer example);
its licence and the terms of its training data;
its knowledge cut-off: nothing after the corpus was collected.

Part 2

Transformer essentials

Tokens, embeddings and attention: the engine inside BERT, GPT and T5.

The pipeline

From text to contextual vectors

1 · Tokens

A tokenizer splits text into pieces from a fixed vocabulary.

theref##undarrived

2 · Ids

Each token becomes an integer index.

19962541686303369

3 · Embeddings

Each id looks up a learned vector (768 numbers in BERT-base), plus a position vector.

4 · Transformer blocks

Attention mixes information between tokens, layer after layer (12 in BERT-base).

Output: one contextual vector per token. “bank” gets a different vector in “river bank” and in “bank loan”.

Positional information: attention on its own ignores word order, so the model adds position vectors: fixed sinusoids in the original Transformer, learned in BERT and GPT-2.

Self-attention

Every token asks: who is relevant to me?

$$ \text{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V $$

Query, key, value

Three learned projections of each token vector: $Q = XW_Q$, $K = XW_K$, $V = XW_V$.

Scores → weights

$QK^{\top}$ scores every pair of tokens; softmax turns each row into weights that sum to 1.

Why $\sqrt{d_k}$?

Dot products grow with the dimension $d_k$. Dividing keeps softmax from saturating, which would leave tiny gradients.

The output for each token is a weighted average of the value vectors of all tokens it attends to (Vaswani et al., 2017).

Simulator · attention map

Where does “it” look?

Causal mask (GPT-style decoder)

Hover or tap a word: its row of weights lights up and the sentence is shaded by attention.

–

Hand-made toy weights for teaching, not extracted from a real model. Each row is a softmax over its keys, so it sums to 1.

Multi-head attention

Many heads, many layers

$$ \text{MultiHead}(X) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)\, W^{O} $$

Each head has its own $W_Q, W_K, W_V$, so it can specialise (coreference, syntax, position…).
BERT-base: 12 layers × 12 heads, hidden size 768, so $d_k = 768 / 12 = 64$.
Each block: attention → add & normalise → feed-forward network → add & normalise.

Encoder vs decoder

Encoder: every token attends to every token (bidirectional). Good for understanding: classification, extraction, embeddings.

Decoder: a causal mask hides future tokens. Good for generating text one token at a time.

Encoder–decoder: the decoder also attends to the encoder's outputs (cross-attention).

Three pretrained families

BERT, GPT and T5

Devlin et al. (2019) · Radford et al. (2019) · Brown et al. (2020) · Raffel et al. (2020).
	BERT	GPT (GPT-2, GPT-3)	T5
Architecture	Encoder	Decoder	Encoder–decoder
Pretraining	Masked LM: 15% of tokens selected and predicted (+ next-sentence prediction)	Causal LM: predict the next token	Span corruption: rebuild masked spans of text
How you use it	The [CLS] vector or pooled token vectors + a small head	Prompting: zero-shot, few-shot (in-context)	Every task as text-to-text: summarize: …
Size	110M (base) · 340M (large)	up to 1.5B (GPT-2) · 175B (GPT-3)	up to 11B

Check · attention

A GPT-style model reads “The customer returned the phone because it was broken”. When it computes the vector for “it”, which tokens can it attend to?

The causal mask sets the scores of future positions to $-\infty$ before the softmax, so their weights are 0. A BERT-style encoder would see all nine tokens.

Tokenization

WordPiece and BPE: sub-words, not words

WordPiece · BERT

Vocabulary of 30,522 pieces; ## marks a continuation. bert-base-uncased also lowercases and strips accents.

ref##und (“refund”)

re##em##bol##sar##em##os (“reembolsaremos”)

Byte-level BPE · GPT-2

Vocabulary of 50,257 pieces built by merging frequent byte pairs; Ġ marks a leading space, and accented letters cost extra bytes.

Ġdays

ĠdÃŃas (“días”)

Frequent words in the training corpus get one token; rare words, and most Spanish words for English-centric tokenizers, get split into several. BPE: Sennrich, Haddow & Birch (2016).

Simulator · token cost

The Spanish token premium

Tokenizer

Words per request –

Requests / month –

Price per 1K tokens –

Ratios measured on the lab's 8 parallel EN/ES sentences. The price is illustrative: use your provider's current price.

Monthly cost · EN – · ES – · premium –

Budgeting in tokens

Measure your tokenizer on your own text

2.2×

tokens for the Spanish version of the same 8 sentences with GPT-2's tokenizer, vs 1.5× with Qwen2.5's (lab, Part 1). Your texts will give different ratios.

NovaTel (fictional) budgets a Spanish support assistant from an English price example. Tokens drive three lines of the business case:

Cost: APIs bill per input and output token.
Context: fewer Spanish words fit in the same window.
Latency: each output token is one more decoding step.

Part 3

Transfer strategies

From a prompt with no examples to continuing pretraining on your own data.

The transfer spectrum

Six ways to reuse a pretrained model

Data sizes are rough orders of magnitude for classification, not rules. Domain-adaptive pretraining: Gururangan et al. (2020).
Strategy	What changes	Labelled data	Compute
Zero-shot	Only the instruction in the prompt	None (a test set to evaluate)	Inference only
Few-shot (in-context)	A few solved examples in the prompt; weights frozen	A handful per class	Inference, longer prompts
Feature extraction	A small head (e.g., logistic regression) on frozen embeddings	Dozens to hundreds per class	CPU is enough
Partial fine-tuning	Top layers + head	Hundreds+ per class	GPU helps
Full fine-tuning	All weights (or LoRA adapters, S04)	Hundreds to thousands per class	GPU
Domain adaptation	Continue self-supervised pretraining on in-domain text, then fine-tune	Much unlabelled text + labels	GPU, hours or more

In-context learning

Zero-shot and few-shot prompting


messages = [
  {"role": "system", "content": "Classify the news article. "
   "Answer with one label: World, Sports, Business, Sci/Tech."},
  # few-shot: solved examples as previous turns
  {"role": "user", "content": "Article: Stocks rallied after…"},
  {"role": "assistant", "content": "Business"},
  {"role": "user", "content": "Article: " + new_article},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True)

No weight updates: the examples live only in this prompt (Brown et al., 2020).
Output is free text: parse it into a label and count failures as unknown.
Results depend on the choice, order and format of the examples (Zhao et al., 2021).
Every example is paid for in tokens and latency, on every request.

Feature extraction

Frozen encoder + a small head


@torch.no_grad()
def embed(texts):
    enc = tok(texts, padding=True, truncation=True,
              return_tensors="pt")
    h = encoder(**enc).last_hidden_state    # (batch, tokens, 384)
    mask = enc["attention_mask"].unsqueeze(-1)
    e = (h * mask).sum(1) / mask.sum(1)     # mean pooling
    return F.normalize(e, dim=1).numpy()

clf = LogisticRegression(max_iter=1000).fit(embed(X_train), y_train)

all-MiniLM-L6-v2: a 6-layer encoder tuned for sentence similarity (Reimers & Gurevych, 2019): 384 numbers per text.
Compute embeddings once; the head trains in seconds on a CPU.
Predictions are fast and deterministic, and the head is easy to audit.
The encoder never learns your task: accuracy has a ceiling.

Changing the weights

Fine-tuning and domain adaptation

Partial fine-tuning

Unfreeze the top layers and the head. Cheaper, less risk of damaging general knowledge. Gradual unfreezing: Howard & Ruder (2018).

Full fine-tuning

Update every weight with a small learning rate. Usually the best accuracy with enough labels; risk of catastrophic forgetting. Session 04.

Domain adaptation

Continue masked-LM pretraining on unlabelled in-domain text (clinical notes, contracts), then fine-tune. Gururangan et al. (2020) report consistent gains from this second pretraining phase.

Same data, same split: compare strategies on the same held-out test set, or the comparison means nothing.

Check · labelled data

You train logistic regression on frozen embeddings with 16, 64, 256 and 1,024 examples per class. What shape do you expect for test accuracy?

Learning curves usually rise quickly and then flatten: the frozen encoder sets a ceiling. In the challenge you measure your own curve and decide whether labelling more data is worth it.

Apply it

Pick a transfer strategy for four scenarios

Choose and justify · 7 min

Scenario	Labelled data	Budget · latency
NovaTel: tag app reviews into 5 topics	None	Very low, no ML team · overnight batch
Andes Bank: 2 M card transactions/month into 20 categories	20,000	Moderate · < 20 ms each, on-premise
Clínica Serena: Spanish clinical notes with in-house abbreviations → 6 services	1,500 + 150,000 unlabelled	GPU available · on-premise
Río Claro (media): news into 4 sections	50 per section	Small · hourly batch

For each: a strategy, one reason, and the first metric you would check. All companies are fictional.

Sit with possible project partners: groups of 3 are registered today.

[0:56 · 7 min + 2 min debrief] Rooms of 3 (suggest they sit with potential project partners). Suggested answers for the debrief (discuss, do not just announce): NovaTel → zero-shot with an LLM, then label ~100 reviews to evaluate; later embeddings + LR. Andes Bank → fine-tune a small encoder (or embeddings + LR as a first baseline): plenty of labels, strict latency and volume rule out an LLM per transaction; on-premise. Clínica Serena → domain-adaptive pretraining of a Spanish encoder (e.g., BETO) on the 150,000 notes, then fine-tune on the 1,500 labels; data stays in the clinic. Río Claro → feature extraction with a multilingual encoder + logistic regression (50 per class is enough for news topics, as the lab shows in English); few-shot as a baseline.

Choosing a model

Read the model card before the leaderboard

Question	Where to look
Size: does it fit my hardware and latency?	Parameters × bytes per parameter
Licence: can I use it commercially?	Card metadata + licence text
Language: was it trained on Spanish?	Training data section
Task fit: encoder, decoder, instruct?	Architecture, intended uses
Limits: known biases and risks	Limitations section

Model cards: Mitchell et al. (2019).

Memory arithmetic (weights only)

32-bit floats: 4 bytes per parameter · 16-bit: 2 bytes.

BERT-base, 110M → ≈ 0.44 GB in fp32
Qwen2.5-0.5B, ≈ 0.5B → ≈ 2 GB in fp32, ≈ 1 GB in fp16

Licences seen today: Apache-2.0 (BERT, MiniLM, Qwen2.5-0.5B), MIT (GPT-2). Some licences forbid commercial use or add usage policies.

Models for Spanish

Spanish and multilingual options

BETO

BERT-base-sized encoder trained on Spanish text by the Universidad de Chile; beat multilingual BERT on most tasks of its Spanish benchmark (Cañete et al., 2020).

dccuchile/bert-base-spanish-wwm-cased

MarIA · RoBERTuito

MarIA: RoBERTa and GPT-2 models trained on web text archived by the National Library of Spain (Gutiérrez-Fandiño et al., 2022). RoBERTuito: Spanish social-media text (Pérez et al., 2022).

Multilingual

XLM-RoBERTa covers 100 languages (Conneau et al., 2020). Multilingual chat LLMs such as Qwen2.5 handle Spanish prompts and tokenize it more efficiently than GPT-2.

Rule of thumb: for Spanish classification, compare a Spanish encoder, a multilingual encoder and an LLM baseline on your test set. Check each licence, including the terms of the training data.

A decision path

Start cheap, measure, then climb

Day 1

Zero-/few-shot baseline on 50–100 labelled test cases.

→

Week 1

Embeddings + logistic regression: cheap, fast, auditable.

→

Only if needed

Fine-tune or adapt the domain when the gap justifies the cost.

Banco Andino (fictional) should ask of every step: how many points of accuracy did it buy, at what cost per 1,000 predictions, and with what latency? The lab's final table answers those three questions on real numbers.

Check · model choice

Andes Bank finds a Spanish model with the best benchmark score for its task. Which line in the model card should stop the rollout until legal reviews it?

A non-commercial licence rules out use in a bank's product, whatever the score. Size and sequence length are engineering constraints; the licence is a hard constraint.

Break

Stretch, and start messaging possible project partners: groups of 3 are registered today.

Hands-on

Guided lab

Four ways to reuse pretrained models, compared on the same news test set.

Lab · 60'

Lab bridge

What we will build

1	Tokenizers: BERT vs GPT-2 (and Qwen2.5) on parallel EN/ES sentences, cost of the ES/EN ratio	12'
2	Fill-mask with BERT: knowledge, bias, and what [CLS] and [SEP] are	8'
3	Feature extraction: MiniLM embeddings + logistic regression on AG News	15'
4	Zero- and few-shot classification with Qwen2.5-0.5B-Instruct	15'
5	Comparison table: accuracy, labels used, latency · project groups	10'

Open the notebook

Kaggle checklist: Internet on · Accelerator GPU T4 x2 recommended for Part 4 (CPU also works, just slower).

Open in Kaggle Colab

Final project · milestone H1

Form your group of 3 today

Groups of 3, ideally mixing business and technical profiles.
One member uploads the group registration to e-Aulas as PF_H1_<codigo> (the uploader's student code) by 23:59 today.
The registration lists members and profiles, a communication channel, where the code will live, a 1–2 line idea and working agreements.
Missing a milestone deducts 0.3 from the technical deliverable. Anyone without a group at the deadline is assigned to one.

Suggested roles

Product & business: problem, users, KPI, ROI, pitch.

Data & evaluation: data, splits, metrics, errors, ethics.

Modelling & demo: training, iterations, Gradio app or Startti agent.

Final-project brief and H1 template

Session challenge

Individual, graded, 40 minutes in class. Submit by 23:59 today.

Challenge de la sesión · evaluado

Challenge S03: ¿cuántos datos necesitas?

Ejecución: embeddings + regresión logística con tu k_train asignado y curva de aprendizaje (16, 64 y 256 ejemplos por clase).
Análisis: few-shot con Qwen2.5-0.5B y tus n_shots sobre tu corte personal de 40 noticias; compara exactitud y latencia.
Explica y decide: estrategia de transferencia y modelo para tu caso de negocio.
Crítica a la IA: encuentra, explica y corrige 3 errores sobre atención y transferencia.

Rúbrica: ejecución 40% · análisis 40% · reflexión y registro de IA 20%.
IA permitida con registro; micro-sustentaciones: si no explicas tu entrega, máximo 3.0.
Entrega: Archivo → Descargar notebook → e-Aulas como S03_<codigo>.ipynb, 23:59.

Abrir en Kaggle Colab

Además: registro del grupo PF_H1_<codigo> hoy, 23:59.

Wrap-up

Key takeaways

Self-supervised pretraining turns unlabelled text into reusable knowledge, biases included.
Attention lets every token weigh every other token; the causal mask makes a decoder.
Tokens are the unit of cost: measure your tokenizer on Spanish text.
Climb the transfer spectrum only as far as your data, budget and latency justify.

Exit question

For your project idea: which transfer strategy would you try first, and what number would make you move to the next one?

Next session: fine-tuning (full fine-tuning, LoRA, chat templates) on a Kaggle GPU. Pre-work: submit the challenge and the H1 group registration by 23:59; check your Kaggle GPU quota.

References

Sources for this session

Vaswani, A., et al. (2017). Attention is all you need. NeurIPS. arXiv:1706.03762.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL.
Radford, A., et al. (2019). Language models are unsupervised multitask learners. OpenAI.
Brown, T. B., et al. (2020). Language models are few-shot learners. NeurIPS.
Raffel, C., et al. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
Cañete, J., Chaperon, G., Fuentes, R., Ho, J.-H., Kang, H., & Pérez, J. (2020). Spanish pre-trained BERT model and evaluation data. PML4DC workshop, ICLR.
Jurafsky, D., & Martin, J. H. Speech and Language Processing (3rd ed. draft), chapters on Transformers and large language models. web.stanford.edu/~jurafsky/slp3
Tunstall, L., von Werra, L., & Wolf, T. (2022). Natural Language Processing with Transformers. O'Reilly.
Alammar, J. (2018). The illustrated transformer. jalammar.github.io/illustrated-transformer

Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. ACL.
Petrov, A., La Malfa, E., Torr, P., & Bibi, A. (2023). Language model tokenizers introduce unfairness between languages. NeurIPS.
Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for text classification. ACL.
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. EMNLP.
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining. ACL.
Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. ICML.
Mitchell, M., et al. (2019). Model cards for model reporting. FAT*.
Conneau, A., et al. (2020). Unsupervised cross-lingual representation learning at scale. ACL.
Gutiérrez-Fandiño, A., et al. (2022). MarIA: Spanish language models. Procesamiento del Lenguaje Natural, 68.
Pérez, J. M., Furman, D. A., Alonso Alemany, L., & Luque, F. (2022). RoBERTuito: A pre-trained language model for social media text in Spanish. LREC.