Introduction to Generative and Pretrained Models

Introducción a modelos generativos y preentrenados

Desarrollo y Evaluación de Modelos Propios

Especialización en IA Generativa y Desarrollo de Negocios

Today

Agenda · 3 hours

Warm-up: who is in the room + two AI-literacy questions 10'
Theory: generative models, model families, build vs buy, tools 60'
Break 10'
Guided lab on Kaggle: PyTorch, Keras and Hugging Face 60'
Session challenge (graded, individual) 40'

Every session follows this same five-block structure.

Warm-up

Who is in the room, and what do we already know about how AI assistants work?

Warm-up · 10'

Warm-up · Diagnostic

Which best describes your role today?

No right answer. Mixed profiles are an asset: the final project is done in groups of 3, formed in Session 03, and the best groups combine business and technical people.

What do you already delegate?

Type one task you already hand to an AI assistant (ChatGPT, Claude, Copilot…) at work.

Warm-up · Q1

When a chat assistant writes an answer, what is the underlying model doing at each step?

A language model writes one token at a time: it scores every possible next token, turns the scores into probabilities and picks one. Products may add search or tools on top (RAG, agents), but the core step is next-token prediction.

Warm-up · Q2

You ask the same question twice and get two different answers. What is the most likely reason?

With sampling (temperature above 0) the model can pick a different token each time, and one different word changes everything after it. Chatting does not change the weights: training is a separate, offline process.

Learning objectives

By the end of this session you can…

Contrast discriminative and generative models; name six model families RAE 1
Explain how temperature and top-k shape the text a model generates RAE 1
Run pretrained models with PyTorch, Keras and Hugging Face on Kaggle RAE 1
Place a business case on the adaptation spectrum and justify it RAE 4
Spot opportunities for generative AI in your own sector RAE 5

RAE numbers refer to the learning outcomes in the course syllabus (guía de asignatura).

Part 1

What is generative AI?

From predicting labels to producing new data.

Theory · 60'

Two kinds of models

Discriminative vs generative

🏷️

Discriminative

Learns $p(y \mid x)$: the label given the input.

“Is this email spam?” · credit scoring · churn prediction · sentiment

Output: a label or a score.

✨

Generative

Learns $p(x)$ or $p(x, y)$: how the data itself is distributed.

Sample new data $x \sim p(x)$: a new email, image, melody or function.

Output: new content.

Bayes' rule links them: $p(y \mid x) = p(x \mid y)\,p(y) / p(x)$. A generative model can also classify (Session 08); a discriminative model cannot generate.

One idea, many modalities

Learn the distribution, then sample from it

📝 Text

Chat assistants, summaries, translation, drafting emails.

🖼️ Images

Text-to-image generation, editing, product mock-ups (e.g., Stable Diffusion; Rombach et al., 2022).

🔊 Audio

Synthetic voices (text-to-speech), music, sound effects.

💻 Code

Code completion and generation from natural language (e.g., Codex; Chen et al., 2021).

Usually the model is conditioned on an input: $p(x \mid \text{prompt})$.

Inside a language model

Writing one token at a time

Text is split into tokens (words or word pieces). An autoregressive model factorises the probability of a text:

$$ p(x_1, \dots, x_T) = \prod_{t=1}^{T} p(x_t \mid x_{<t}) $$

prompt → model → logits → softmax(T) → top-k → sample → next token ↺

Softmax with temperature

$ p_i = \dfrac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} $

Top-k

Keep only the $k$ most likely tokens, renormalise, then sample. $T \to 0$ or $k = 1$ is greedy decoding.

Simulator · next-token sampling

Temperature & top-k at Andes Bank

Prompt: “Thank you for contacting Andes Bank. Your request has been …”

Temperature 1.00

Top-k 8

Most likely: – · Entropy: – bits · Tokens in play: –

Check · decoding

Andes Bank's assistant must answer regulatory questions with consistent wording. Best starting point?

Low temperature concentrates probability on the most likely tokens, so answers are consistent. But consistent is not the same as correct: factual accuracy needs grounding in the bank's documents (RAG) and evaluation (Session 05).

Customer-facing risk

Fluent is not the same as correct

NovaTel (a fictional telecom) lets a model draft replies to billing complaints. The model optimises plausibility, not truth: it can invent a refund policy that sounds exactly right.

What a manager should ask

Which answers create commitments (refunds, prices, legal terms)? Keep a human in the loop for those.
Which settings for which task? Low temperature for policy answers, higher for ideas.
Did we sample several outputs per prompt before deciding? One good demo proves little.

Part 2

Model families

Six architectures behind today's generative AI, and the idea that unites them: pretrain once, adapt many times.

Three Transformer families

Same building block, different objectives

All three are built from the Transformer (Vaswani et al., 2017). BERT: Devlin et al. (2019) · GPT-2: Radford et al. (2019) · T5: Raffel et al. (2020).
Family	Pretraining objective	Context it sees	Typical use	Examples
Autoregressive decoder-only	Predict the next token	Left side only	Generation, chat, code	GPT-2, GPT-3, SmolLM2, Qwen2.5
Autoencoding encoder-only	Recover masked tokens (masked language modelling)	Both sides	Classification, extraction, embeddings	BERT, DistilBERT
Seq2seq encoder–decoder	Rebuild corrupted text; map an input text to an output text	Encoder: both sides · decoder: left	Translation, summarisation	T5, FLAN-T5, MarianMT

In today's lab you will use an autoencoding classifier, BERT's masked-word predictor and an autoregressive chat model.

Beyond text

VAEs, GANs and diffusion models

VAE

An encoder compresses $x$ into a latent code $z$; a decoder rebuilds $x$. Trained with reconstruction loss plus a KL term. Sample a new $z$ → new data.

Kingma & Welling (2014) · you train one in S02

GAN

A generator forges samples; a discriminator tries to tell real from fake. They improve by competing. Sharp images, but training can be unstable.

Goodfellow et al. (2014)

Diffusion

Learns to reverse a gradual noising process: generation starts from pure noise and removes it step by step. Used in many text-to-image systems.

Ho, Jain & Abbeel (2020)

Reference text: Goodfellow, Bengio & Courville (2016), Deep Learning, ch. 20 covers deep generative models in depth.

Self-check · name the family

Which model family is this?

Predicts the next token using only the text on its left.

Autoregressive (GPT-style decoder)

Fills in [MASK] tokens using context on both sides.

Autoencoding (BERT-style encoder)

Reads a whole input, then writes a new text such as a translation.

Seq2seq (T5-style encoder–decoder)

Two networks: one forges samples, the other spots fakes.

GAN: generator vs discriminator

Compresses data into a smooth latent space and decodes new samples from it.

VAE: variational autoencoder

Starts from pure noise and removes it step by step.

Diffusion model

Check · model families

Andes Bank wants to route 10,000 complaint emails a month into 12 categories. Most natural starting point?

Routing is classification, where encoders are strong and cheap to run. A decoder LLM can also classify by prompting (zero-shot), which makes a good baseline; Session 08 compares both.

Foundation models

Pretrain once, adapt many times

Broad data

Web text, books, code; mostly unlabelled.

→

Pretraining

Self-supervised, at scale: next token or masked token.

→

Adaptation

Prompting, RAG, fine-tuning → many downstream tasks.

A foundation model is trained on broad data, usually with self-supervision at scale, and can be adapted to a wide range of tasks (Bommasani et al., 2021).

Emergence: capabilities appear that nobody trained for explicitly, such as following instructions given only in the prompt.

Homogenization: many applications share the same few models, so they also share their defects.

Size matters, differently

The models you will actually run

≈1,300×

fewer parameters in SmolLM2-135M (today's lab) than in GPT-3, which has 175 billion (Brown et al., 2020). Arithmetic: 175,000 M ÷ 135 M ≈ 1,296.

Small open models run on a free Kaggle GPU, sometimes on a laptop CPU.
They are weaker, but cheap, fast, private and fully under your control.
Parameter count is not quality: it drives cost, latency and where the model can run.

SmolLM2: Allal et al. (2025).

Vendor dependence

Homogenization is a supplier risk

Andes Bank (fictional) builds its chatbot, its email summaries and its fraud-alert explanations on the same foundation model from one provider.

One provider update, or one deprecated model version, changes all three products at once.

Questions to ask any model provider

How are model versions pinned, updated and retired?
Is our data stored, used for training, or processed abroad?
Can we test a new version on our own cases before switching?

Your own evaluation set is your insurance (Sessions 05 and 11).

Part 3

Own models vs APIs

Where on the adaptation spectrum should your use case sit?

The adaptation spectrum

Four ways to make a model yours

1 · Prompting

Change: instructions and examples in the prompt.

Needs: a good prompt and test cases. Hours to days.

2 · RAG

Change: the context: retrieve your documents and add them to the prompt.

Needs: clean, searchable documents. Days to weeks.

3 · Fine-tuning

Change: the weights, starting from a pretrained model (full or LoRA).

Needs: curated examples and a GPU. Weeks.

4 · From scratch

Change: everything; the weights start random.

Needs: very large data, compute and a specialised team. Months.

← faster · cheaper · less datamore control · specialisation · cost →

They combine: a fine-tuned model can also use RAG. RAG: Lewis et al. (2020) · LoRA: Hu et al. (2022).

Build vs buy

Four lenses: cost, control, privacy, IP

	Buy: vendor API	Build: own or open-weights model
Cost	Pay per token; no infrastructure; the bill grows with volume	Upfront work, GPUs or servers, operations; low cost per extra request
Control	The vendor decides updates, limits and retirements	You pin versions, decide when to update and can change the weights
Privacy	Data leaves your perimeter; you rely on contract terms	Data can stay on your own infrastructure
IP & licences	Read the provider's terms on data and outputs	Read the model licence: open weights ≠ unrestricted use; your fine-tuned weights are an asset

The options are not exclusive: you can call an API for one task and run your own model for another.

Simulator · build vs buy

Where should this use case sit?

Data sensitivity –

Requests / month –

Customization need –

Internal knowledge –

A teaching heuristic to structure the discussion, not a decision rule.

Recommended position: –

Apply it

Place five scenarios on the spectrum

Classify each scenario · 6 min

NovaTel: draft polite replies to about 200 routine customer emails a day.
Andes Bank: agents need answers from 2,000 pages of product manuals that change monthly.
Clínica Serena: extract fields from clinical notes full of in-house abbreviations; data cannot leave the clinic.
Café Alto Montaña: 20 social-media captions a week; no sensitive data; no tech team.
GeoSens: generate realistic readings for a new sensor no public model has seen; millions of unlabelled recordings.

For each: prompting, RAG, fine-tuning or from scratch, plus one reason. All companies are fictional.

Rooms of 3–4. One person per room reports one scenario in 30 seconds.

Total cost of ownership

The token bill is only one line

50 M

tokens per month for 50,000 conversations of about 1,000 tokens each (illustrative arithmetic). Multiply by your provider's current price per million tokens, then add the rest.

Beyond tokens, budget for:

Data preparation and labelling
Evaluation before launch and after every change
Monitoring, incidents and human review
People, training and change management
Switching costs if the provider changes terms

Check · build vs buy

Andes Bank's policy forbids sending customer data to third parties. It needs call summaries in its own style. Best first option?

The policy rules out sending data outside; the "own style" requirement suggests adapting the weights. A small open model fine-tuned in-house meets both at a fraction of the cost of training from scratch. A private deployment with contractual guarantees could also be discussed with legal.

Part 4

Tools, sectors and the road ahead

The toolbox for the lab, where generative AI is being applied, and how this course works.

Tools & frameworks

The open-source toolbox

PyTorch

Tensors, automatic differentiation and GPU support; the main backend of Hugging Face transformers.

TensorFlow · Keras 3

Keras 3 is multi-backend: the same model code runs on JAX, TensorFlow or PyTorch.

JAX

NumPy-style arrays with composable transformations: grad, jit, vmap.

Hugging Face Hub

Models, datasets and demo apps (Spaces), each with a card and a licence.

transformers · datasets

Load pretrained models, tokenizers and datasets in a few lines.

PEFT

Parameter-efficient fine-tuning such as LoRA: train a small fraction of the weights (S04).

Lab preview

Two pretrained models in a few lines


from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

clf = pipeline("sentiment-analysis",
               model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
print(clf("The new app is fast and easy to use"))   # [{'label': 'POSITIVE', ...}]

name = "HuggingFaceTB/SmolLM2-135M-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
messages = [{"role": "user", "content": "Write a slogan for a coffee shop."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.7)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Compute

Where the code runs: Kaggle and Colab

Kaggle Notebooks (default)

Free GPU (T4 x2 or P100) with a weekly quota, about 30 h; Kaggle may change it.
GPU and Internet require a verified phone number.
“Open in Kaggle” creates a private copy in your account.

Google Colab (fallback)

Same notebooks, one click from each session guide.
Free GPU availability varies.
Use it if Kaggle is down or your quota runs out.

Today: the lab and the challenge run on CPU. Keep Internet on; save your GPU quota for later sessions.

Applications

Generative AI across sectors

🏦 Banking & insurance

Summarise complaint calls, draft customer letters, explain products in plain language.

🛒 Retail

Product descriptions, campaign variants, analysis of customer reviews.

🏥 Health

Draft clinical documentation and patient-friendly summaries, always reviewed by professionals.

🏛️ Public sector

Answer citizen questions about procedures; route and summarise requests.

📡 Telecom

Support assistants, call summaries, agent coaching.

💻 Software & IT

Code assistants, test generation, documentation.

Value at stake

Where the value is expected

USD 2.6–4.4 T

estimated annual value across 63 use cases analysed by McKinsey (2023); about 75% of it falls in four areas: customer operations, marketing and sales, software engineering, and R&D.

These are scenario estimates, not measured results. Use them to decide where to look first, then build your own business case with a baseline you can measure.

Your process

One process in your organisation that generative AI could improve, and how you would measure it.

Course roadmap

Twelve sessions, three stages

Foundations

Intro to generative & pretrained models
Training generative models
Transfer learning
Fine-tuning
Evaluating generative models

Applications

Basic AI applications
Chatbots and agents
Generative classifiers
Real-world use cases

Responsible & continuous

Ethics and sustainability
Continuous improvement
Integrative project: demos

Every session: theory with simulators → guided lab on Kaggle → graded challenge. The final project runs in parallel, with milestones at S03 (H1), Wed 14 Oct (H2), S09 (H3) and the final delivery on Fri 23 Oct (H4).

Final project · 30%

Your own model, from idea to demo

Groups of 3 (mix business and technical profiles). The model must be fine-tuned or trained by the group, then:

compared against a simple baseline with at least 2 justified metrics;
analysed for errors and improved in at least one documented iteration;
reviewed for ethics and sustainability, with a model card;
shown in a working demo (Gradio app or Startti agent).

Milestones

H1 · S03	Groups formed
H2 · Wed 14 Oct	One-page canvas (feedback in S07)
H3 · S09	Use-case canvas and baseline
H4 · Fri 23 Oct	Improvement iteration (report §5, with the final delivery)
Final · Fri 23 Oct / S12	Deliverables (Fri 23 Oct); pitch and demo in S12

Evaluación del curso

¿Cómo se calcula tu nota?

70% · Challenges de sesión

11 challenges individuales (S01–S11); cuentan tus mejores 10, 7% cada uno. Rúbrica: ejecución técnica 40% · análisis e interpretación 40% · reflexión y registro de IA 20%.

30% · Proyecto final

Entregable técnico 15% · socialización y demo 10% · defensa individual 5%, multiplicado por el factor de coevaluación (0.7–1.0).

Política de uso de IA

Puedes usar IA y debes declararla en el registro: sin registro, ese 20% vale 0.0.
Eres responsable de todo lo que entregas.
Micro-sustentaciones al azar: si no puedes explicar tu entrega, la nota máxima es 3.0.

Escala 0.0–5.0 · se aprueba con 3.0 · entrega del challenge el mismo día, 23:59.

Evaluación · Política de uso de IA

Break

Stretch, refill your coffee, and check that your Kaggle phone number is verified.

Hands-on

Guided lab

From a line fitted by hand to a chat model: PyTorch, Keras and Hugging Face on Kaggle.

Lab · 60'

Lab bridge

What we will build

1	Kaggle tour · PyTorch tensors and autograd: fit $y = 3x + 2$ by gradient descent	15'
2	The same model in Keras 3, in three lines	10'
3	Hugging Face: sentiment, fill-mask and a SmolLM2 chat	20'
4	Tokens (English vs Spanish) and generation parameters	15'

Open the notebook

Kaggle checklist: Internet on · Accelerator none (CPU is enough today).

Open in Kaggle Colab

How to follow: run each cell right after the instructor (Shift + Enter). ✋ Checkpoints are answered in the Zoom chat; 🧪 “Try it” exercises are stretch goals with a hidden solution.

Session challenge

Individual, graded, 40 minutes in class. Submit by 23:59 today.

Challenge de la sesión · evaluado

Challenge S01: temperatura, diversidad y decisión

Ejecución: genera 5 muestras por prompt × temperatura con SmolLM2 y mide distinct-1, distinct-2 y longitud.
Análisis: ¿cómo cambiaron la diversidad y la calidad con tus números? Elige la mejor temperatura por prompt.
Explica y decide: ubica tu escenario de negocio en el espectro de adaptación.
Crítica a la IA: encuentra, explica y corrige 3 errores sobre BERT y GPT.

Rúbrica: ejecución 40% · análisis 40% · reflexión y registro de IA 20%.
IA permitida con registro; micro-sustentaciones: si no explicas tu entrega, máximo 3.0.
Entrega: Archivo → Descargar notebook → e-Aulas como S01_<codigo>.ipynb, 23:59.

Abrir en Kaggle Colab

Wrap-up

Key takeaways

Generative models learn $p(x)$ and sample from it; language models do it one token at a time.
Temperature and top-k trade consistency for diversity; neither guarantees truth.
Pick the model family by the task: encoders understand, decoders generate.
Prompting → RAG → fine-tuning → from scratch: more control, more cost.

Exit question

For your own use case: where on the adaptation spectrum would you start, and what would make you move one step to the right?

Next session: Training generative models (loss functions, training loops, a VAE), Saturday 7:00, followed by S03 at 10:00. Pre-work: finish the challenge by 23:59 and prepare both S02 and S03 (session guides) before Saturday 7:00.

References

Sources for this session

Vaswani, A., et al. (2017). Attention is all you need. NeurIPS. arXiv:1706.03762.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL.
Radford, A., et al. (2019). Language models are unsupervised multitask learners. OpenAI.
Raffel, C., et al. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. ICLR. arXiv:1312.6114.
Goodfellow, I., et al. (2014). Generative adversarial nets. NeurIPS.
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS.
Rombach, R., et al. (2022). High-resolution image synthesis with latent diffusion models. CVPR.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.

Bommasani, R., et al. (2021). On the opportunities and risks of foundation models. arXiv:2108.07258.
Brown, T. B., et al. (2020). Language models are few-shot learners. NeurIPS.
Chen, M., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374.
Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS.
Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. ICLR.
Allal, L. B., et al. (2025). SmolLM2: When smol goes big. arXiv:2502.02737.
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots. FAccT.
McKinsey & Company (2023). The economic potential of generative AI: The next productivity frontier.
Hugging Face. LLM Course (Spanish edition): huggingface.co/learn/llm-course/es