Two lessons from last session's real-world cases, and a question about your own use case.
Warm-up · 10'Warm-up · Q1
Dell'Acqua et al. (2023) gave consultants GPT-4 for a task designed to fall outside the AI's capability frontier. What happened?
The "jagged technological frontier": inside it, AI users were faster and produced higher-quality
work; on the task outside it, they were about 19 percentage points less likely to be correct. Fluent output
invites over-reliance, which is an ethical risk as much as a quality one.
Warm-up · Q2
In Moffatt v. Air Canada (2024), the airline's website chatbot gave a customer wrong information about a fare policy. What did the tribunal decide?
The British Columbia Civil Resolution Tribunal rejected the idea that the chatbot was a separate
entity and found the airline liable for negligent misrepresentation. Deploying a model means owning what it says.
From value to risk
Your use case, seen from the other side
One line in the chat
Take the use case you analysed in S09 (or your final project). What is the worst thing that could
happen to a person because of the model? Be concrete: who, what, how often.
Today we turn that worry into:
numbers (bias metrics, redaction recall, kWh and kg CO₂e),
legal categories (EU AI Act tiers, Ley 1581 de 2012),
and documents (model cards, datasheets).
Learning objectives
By the end of this session you can…
Map the main risks of generative models and name a control for each RAE 6
Measure bias with counterfactual tests, demographic parity and equal opportunity RAE 2
Detect and redact personal data, and explain what Ley 1581 de 2012 requires RAE 6
Classify a use case under the EU AI Act and the Colombian framework, and propose controls RAE 6
Estimate energy and carbon, and document a model with a model card RAE 2RAE 6
RAE numbers refer to the learning outcomes in the course syllabus (guía de asignatura).
Part 1
What can go wrong?
A map of the risks of generative models, and who owns each one.
Theory · 60'
Risk taxonomy
Eight families of risk
⚖️ Bias
Unequal quality or stereotypes for groups of people.
A CV ranker that scores some names lower.
🔒 Privacy
Personal data memorised, leaked or re-identified.
A model that repeats a phone number from training data.
Grouping adapted from Weidinger et al. (2021) and Bender et al. (2021). Real systems combine several of these at once.
Two papers every AI team should read
Stochastic parrots and a taxonomy of harms
Bender, Gebru, McMillan-Major & Shmitchell (2021)
On the dangers of stochastic parrots: Can language models be too big? (FAccT)
Environmental and financial costs grow with scale, and are not paid by those who benefit.
Web-scale data is too large to document ("documentation debt") and over-represents dominant views.
Fluent text is easily mistaken for understanding.
Recommendation: weigh costs first, curate and document data, run pre-mortems.
Weidinger et al. (2021)
Ethical and social risks of harm from language models (arXiv:2112.04359)
Discrimination, exclusion and toxicity
Information hazards (e.g., private data leaks)
Misinformation harms
Malicious uses
Human–computer interaction harms
Automation, access and environmental harms
Security
When the input is the attack
Prompt injection
Direct: the user tries to override the instructions. Indirect: instructions hidden in a web page, email or file that the model reads (Greshake et al., 2023).
Riskier with tools and agents (S07).
Data poisoning
An attacker plants examples in training, fine-tuning or RAG data to create a backdoor or bias. Poisoning web-scale data has been shown to be practical (Carlini et al., 2024).
Training-data extraction
Carlini et al. (2021) extracted verbatim training text from GPT-2, including names, phone numbers and emails.
Controls: least privilege for tools · keep instructions and data separate · human confirmation
for irreversible actions · data provenance and validation · red teaming before launch (see the OWASP Top 10 for LLM Applications).
Poll · your project
Which risk could hurt the users of your final project the most?
No single right answer: the risk profile depends on the use case. Keep your answer: it becomes the
first line of section 8 (ethical considerations) of your project's model card.
Risk register
Every risk needs an owner and a control
Risk (Andes Bank assistant)
Likelihood
Impact
Owner
Control
Invents a fee or an interest rate
High
High
Product owner
RAG over official tariffs, citations, human review of commitments
Customer data sent to an external API
Medium
High
Data protection officer
Redaction, contract with the provider, data minimisation
Prompt injection through uploaded files
Medium
Medium
Security
No autonomous transfers, tool permissions, logging
Worse answers for some regions or groups
Medium
Medium
Data science
Disaggregated evaluation before every release
Andes Bank (a fictional digital bank): a risk without an owner is a wish. Rate each row, assign one
accountable person, and define a control you can measure.
Part 2
Bias and fairness
Where bias comes from, how to measure it, and what you can do about it.
Bias along the lifecycle
Bias enters at every step, not only in the data
1 · Data
Historical: the world as it was.
Representation: who is missing.
2 · Labels
Measurement: proxies that mean different things for different groups.
3 · Training
Aggregation: one model for all.
Learning: an objective that trades groups off.
4 · Evaluation
Evaluation: a benchmark that does not look like your users.
5 · Deployment
Deployment: used for something else, feedback loops.
For generative models: web-scale pretraining text, the pool of people who rate outputs
during alignment, and the prompts themselves add new entry points. Framework: Suresh & Guttag (2021).
Evidence
Aggregate accuracy hides who fails
Gender Shades (Buolamwini & Gebru, 2018) audited three commercial gender classifiers by skin type and gender:
34.7% vs 0.8%
maximum error rate for darker-skinned women vs lighter-skinned men.
Lesson: disaggregate every metric by the groups your system affects, including intersections.
What BERT learned: share of "he" vs "she" in "[MASK] works as a …" and "the … said that [MASK] would be late".
Real outputs of bert-base-uncased, computed for this course. 0.5 = no preference. You reproduce this in the lab.
Fairness metrics
Two common definitions of "fair"
Demographic parity
\( P(\hat Y = 1 \mid A = a) = P(\hat Y = 1 \mid A = b) \)
Same rate of favourable decisions in every group.
Credit: both groups are approved at the same rate.
Equal opportunity
\( P(\hat Y = 1 \mid Y = 1, A = a) = P(\hat Y = 1 \mid Y = 1, A = b) \)
Same true positive rate: people who deserve the favourable outcome get it equally often (Hardt, Price & Srebro, 2016).
Credit: among applicants who would repay, approval rates are equal.
You cannot have everything: when base rates differ between groups, a useful classifier cannot satisfy
calibration and equal error rates at once (Kleinberg, Mullainathan & Raghavan, 2017; Chouldechova, 2017).
Choose the metric from the harm you want to avoid, and document the choice.
Counterfactual testing
Change only the person, keep everything else
Same sentence, one identity term swapped:
Carlos is a new customer of the bank.
María is a new customer of the bank.
A fair model gives (almost) the same output. We measure:
Mean score gap between group A and group B
Flip rate: share of pairs whose decision changes
DP and EO differences over the whole template set
Kiritchenko & Mohammad (2018) used templates like these to test 219 sentiment systems and found
several with statistically significant gender or race bias. See also Garg et al. (2019).
Limits: templates are artificial; names are imperfect proxies for gender or origin;
binary pairs leave people out; results in English do not transfer to Spanish.
Simulator · counterfactual bias
Same sentence, different person
Mean A: – · Mean B: –
Gap A − B: – · Flips: –
Positive rate A / B: –
Real P(positive) from distilbert-base-uncased-finetuned-sst-2-english, precomputed.
–
Group A
Group B
Check · fairness metrics
Andes Bank audits its loan model. Among applicants who actually repaid, group B was approved less often than group A. Which metric captures this problem?
Equal opportunity compares true positive rates: among people who deserve the favourable outcome
(they repaid), each group should be approved equally often. Demographic parity ignores the true outcome; overall
accuracy can look excellent while one group carries most of the errors.
Mitigation
Three places to intervene, plus the product
Before training
Rebalance or reweight data; counterfactual data augmentation: add copies with the attribute swapped (Zmigrod et al., 2019, studied it for languages with rich morphology such as Spanish).
During training
Fairness constraints or penalties in the loss; adversarial debiasing (Zhang, Lemoine & Mitchell, 2018).
After training
Group-specific thresholds (Hardt et al., 2016); output filters; averaging over counterfactual versions of the input.
Product controls: human review of adverse decisions, appeals, monitoring by group.
Group thresholds need the protected attribute at decision time (sensitive data, unequal treatment):
get legal review and record the fairness–accuracy–cost trade-off.
Part 3
Privacy and regulation
Personal data, Colombian law, international frameworks and the EU AI Act.
Privacy
Where personal data leaks into a generative system
Training and fine-tuning data: web scrapes, customer messages, tickets.
Prompts and files sent to external APIs; RAG knowledge bases; logs.
Memorisation: models can reproduce rare training strings verbatim (Carlini et al., 2021).
Re-identification: ZIP code + birth date + sex were enough to single out about 87% of the US population (Sweeney, 2000).
Truly anonymised data cannot be linked back to a person by any reasonably likely means, so it is no longer
personal data (the anonymisation step itself is still processing). Pseudonymised data (names replaced by codes, identifiers removed but re-identification still possible)
is still personal data (GDPR, Recital 26).
Toolbox: minimise what you collect · redact with regex + NER (lab) · keep pseudonym keys
separate · aggregate · review samples by hand.
Colombia
Ley 1581 de 2012: habeas data in practice
Applies to personal data: information linked to an identified or identifiable person. Regulated by Decreto 1377 de 2013.
Prior, express and informed authorisation for a stated purpose (principle of finality).
Sensitive data (health, biometrics, ethnic origin, political views, religion, sex life…) get reinforced protection.
People can know, update, rectify and request deletion of their data, or revoke authorisation.
International transfers only to countries with adequate protection, unless an exception applies.
Authority: the SIC. Its Circular Externa 002 de 2024 gives guidance on personal data in AI systems.
GDPR (EU) in comparison: similar principles, plus explicit data protection impact assessments (Art. 35)
and rules on decisions based solely on automated processing (Art. 22).
NovaTel (fictional) wants to send support chats to a foreign LLM API. Before the pilot:
purpose covered by the authorisation? Cross-border transfer allowed? Redaction in place? A contract that forbids the
provider from reusing the data?
Frameworks
From principles to obligations
Framework
Type
What to take from it
EU HLEG Ethics Guidelines for Trustworthy AI (2019)
Shared values-based principles and recommendations for governments
UNESCO Recommendation on the Ethics of AI (2021)
Global standard
Human rights first, incl. environment and gender; impact assessments
NIST AI Risk Management Framework 1.0 (2023)
Voluntary (US)
Four functions: Govern · Map · Measure · Manage; a generative AI profile followed in 2024
EU AI Act, Regulation (EU) 2024/1689
Binding law
Obligations by risk tier (next slide)
CONPES 4144 (Colombia, Feb 2025)
National policy
Política Nacional de IA: six axes, incl. ethics & governance and risk mitigation
EU AI Act · Regulation (EU) 2024/1689
Four risk tiers
Unacceptable → prohibitedSocial scoring · emotion recognition at work or school · untargeted face scraping · manipulation causing significant harm
High risk → strict obligationsAnnex III areas: biometrics, critical infrastructure, education, employment, access to essential services (e.g., credit scoring), law enforcement, migration, justice
Limited risk → transparencyTell people they talk to an AI; label deepfakes and AI-generated content
Minimal risk → no new obligationsSpam filters, games, most internal productivity uses
High-risk obligations: risk management, data governance, technical documentation, logging, human oversight,
accuracy and robustness, conformity assessment before market.
General-purpose AI models have their own duties (documentation, copyright policy, training-data summary), with more for models with systemic risk.
Not every face system is banned: prohibitions are specific (e.g., real-time remote biometric identification in public spaces for policing, with narrow exceptions); other biometric systems are high-risk.
EU AI Act · timeline and scope
Phased application: check the current status
1 Aug 2024Entry into force
2 Feb 2025Prohibited practices and AI literacy apply
2 Aug 2025Obligations for general-purpose AI models
2 Aug 2026Transparency obligations (Art. 50)
Aug 2026 → Dec 2027High-risk systems in Annex III (e.g., credit, hiring, education)
Aug 2027 → Aug 2028High-risk AI in regulated products (Annex I)
The high-risk dates were postponed by an amendment adopted in 2026 (the "Digital Omnibus on AI"),
first proposed in late 2025. Dates can change again: always check the current consolidated text before advising anyone.
Does it apply to a Colombian company? Yes, if it places an AI system on the EU market or the
system's output is used in the EU. Otherwise it is a useful reference, while Colombian law (e.g., Ley 1581) applies.
Check · EU AI Act
TalentoAndino (fictional) uses a model to rank CVs and automatically discard candidates before a person sees them. Under the EU AI Act, this system is…
Recruitment and selection, including filtering applications and evaluating candidates, is listed in Annex III
(employment). The provider needs risk management, data governance, documentation, logging and effective human oversight.
The technology (text only) does not lower the tier; the use does.
Apply it
Classify five use cases
Risk tier + one control · 7 min
NovaTel: call-centre software infers agents' emotions from their voice to score their performance.
SegurVida: a model sets life and health insurance premiums for individual customers.
AguaClara: an AI component controls valves in a city's water-supply network.
MetroCali: a website chatbot answers questions about timetables and fares.
Café Alto Montaña: a model drafts Instagram captions for new drinks.
For each: prohibited, high, limited or minimal, the reason, and one control. All companies are fictional.
Rooms of 3–4. Each room reports one case in 30 seconds.
Debrief
Flip to check your room's answers
1 · Emotion recognition of call-centre agents
Prohibited (Art. 5): emotion recognition at work, unless for medical or safety reasons.
2 · Life and health insurance pricing
High risk (Annex III): control = human review, bias audit, explanation to customers.
3 · Valves in the water network
High risk (Annex III, critical infrastructure): control = fail-safe, human override, logging.
4 · Timetable chatbot
Limited risk (Art. 50): tell users they talk to an AI; easy handover to a person.
5 · Instagram captions
Minimal risk: voluntary codes; label realistic synthetic images if used.
In Colombia
Ask too: does it process personal data (Ley 1581)? Sensitive data (health, biometrics)? Is there a sector supervisor?
Part 4
Carbon, documentation and governance
What a model costs the planet, and how to write down everything we measured.
Large, one-off, and repeated for every experiment. Strubell, Ganesh & McCallum (2019) made the energy cost of NLP visible and asked for reporting.
Inference
Small per request, huge at scale. Generative tasks use far more energy per request than classification, and multi-purpose models more than task-specific ones (Luccioni, Jernite & Strubell, 2024).
Beyond electricity
Manufacturing the hardware (embodied emissions), water for cooling, and the data centre's overhead (PUE = total energy ÷ IT energy).
Simulator · carbon calculator
How much CO₂e does a training run emit?
Energy
–
Emissions
–
≈ – km in a gasoline car · ≈ – hours of a 10 W LED bulb
Illustrative ranges, not official factors: look up your grid's factor (Colombia: UPME). Nominal power overstates real draw. Car ≈ 0.2 kg CO₂e/km.
Check · carbon
A team fine-tunes a model for 100 GPU-hours. All else equal, which single change cuts emissions the most?
Emissions scale linearly with each factor: 0.15 / 0.8 is an 81% cut, PUE 1.1 / 1.2 about 8%, and 10% less time
10%. Doubling power for the same hours doubles energy. Often the biggest lever of all is not training a new model:
a smaller or existing model may be enough.
Green AI
Levers you control
Right-size: small, distilled or task-specific models; LoRA instead of full fine-tuning (S04).
Fewer, smarter experiments: small subsets first, early stopping, efficient search (S11).
Where and when: low-carbon regions, efficient data centres.
Measure and report: CodeCarbon or a cited method; report efficiency next to accuracy (Schwartz et al., 2020).
Energy is also cost. Every kWh saved is money saved. Put Wh per 1,000 requests next to
accuracy and latency on the dashboard of NovaTel's assistant, and review it with the same seriousness.
Documentation
Model cards and datasheets
Model cards · Mitchell et al. (2019)
Model details · intended use (and out-of-scope uses) · factors · metrics · evaluation data · training data ·
quantitative analyses disaggregated by group · ethical considerations · caveats and recommendations.
Your project template adds environmental impact and how to use the model.
Datasheets for datasets · Gebru et al. (2021)
Motivation · composition · collection process · preprocessing and labelling · uses · distribution ·
maintenance.
Answer them for every dataset you create or scrape, including synthetic data.
Project: the model card is a graded deliverable (proyecto-final/plantillas/model-card.md). Start the draft today with the lab's generated card.
Governance
Lightweight AI governance for a company
Practice
NIST AI RMF
Inventory of AI systems, each with an owner
Govern
Risk classification at intake (AI Act tier, personal data)
Map
Impact assessment: privacy and fairness before launch
Map · Measure
Testing: evaluation, bias audit, red teaming
Measure
Human oversight, appeals, incident response, rollback triggers
Manage
Documentation: model cards, datasheets, logs
Govern
Start small. A one-page AI policy, an inventory spreadsheet and a pre-launch checklist that is
actually used beat a 100-page policy nobody reads. At Andes Bank, no model goes live without an owner,
a tier and a model card.
Break
Stretch, refill your coffee, and open Kaggle with Internet switched on.
Hands-on
Guided lab
Audit NovaTel's sentiment model: bias, personal data, energy and a model card.
Lab · 60'
Lab bridge
What we will build
1
Fill-mask probing: "he" vs "she" by occupation with bert-base-uncased
10'
2
Counterfactual test of the SST-2 classifier: gap, DP, EO, flip rate
15'
3
PII redaction: regex (cédula, celular, correo) + dslim/bert-base-NER, with recall
15'
4
CodeCarbon: inference vs training, scaled to a month
10'
5
A model card filled with your own numbers
10'
Open the notebook
Kaggle checklist: Internet on · Accelerator none (CPU is enough today) · no secrets needed.
How to follow: run each cell right after the instructor (Shift + Enter).
✋ Checkpoints are answered in the Zoom chat; 🧪 "Try it" exercises are stretch goals with a hidden solution.
Session challenge
Individual, graded, 40 minutes in class. Submit by 23:59 today.
Challenge de la sesión · evaluado
Challenge S10: auditoría de sesgo y clasificación regulatoria
Auditoría: brecha, DPD, EOD y tasa de cambio con tu atributo, tus 6 plantillas y tu modelo.
Mitigación: umbral por grupo y aumento contrafactual; mide de nuevo, también con pares reservados.
Explica y decide: clasifica tu caso de uso (Reglamento de IA de la UE y marco colombiano), propone controles y una mini model card.
Crítica a la IA: encuentra, explica y corrige 3 errores de una declaración ética.
Rúbrica: ejecución 40% · análisis 40% · reflexión y registro de IA 20%.
IA permitida con registro; micro-sustentaciones: si no explicas tu entrega, máximo 3.0.
Entrega: Archivo → Descargar notebook → e-Aulas como S10_<codigo>.ipynb, 23:59.
Risks are specific: name them, give each an owner and a measurable control.
Bias can be measured: counterfactual pairs, DP and EO; the threshold is a fairness decision too.
Removing names is pseudonymisation, not anonymisation: Ley 1581 still applies.
The risk tier follows the use, not the technology; check current regulatory dates.
Energy = power × time × PUE; emissions depend on the grid. Measure, report and document it in a model card.
Exit question
Which single control would you add to your final project before tonight's deadline, and which metric will tell you it works?
Tonight by 23:59: the challenge, and your group's final deliverable with H4 (the improvement iteration, §5 of the technical report). Use today's tools to close the ethics section and the model card.
Next sessions: S11 (evaluation and continuous improvement) and S12 (final presentations), back to back on Saturday 24 Oct, 7:00–10:00 and 10:00–13:00.
Prepare both before Saturday 7:00: S11's readings and your pitch and demo for S12.
References
Sources for this session
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? FAccT.
Weidinger, L., et al. (2021). Ethical and social risks of harm from language models. arXiv:2112.04359.
Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. FAT*, PMLR 81.
Suresh, H., & Guttag, J. (2021). A framework for understanding sources of harm throughout the machine learning life cycle. EAAMO.
Hardt, M., Price, E., & Srebro, N. (2016). Equality of opportunity in supervised learning. NeurIPS.
Barocas, S., Hardt, M., & Narayanan, A. Fairness and machine learning: Limitations and opportunities. MIT Press (2023) · fairmlbook.org.
Kiritchenko, S., & Mohammad, S. M. (2018). Examining gender and race bias in two hundred sentiment analysis systems. *SEM.
Kurita, K., et al. (2019). Measuring bias in contextualized word representations. Workshop on Gender Bias in NLP.
Mitchell, M., et al. (2019). Model cards for model reporting. FAT*.
Gebru, T., et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12).
Carlini, N., et al. (2021). Extracting training data from large language models. USENIX Security.
Greshake, K., et al. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. AISec.
Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. ACL.
Luccioni, A. S., Jernite, Y., & Strubell, E. (2024). Power hungry processing: Watts driving the cost of AI deployment? FAccT.
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12).
Sweeney, L. (2000). Simple demographics often identify people uniquely. Carnegie Mellon University.
High-Level Expert Group on AI (2019). Ethics guidelines for trustworthy AI. European Commission.
Regulation (EU) 2024/1689 (Artificial Intelligence Act), and its 2026 amendment, Regulation (EU) 2026/1744 ("Digital Omnibus on AI").
NIST (2023). AI Risk Management Framework (AI RMF 1.0). NIST AI 100-1.
UNESCO (2021). Recommendation on the ethics of artificial intelligence.
OECD (2019, updated 2024). Recommendation of the Council on Artificial Intelligence.
DNP (2025). Documento CONPES 4144: Política Nacional de Inteligencia Artificial.
Congreso de Colombia. Ley 1581 de 2012 · SIC (2024). Circular Externa 002 de 2024.
Dell'Acqua, F., et al. (2023). Navigating the jagged technological frontier. HBS Working Paper 24-013.