Casos de uso: modelos generativos en contexto real
Desarrollo y Evaluación de Modelos Propios
Especialización en IA Generativa y Desarrollo de Negocios
Universidad del Rosario · Rosario GSB
Today
Agenda · 3 hours
Warm-up: three questions on generative classifiers (S08) 10'
Theory: frameworks, evidence, three sectors, failures and your canvas 60'
Break 10'
Guided lab: finance, health and education prototypes + a sector agent 60'
Session challenge (graded, individual) 40'
Project milestone H3 (use-case canvas) is due today at 23:59.
Warm-up
Three questions on last session: building classifiers with generative models.
Warm-up · 10'Warm-up · Q1
You only have 30 labelled examples. Which classifier usually gets closer to its best possible error first?
Ng & Jordan (2002): generative classifiers such as Naive Bayes approach their
(higher) asymptotic error with fewer examples; logistic regression usually wins once data is plentiful.
That is why the learning curve in the S08 lab crossed over.
Warm-up · Q2
In zero-shot classification by label likelihood, what does the LLM compute for each label?
We score \( \log p(\text{label tokens} \mid \text{prompt}) \) for every verbalized label and
pick the arg max. No training, but the choice of verbalizer (“bullish” vs “positive”) matters. We use it
again today in the finance part of the lab.
Warm-up · Q3
A teammate fitted TF-IDF on the whole dataset and tuned the decision threshold on the test set. The report says 94% accuracy. What do you answer?
Both are data leakage. Unsupervised statistics (vocabulary, IDF) computed on test data and a
threshold tuned on test data both make the reported number look better than what production will see.
Today this matters twice: a business case built on a leaky metric overstates the ROI.
Learning objectives
By the end of this session you can…
Analyse a use case with value × feasibility, risk × impact and a KPI ladder RAE 4
Estimate ROI and payback with explicit assumptions and decide go / no-go RAE 4RAE 5
Read the evidence critically: economic estimates, field experiments and documented failures RAE 4
Prototype and measure sector solutions in finance, health and education RAE 5
Anticipate sector risks and choose controls: liability, safety, privacy RAE 6
RAE numbers refer to the learning outcomes in the course syllabus (guía de asignatura).
Part 1
From idea to use case
Four lenses to decide where generative AI creates value, and at what risk.
Unit of analysis
A use case is a task, not a technology
Break a role into tasks
Claims analyst at an insurer (illustrative):
Read claim documents
extract
Check the policy's coverage
retrieve + answer
Flag unusual claims
classify
Write the decision letter
draft
Call the customer
human
Use-case statement
For [user] who [task and pain], a model that
[capability] produces [output], measured by [KPI],
with [human control].
Why tasks? Jobs mix tasks where AI helps a lot with tasks where it
hurts. Value, risk and ROI are all computed per task.
Handling time, first-contact resolution, time to first draft, escalations
Process owner
Business outcome
Cost per case, revenue, satisfaction, learning gains
Sponsor
Guardrail
Complaints, incidents, unsafe answers, gaps between user groups
Risk & compliance
Rule: measure the baseline before the pilot,
and define kill criteria upfront (e.g., “stop if escalations rise”).
ROI and payback
A transparent formula beats a big number
Hours saved per month, net of rework:
$$ H = \frac{V \cdot a \cdot \big(q\,m - (1-q)\,r\big)}{60} $$
$$ B = H \cdot c - C_{\text{run}} \qquad \text{payback} = \frac{C_{\text{impl}}}{B}\ \ (B > 0) $$
$$ \text{ROI}_{12} = \frac{12\,H\,c - (C_{\text{impl}} + 12\,C_{\text{run}})}{C_{\text{impl}} + 12\,C_{\text{run}}} $$
\(V\) cases per month · \(a\) adoption · \(q\) share usable without rework ·
\(m\) minutes saved per usable output · \(r\) rework minutes per unusable output · \(c\) loaded cost per hour ·
\(C_{\text{impl}}\) one-off cost · \(C_{\text{run}}\) monthly cost (compute, licences, monitoring, review).
Simulator · ROI calculator
When does the pilot pay back?
Illustrative defaults, not benchmarks. Same formula as the challenge.
Net / month–
Payback–
ROI 12 m–
Check · payback
Hours saved are worth USD 8,000 per month, run costs are USD 3,000 per month and the one-off cost is USD 40,000. What is the payback?
Net monthly benefit = 8,000 − 3,000 = 5,000; payback = 40,000 / 5,000 = 8 months.
Answer (a) forgets the run cost, the most common mistake in business cases.
Part 2
What the evidence says
Top-down estimates, field experiments and what transfers to your case.
Top-down estimate
The economic potential, and how to read it
USD 2.6–4.4 T
added value per year across the 63 use cases analysed by McKinsey (2023); about 75% of it
in customer operations, marketing and sales, software engineering and R&D.
How to read it: a scenario estimate of potential at scale,
not measured results. It says where to look, not what your pilot will return.
Use top-down numbers to pick where to search; use your own baseline and
pilot to build the business case.
Field evidence · customer support
Generative AI at work: support agents
+14%
issues resolved per hour, on average, after 5,179 support agents got an AI assistant
(NBER working paper, 2023). The 2025 journal version reports about +15%.
Biggest gains for novices and lower-skilled agents (+34% in the working paper); little
effect for the most experienced.
The assistant spreads the practices of top performers to everyone else.
Customers were more positive and asked less often for a manager.
Brynjolfsson, Li & Raymond (2023, NBER w31161; QJE, 2025).
Field evidence · knowledge work
The jagged technological frontier
Inside the frontier
758 BCG consultants with GPT-4 completed 12.2% more tasks,
25.1% faster, with more than 40% higher quality ratings.
Outside the frontier
On a task designed to fall outside the model's capabilities, consultants using AI were
19 percentage points less likely to reach the correct answer.
Takeaway: tasks that look equally hard can sit on opposite
sides of the frontier. Map it with your own tests before you scale. “Centaurs” split tasks between human and AI;
“cyborgs” interleave them.
Dell'Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon & Lakhani (2023), HBS Working Paper 24-013.
Check · evidence
Andes Bank (fictional) wants to cite Brynjolfsson et al. to justify an assistant for its most senior advisors. What is the best reply?
The effect was concentrated among novice and lower-skilled agents, in customer support.
Transfer the finding only after checking task, users and quality measure, ideally with a pilot and a comparison group.
From their study to your case
Does the evidence transfer?
Four transfer questions
Same task? Support chats ≠ legal opinions.
Same users? Novices gain more than experts.
Same quality measure? Speed alone can hide errors.
Same tool and context? Language, data, integration.
Budget a pilot with a comparison group and a baseline. It is the
cheapest way to replace borrowed numbers with your own.
Your turn
Which of the three studies is closest to your project, and which transfer question worries you most?
Part 3
Three sectors up close
Health, finance and education: use cases, evidence and the risks that shape them.
Sector · health
Health: high value, high stakes
Use cases
Clinical documentation drafts
Replies to patient messages
Intake and routing of requests
Coding and admin work
Evidence
Med-PaLM work (Singhal et al., 2023, Nature): 67.6% on USMLE-style
MedQA with Flan-PaLM, yet clinician review still found gaps in the answers. Benchmark scores are not safety.
Risks
Under-triage of urgent cases
Automation bias in staff
Health data are sensitive data (Ley 1581 de 2012)
Lab framing: our symptom-retrieval prototype routes requests;
it is not medical advice and a clinician always decides.
Sector · finance
Finance: documents, service and signals
Use cases
Customer-service assistants
Document processing (KYC, invoices, contracts)
Knowledge assistants for advisors
News and sentiment triage
Examples
Morgan Stanley launched a GPT-4-based assistant for its financial advisors
to query internal research (2023). FinBERT (Araci, 2019) adapted BERT to financial sentiment.
Risks
Advice rules and supervision
Explainability of decisions
Fraud: deepfakes and phishing also use GenAI
In fraud, generative AI cuts both ways:
it helps analysts summarise alerts, and it helps criminals produce convincing fakes. Budget for both sides.
Sector · education
Education: help that does not replace learning
Use cases
Tutoring: Khan Academy's Khanmigo guides with questions instead of giving answers
Feedback on drafts; question generation; teacher admin
Risks
Over-reliance, wrong content, unequal access, student data
Evidence · Bastani et al. (2025, PNAS)
Field experiment with about 1,000 high-school math students in Turkey:
Plain GPT-4 interface: practice grades +48%, but −17% on exams without AI
Tutor with guardrails: practice +127%, no significant harm on exams
Check · controls
A night-time telemedicine chat (fictional) wants an assistant that suggests possible conditions to order the queue. Which control matters most?
High harm needs layered controls: deterministic rules for emergencies, a human decision,
and a KPI that watches the costliest error. A disclaimer alone does not transfer responsibility, as the next
part shows.
Breakout · by sector
Analyse one case with the canvas
Case canvas · 8 min
Room 1 · IPS Vida Plena: draft discharge summaries for doctors.
Room 2 · NovaCrédito: check documents in loan applications.
Room 3 · Colibrí Online University: a tutor for Statistics 101.
Use-case statement (user, task, output, KPI, human control)
Value and feasibility, 1–5 each, with one reason
Top two risks and one control each
One KPI, one guardrail, and go / pilot / no-go
All organizations are fictional. Rooms with more people get the same case twice.
One person per room reports the decision and its main reason in 30 seconds.
Part 4
When it fails, and what makes it work
Documented failures, success factors, the Colombian context and your H3 canvas.
Documented failures
Your chatbot speaks for your organization
Moffatt v. Air Canada (2024 BCCRT 149)
The airline's website chatbot told a customer he could request a bereavement fare after
travelling; the airline's policy said otherwise. Air Canada argued the chatbot was responsible for its own
actions. The British Columbia Civil Resolution Tribunal rejected this, found negligent
misrepresentation and ordered the airline to pay the fare difference plus interest and fees.
NYC MyCity chatbot (The Markup, 2024)
New York City's official AI chatbot for business owners gave answers that contradicted the law,
for example that employers could take a cut of workers' tips and that landlords could refuse housing vouchers.
The city kept the tool online with added disclaimers.
From failure patterns to controls
What would have prevented it?
Failure pattern
Control
Answers beyond its knowledge base
Grounding (RAG) + “I don't know” + handoff to a person
Bot contradicts the official policy
One source of truth; test set built from the policies; versioning
Confident legal or medical advice
Scope limits, refusals, deterministic rules for red flags
Ley 1581 de 2012: personal data protection; health data are sensitive data
SIC, Circular Externa 002 de 2024: guidance on personal data in AI systems
CONPES 4144 (2025): national AI policy
ILIA (CENIA and CEPAL): regional AI index; Colombia is among the “adopter” countries
Data availability
Most open datasets and benchmarks are in English, like today's lab
Spanish and local terms need your own test set
Open public data: datos.gov.co
Spanish models exist (BETO, S03), but domain data are scarce
Data owner
Who owns the data your project needs in Colombia, and what authorization would you need?
Final project · milestone H3
Your use-case canvas, due today
Part A updated (1 page, mark changes with [CAMBIO]) + Part B (up to 2 pages):
B1 Changes since the proposal
B2 Baseline and first results
B3 Value × feasibility, with evidence
B4 KPIs · B5 ROI (base and pessimistic)
B6 Risks · B7 Continue / adjust / pivot
Submission
One member uploads PF_G<NN>_H3_canvas (PDF or Markdown) to e-Aulas before 23:59 today.
Formative, but a missing milestone costs 0.3 on the technical deliverable.
No Startti account? Part 4 runs a Python agent over the same document and is scored
with the same test questions. Both paths are equally valid in the challenge.
Session challenge
Individual, graded, 40 minutes in class. Submit by 23:59 today.
Challenge de la sesión · evaluado
Challenge S09: tu caso, tus números, tu decisión
Tu código te asigna un sector (salud, finanzas o educación), un caso ficticio y tus parámetros de ROI.
Prototipo: corre el prototipo del sector con tu muestra y el asistente del caso (Startti o Python) y mide.
Canvas: valor, factibilidad, riesgos y KPI, apoyados en tus números.
Explica y decide: ROI y payback con tus parámetros → go / no-go.
Crítica a la IA: un resumen escrito por IA con una estadística inventada y una afirmación legal falsa: verifica y corrige.
Rúbrica: ejecución 40% · análisis 40% · reflexión y registro de IA 20%.
IA permitida con registro; micro-sustentaciones: si no explicas tu entrega, máximo 3.0.
Entrega: Archivo → Descargar notebook → e-Aulas como S09_<codigo>.ipynb, 23:59.
Hoy también: hito H3 del proyecto (canvas de caso de uso), 23:59.
Wrap-up
Key takeaways
Analyse tasks, not technologies: value, feasibility, risk and KPIs, each with evidence.
ROI = volume × adoption × (savings − rework) − costs; quality and adoption drive it.
Field evidence is real but uneven: novices gain most, and some tasks fall outside the frontier.
Your assistant speaks for you: grounding, scope limits and humans where harm is high.
Exit question
What single number would convince your sponsor to fund the next step of your project, and how will you measure it?
Next session: S10, Friday 23 Oct, 18:00–21:00: ethics and sustainability (bias, privacy, regulation, carbon footprint).
Before: submit the challenge and the H3 canvas by 23:59.
References
Sources for this session
McKinsey & Company (2023). The economic potential of generative AI: The next productivity frontier.
Brynjolfsson, E., Li, D., & Raymond, L. R. (2023). Generative AI at work. NBER Working Paper 31161; Quarterly Journal of Economics, 140(2), 2025.
Dell'Acqua, F., et al. (2023). Navigating the jagged technological frontier. HBS Working Paper 24-013; Organization Science, 2026.
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., et al. (2023). Large language models encode clinical knowledge. Nature, 620, 172–180.
Singhal, K., Tu, T., Gottweis, J., et al. (2025). Toward expert-level medical question answering with large language models. Nature Medicine, 31, 943–950.
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning. PNAS, 122(26).
Araci, D. (2019). FinBERT: Financial sentiment analysis with pre-trained language models. arXiv:1908.10063.
Ng, A. Y., & Jordan, M. I. (2002). On discriminative vs. generative classifiers. NeurIPS.
Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia).
Lecher, C. (2024, March 29). NYC's AI chatbot tells businesses to break the law. The Markup.
Porter, M. E. (1985). Competitive advantage. Free Press.
Khan Academy. Khanmigo (AI-powered tutor and teaching assistant), described qualitatively.
Congreso de Colombia (2012). Ley 1581 de 2012, protección de datos personales.
Superintendencia de Industria y Comercio (2024). Circular Externa 002 de 2024: datos personales en sistemas de IA.
CONPES 4144 (2025). Política Nacional de Inteligencia Artificial. DNP.
CENIA & CEPAL (2025). Índice Latinoamericano de Inteligencia Artificial (ILIA).