Introducción al desarrollo de aplicaciones básicas
Desarrollo y Evaluación de Modelos Propios
Especialización en IA Generativa y Desarrollo de Negocios
Universidad del Rosario · Rosario GSB
Today
Agenda · 3 hours
Warm-up: three questions on evaluation (S05) 10'
Theory: from model to product, UX for AI, thresholds and humans in the loop, interfaces 60'
Break 10'
Guided lab: a Gradio app and your first Startti agent 60'
Session challenge (graded, individual) 40'
Project checkpoint: your H2 proposal canvas is due Wednesday 14 Oct, 23:59; written feedback comes in S07.
Warm-up
Three questions to reactivate Session 05: evaluating generative models.
Warm-up · 10'Warm-up · Q1
BLEU scores a machine translation against references. What is it built from?
BLEU is precision-oriented: the geometric mean of clipped 1- to 4-gram precisions,
multiplied by a brevity penalty so that very short outputs are not rewarded (Papineni et al., 2002).
ROUGE is the recall-oriented family; BERTScore uses embeddings.
Warm-up · Q2
In the S05 threshold explorer, you raise the decision threshold of a classifier. What usually happens?
A stricter threshold accepts fewer cases, so the ones it accepts are usually right more
often (precision up), and more true cases are missed (recall down). Today the same idea decides which customer
messages a bot answers and which go to a person.
Warm-up · Q3
You tuned hyperparameters with 5-fold cross-validation on the training data. Do you still need a held-out test set?
Every decision taken with a dataset "uses it up". Choose on validation (or CV), then report
once on the test set. Today you will choose a confidence threshold exactly this way.
Learning objectives
By the end of this session you can…
Frame an AI feature as a product: user, job to be done, inputs and outputs, constraints RAE 4
Sketch the conceptual design and architecture of a classifier app and a chatbot RAE 3
Apply UX-for-AI guidelines: expectations, confidence, graceful failure, human handoff RAE 3
Choose a confidence threshold that minimises expected cost with a human in the loop RAE 4
Build a Gradio app, and publish and call a Startti agent through its API RAE 3
RAE 3: develop functional prototypes of AI applications · RAE 4: analyse real use cases to integrate models.
Part 1
From model to product
Who uses it, for which job, with which inputs and outputs, under which constraints.
Theory · 60'
Framing
A model is one box in a product
👤 User
Who, in which moment, through which channel? Customer, advisor, analyst?
🎯 Job to be done
The progress the user wants to make, not the feature you want to ship.
↔️ Inputs → outputs
Free text in; a label + confidence, or an answer, out. What happens next?
⚙️ Constraints
Latency, cost per request, language, privacy (Ley 1581 de 2012), regulation.
Job-to-be-done statement (Christensen et al., 2016) for Andes Bank, a fictional digital bank:
“When my card is declined abroad, I want to know why and fix it in minutes, so I can pay without calling the bank.”
· “When hundreds of messages arrive on Monday morning, I want each one in the right queue, so no urgent case waits.”
In real ML systems the model code is a small part of the whole (Sculley et al., 2015).
Conceptual design
Two basic designs: classifier or chatbot
Classifier app
Chatbot (LLM or agent)
Output
A label from a closed list + a confidence
Free text in a multi-turn conversation
Typical failure
Wrong label; always answers something, even out of scope
Fluent but wrong or invented answer
How you control it
Threshold, human queue, sensitive classes
Instructions, knowledge base, guardrails, handoff
How you evaluate it
Accuracy, coverage vs accuracy, cost (S05, today)
Test conversations, task success (S07)
Latency · cost
Milliseconds on a CPU; almost free per request
Seconds; paid per token or per plan
They combine well: a classifier routes and detects sensitive cases; an LLM drafts the answer.
Check · design
NovaTel (a fictional telecom) wants each incoming complaint email sent to one of 12 teams. No reply is sent to the customer. Best first design?
The job is routing, and the output is one of 12 labels: a classifier fits, is cheap and is easy
to evaluate. Keyword rules are a fair baseline to beat, not a destination; a chatbot adds friction the customer did not ask for.
Architecture pattern
Model service, UI, router, logs, feedback
👤 UI
Web chat, WhatsApp, a Gradio app
⇄
🧭 App + router
Session, business rules, confidence threshold
⇄
🧠 Model service
Classifier, LLM or agent behind an API
↓ every request↓ if confidence < threshold↓ 👍 / 👎, corrections
🗂️ Logs
Input, output, confidence, latency, model version
←
🙋 Human queue
The advisor sees the message and the top-3 suggestions
→
🔁 Feedback & evaluation
New labels, error analysis, retraining (S11)
In today's lab: scikit-learn model service · gr.Blocks UI · a threshold rule · a CSV feedback log.
Where the value is
Value comes from the workflow, not the model
Andes Bank's router only creates value if the queues, service-level agreements and approved answers exist behind it.
A great model feeding an unstaffed queue saves nothing.
Name the KPI before the model
Time to first response · time to resolution
Cost per contact · share of contacts automated
First-contact resolution · customer satisfaction
Complaints and errors caused by the assistant
Part 2
UX for AI
Designing for a system that will be wrong some of the time.
Why it is different
Probabilistic systems need honest interfaces
Errors are guaranteed, not exceptional: design the failure path first.
Outputs vary with the input wording and, for generators, between runs.
Users build a mental model of what the system can do; the interface shapes it.
Trust must be calibrated: over-trust leads to misuse, under-trust to disuse (Parasuraman & Riley, 1997).
People + AI Guidebook (Google PAIR)
Chapters we use today:
User needs and defining success
Mental models
Explainability and trust
Feedback and control
Errors and graceful failure
Amershi et al. (2019), CHI
18 guidelines for human–AI interaction
Initially
Make clear what the system can do
Make clear how well it can do it
During interaction
Time services based on context
Show contextually relevant information
Match relevant social norms
Mitigate social biases
When wrong
Support efficient invocation
Support efficient dismissal
Support efficient correction
Scope services when in doubt
Make clear why the system did what it did
Over time
Remember recent interactions
Learn from user behaviour
Update and adapt cautiously
Encourage granular feedback
Convey the consequences of user actions
Provide global controls
Notify users about changes
In bold: the guidelines your app and your challenge must show (G1, G2, G8–G10, G15).
Four practices for today
Expectations, confidence, failure, handoff
1 · Set expectations G1 · G2
❌ “Hi! I can solve anything you need.”
✅ “I'm Andes Bank's virtual assistant (AI). I can help with cards and transfers.”
2 · Show confidence usefully G2 · G11
❌ “Intent: card_arrival (p = 0.4127)” shown to a customer
✅ Customers see “I think you mean…”; advisors see the top-3 with scores.
3 · Fail gracefully G9 · G10
❌ “Error 500: NoneType has no attribute 'text'”
✅ “I couldn't answer that. Do you mean A or B? Or I can pass you to a person.”
4 · Hand off to a human G8 · G10
❌ Type AGENT three times to reach a person.
✅ A visible “Talk to a person” option, plus automatic handoff for low confidence and sensitive cases.
Simulator · UX anti-pattern spotter
Name the anti-pattern, then flip
“Hi, I'm Laura, your personal advisor. I can solve anything, instantly.”
Fake human + over-promise. Disclose it is an AI and state its scope. G1 · G2
Confidence 0.38, below the threshold: the bot answers anyway, in the same assertive tone.
Hidden uncertainty. Ask a clarifying question or hand off. G10 · scope services when in doubt
“To talk to a person, type AGENT three times.”
Buried human handoff. Offer a visible way out, always. G8 · efficient dismissal
The customer sees “Error 500: NoneType object has no attribute 'text'”.
Raw error, no way forward. Apologise in plain words and offer a next step. G9 · PAIR: errors
No way to say an answer was wrong: “we will learn from complaints”.
No feedback loop. Add 👍/👎 and “what was wrong?”, and log it. G15 · granular feedback
The model was replaced overnight; answers changed and nobody was told.
Silent change. Announce updates and re-test before switching. G14 · G18
Revealed: 0 / 6 · Guidelines covered: –
Check · UX for AI
Andes Bank's router is only 55% confident that a customer's message is about a lost or stolen card. Best behaviour?
Some intents are too costly to get wrong or too urgent to delay: route them to a person (or a
secure flow) whatever the confidence. That is G10 plus a business rule; the threshold is not the only lever.
Logs are data about people
Feedback loops need a privacy plan
Every conversation log and every 👎 with a comment can contain personal data. In Colombia, processing personal
data is regulated by Ley 1581 de 2012.
Decide before launch: what you store, for how long, who can read it, and how you anonymise it before retraining.
Trust is a business asset
A bot that pretends to be human or hides its limits damages the brand when it fails.
An easy handoff costs advisor time but protects customers and reputation.
Clear expectations reduce complaints about “the bot that doesn't understand”.
Part 3
Confidence thresholds and humans in the loop
When should the system answer, and when should it hand the case to a person?
Selective prediction
Answer or abstain: coverage, accuracy, cost
message → classifier →
confidence p → p ≥ t ?
✅ yes → automatic answer · 🙋 no → human queue
Coverage \(c(t)\): share of messages answered automatically.
Selective accuracy \(a(t)\): share of automatic answers that are right.
Raising \(t\): lower coverage, usually higher accuracy. This is the reject option (Chow, 1970).
Automated: – · To humans: –
Accuracy of automated part: –
Wrong automatic answers: – per 1,000
1,000 synthetic tickets · illustrative costs.
Cost per 1,000: – · all to humans: –
Saving vs all to humans: –
Cheapest threshold: – · calibrated rule \(t^*\): –
Calibration
Is a confidence of 0.8 really 80%?
A model is calibrated if, among the answers it gives with confidence ≈ 0.8, about 80% are right.
Modern neural networks are often overconfident (Guo et al., 2017).
In today's lab, TF-IDF + logistic regression turns out underconfident: its automatic answers are right more often than their confidence says.
Either way the rule \(t^*\) is off: choose the threshold empirically on validation data and report it on test.
How to check and fix it
Reliability table: group by confidence, compare mean confidence with accuracy.
Recalibrate on validation data: Platt scaling, isotonic regression or temperature scaling
(CalibratedClassifierCV in scikit-learn).
Re-check after every model update: calibration can drift.
Check · threshold
A human costs 2.5 USD per ticket, a wrong automatic answer costs 10 USD and automation is almost free. With a calibrated model, from which confidence should you automate?
\(t^* = 1 - C_{hum}/C_{err} = 1 - 2.5/10 = 0.75\). Above 0.75, the expected error cost
\((1-p)\times 10\) is below the 2.5 USD of a human.
Human in the loop
Design the human side too
What the advisor sees: the message, the top-3 suggestions, the customer's context; one click to accept or correct.
Sensitive intents skip automation: fraud, lost cards, legal complaints, whatever the confidence.
Capacity: the threshold also sets the human workload. Check it against staff and service levels.
Corrections become labels: the queue is your best source of new training data.
Automation bias: people tend to accept a system's suggestion even when it is wrong
(Parasuraman & Riley, 1997). Audit a sample of accepted suggestions.
Aim for high automation and high human control at the same time, not a trade-off
between them (Shneiderman, 2020).
Tool for the breakout
The AI Product Canvas
1 · User & job
Who, when, and the job-to-be-done statement
2 · Inputs → outputs
What comes in, what comes out, what happens next
3 · Model & baseline
Classifier, LLM or agent; the simplest alternative to beat
4 · Confidence & handoff
Threshold, sensitive cases, who takes over
5 · UX copy
First message (G1, G2) and failure message (G9, G10)
6 · Feedback & logs
What you log, how users correct it (G15), privacy
7 · Metrics
One model metric, one business KPI, and the target
8 · Constraints & risks
Latency, cost per request, language, regulation
Template in the session folder: data/ai-product-canvas.md. It reuses the structure of your project proposal.
Apply it
Fill the canvas for your assigned case
AI Product Canvas · 7 min
Your room number decides the case (rooms 5+ restart at A). All companies are fictional.
NovaTel: web chatbot for billing questions.
Andes Bank: router for complaint emails to 12 teams.
Clínica Serena: WhatsApp assistant to book, move or cancel appointments (no medical advice).
Banco Andino: internal assistant that answers advisors' questions about product policies.
Fill boxes 1, 4, 5 and 7 at least. One person reports the threshold rule and the failure message in 30 seconds.
Tip: write the failure message first.
Who sets the threshold?
Pick the threshold with finance and operations
The three unit costs are business inputs, not model outputs: the cost of a human contact comes from
operations; the cost of an error includes rework, repeat contacts, compensation, churn and regulatory exposure.
Agree on them (and on the human capacity) before the data scientist optimises anything.
From the simulator to a budget
Cost per 1,000 × monthly volume ÷ 1,000 = monthly cost of the process.
Compare with “everything to humans” and with “everything automatic”.
Re-estimate when volumes, prices or the model change.
Part 4
Building it: interfaces and platforms
Gradio, Streamlit and a no-code agent platform; latency, cost and the prototyping loop.
Three ways to put a model in front of users
Gradio, Streamlit or a no-code agent platform
Gradio
Streamlit
No-code agent platform (Startti ADP)
You write
Python functions + components and events
A Python script that re-runs top to bottom on each interaction
Little or no code: nodes on a visual canvas
Best for
ML demos and prototypes, chat UIs, Hugging Face Spaces
Data apps and dashboards
LLM agents with knowledge bases, tools and sub-agents
Sharing
share=True: temporary public link from a notebook
Deploy the app to a server or a hosting service
Publish the agent, call it by API
Who maintains it
Developers
Developers, analysts
Trained business and tech teams
Gradio: Abid et al. (2019) · docs at gradio.app · Streamlit docs at docs.streamlit.io · Startti ADP docs at adp.startti.ai/docs
Lab preview · Gradio Blocks
A router app in one cell
import gradio as gr
with gr.Blocks(title="Andes Bank · support router") as demo:
gr.Markdown("I suggest one of 77 banking topics. When unsure, a person reviews it.")
msg = gr.Textbox(label="Customer message (English)")
threshold = gr.Slider(0.0, 1.0, value=0.6, step=0.05, label="Confidence threshold")
button = gr.Button("Route message", variant="primary")
top3, decision = gr.Label(num_top_classes=3), gr.Markdown()
button.click(route_message, inputs=[msg, threshold],
outputs=[top3, decision])
demo.launch(share=True) # never in the automatic test (FAST_DEV_RUN)
No-code · Startti ADP
Agents on a canvas, published behind an API
Build on a canvas (run by the Colmena engine): chat input → language-model call (model + system instructions) → chat output.
Extend with knowledge bases, tools/APIs and sub-agents (S07, S09).
Test in the preview chat next to the canvas.
Publish: an explicit step, different from saving. Unpublished agents answer the API with 409.
CallPOST https://api.startti.ai/v1/run with a Bearer key adp_…; sessionKey keeps a conversation.
AGENT_ID = "paste-your-agent-id" # not secret; the API key is (Kaggle Secrets)
if STARTTI_API_KEY and AGENT_ID:
print(startti_run(AGENT_ID, "I want to block my card", session_key="demo-01"))
print(startti_run(AGENT_ID, "It's the credit card", session_key="demo-01"))
else:
print("No key or agent ID yet: the Startti cells are skipped.")
Never paste the key in a cell, a chat, a screenshot, a slide or a repository. If it leaks, revoke it and create a new one.
Frequent errors: 401 invalid key · 403 key cannot access this agent · 404 wrong agent ID · 409 agent not published.
Latency & cost
Measure both before you choose
⏱️ Latency
Report the median (p50) and the slow tail (p95), not only the mean.
A small classifier on a CPU answers in milliseconds; a generative model or an API call takes seconds.
Perceived latency matters: typing indicators, streaming, “this may take a few seconds”.
💵 Cost per request
Own model: hosting and maintenance, almost nothing per extra request.
LLM or agent: tokens in + tokens out × price, or the platform plan.
Monthly cost ≈ conversations × turns × cost per turn. Look up current prices and date them.
In the lab you will print both latencies: the TF-IDF router and SmolLM2 or your Startti agent.
Prototyping loop
Build the thinnest slice, then learn from users
1 · Sketch
Fill the AI Product Canvas; write the failure message first.
2 · Fake it
A “Wizard of Oz” test: a person plays the AI to test the flow (Kelley, 1984).
3 · Thin slice
Gradio app or Startti agent with the simplest model that could work.
4 · Five users
Watch real people use it; log everything (with consent).
5 · Evaluate
S05 metrics + a business KPI → next iteration (S11).
Each loop should take days, not months. Change one thing per iteration so you know what helped.
Check · code or no-code
Banco Andino's service team wants an FAQ assistant next week, maintained by non-developers and reachable by API from its web chat. Best first prototype?
Time, maintainers and integration point to a no-code agent. A Gradio share link is temporary and meant for
demos. Whatever the tool, evaluate it with test conversations before launch (Session 07).
Build, configure or buy
No-code speeds you up; know what you depend on
A no-code platform shortens time to value and lets business teams own the assistant. In exchange, you depend on the
platform's plans, limits and roadmap.
Ask: where do our data and prompts live? Can we export them? What does one more conversation cost? Who evaluates changes?
Your final project
The demo can be a Gradio app or a Startti agent.
The model must be yours (fine-tuned or trained) and play a verifiable role in the demo.
An API or agent can be the baseline or the conversational layer.
Break
Stretch, and if you have your Startti licence, sign in at app.startti.ai before the lab.
Hands-on
Guided lab
The same assistant twice: in code with Gradio, and without code with a Startti agent.
Lab · 60'
Lab bridge
What we will build
1
Banking77 intent classifier (TF-IDF + logistic regression) with predict_proba
10'
2
Coverage, accuracy and cost per threshold; calibration check
12'
3
Gradio Blocks app: router, feedback log, SmolLM2 text generator
18'
4
First Startti agent: publish, API key in Secrets, startti_run, ChatInterface
17'
Open the notebook
Kaggle checklist: Internet on · Accelerator none (CPU is enough) · Add-ons → Secrets → STARTTI_API_KEY (Part 4).
Startti es opcional: el camino en Python vale lo mismo.
Wrap-up
Key takeaways
Start from the user and the job to be done; the model is one box among UI, router, logs and feedback.
Design the failure path: set expectations, show confidence usefully, fail gracefully, make the human easy to reach.
A threshold turns a classifier into a human-in-the-loop system: choose it by expected cost on validation data, and check calibration.
Gradio for fast prototypes in code; a no-code agent platform for published assistants. Measure latency and cost for both.
Exit question
In your own project, what should happen when the model is not sure? Write the exact message the user would see.
Next session: Implementing basic chatbots and agents (S07).
Before S07: save your Startti API key in Kaggle Secrets as STARTTI_API_KEY
(guide) ·
Project: finish and submit the H2 proposal by Wed 14 Oct, 23:59.
References
Sources for this session
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. CHI '19. doi:10.1145/3290605.3300233
Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction, 36(6), 495–504.
Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.
Chow, C. K. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1), 41–46.
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML.
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. ACL.
Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., & Vulić, I. (2020). Efficient intent detection with dual sentence encoders. NLP4ConvAI workshop, ACL. arXiv:2003.04807 (Banking77).
Sculley, D., et al. (2015). Hidden technical debt in machine learning systems. NeurIPS.
Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Know your customers' “jobs to be done”. Harvard Business Review, 94(9), 54–62.
Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. ACM TOIS, 2(1), 26–41.
Abid, A., Abdalla, A., Abid, A., Khan, D., Alfozan, A., & Zou, J. (2019). Gradio: Hassle-free sharing and testing of ML models in the wild. arXiv:1906.02569.