Consumer support ran as an efficiency function inside a global product used by hundreds of millions of people. Volume was the metric, and the levers were staffing and macros. Meanwhile the same twenty questions arrived every day, the help content had drifted from the product, and nobody owned the experience end to end.
The request was
“Add an AI chatbot to deflect tickets.”
Discovery showed
Deflection was the wrong target. The recurring contacts were product signal, evidence of confusion the product itself was creating. Automating the responses without fixing the source would have automated the mess at scale.
A bot bolted onto a broken content layer produces confident wrong answers, which is worse than a queue. The real work was defining what the AI was allowed to be confident about, rebuilding support as an enablement surface rather than a complaint desk, and instrumenting the whole thing so investment decisions could follow evidence.
- Clustered contact drivers to separate genuine product defects from documentation gaps and expectation mismatches.
- Audited help content against current product behavior to quantify drift.
- Modeled confidence thresholds against answer accuracy to find where automation earned trust and where it did not.
- Built the baseline analytics (containment, CSAT/BSAT, deflection, content gaps) before proposing any investment.
Exhibit
AI-First Support Entry
- 01
Intent capture
Consumer describes the problem in their own words
- 02
Confidence gate
Answer only above threshold; below it, escalate or educate
- 03
Resolve or enable
Direct answer, guided troubleshooting, or product education
- 04
Warm handoff
Full context passed to a human, no repetition
- 05
Signal loop
Every gap routed back as content or product work
The signal loop is the part most teams skip. Without it, support learns nothing.
Exhibit
Automation Trust Standards
| Confidence required | Human review | Failure mode | |
|---|---|---|---|
| 01Informational answer | Moderate | Sampled | Low, correctable |
| 02Troubleshooting steps | High | Sampled + flagged | Medium, wasted user time |
| 03Account or billing action | Very high | Always | High, trust damage |
| 04Safety or legal topic | Never automated | Human only | Severe |
- 01A product-led support model with named ownership, operating principles, and published metrics.
- 02An AI-first support entry positioning the AI agent as the primary consumer interface.
- 03A unified enablement hub combining troubleshooting, education, and product value in one surface.
- 04Automation confidence thresholds, AI operating procedures, and trust standards.
- 05The analytics foundation for containment, CSAT/BSAT, deflection, and content gaps.
- Product
- Owned the support product strategy and roadmap.
- Research
- Ran contact-driver analysis and content audits personally.
- Standards
- Authored the automation confidence and trust framework.
- Analytics
- Defined the measurement model and reporting cadence.
- Support shifted from a throughput function to a product with owners, metrics, and a roadmap.
- Self-service scaled on evidence rather than optimism, with explicit thresholds gating expansion.
- Content and product gaps became a routed, prioritized backlog instead of anecdote.
- Executives gained a single view of containment, satisfaction, and deflection.
Six months is not long. I chose to spend the first third of it on measurement instead of shipping, which felt slow and was the only reason the later decisions held up. I would make that trade again, and I would make the case for it earlier and louder.