If you ship an AI product, your users are telling you exactly what's wrong with it, in plain language, thousands of times a week. Every chat with your agent contains sentiment (were they delighted or grinding their teeth), failure modes (where the agent hallucinated, looped, or got corrected), and intents (what they were actually trying to do, including things you never designed for). Almost nobody mines this systematically, which is strange, because for an AI product it's the single richest discovery signal that exists.

Can you build this with Claude Code? Yes, and this playbook is the complete build. But be clear going in: this is a harder system than churn analysis. Churn works on structured events, so SQL does the heavy lifting and no model ever reads raw data. Conversations are text. Something has to read them, and "something reads 30 million tokens of chat logs every week" is a problem you solve with architecture, not with a bigger context window.

This is also the anchor playbook for every text-at-scale workflow: support ticket radars, voice-of-customer rollups, interview synthesis, and review mining are all this same machine pointed at a different source.

What you're building

Every Monday, without you touching anything:

Sentiment by segment: how frustrated or happy users were this week, per plan tier or persona, trended against previous weeks.
A failure-mode leaderboard: the top clusters of agent failures (hallucinated data, refused valid requests, looped, needed user correction), each with volume, trend, and example conversations.
An emergent-intent report: what users tried to do that your product doesn't support yet, clustered and ranked by frequency. This is roadmap input you cannot get anywhere else.
Only what's new: the report diffs against last week. A new failure cluster gets flagged loudly; a known one just gets a trend line.

Why the naive version dies

Say 10,000 users, 30% weekly active, three conversations each, averaging 3K tokens. That's roughly 9,000 conversations and ~27 million tokens per week. Against a 200K context window, reading it directly is out by two orders of magnitude, every single week, forever.

And the batch trick from the churn playbook doesn't transfer. SQL can compute sessions-per-week; it cannot compute "the user got sarcastic after the third failed retry." For text, the aggregation step itself needs a model. So the architecture becomes three layers:

A map pass with a cheap model. Every conversation gets read once, by the cheapest model that can do the job, producing a tiny structured record. A 3K-token chat becomes a ~100-token row.
Embeddings and clustering on the records. Numerical work again: embed the extracted themes, cluster them, maintain cluster identity over time.
A big model reads only cluster summaries. Claude's analyst agents see "cluster 14: 212 conversations this week, +40% vs last week, 5 exemplars," never the raw firehose.

One more thing the churn playbook didn't need: continuous processing. You process only new conversations each run and assign them to existing clusters. Re-clustering everything weekly isn't just wasteful, it breaks your trend lines, because this week's "cluster 3" would no longer be last week's "cluster 3." Stable cluster identity is a thing you must engineer, and most of this playbook's difficulty lives there.

Phase 0: Prerequisites

Claude Code installed and authenticated, Pro/Max/Team/Enterprise for the scheduling and publishing steps.
An Anthropic API key with billing set up, separate from your subscription. This is the first workflow in the series where the subscription is not enough: the extraction pass calls the API programmatically (ideally the Batch API, which is discounted ~50%), and that's metered API spend, not plan usage. Budget for it explicitly; numbers below.
Access to your conversation store. Your agent's chats live in your own database, or in an LLM-observability tool. If you're on PostHog, its LLM analytics and MCP server cover the drill-down side; bulk export still comes from your store.
Python 3.11+ with duckdb, pandas, scikit-learn (HDBSCAN), plus an embeddings provider (a local sentence-transformers model is fine to start and costs nothing per token; a hosted embeddings API scales with less setup).
A dedicated git repo. Same rule as churn: run outputs and cluster state get committed; raw conversations never do.
A privacy decision, made up front. Conversations contain PII and customer content. Decide what leaves your infrastructure: the extraction pass sends conversation text to the API, so this needs the same data-processing sign-off as any vendor. Strip emails/names in the export script if your policy requires it.

Phase 1: The repo

convo-intelligence/
├── CLAUDE.md
├── .claude/
│   ├── agents/
│   │   ├── failure-analyst.md      # digs into one failure cluster
│   │   ├── intent-analyst.md       # digs into one emergent-intent cluster
│   │   ├── synthesizer.md          # merges into the weekly report
│   │   └── critic.md               # judge loop before publishing
│   └── skills/
│       └── convo-intel/
│           └── SKILL.md            # the orchestration recipe
├── scripts/
│   ├── export_convos.py            # new conversations since last run → data/
│   ├── extract.py                  # Batch API map pass → data/records.parquet
│   ├── embed_assign.py             # embed records, assign to clusters
│   └── recluster.py                # monthly full re-cluster + ID remapping
├── state/                          # committed: the system's memory
│   ├── clusters.json               # cluster IDs, centroids, names, birth date
│   └── watermark.json              # last processed conversation timestamp
├── data/                           # gitignored: convos, records, embeddings
└── runs/                           # committed: weekly reports + diffs
    └── 2026-07-06/
        ├── cluster_summaries/
        ├── report.md
        └── state_snapshot.json

Two files carry the whole recurring design. state/watermark.json records the timestamp of the last processed conversation, so every run is incremental by construction. state/clusters.json is the cluster registry: stable IDs, centroids, human-readable names, and birth dates. Both are committed, which means every scheduled run inherits them by cloning the repo, and every change to them has git history. When a trend line looks wrong in October, you can see exactly when cluster 14 was born and what it absorbed.

CLAUDE.md follows the same pattern as the churn playbook (definitions, data flow, "never read raw conversations into context"), with one addition worth writing down explicitly:

## Cluster identity rules
- Cluster IDs in state/clusters.json are STABLE. Never renumber.
- New conversations get assigned to existing clusters by embedding
  distance; only distance > threshold creates a candidate cluster.
- Candidate clusters are provisional until a human confirms the name.
- Full re-clustering happens only via scripts/recluster.py, which
  writes an old-ID → new-ID mapping. Reports must use it.

Phase 2: The extraction pass (where the real money goes)

extract.py is the heart of the system. It takes each new conversation and gets back a structured record. Have Claude Code write it against the Batch API, not the streaming API: extraction is embarrassingly parallel, nothing is latency-sensitive, and the batch discount roughly halves the biggest line item in this whole system.

The extraction schema is a product decision. A starting point:

{
  "conversation_id": "...",
  "user_segment": "pro_plan",
  "sentiment": -0.6,
  "sentiment_trajectory": "declined",
  "frustration_events": ["repeated_correction", "gave_up"],
  "outcome": "abandoned",
  "intents": ["export report to slides"],
  "unsupported_intent": true,
  "failure_modes": ["agent_looped"],
  "representative_quote": "why do you keep giving me last month's numbers",
  "quote_is_verbatim": true
}

Three hard-won rules for this step:

Use the cheapest model that passes your spot-checks. Extraction is classification plus quoting, not reasoning. Start with the small model tier, hand-grade 50 extractions against the raw chats, and only upgrade the model if the grading fails. Every tier you go up multiplies the largest cost in the system.
Force verbatim quotes. quote_is_verbatim exists because models paraphrase, and a paraphrased "user quote" in front of your exec team is how the whole report loses credibility. The critic agent spot-checks quotes against source later; give it the flag to check.
Version the schema. Put schema_version in every record. You will change this schema, and unversioned records make trend analysis quietly wrong.

Claude Code's role here is worth stating plainly: it writes extract.py, the batch submission and retry logic, and the schema validation. It does not run 9,000 extractions inside its own context. The session that builds this script is normal interactive Claude Code work; the script then runs on its own, headlessly, forever.

Phase 3: Clustering with stable identity

embed_assign.py runs per batch of new records:

Embed each record's intents, failure modes, and quote.
For each embedding, compute distance to every centroid in state/clusters.json.
Within threshold: assign to that cluster, update its running centroid and count.
Beyond threshold: hold out. When held-out records themselves form a dense group (HDBSCAN over the holdouts), create a candidate cluster with a provisional ID.
Write per-cluster weekly summaries (counts, trend, exemplar records) to the run directory.

Candidate clusters are the emerging-issue detector. "Nine conversations this week where users asked the agent to export to slides, no existing cluster fits" is precisely the signal the whole system exists to produce, so candidates get a prominent section in the report, not silent auto-adoption. A human confirms the cluster's name; the confirmation is a one-line edit to clusters.json in a PR.

recluster.py runs monthly, because incremental assignment slowly degrades: centroids drift, clusters that should merge don't, and the threshold accumulates wrong judgment calls. A full re-cluster fixes the geometry, then remaps old IDs to new ones by centroid overlap and writes the mapping into clusters.json, so October's report can still say "this was cluster 14 in July." Skipping the remap is the classic mistake; it resets every trend line to zero and nobody notices until an exec asks why the chart starts fresh.

Phase 4: The agents

Same three-role pattern as churn (analyst fan-out, synthesizer, critic), with the analysts specialized by cluster type. failure-analyst.md gets a failure cluster's summary and 5 exemplar conversations (full text, this is the only place raw conversations enter a context, ~15K tokens per analyst), and answers: what exactly goes wrong, what does the user do next, is this a prompt bug, a tool bug, or a capability gap? intent-analyst.md gets an emergent-intent cluster and answers: what job are these users hiring us for, what's the nearest feature we have, how are they working around us today?

The critic's checklist earns its place even more here than in churn:

1. Every quote in the report: grep it against the exemplar files.
   Not found verbatim → FAIL.
2. Every "new this week" claim: verify the cluster ID is absent
   from last week's state_snapshot.json.
3. Every trend number: recompute from cluster_summaries.
4. Sentiment claims must cite segment and sample size.
Return PASS or FAIL with fixes.

Fabricated quotes and phantom "new issues" are the two failure modes that kill trust in exactly one meeting. The critic is cheap insurance.

Phase 5: The skill, the schedule, and passing state forward

The /convo-intel skill mirrors the churn skill's shape: read prior state, run the pipeline scripts, fan out analysts over the top clusters by volume and all candidate clusters, synthesize with last week's report in hand, critique, persist, publish. The scheduling mechanics are also the same, so briefly:

Managed path: a weekly cloud routine via /schedule. The routine clones the repo, so it inherits state/ automatically; that's the state-passing mechanism, and it's why state lives in git rather than on someone's laptop. Configure the API key for extract.py in the routine's environment variables.
Self-hosted path: cron plus claude -p "/convo-intel" on a machine you own, or a GitHub Actions cron with claude-code-action. Your conversations then flow through whatever infrastructure runs the job; for many teams that's the deciding factor, given the privacy decision from Phase 0.
Failure alerting: a Stop hook posting run status to Slack. A conversation-intelligence system that silently stopped three weeks ago is worse than none, because everyone believes it's still watching.

One scale note: the extraction pass is decoupled from the analysis session on purpose. If your volume grows 10x, extract.py and the Batch API absorb it without touching the agent layer; the analysis session's token cost barely moves, because it's a function of cluster count, not conversation count. That decoupling is what "setting this up at scale" actually means.

Phase 6: Sharing with the team

Same two channels as the churn playbook, with one addition specific to this workflow:

Artifact publish for the weekly report page (stable URL, org-scoped visibility on Team/Enterprise plans, publish step runs from CLI/desktop rather than every automated surface).
Slack delivery of the top three movers plus candidate clusters, linking to the full page.
Route clusters to owners. Failure-mode clusters should file into the eng tracker (Linear's official MCP server makes this a one-line step in the skill: create or update a ticket per confirmed failure cluster, tagged with volume). Emergent intents go to whoever owns roadmap intake. A report everyone reads is good; clusters that become tickets with owners are the version that survives quarter three.

Phase 7: Maintenance (what you own now)

Everything from the churn playbook's maintenance list applies (prompt drift, secrets rotation, cost watch). This system adds its own, mostly downstream of the fact that both your product and your users keep changing:

Threshold tuning. The assignment distance threshold is a dial between "everything merges into mush" and "every conversation is its own cluster." You will adjust it in the first weeks, and each adjustment shifts what counts as new. Log the threshold value into every run's snapshot.
The monthly re-cluster review. Someone looks at the remapping diff and confirms it's sane. Twenty minutes when it's fine; a real investigation when it isn't.
Extraction quality audits. Re-grade 50 random extractions quarterly, because your product's conversations drift (new features, new user types) and the extraction prompt silently ages. This is an eval, and yes, you now own an eval.
Schema migrations. Adding a field to the extraction schema means deciding whether to backfill (re-extract history, real API cost) or fork the trend line (annotate the report). Neither is free; pick knowingly.
Candidate-cluster hygiene. If nobody confirms or rejects candidates, they pile up and the report's "new this week" section becomes noise. This is a weekly ten-minute human duty; assign it to a person by name.

What this costs

This is the workflow where cost stops being a footnote, so numbers with stated assumptions (9,000 conversations/week averaging 3K tokens; check them against your volume):

Extraction (the big line): ~27M input tokens/week through a small model. At small-model API pricing with the Batch API discount, that lands in the tens of dollars per week, roughly $15-30 at current published rates. On a mid-tier model it's several times that; on a frontier model it's 10x and pure waste for this task. This is metered API spend on top of your subscription, and it scales linearly with conversation volume, forever.
Embeddings: cents per week hosted, or zero marginal cost with a local sentence-transformers model. Not a real line item either way.
The analysis session: cluster summaries plus a handful of exemplar-reading analysts, well within subscription usage. Like churn, the risk isn't the average run but the agent that wanders into data/.
Plan tier: routines and org-scoped artifact sharing need the same Pro-and-up / Team-and-up tiers as every playbook in this series.
The invoice nobody prints: an extraction schema that's really a taxonomy of your product's failure modes, a clustering threshold someone tunes, a monthly re-cluster review, a quarterly extraction eval, and a weekly candidate-triage duty. That's a standing analytical system with a named owner. For an AI product the signal is worth it; just don't let anyone tell you the system is "just a cron job and some prompts."

The four practical challenges

Setup at scale: a three-layer architecture (cheap-model map pass, embedding clustering with stable IDs, agents on summaries only), because 27M tokens a week must never meet a context window directly.
Sharing with the team: artifact publish plus Slack, and cluster-to-ticket routing so findings land in owners' queues instead of a channel scroll.
Maintenance: threshold tuning, re-cluster reviews, extraction evals, schema migrations, candidate hygiene. The recurring human duties are the system.
Cost: a real, linear, weekly API line item for extraction on top of the subscription, cheap only if you resist the urge to over-model the map pass.

The managed alternative

The closest managed category is LLM observability: platforms like Langfuse and LangSmith trace your agent's conversations, and PostHog's LLM analytics sits alongside your product analytics. They're excellent at the engineering half (traces, latencies, token costs, error rates). The gap is the product half: sentiment by segment, emergent-intent discovery, and roadmap-shaped reporting are thin or absent, because these tools are built for debugging agents, not for discovering what users wish your product did. You'd still be exporting their data into something like this playbook for the PM questions.

Doing this with Second Axis

Everything above is the do-it-yourself path. On Second Axis, conversation intelligence is a workflow in the marketplace: connect your conversation store or LLM analytics, choose the workflow, and set the goal ("weekly sentiment, failure modes, and emergent intents, with new clusters flagged"). It runs on schedule with the extraction pass, continuous clustering, and stable cluster identity handled for you, and it monitors itself: broken exports, failed runs, or a cluster registry that stops making sense get caught and alerted, not silently skipped.