A Product FAQ Bot That Stays Current with Claude Code, Step by Step
Date: July 6, 2026 • Author: Second Axis
Every PM knows this tax. Sales asks whether SSO supports SAML yet. CS asks why the export button disappeared for one customer. A solutions engineer asks, again, how billing rounds partial months. One PM on Reddit estimated that up to 90% of the questions sales asks are answerable from documentation that already exists. Another got tired enough to build a Slack bot so that "the slack bot just replies them with a link instead of me." The same thread has the reason you can't just ignore the questions: "if I don't answer them, then it will be built wrong."
Can you build that bot with Claude Code? Yes, and the bot is the easy half. The hard half, the part this playbook is really about, is freshness. A bot grounded in documentation that was true two releases ago answers confidently and wrongly, and a confident wrong answer to a sales rep mid-deal is worse than no answer at all. What you're really building is a knowledge base with a maintenance loop; the bot is just its front door.
What you're building
Running continuously, without you in the reply chain:
- A #ask-product answering flow: teammates ask in Slack, an agent answers from a committed knowledge base, and every answer cites the exact file and section it came from.
- Refusal as a feature: if the KB doesn't contain the answer, the agent says so and files a gap instead of improvising.
- A gap loop: every unanswered or human-corrected question becomes a KB backlog item, so the knowledge base grows exactly where demand is.
- A freshness pipeline: a scheduled run diffs shipped tickets and release notes against KB claims, updates the feature-status registry, and flags contradictions before a human repeats them.
- A weekly rollup of the top questions asked, which turns out to be free roadmap signal.
Why naive answer bots die
The standard failure sequence: someone wires a retrieval bot over the docs folder, and for two weeks it's magic. Then a release changes how invoicing works, the docs don't, and the bot spends a month telling sales the old behavior with total fluency. Someone gets burned in front of a customer, nobody trusts the bot again, and everyone resumes asking you.
Trust is the scarce resource here, and it's asymmetric: one confident wrong answer costs more than fifty correct ones earn. That dictates the architecture:
- The KB is the only source of answers. Committed markdown, versioned, with named owners. The agent cites file and section or it does not answer.
- Refusal is cheap, so make it the default. "Not in the KB yet, I've filed it" preserves trust. A plausible guess destroys it.
- Freshness is a pipeline, not a policy. "Please keep docs updated" has never once worked. A scheduled job that reads what actually shipped and flags contradictions is what works.
Everything below builds those three properties.
Phase 0: Prerequisites
- Claude Code installed (
npm install -g @anthropic-ai/claude-code) and authenticated. Scheduled routines require a Pro, Max, Team, or Enterprise plan; the always-on option may push you to metered API use (see the cost section). - A Slack path. There is no official Slack MCP server. Your options are the community korotovsky/slack-mcp-server (~1.7k stars) or a small webhook script that Claude Code writes for you: an endpoint that receives channel messages and posts replies via a bot token. The script is less magic and easier to audit; either works.
- Access to your issue tracker and release notes, because the freshness pipeline reads what shipped. Linear has an official MCP server; Jira is covered by Atlassian's official server.
- A dedicated git repo for the knowledge base. Non-negotiable: citations point into it, the gap backlog lives in it, scheduled runs clone it. Anthropic's knowledge-work-plugins include Product Management and Customer Support plugins that are useful scaffolding for the writing side; the answering flow and freshness pipeline below are not in them.
Phase 1: The knowledge repo
product-kb/
├── CLAUDE.md # rules for every session
├── .claude/
│ ├── agents/
│ │ └── answerer.md # the citation-or-refuse agent
│ └── skills/
│ ├── answer-question/ # the answering recipe
│ └── freshness-check/ # the scheduled diff recipe
├── docs/ # how features work, one file per area
├── status/
│ ├── feature-registry.md # every feature: shipped / beta / building / not planned
│ └── not-supported.md # explicit "we do not do this" list
├── decisions/ # decision log: why things are the way they are
├── gaps/
│ └── backlog.md # questions the KB could not answer
└── state/
├── staleness.json # per-page freshness scores
└── qa-log/ # every Q&A pair, one file per week
Three choices carry the system:
The feature-status registry is separate from the docs. Docs explain how things work; the registry says what exists right now, in one place, in a rigid format (feature, status, since-version, owner). It's the file the freshness pipeline updates automatically, and where most sales questions actually resolve.
The "not supported yet" list is load-bearing. Half the questions PMs field are really "can we do X," and the trustworthy answer to most is a clear no with a date it was last confirmed. An agent can only give that answer if the no is written down; absence of evidence in the docs isn't evidence of absence, but this file is.
Every section has an anchor and an owner. Citations are file#section, so headings are stable IDs, and each file's frontmatter names a human owner. When the gap loop files an item, it lands on someone's plate, not "the team's."
CLAUDE.md states the rules binding every session, interactive or scheduled: answer only from files in this repo, always cite, never edit status/ during an answering session, and treat not-supported.md as authoritative for negative answers.
Phase 2: The answering flow
The agent definition is where citation-or-refuse behavior gets engineered.
.claude/agents/answerer.md
---
name: answerer
description: Answers product questions from the KB with citations
tools: Read, Grep, Glob
---
You answer ONE question, passed in your task prompt.
Search docs/, status/, and decisions/ for relevant sections.
You may answer ONLY if you can cite at least one file#section
that directly supports the answer. Format:
Answer (2-4 sentences), then "Source:" with repo links.
If the KB does not contain the answer: reply exactly that it
is not in the KB, and append the question to gaps/backlog.md
with today's date and the asker's channel.
ALWAYS escalate to a human instead of answering when the
question involves: pricing or discounts, legal or compliance
commitments, roadmap dates, or anything a customer will be
promised. Escalation = name the owning human, do not guess.
Confidence rules: if two KB files disagree, refuse, cite both,
and file the contradiction as a gap. Never reconcile silently.
The escalation list deserves defense. Pricing, legal, and roadmap commitments are categories where even an accurate-sounding answer creates liability, because context a bot can't see (this customer's contract, this quarter's discount policy) changes the right answer. Routing those to a named human is not a limitation; it is the system working.
The delivery wrapper is thin: the webhook script (or the community Slack MCP server) receives a message in #ask-product, invokes Claude Code headlessly (claude -p with --allowedTools scoped to read-only plus the backlog append), and posts the reply in thread. Log every Q&A pair into state/qa-log/; the measurement below depends on it.
Run this interactively for a week before wiring Slack: ask it the twenty questions you answered last week and grade the citations by hand. Every weakly-cited answer at this stage is a KB fix, and fixing them now is how launch goes well.
Phase 3: The gap loop
This mechanism stops the KB from freezing at whatever state it launched in.
Every refusal appends to gaps/backlog.md. So does every human correction: when someone replies in thread "actually that changed last sprint," a lightweight rule (a reaction emoji the webhook watches for, or the PM pasting the thread link) files it as a gap with the correction attached. Entry format: question, date, asker, frequency count, correcting context if any.
The loop closes on a human cadence: weekly, someone triages the backlog, writes or fixes the KB sections, and commits. The commit is the resolution; the next identical question gets a citation instead of a refusal.
In aggregate, the KB grows in exact proportion to real demand; you never write documentation speculatively again. The refusal rate is your progress bar: it should fall week over week, and if it doesn't, triage is being skipped.
Phase 4: The freshness pipeline
The gap loop handles what the KB never knew. The freshness pipeline handles what the KB used to know correctly, the more dangerous category, because nothing looks wrong.
.claude/skills/freshness-check/SKILL.md runs on a schedule (a cloud routine via /schedule, or cron plus claude -p "/freshness-check" on your own machine) and does four things:
- Pull what shipped since the last run: closed tickets tagged as released, plus release notes, via the issue tracker's MCP server. The watermark of the last processed release lives in
state/, committed, so every run knows where the previous one stopped. Continuity across scheduled runs isn't automatic; it exists because the run reads state the last run committed. - Update the feature-status registry mechanically: a ticket flagged as shipping feature X moves X's row from building to shipped, with version and date. Safe to automate because the registry's format is rigid.
- Diff shipped changes against KB claims. For each shipped ticket, grep the docs for sections describing the touched feature and ask: does this page still describe current behavior? Contradictions get filed as high-priority gaps with the ticket linked, not auto-edited, because rewriting prose about product behavior is a judgment call for the human owner.
- Update staleness scores.
state/staleness.jsontracks, per page, when the feature it describes last changed versus when the page was last edited. A page whose subject shipped three releases of changes since its last edit scores high, and the answerer adds a "last verified" note when citing anything above threshold. Freshness is computed from what the page is about, not from the page's own timestamp.
Add a Stop hook posting run status to Slack; a freshness pipeline that silently died is precisely the stale-KB failure it exists to prevent.
Phase 5: Sharing with the team
Slack-first, unlike the report-shaped playbooks in this series, because the product is answers where questions happen. Two additions beyond the in-thread replies:
- The weekly top-questions rollup. A scheduled run reads
state/qa-log/, ranks questions by frequency, splits answered from refused, and posts a digest: top ten questions, refusal rate trend, gaps opened and closed. This is roadmap signal in disguise; the questions sales asks most are the confusions your product ships. - An artifact page for the rollup. Claude Code can publish the digest as a hosted page on claude.ai with a stable URL across republishes. Caveats: publishing works from the CLI and desktop app, not the Agent SDK or GitHub Actions, so an autonomous pipeline may keep a human on the publish step, and org-scoped visibility requires a Team or Enterprise plan. If those bite, a pinned Slack message is a fine substitute.
Phase 6: Maintenance (what you own now)
- KB governance. Every section has an owner in frontmatter; ownerless pages rot first. Review ownership when the org chart changes, which is more often than anyone plans for.
- Gap backlog triage. The weekly duty from Phase 3, assigned to a person by name. This is the single point of failure: skip it for three weeks and the refusal rate climbs, and people quietly go back to DMing you.
- Answer-quality audits. Monthly, sample twenty logged Q&A pairs and grade them: citation correct, tone right, escalations actually escalated. Tone matters; a correct answer that reads as curt to a customer-facing team erodes adoption. This is a judge loop with a human judge, and its findings become edits to the answerer prompt.
- Escalation list upkeep. The always-escalate categories change (a new compliance regime, a pricing revamp). The list is a prompt file in git; changes get PR review, because an edit here changes what your company tells customers.
- Secrets and connectors. The Slack token and tracker credentials rotate; the routine's environment variables are one more place they live.
What this costs
Assumptions to check against your reality: 50 questions a day, and per question the agent greps the repo and reads a handful of sections, call it 5-15K input tokens and a few hundred output tokens per answer. Roughly 250-750K input tokens a day of small, bursty sessions.
- Subscription versus API. A real-time responder is always-on: something must invoke Claude Code within seconds of each question, all day. Headless runs draw on plan usage, and at 50 questions a day a Pro or Max plan may absorb it, but you risk hitting plan limits at exactly the moment a sales rep is waiting. Metered API keys make throughput a billing question instead of an availability question, at the price of a line item that scales with question volume. The token math above is small enough that at current published mid-tier rates this is a modest daily spend; run your own numbers against the pricing page.
- The digest compromise. If the API line item isn't approved yet, run the whole thing as a scheduled digest instead of real-time: questions accumulate in the channel, a routine sweeps them and answers in batch. Routines stay within subscription usage, and cloud routines have a minimum one-hour interval, which is exactly the latency you're accepting. An answer within the hour still beats an answer whenever the PM surfaces.
- The freshness pipeline is cheap: one scheduled run reading tickets and grepping markdown, within subscription usage.
- The invoice nobody prints: a weekly triage duty, a monthly audit, and section ownership across the KB. That's the price of trustworthy answers, paid in PM hours, just far fewer than answering everything yourself.
The four practical challenges
- Setup at scale: not data volume this time but trust engineering: the citation-or-refuse agent, the escalation rules, and a KB structured so citations are possible (Phases 1-2).
- Sharing with the team: Slack-first delivery with no official Slack MCP server, so a community server or your own webhook script, plus artifact caveats for the rollup (Phase 5).
- Maintenance: gap triage, ownership governance, and answer-quality audits (Phase 6). The bot is software; the KB is a garden.
- Cost: modest tokens per question, but always-on delivery forces a real subscription-versus-API decision, with the scheduled digest as the middle path.
The managed alternative
If you want this without building it, the adjacent categories are knowledge tools like Guru, which push verified answer cards into Slack, and enterprise search like Glean, which answers questions across your connected tools. What you get is speed, polished Slack UX, and someone else's connectors. The fair trades: another per-seat subscription, their connectors rather than your registry, and the loop that matters most here, from product reality to knowledge base, is still yours to run. No tool knows a feature shipped unless something reads your tracker and updates the source of truth, and that pipeline is the part this playbook spent the most words on.
Doing this with Second Axis
Everything above is the do-it-yourself path. On Second Axis, the product FAQ bot is a workflow in the marketplace: connect Slack, your docs, and your issue tracker, choose the workflow, and set the goal ("answer product questions in #ask-product with citations, escalate what it cannot source"). It runs continuously from there, keeps the KB fresh from your release data, and monitors itself: broken sources, failed runs, and a rising unanswered-question rate get found and alerted, not discovered three weeks later as a wrong answer in a deal thread.