Build an AI Customer Support Chatbot

Before beginning the implementation, confirm the following access, technical conditions, and decisions are in place.

Decisions

Confirm the following before beginning:

1. 1

   Purpose

   The chatbot can only answer well when it has clean, retrievable content. This step converts your existing support content into a structured collection of question–answer or passage units the retrieval layer can search.

2. 2

   Actions

   Export the support content you plan to use. Prefer a structured export (CSV, JSON, or a documentation export) over scraping rendered pages.
   Split content into discrete units. Each unit should be a single coherent passage or FAQ entry that can stand alone as an answer source. A typical unit is one question–answer pair or one documentation section.
   Normalize each unit into a consistent schema. A minimal schema contains:

3. 3

   Review

   Remove or flag content that is outdated, contradictory, or internally inconsistent. Contradictory units degrade retrieval quality.
   Review the units with a subject-matter expert. Confirm the answers are correct and current before proceeding.

4. 4

   Considerations

   The quality of retrieval is bounded by the quality of the source content. Do not proceed with content you cannot verify.
   If answers frequently depend on user account state (for example, order status or subscription tier), plan to escalate those cases rather than answering from static content.
   Confirm whether any content contains personally identifiable information or data that should not be exposed through the chatbot.

Actions

   {
     "id": "unique-identifier",
     "source": "url-or-document-name",
     "question": "the question this unit answers",
     "answer": "the authoritative response text",
     "topics": ["optional", "categorization", "tags"]
   }

Important considerations

1. 1

   Purpose

   The chatbot needs credentials to call the model provider and, if applicable, to read the knowledge base or post messages to the support platform. This step creates narrowly scoped credentials and keeps them out of source code.

2. 2

   Actions

   Create a service account or API key for the AI model provider with permission limited to invoking the model. Do not reuse a broader administrative credential.
   Create a separate credential for the retrieval source if it requires authentication. Scope it to read-only access over the knowledge content only.
   Store all credentials in the environment's secret manager, not in application code or configuration files committed to source control.
   Confirm the credentials work with a single test request outside the deployment environment.

3. 3

   Considerations

   Apply least privilege. The chatbot does not need write access to the knowledge base or administrative access to the model account.
   If the production channel posts messages on behalf of users, confirm whether the platform requires delegated user authentication or supports a service identity. This decision depends on the platform you selected in Prerequisites.
   Keep staging and production credentials separate so test traffic never writes to production queues.

Actions

Example of referencing secrets rather than embedding them:

export MODEL_API_KEY="<MODEL_API_KEY>"
export KNOWLEDGE_BASE_TOKEN="<KNOWLEDGE_BASE_TOKEN>"

Important considerations

1. 1

   Purpose

   This is the core orchestration layer. It takes a user message, finds the most relevant knowledge units, passes them to the model with the user question, and returns a grounded answer or an escalation instruction.

2. 2

   Actions

   Decide the retrieval approach for your content volume:

3. 3

   Considerations

   Do not allow the model to answer from general knowledge when retrieval returns nothing relevant. This is the most common cause of fabricated support answers.
   Log the retrieved units alongside each generated response. This is required for debugging why a particular answer was produced.
   The prompt structure above is illustrative. The exact wording will depend on the model provider and the tone you want for the channel.

Actions

• For a small fixed set of FAQs (under a few hundred units), keyword or embedding-based retrieval over the full set is sufficient.
• For larger or continuously growing content, use an embedding-based vector store where each knowledge unit is indexed once and queried by similarity.

1. Build the retrieval step. Given a user message, return the top k most relevant knowledge units. Start with k = 3 and tune later.
2. Build the prompt that generates the response. It must satisfy three constraints:

• Answer only from the retrieved context.
• State when the context is insufficient rather than guessing.
• Return a machine-readable escalation signal when the context does not support a confident answer.

A minimal prompt structure:

   You are a customer support assistant.

   Answer the user's question using ONLY the context below.

   Context:
   {retrieved_units}

   If the context does not contain the answer, respond with "ESCALATE" and
   do not attempt to answer.

   User question: {user_message}

1. Add an escalation path. When the model returns the escalation signal, route the conversation to a human queue in the support platform rather than replying to the user with an ungrounded answer.
2. Set a guardrail on response generation. If the model call fails, times out, or returns malformed output, reply to the user with a neutral fallback message and log the event.

Important considerations

1. 1

   Purpose

   Before deploying to real users, measure whether the chatbot answers correctly on a representative set of questions and behaves safely when it does not know the answer.

2. 2

   Actions

   Assemble an evaluation set of at least 30–50 real or representative customer questions. Include:

3. 3

   Considerations

   A small evaluation set is directional, not statistically conclusive. Use it to catch egregious failures, not to certify quality.
   Do not tune on a handful of examples you like and then declare success. The evaluation set must reflect the variety of real traffic.
   Expectations differ by channel. A low-stakes website widget may tolerate more escalations than a support channel where users expect immediate resolution.

Actions

• questions the knowledge base can answer;
• questions that are close to but not answered by the knowledge base;
• questions entirely outside the knowledge base;
• malformed or empty inputs.

1. Define the expected outcome for each question: a correct answer, an escalation, or a refusal.
2. Run the full pipeline over the evaluation set. Record the retrieval result, the generated response, and the escalation decision for every question.
3. Compare actual outcomes to expected outcomes. Group failures into retrieval failures (wrong context found) and generation failures (right context, wrong answer).
4. Fix the highest-impact failures first:

• retrieval failures: add missing units, improve unit splitting, or adjust the number of retrieved units;
• generation failures: tighten the prompt, add an instruction not to exceed the context, or move the failure case to escalation.

1. Re-run the evaluation set after each change until the error rate is acceptable for your support context.

Important considerations

1. 1

   Purpose

   This step makes the pipeline reachable from the production channel and switches real traffic to it, ideally behind a controlled rollout.

2. 2

   Actions

   Deploy the orchestration service to your chosen runtime. Confirm environment variables and secrets are set in the deployment environment, not baked into the image or bundle.
   Create the channel integration in the support platform pointing at the deployed service endpoint.
   Run a smoke test from the production channel: send one known question and one out-of-scope question, and confirm both routes behave correctly.
   Roll out to a limited audience first if the platform supports it:

3. 3

   Considerations

   Deployment and rollout are separate activities. Deploying the service does not automatically expose it to all users; control that exposure deliberately.
   The initial deployment should default to escalating when uncertain. Expanding an overconfident chatbot is harder than relaxing an escalation-heavy one.

Actions

• A percentage-based rollout of website traffic.
• A single support category or team receiving chatbot-assisted handling before wider exposure.

1. Monitor the rollout before expanding. Look at escalation rate, fallback rate, and whether answers are being rejected or corrected by users.
2. Expand exposure gradually only when the monitored behavior is acceptable.

Important considerations

Verify the complete implementation as a system, not just each component.

Rollback

The safest rollback is to disable the channel integration, which immediately routes all traffic back to the previous human-handled flow.

1. In the support platform, disable or disconnect the chatbot integration. User messages return to the normal queue.
2. Leave the deployed service running so logs remain available for diagnosis.
3. If the issue is in the model or retrieval layer rather than the channel, fix the pipeline and re-point the integration after re-running the evaluation set.

Edge cases

• Empty knowledge base: the deployment must not happen; validation will fail every question. Check the knowledge base has content before enabling the channel.
• Duplicate knowledge units: the same question answered by multiple units produces retrieval noise. De-duplicate during Step 1.
• Model timeout: the user should receive a fallback message, not an indefinite wait. Set a timeout on the model call and log the event.
• Rate limits: if the model provider rate-limits the account, configure retry with backoff for transient limits and alert when persistent limits occur.
• Expired credentials: the secret manager should support rotation without redeploying the service. Confirm rotation is possible before you need it.
• Retrieved content is outdated: if the knowledge base changes, re-run the evaluation set. Content drift is the most common cause of silently degrading answer quality.
• User asks a time-sensitive question: static content cannot answer questions about order status, outages, or account state. These should escalate unless you integrate live data sources, which is a separate implementation.

After the chatbot is live and escalation rates are stable, review the logged conversations where users corrected the chatbot or where the chatbot escalated. Those logged cases are the most direct signal for which knowledge units are missing or which answers are misleading. Expand the knowledge base from that evidence, re-run the evaluation set, and repeat. When the chatbot is handling a meaningful share of traffic without user correction, review the architecture with Octacer to identify whether a cross-capability implementation — such as adding deterministic workflow automation for routine requests or integrating live account data — would further reduce the support queue.