What AI model powers the support agent?

We used Google Gemini 2.5 Flash for generation and text-embedding-004 for semantic retrieval. The model choice was driven by latency requirements — responses under 2 seconds — and strong performance on technical documentation.

How did you prevent the AI from giving wrong answers?

Every answer is grounded in retrieved documentation chunks. If the retrieval score for the top result falls below our confidence threshold, the agent acknowledges uncertainty and routes to a human agent rather than guessing.

Can the AI handle billing and account queries?

Yes. We integrated the Stripe API so the agent can look up billing status, invoice history, and subscription tier in real-time — securely and only after verifying the user's session token.

How long did the deployment take?

14 weeks from kickoff to production: 3 weeks for knowledge base architecture and embedding, 5 weeks for the RAG pipeline and guardrail layer, 3 weeks for the frontend widget and Intercom integration, and 3 weeks of supervised production testing before full rollout.

Home Case Studies

AI & ML Solutions

Case Study · AI & ML Solutions

A Support Agent That Retrieves Before It Answers.

How we built a production RAG support agent for a US B2B SaaS company — grounded in their own documentation, with a hard escalation path to a human when confidence drops.

Service

AI & ML Solutions

Location

USA

Duration

14 weeks

Year

2025

Industry

SaaS / Customer Support

Project Overview

A B2B SaaS company in the USA was losing most of its support capacity to repetitive first-line queries — password resets, billing questions, feature explanations — while genuinely technical issues waited behind them. They approached Futurise Solutions to build an AI layer that could answer the routine questions without degrading the experience for customers who needed a person.

Key Insight: Chunking Beat Model Size

The largest gains in answer quality came from how the documentation was split and how context was assembled in the prompt — not from reaching for a bigger, more expensive model. Character-count chunking split code blocks and numbered procedures mid-instruction, so retrieval returned fragments that read as complete but were not. Switching to syntax-aware chunking that respects code fences and step boundaries, plus a metadata taxonomy that keeps release notes distinct from SOPs, did more for grounding than any model upgrade we tested. Cheap, well-structured context outperforms expensive, poorly-structured context.

The Challenge

The Problem

The company had a large help centre but most customers opened a ticket without finding the answer already published in it. An earlier chatbot built on keyword matching was brittle and frequently wrong, which had trained users to skip it entirely. The team needed something that could reason over their documentation, handle nuance, and stay inside a hard guardrail against inventing answers.

A large documentation corpus with no unified retrieval layer

Previous keyword chatbot was unreliable enough that users had abandoned it

First-line queries consumed the support queue, delaying L2 and L3 resolution

No visibility into which questions were being asked most

GDPR and SOC 2 compliance requirements for data handling

Our Solution

Our Approach

We designed and built a full RAG (Retrieval-Augmented Generation) pipeline backed by a purpose-built knowledge base. Rather than connecting an off-the-shelf chatbot, we engineered the system to understand the client's specific product context — integrating live ticket data, documentation, and CRM metadata into a unified vector store queried in real-time.

Knowledge Architecture

Chunked and embedded the documentation, release notes and SOPs into a MongoDB Atlas vector index using `text-embedding-004`. Built a metadata taxonomy so retrieval could distinguish reference material from procedure.

RAG Pipeline

Semantic search retrieves the top-5 most relevant chunks per query. A Gemini 2.5 Flash model assembles grounded answers with strict context windows to prevent hallucination.

Guardrails & Escalation

Confidence scoring below a 0.75 threshold triggers a live-agent handoff. A custom intent classifier routes billing and account queries to the appropriate internal API.

Analytics Dashboard

Built a real-time dashboard showing deflection rate, query clusters and missed topics, so the support team could see which gaps in the knowledge base were generating tickets.

Execution Timeline

Our Step-by-Step Process

Weeks 1-3

Discovery & Data Prep

Auditing help center docs, cleaning metadata, and formatting data for retrieval models.

Weeks 4-7

RAG & LLM Prototyping

Setting up MongoDB Atlas Vector search, building the semantic search layer, and refining Gemini prompts.

Weeks 8-11

Guardrails & Integration

Creating confidence threshold routing, implementing fallback to live agents, and integrating Stripe API.

Weeks 12-14

Testing & Rollout

Shadow-testing alongside human agents, tuning responses, then moving to customer-facing traffic.

Tech Stack

Google Gemini 2.5 Flash
text-embedding-004
MongoDB Atlas Vector Search
Node.js / Express
React
Vercel
Intercom API
Stripe API (billing queries)

Results

Measurable impact — verified outcomes

RAG

Retrieval architecture

Every answer is generated over documentation chunks retrieved at query time, not from model memory

Top-5

Chunks retrieved per query

Semantic search across a MongoDB Atlas vector index built with text-embedding-004

0.75

Confidence threshold

Below this score the agent declines to answer and hands off to a live agent

14 wks

Kickoff to production

Knowledge architecture, RAG pipeline, guardrails, then supervised rollout

Retrospective

Lessons Learned & Refinements

Dynamic Chunking Wins

Standard character-count chunking broke code blocks and step-by-step instructions. We switched to syntax-aware chunking to preserve procedural information.

Human-in-the-Loop Backup

Initially the agent tried to handle too much. Adding an early handoff trigger on frustrated sentiment mattered more than raising raw answer coverage.

Frequently Asked

Let's build your success story.

Book a free 45-minute strategy call. We'll scope your project, recommend the best approach, and give you a clear roadmap — no strings attached.

business@futurisesolutions.com