AI Customer Support Agent

Overview

Built an AI-powered customer support agent that handles Tier-1 support tickets autonomously. The agent uses retrieval-augmented generation (RAG) to search internal knowledge bases, draft responses, and resolve common issues without human intervention.

The Problem

Our support team was spending 60% of their time on repetitive questions — password resets, billing inquiries, feature how-tos. These tickets had clear patterns and documented solutions, but still required a human to copy-paste the right answer.

How It Works

1. Ingestion pipeline — Internal docs, FAQ pages, and past resolved tickets are chunked, embedded, and stored in a vector database (Pinecone).
2. Query understanding — When a new ticket arrives, the agent classifies intent and extracts key entities (account ID, product name, error code).
3. Retrieval — Relevant knowledge base chunks are retrieved using hybrid search (semantic + keyword).
4. Response generation — An LLM synthesizes the retrieved context into a natural, on-brand reply.
5. Confidence scoring — If the agent's confidence is below threshold, the ticket is escalated to a human with the draft response attached.

Key Learnings

• Chunking strategy matters more than model choice. Switching from fixed-size to semantic chunking improved answer quality by ~30%.
• Confidence calibration is hard. The model's self-reported confidence didn't correlate well with actual correctness. We ended up using a separate classifier trained on human feedback.
• Start with the escalation path. Building the human-in-the-loop fallback first gave us a safety net that let us iterate faster on the autonomous path.

Results

• 72% auto-resolution rate on Tier-1 tickets within the first month
• 4.2/5 average CSAT on AI-handled tickets (vs. 4.3/5 for human-handled)
• 3x faster median response time compared to the human baseline