Case studies

ThirdSlate: Agentic RAG for Course-Grounded Learning

Co-founder. I designed the product and built the AI backend, from document ingestion and retrieval to evaluation and deployment.

1,000+
students
real product usage
94%
response relevance
answer/query relevance
−55%
manual QA
automated evaluation
89%
RAGAS faithfulness
evidence support
−38%
hallucinations
grounding + review
−60%
processing time
optimized ingestion
Approach
[01]Course material[02]Scoped retrieval[03]Agent orchestration[04]Grounding[05]Evidence-backed response

[1] System

Product problem, ownership & system architecture

How the student experience, identity, AI services and data fit together.

Student product loop

From uploaded course material to a grounded, reviewable answer.

Fig. 1 · Upload, retrieve, ground, then cite or review.

System architecture

Fig. 2 · Specialized services: Doc Processing, RAG Chat, Practice.

Dependency map

AreaServiceDepends on
App dataServer / APIPostgres · Storage
DocumentDoc ServiceStorage → OCR → Embeddings → pgvector
Chat / RAGRAG Chatpgvector · Memory · Gemini · Grounding · Postgres
PracticePracticepgvector · Gemini · Postgres

Ownership ledger

AreaDetailOwnership
UI / UXComplete UI / UXPrimary owner
GroundingCritic + human reviewPrimary owner
IngestionPDF + DOCX + OCRPrimary owner
Domain services3 FastAPI servicesPrimary owner
Agent backendLangGraph + RAGPrimary owner
Evaluation500+ cases / 8 domainsPrimary owner
Auth boundarySupabase + JWTPrimary owner
DeploymentDocker + EKS + CI/CDPrimary owner

Problems solved

Course groundingCourse material as evidence.
Data isolationUser · course · material scope.
Traceable answersInline source citations.
Deployable AIContainerized Kubernetes path.

[2] Agentic RAG

Agentic RAG, grounding & memory

A state machine that handles retrieval, generation, grounding, citations, human review and resuming.

Retrieval context quality

Scoped evidenceUser · course · material.
Up to 5 documentsRetrieved per query, with history and preferences as context.
94% relevanceEvaluated responses.

LangGraph workflow

Parallel retrieval → generation → grounding check → citation or human review → saved state.

Fig. 3 · Low grounding routes to human review instead of a citation.

Memory layers

Fig. 4 · Four persistence roles kept separate.

Engineering decisions

LangGraphExplicit state + routing.
Grounding + review−38% hallucinations.
Memory layersSeparate persistence roles.
Inline citationsExpose evidence path.

[3] Evaluation

Evaluation & ingestion engineering

Quality gates made retrieval and prompt changes repeatable, and async ingestion turned student files into searchable evidence.

Evaluation system

Every retrieval or prompt change ran through a 500+ case regression suite across 8 domains.

Fig. 5 · Release the change, or revise and re-run.

Quality signals

500+ test casesEvaluation coverage.
8 domainsEvaluation breadth.
89% faithfulnessRAGAS.
−55% manual QAAutomated evaluation.

Async ingestion (Document AI)

PDF, DOCX and image/OCR content moved off the upload request, with saved progress, retries, timeouts, cleanup and bounded vector writes.

Document processing

Fig. 6 · From upload to pgvector.
−60%
document-processing time
optimized extraction, OCR, chunking, embedding, ingestion

[4] Deployment

Microservices, security & deployment

Three FastAPI services behind an authenticated FastAPI gateway, deployed to AWS EKS.

Three services plus a gateway: document processing, RAG chat and practice each had their own scope, and the signed-in identity passed from Supabase through the server and FastAPI layers.

Identity path

Identity and delivery were explicit boundaries, not trust in the client.

Fig. 7 · Authenticated identity scopes every downstream request.

Delivery path

Fig. 8 · From GitHub Actions to AWS EKS.

Production controls

SecurityScoped identityServer-side auth and JWT passing limited retrieval to the signed-in student.
ArchitectureService boundariesIngestion, RAG and practice each had their own job and their own ways to fail.
ReliabilityRuntime controlsStreaming, async processing, retries, timeouts, bounded writes, checkpoints, health checks and structured logs.

Production delivery (AWS EKS)

The Python and FastAPI services ran in Docker containers and shipped through a GitHub Actions CI/CD pipeline. Tooling included Pytest, Pydantic, Black, Ruff and health and readiness checks.

[5] Results

Results, engineering decisions & lessons

The outcomes, architecture choices and reliability lessons from building ThirdSlate end to end.

SystemResultFraming
Product1,000+ studentsReal product usage
AI quality89% faithfulness | 94% relevance500+ cases
Grounding / QA−38% hallucinations | −55% manual QABaseline + automated QA
Ingestion−60% processing timeOptimized multimodal pipeline

Architecture decisions

LangGraphExplicit state and routing for retrieval, grounding, review, citations and resuming.
Service boundariesIngestion, chat and practice are separate because their load and failures differ.
Memory layersKnowledge, session recall, preferences and graph state are stored separately.
Eval gatesPrompt and retrieval changes pass repeatable evaluation, not spot checks.

Engineering lessons

01
Grounding is control.

Reliability improved once grounding was an explicit decision step, not just a prompt instruction.

02
Eval before release.

A maintained benchmark made changes comparable and cut repetitive manual QA.

03
AI is systems.

Good model behavior depended on ingestion, identity, memory, routing, storage and deployment all working together.

Note on scope

High-level architecture · reported metrics · no confidential data · backed by production use.

Want to talk about any of this?

Happy to go deeper on grounding, evaluation gates, or the ingestion pipeline.

Email LinkedIn GitHub
© 2026 Tharun Pranav SakthivelHome