Tekkscope: Full-Stack AI Research Platform

I built the backend behind a research dashboard, public API and MCP tools: search, extraction, streaming, reports and deployment.

ordered LLM providers

sync · async · tool calls

streaming workers

Redis cross-worker delivery

MCP tools

JSON-RPC over stdio

report formats

PDF · DOCX · Markdown

TechnologiesPython,FastAPI,LangChain,LangGraph,SearXNG,Crawlee,Playwright,Redis,Supabase,Next.js,React,Docker,Kubernetes,EKS,GitHub Actions

[1] Surfaces

Product surfaces & engineering ownership

Shared core behind browser, API, and MCP workflows.

Shared research core

Three ways in, one source-backed research system.

Fig. 1 · Dashboard, API, and MCP converge on one research core.

I own the FastAPI research routes, model routing, retrieval and extraction, SSE and Redis streaming, MCP, reports, natural language to structured output (NLTS), Supabase data flows, deployment and backend tests. I built the core research platform.

Three access surfaces, one system

Web users, API clients and outside agents all use the same platform logic.

Fig. 2 · One platform logic, three entry points.

Ownership ledger

AreaDetailOwnership

Research APITyped routes + workflow orchestrationPrimary owner

Search retrievalSearXNG · query expansion · dedupePrimary owner

Real-time streamingSSE · asyncio queues · Redis Pub/SubPrimary owner

Artifacts & dataReports · structured JSON · Supabase persistencePrimary owner

Model routingOrdered failover · sync / async / toolsPrimary owner

Browser extractionCrawlee / Playwright · bounded fan-outPrimary owner

Agent surfaceMCP JSON-RPC server · 7 toolsPrimary owner

Delivery & testsDocker · EKS · CI/CD · backend testsPrimary owner

Primary capabilities

Link search · Extracted search · Lens · DeepLens · ReportLens · Tek Chat · NLTS. Seven features share retrieval, extraction, model, streaming and storage code instead of duplicating it.

[2] Retrieval

Search, extraction & research pipeline

Finding sources and extracting pages in a real browser are shared by every research mode.

Source-backed retrieval

Optional query expansion fans out into concurrent search, deduplication, a capped browser run and cleaned source output.

Fig. 3 · Each browser scrape has an 8-second extraction deadline.

Reused across modes

Discovery onlyLink searchSearXNG search, optional expansion, concurrent result pages, URL deduplication and a capped result set.

Discovery + extractionExtracted searchBrowser-rendered extraction returns cleaned source text and metadata.

Research modesLens · DeepLens · ReportLensQuick, deep and report workflows all reuse the same search and extraction.

Engineering decisions

Browser-rendered extractionCrawlee + Playwright for pages that require rendering.

Bounded browser concurrencySemaphore fan-out + 8-second scrape deadline.

Link-only search stops after finding sources. Extracted search and research continue through browser extraction before the model writes an answer or report.

[3] Routing & streaming

LLM routing, failover & real-time streaming

Provider failover and streaming are separate layers inside one research request.

Ordered model routing

The same ordered routing serves synchronous, asynchronous and tool-enabled calls.

Fig. 4 · Provider failure falls through within the same request.

Real-time SSE streaming

Each user has a bounded local queue, and Redis carries events across the 8 workers.

Fig. 5 · Redis carries events between workers; queues serve the client.

If Redis is off or down, local delivery still works. Stream events cover progress, response chunks, citations, downloads and end of stream.

Streaming runs across 8 workers in production. Redis delivers events between workers, and the connection manager de-duplicates messages and bounds queues.

[4] Agents & output

Agent access, structured output & stored artifacts

The same core serves outside agents, JSON for machines and downloadable long-form reports.

MCP agent surface (JSON-RPC over stdio)

Seven tools use the same search, extraction, research, saved-source and report code as the product.

Fig. 6 · External agents reach the shared core through MCP.

Report generation (ReportLens)

Reports are saved as files you can download.

Fig. 7 · From report research to a download URL.

Natural language → structured JSON

A caller-supplied or inferred schema feeds quick research and constrained extraction.

Fig. 8 · Provided or inferred schema constrains the JSON output.

NLTS returns structured_output, schema_used and research_summary. Report workflows save the generated files and create download links.

[5] Infrastructure

Infrastructure, reliability & engineering decisions

How it's deployed, what keeps it reliable, and what this write-up does not claim.

Production delivery (Docker + Kubernetes on EKS)

GitHub Actions deploys the backend and Redis manifests, with the backend image pinned to the CI commit.

Fig. 9 · From GitHub Actions to EKS.

Reliability mechanisms

BoundedPer-user queues: stream backpressure boundary.

8 secScrape deadline: browser extraction limit.

OrderedProvider failover: same-request fallback.

8 workersStreaming topology: Redis cross-worker delivery.

Verified results

SystemVerified factFraming

Model routing8-provider ordered failoverAll call types

Streaming8-worker SSE topologyRedis + fallback

MCP7 JSON-RPC tools over stdioShared research core

Engineering decisions

Shared retrieval primitivesSame source discovery / extraction across modes.

Shared core across surfacesWeb, API, and MCP use one research system.

Note on scope

High-level architecture and named technologies only. No secrets, customer data, or unverified performance, uptime or quality numbers.

Want to talk about any of this?

Happy to go deeper on provider failover, streaming, or browser extraction.

Email LinkedIn GitHub