Vibe Coding Forem: Anmol Baranwal

Open Source Toolkit for Building AI Agents in 2026

Anmol Baranwal — Thu, 21 May 2026 14:59:12 +0000

It used to take a lot of effort to get your first PR merged in open source. Now you can ship something real in a weekend thanks to coding agents like Claude Code.

But that also means the noise went way up. I saw the wave of openclaw repos a while back but the ratio of hype to actual maintenance is quite different.

I have a habit of exploring new projects almost every day, been doing that for 2+ years. Lately I have been deep in the AI agents space and these are the repos that actually stuck.

These aren't random projects, I have come across these in my journey and built with some of them. Agent harness, frontend stack, engineering skills for coding agents, voice agents, browser automation, computer use and a lot more. If you are building AI agents in 2026, this list is for you.

If you are new to open source, check out this free guide I made a while back. For any project you're considering: look for a CONTRIBUTING.md and a healthy community profile.

Let's jump in.

1. Frontend & UI Layer - CopilotKit

CopilotKit is the frontend stack for Agents. Most agent stacks give you the backend and leave the user-facing layer entirely to you.

CopilotKit is the layer that provides all the building blocks. Chat components, hooks, headless UI for custom agent interfaces, persistent threads, human-in-the-loop, shared state and a built-in Inspector for debugging.

They support all three generative UI patterns in one runtime. Basically, it allows the agent to show components rather than just describing them (A2UI by Google is one pattern).

The part I really like is that they have a mcp server for coding agents that lets coding agents fetch live docs without any usage limit.

You can connect directly to any LLM in just a few lines without any agent framework on the backend and make it context-aware about your app. They also have 13+ first-party integration support with major frameworks.

import {
  CopilotRuntime,
  copilotRuntimeNextJSAppRouterEndpoint,
} from "@copilotkit/runtime";
import { BuiltInAgent } from "@copilotkit/runtime/v2"; 
import { NextRequest } from "next/server";

const builtInAgent = new BuiltInAgent({ 
  model: "openai:gpt-5.5",
});

const runtime = new CopilotRuntime({
  agents: { default: builtInAgent }, 
});

export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    endpoint: "/api/copilotkit",
  });

  return handleRequest(req);
};

You can enable multimodal attachments like images, PDFs, audio, video & pass reasoningEffort to control how hard the model thinks.

The framework-agnostic design is what makes it practical. It's built on AG-UI protocol, the open event protocol for agent-user interaction now adopted by Google, AWS, Microsoft, LangChain, and many more. So if you switch any framework or protocol, everything on the frontend stays exactly the same.

I have used it for a couple of my recent projects, including a job search assistant I built using LangChain Deep Agents & it helped me surface all the things it was doing under the hood.

CopilotKit has 31.5k stars on GitHub.

Star CopilotKit ⭐️

Alternatives

TanStack AI - Framework-agnostic, vendor-neutral AI SDK from the TanStack team. Strong TypeScript support, modular adapters per provider. Direct alternative to Vercel AI SDK without the Next.js coupling.
Vercel AI SDK - Good for streaming and tool calling in Next.js. Stateless and tied to the Vercel ecosystem.
Tambo - React SDK focused purely on generative UI. Still early and not the full agent-chat stack.
Assistant UI - Headless React primitives for building chat UIs.
agent-native - Framework from Builder.io where agent and UI share the same action model. Define actions once, expose them to both. No separate agent API - if the UI can do it, the agent can and vice versa.

2. Skills & Plugins - agent-skills

Anthropic shipped the Skills format and the ecosystem took off fast. A lot of people are even saying MCP is dead because of Skills (I don't really believe so).

A skill is basically a directory containing a SKILL.md file that contains organized folders of instructions, scripts, and resources that give agents additional capabilities.

The official repo has 138k stars & it's worth reading the engineering blog to understand how progressive disclosure works in practice.

agent-skills by Addy Osmani is the gold. 23 production-grade engineering skills with 7 slash commands that map to the full dev lifecycle (/spec, /plan, /build, /test, /review, /ship).

Hard exit criteria, anti-rationalization tables, progressive disclosure. Encodes Google engineering culture -- Hyrum's Law, the Beyonce Rule, trunk-based development.

It has 43.8k stars on GitHub. Here are all the skills included.

Star Agent Skills ⭐️

agent-skills/
├── skills/                            # 23 skills (22 lifecycle + 1 meta)
│   ├── interview-me/                  #   Define
│   ├── idea-refine/                   #   Define
│   ├── spec-driven-development/       #   Define
│   ├── planning-and-task-breakdown/   #   Plan
│   ├── incremental-implementation/    #   Build
│   ├── context-engineering/           #   Build
│   ├── source-driven-development/     #   Build
│   ├── doubt-driven-development/      #   Build
│   ├── frontend-ui-engineering/       #   Build
│   ├── test-driven-development/       #   Build
│   ├── api-and-interface-design/      #   Build
│   ├── browser-testing-with-devtools/ #   Verify
│   ├── debugging-and-error-recovery/  #   Verify
│   ├── code-review-and-quality/       #   Review
│   ├── code-simplification/          #   Review
│   ├── security-and-hardening/        #   Review
│   ├── performance-optimization/      #   Review
│   ├── git-workflow-and-versioning/   #   Ship
│   ├── ci-cd-and-automation/          #   Ship
│   ├── deprecation-and-migration/     #   Ship
│   ├── documentation-and-adrs/        #   Ship
│   ├── shipping-and-launch/           #   Ship
│   └── using-agent-skills/            #   Meta: how to use this pack
├── agents/                            # 3 specialist personas
├── references/                        # 4 supplementary checklists
├── hooks/                             # Session lifecycle hooks
├── .claude/commands/                  # 7 slash commands (Claude Code)
├── .gemini/commands/                  # 7 slash commands (Gemini CLI)
└── docs/                              # Setup guides per tool

Alternatives

skills.sh - The npm for agent skills (marketplace). Install any skill with npx skills add <owner/repo>. The leaderboard surfaces what developers are actually using rather than what got hyped on launch day.
taste-skill - Portable design taste skills (minimalist, brutalist, GPT-tuned) that fix the generic-looking AI slop. One of the few skills that visibly changes what the agent produces. I have been using it for a couple of months.
Repomix - Packs an entire repo into one AI-friendly file. Pick when you need the agent to see the whole codebase at once.

3. Computer Use - UI-TARS Desktop

Most computer-use agents take a screenshot and ask a generalist VLM to guess pixel coordinates. UI-TARS was trained end-to-end on GUI grounding - it understands UI elements as a first-class concept rather than image regions to click on.

What I find really interesting is the "System-2 reflection" -- after each action, it compares before/after screenshots and generates a corrective plan if something didn't land right, instead of just going through a broken sequence.

It scores higher on OSWorld vs Claude Computer Use. Personally, I believe practical usage matters a lot more than benchmarks. 😅

They also ship Agent TARS - a CLI and Web UI that brings the same vision + MCP tool integration to your terminal and browser.

You can say something like: Please help me book the earliest flight from San Jose to New York on September 1st and the last return flight on September 6th on Priceline. All the demos are in the readme.

It has 34k stars on GitHub.

Star UI-TARS Desktop ⭐️

One very interesting repo I found is Sutando. It's a personal AI agent for macOS and runs on your Claude Code subscription with minimal extra costs.

The use cases are pretty wild. You can say "join my 2pm call" - it reads your calendar, joins Zoom via the desktop app or Google Meet via browser, takes screenshots to identify participants, does live research when someone asks a question and writes you a summary when the call ends.

Or you can call it from your phone, say "summon" - it opens Zoom with screen sharing and you control your computer by voice while walking around.

When you're not giving it tasks, Sutando runs an autonomous build loop - it monitors its own health, detects patterns in how you work, discovers new skills, and builds missing capabilities, which is crazy.

It has like 300 stars on Github but it is genuinely interesting.

Alternatives

Midscene - Also from ByteDance's Web Infra team. Vision-driven UI automation across web, Android, and iOS from one API. Integrates with Playwright and Puppeteer, ships a Chrome extension, CLI, and MCP server.
Agent-S - Hierarchical planning approach that builds a knowledge base from past interactions and uses it to plan future tasks. Benchmarks well on OSWorld and WindowsAgentArena.
Bytebot - Self-hosted AI desktop agent in a containerized Linux environment. The agent gets its own full virtual desktop -- browser, file system, password manager, any app. docker-compose up and it's running.
cua - macOS/Linux VM sandbox so the agent runs on a virtual machine, not your real machine.
OpenHands - Full developer environment that can browse, write code, run tests, and commit PRs. Covered again under Coding Agents.

4. Agent Orchestration - LangGraph

LangGraph is the stateful graph runtime built on LangChain. It is the most mature framework for building, managing, and deploying long-running, stateful agents.

The loop is a graph. Every step is a node. State is typed and checkpointed. You can pause at any node, serialize the entire state to disk, resume on a different machine days later.

They also ship Deep Agents - a coding agent harness on top of LangGraph with planning, filesystem tools, sub-agents, and context compression, if you want to skip writing the graph yourself.

Combining it with other products like LangSmith Engine, LangChain, Deep Agents gives developers a full suite of tools for building agents. And it's super useful for debugging.

If you're confused, here's a simple distinction:

LangChain - agents via chains and create_agent. Simple, fast to get started, less control over the state. The foundation on which everything else is built. If the process dies, the agent starts over.
LangGraph - stateful graph runtime built on LangChain. You can replay from any checkpoint to debug what went wrong.
Deep Agents - harness built on top of LangGraph.

LangGraph has 32.3k stars on GitHub.

Star LangGraph ⭐️

Alternatives

Agno - lightweight support for agents needing persistent memory and multimodal inputs. Ships with AgentOS, a pre-built FastAPI server with sessions, streaming, RBAC, and observability. Claims 529x faster instantiation than LangGraph.
Mastra - TypeScript-first with RAG, observability, MCP, and workflows baked in. Pick if your team lives in JS/TS rather than Python.
Pydantic AI - Type-safe agent framework from the Pydantic team. Pick when you want validated structured outputs without writing the validators yourself.
Google ADK - Google's official agent dev kit with native Vertex AI integration. Pick if you're building on Google Cloud.
PocketFlow - A 100-line LLM framework. Genuinely minimal. Pick when LangGraph feels like too much infrastructure.

5. Coding Agent Harness - Deep Agents

A harness is everything around the model that makes it an agent - tools, state, planning, memory, feedback loops, guardrails.

You can say: "Agent = Model + Harness". LangChain proved this matters more than most teams expect: harness-layer changes alone moved the same model from 52.8% to 66.5% on Terminal Bench 2.0, jumping from Top 30 to Top 5. No model change.

Deep Agents is LangChain's batteries-included harness built on LangGraph. Planning, filesystem tools, sub-agents, and context compression out of the box.

User goal
  ↓
Deep Agent (LangGraph StateGraph)
  ├─ Plan: write_todos → updates "todos" in state
  ├─ Delegate: task(...) → runs a subagent with its own tool loop
  ├─ Context: ls/read_file/write_file/edit_file → persists working notes/artifacts
  ↓
Final answer

The core problem with long-running agents is that they accumulate tool call results until the context window fills -- causing context poisoning, distraction, and confusion.

Their fix:

Large tool outputs go to a virtual filesystem instead of the prompt
Skills load only frontmatter at startup, full content on demand
Conversation history gets compressed as sessions grow
Sub-agents run in their own context window, the main agent only gets the final result

You can build lots of stuff around this like Deep Research Assistant.

Deep Agents has 23.1k stars on GitHub.

Star Deep Agents ⭐️

Alternatives

Hive - Outcome-driven agent development framework. Agents evolve based on whether they actually achieved the goal, not just whether they completed steps.
Browser Harness - From the Browser Use team. Self-healing harness that gives the LLM maximum freedom -- instead of wrapping Chrome with thousands of lines of heuristics, it lets the LLM use CDP directly and add its own tools when it needs them. Different philosophy from most browser frameworks.
Archon - Open-source harness builder for AI coding. Describe what you want and it generates a deterministic, repeatable agent harness for you.

6. Open-Source Coding Agents - OpenCode

I have used Claude Code and Codex extensively. Both are great but locked to their respective ecosystems.

OpenCode is the open-source alternative - terminal-native, 75+ provider support, LSP integration, multi-session (run multiple agents in parallel on the same project), privacy-first.

What makes it stand out: genuinely provider-agnostic from day one. You can switch between Claude, Gemini, GPT-5 and local models in the same session without reconfiguring anything. Most other coding agents have a preferred model set up in the defaults.

You can also share a link to any session for reference or to debug. It is available as a terminal interface, desktop app, and IDE extension - though I have only used the terminal.

OpenCode has 162k stars on GitHub.

Star OpenCode ⭐️

Alternatives

Codex (OpenAI) - OpenAI's official terminal coding agent. Pick when you want first-party support and the cleanest GPT-5 integration.
Gemini CLI - Google's official terminal agent with 1M token context. The free tier is hard to beat for experimentation.
Cline - VS Code extension with per-step approval. Pick if you want IDE-native control rather than a terminal.
Aider - Git-native terminal pair programmer. 70%+ of Aider's own code is now written by Aider. Fast and model-agnostic.
OpenHands - Full agentic developer environment that can browse, run shell, and commit PRs. Heavier than the others.
Goose - Block's extensible coding agent with first-class MCP and a clean extension model.

7. Browser Automation - Browser Use

Browser Use gives your agent a browser. Point it at a URL, describe what you want done, and it clicks, types, and navigates. You write intent, not selectors -- the agent reads the DOM and figures out the interaction itself.

The LLM-first design means you describe intent, not selectors, and the agent figures out the DOM.

The reason they are so good is that they have built purpose-built LLMs specifically for browser tasks. Their bu-ultra model scores 97% on Mind2Web vs 62% for claude-opus-4-6.

The open source library works with any model but their custom ones are what the benchmarks run on.

They also have a desktop app that controls your local Chrome directly and Browser Use Box - a 24/7 Claude Code agent you can deploy on any $5 VPS and control via Telegram.

# pip install browser-use-sdk

from browser_use_sdk.v3 import AsyncBrowserUse

client = AsyncBrowserUse()

result = await client.run(
  "Go to amazon.com, extract 200 products with name, price and reviews, save to products.csv"

It has 94k stars on GitHub.

Star Browser Use ⭐️

Alternatives

Stagehand - Four primitives: act, extract, observe, agent. Deterministic step-by-step control when you need it, autonomous execution when you don't. Self-healing -- "click submit" survives page redesigns because it's resolved by AI at runtime, not hardcoded selectors.
Playwright MCP - Microsoft's MCP server wrapping Playwright. Pick if you already write Playwright tests and want your agent to drive the same browser.
Skyvern - Uses a swarm of agents + computer vision to operate on sites it's never seen before. No XPaths, no selectors -- maps visual elements to actions in real-time. Also ships a no-code workflow builder.
Scrapling - Adaptive scraper that survives selector drift. Bypass anti-bot systems like Cloudflare Turnstile out of the box. Allows concurrent, multi-session crawls with automatic proxy rotation.

8. Web Scraping & Ingestion - Firecrawl

Agents need to pull content from the web constantly - research, monitoring, competitive intel, RAG pipelines.

Most scrapers give you raw HTML full of nav menus, ads, and cookie banners that burn tokens and confuse the model.

Firecrawl converts any website into clean LLM-ready Markdown or structured JSON. Three core endpoints that cover everything:

/search for web search with content already extracted
/scrape for full page Markdown
/extract for structured JSON via natural language prompt.

They also have an /agent endpoint where basically you describe what you want in natural language and it searches, navigates, and extracts across multiple sites autonomously. No URLs needed.

import Firecrawl from '@mendable/firecrawl-js';
import { z } from 'zod';

const firecrawl = new Firecrawl({ 
  apiKey: 'fc-YOUR-API-KEY' 
});

const schema = z.object({
  companies: z.array(z.object({
    name: z.string(),
    founders: z.array(z.string()),
    funding: z.string().optional(),
    website: z.string()
  }))
});

const result = await firecrawl.agent({
  prompt: 'Get all YC W24 companies',
  schema: schema
});

Their FIRE-1 navigation agent (beta) can autonomously navigate complex sites, clicks, scrolls, fills forms and handles multi-step flows before extracting. Pages behind login or pagination are no longer a blocker.

There's a lot more so feel free to explore. It has 122k stars on GitHub.

Star Firecrawl ⭐️

Alternatives

Gitingest - Replace 'hub' with 'ingest' in any GitHub URL and get a prompt-friendly extract of the codebase. Filter by file size, include/exclude specific paths, and support private repos too.
Crawl4AI - Open-source, self-hosted, no API key needed. Built specifically for RAG pipelines - LLM-aware chunking, BM25 content filtering, full-site crawling with depth control. Pick when you want full control without per-request fees.
Jina Reader - Prepend r.jina.ai/ to any URL and get clean Markdown. Zero setup, zero SDK. Pick for quick one-off page conversion or prototyping, where you don't want any configuration.
ScrapeGraphAI - Prompt-driven scraping. Describe what you want extracted in natural language and it builds the scraping workflow. Pick when you need structured JSON extraction, not just Markdown.

9. Multi-Agent Frameworks - CrewAI

CrewAI is the most widely adopted multi-agent framework right now. You define a crew of agents with specific roles, goals, and tools and CrewAI handles how they collaborate to finish a task.

The role-based model (PM, researcher, engineer) is intuitive because it maps to how you would actually divide work between people.

It's the easiest entry point into multi-agent systems. Well-documented, large community, with examples for almost everything.

The tradeoff is control - agent-to-agent communication is mediated through task outputs, not direct messaging and there's no built-in checkpointing for long-running workflows.

CrewAI Flows (the newer event-driven mode) addresses some of this for predictable pipelines. For cyclical workflows with feedback loops, most teams end up moving parts to LangGraph.

CrewAI has 51.6k stars on GitHub.

Star CrewAI ⭐️

Alternatives

AG2 - The community fork of AutoGen (Microsoft moved AutoGen to maintenance mode). Conversable agents that talk to each other in group chats, swarms, and nested chats. Better for research and custom communication patterns.
Microsoft Agent Framework - The enterprise successor to AutoGen. Stable APIs, long-term support, A2A and MCP built in. Pick when you need long-term support guarantees.
OWL - Multi-agent automation framework built on CAMEL. Uses a planning agent + execution agent model for long-horizon real-world tasks. Ranked #1 on the GAIA benchmark among open-source frameworks.
MetaGPT - Simulates a software company with PM, architect, and engineer agents. Best for code-generation pipelines.
AgentScope - Alibaba's production framework with realtime voice, MCP, A2A, and OTel built in.

10. Document Processing - Docling

Feeding a PDF to an agent and watching it miss things that were right there in the table is frustrating. Most parsers extract raw text and lose the structure - tables flatten, multi-column layouts collapse, equations come out unreadable.

Docling is IBM Research's document conversion engine, now donated to the Linux Foundation. It uses Granite-Docling-258M - a parameter vision-language model purpose-built for document conversion that rivals systems several times its size.

The output uses DocTags, a universal markup format IBM Research developed that captures every page element, its type, position and reading order. Not only Markdown. This is what makes the RAG downstream more accurate.

Red Hat called it "the number one open source repository for document intelligence". Sharing this for no reason 😅

from docling.document_converter import DocumentConverter

source = "https://arxiv.org/pdf/2408.09869"  # file path or URL
converter = DocumentConverter()
doc = converter.convert(source).document

print(doc.export_to_markdown())  # output: "### Docling Technical Report[...]"

Docling has 60.1k stars on GitHub.

Star Docling ⭐️

Alternatives

LlamaIndex - Full RAG framework with 160+ data connectors. Pick when you need orchestration across many data sources, not just document parsing.
MinerU - PDF parser with SOTA table and formula extraction. Pick for technical or scientific documents where math and equations dominate.
RAGFlow - DeepDoc pipeline that handles parsing, chunking, and retrieval end-to-end. Pick when you want the full RAG stack, not just the parser.
Marker - PDF, EPUB, PPTX to Markdown with high fidelity. Faster than Docling, lower GPU requirements. Pick for clean plain-text output at scale.
PaddleOCR - The OCR engine powering MinerU, RAGFlow, and OmniParser under the hood. 100+ languages, browser SDK. Pick when you need raw OCR power directly.
Unstructured - 65+ file formats including emails, spreadsheets, and images. Pick for mixed-input pipelines beyond just PDFs.

11. Voice Agents - Pipecat

Building voice agents is still harder than it should be. The AI part is mostly solved in my opinion. The hard part is everything around it, like interruption handling, transport, latency budgets across STT/LLM/TTS boundaries.

Pipecat is a Python framework from Daily for real-time voice and multimodal AI agents. STT, LLM, and TTS are composable frame processors like Unix pipes, but for voice.

The reason I picked it: you can swap any STT, LLM, or TTS without rewriting the pipeline. Most devs don't want to be locked into one provider, especially when voice model quality is still changing fast.

Silero VAD (voice activity detection) handles interruption at the framework level - voice activity mid-response stops the audio and re-engages the LLM automatically.

Pipecat Flows then adds predefined conversation paths with dynamic transitions so agents stay on task without going off-script mid-call.

import {
  PipecatAppBase,
  ConnectButton,
  UserAudioControl,
} from "@pipecat-ai/voice-ui-kit";

<PipecatAppBase
  transportType="smallwebrtc"
  connectParams={{ webrtcUrl: "/api/offer" }}
>
  <div>
    <ConnectButton />
    <UserAudioControl />
  </div>
</PipecatAppBase>

You can go to the website and try it out live. As you can see, I asked How can I bring it to my app & it opened the code panel as well as gave me a proper explanation.

There were multiple panels of code (install, pipeline, final implementation). The part that surprised me was that it was able to detect that I played around with the code when I stopped it, which makes it context-aware to the app.

It has 12k stars.

Star Pipecat ⭐️

The other day I was in a project where the agent would create an appropriate form based on the call and it worked. Generally, it's very hard to collect data from just voice.

Alternatives

LiveKit Agents - The WebRTC stack that OpenAI (ChatGPT Voice) and Meta run on. Room-first, not pipeline-first - your agent joins as a participant. Native telephony, semantic turn detection, MCP support. Pick when you need production infrastructure bundled with the framework.
fish-speech - SOTA open TTS with multilingual zero-shot voice cloning. Pick when output quality matters more than orchestration.
Moonshine - Very low-latency on-device STT. Pick when latency is the constraint and cloud isn't an option.
Whisper - OpenAI's STT model. The default transcription layer for custom voice stacks.
GPT-SoVITS - Few-shot voice cloning from 1 minute of audio, zero-shot from 5 seconds. Chinese, English, Japanese, Korean. Pick for cloned-voice TTS with minimal training data.
CosyVoice - Alibaba's multilingual zero-shot voice generation with 150ms latency. Pick for non-English voice quality.

12. Visual Builders - Langflow

Langflow is a drag-and-drop agent pipeline builder, IBM-backed. Connects to any LLM, any vector DB, any tool.

The way the CTO puts it: "Langflow is essentially an API designer. The flow becomes an endpoint you can call from anywhere."

What makes it interesting: the flow becomes a callable REST endpoint. You prototype visually, then your engineers call it an API from any codebase.

Every component exposes its Python class so you can extend with custom logic without leaving the tool.

You can find a lot of pre-built templates to get started. Here is a Supply chain risk monitoring workflow that scores disruption risk by linking supplier and route data with live news signals.

It has 149k stars on GitHub.

Star Langflow ⭐️

Alternatives

Dify - LLM app platform with a visual agent builder, RAG pipeline, and plugin marketplace. Pick the more polished UI and built-in app shell.
Sim - Drag-and-drop agent orchestration, very actively developed. The cleanest 2026-native alternative to Langflow.
n8n - Workflow automation with 400+ integrations and strong AI nodes. Pick when the workflow is mostly cross-service automation with an agent at the center.
Flowise - Simpler no-code LangChain builder. Pick for non-technical users who need to build agent workflows without writing code.
Coze Studio - ByteDance's open-source visual agent platform. Agent builder with RAG and plugins, workflow engine with loops and custom Python execution.

13. MCP & Tool Integration - Composio

MCP servers solved how agents connect to tools. But managing OAuth, token refresh, and keeping 1000+ integrations working is still your problem.

Composio is the integration layer between your agent and real-world tools with managed auth.

The part I liked the most is the Tool Router - one MCP endpoint that dynamically discovers and loads the right tools based on what the agent is trying to do. Instead of pre-loading every tool and bloating the context, it surfaces only what's relevant.

They have 28.4k stars on GitHub.

Star Composio ⭐️

Alternatives

LiteLLM - One unified API across 100+ LLM providers. Azure, Bedrock, Anthropic all look like OpenAI to your code. Cost tracking, routing, fallback. Pick the model gateway layer.
MindsDB - SQL access to 200+ data sources from a single interface, no ETL. Also ships as an MCP server. Pick when your agent needs to read data, not take actions.
ACI - 600+ tool integrations via a unified MCP server. Self-hosted Composio alternative.
Portkey AI Gateway - 1,600+ models with guardrails built in. Pick when gateway latency matters.

14. Sandboxing & Code Execution - E2B

When an agent generates and runs code, that code needs somewhere safe to execute. Regular Docker containers share the host kernel - one vulnerability and untrusted code breaks out.

E2B provides isolated sandboxes that let agents safely execute code, process data, and run tools. Their SDKs make it easy to start and manage these environments.

It runs agent-generated code in Firecracker microVMs - each sandbox gets its own dedicated kernel. ~150ms boot, full Linux filesystem, Python, Node, and common packages pre-installed.

import { Sandbox } from 'e2b'

const sandbox = await Sandbox.create() // Needs E2B_API_KEY environment variable
const result = await sandbox.commands.run('echo "Hello from E2B Sandbox!"')
console.log(result.stdout)

Nothing touches the host machine. Manus uses it to run 27 different tools. Perplexity uses it for data analysis. Hugging Face uses it to replicate DeepSeek-R1.

You can check some of the examples in their cookbook repo on how to use all of this.

It has 12k stars on GitHub.

Star E2B ⭐️

Alternatives

OpenSandbox - Alibaba's internal sandbox infrastructure. Covers coding agents, GUI agents, browser automation, VNC desktops, and RL training from one API. Docker and Kubernetes runtimes. Broader scope than E2B.
Daytona - ~90ms spin-up, open-source, persistent environments with Git integration and LSP. Pick when you need a state that persists across sessions.
microsandbox - Local, programmable sandboxes. Pick when you want sandboxes on the dev machine without cloud dependency.
Firecracker - The microVM tech under E2B, Lambda, and Fargate. Pick when you need the raw layer with full control.

15. Agent Memory - mem0

Memory is where most agent demos fall apart in production. The agent knows things in one session and forgets them in the next. We seriously don't want agents to die after closing the tab haha.

mem0 is the most widely deployed standalone memory layer right now. Instead of storing raw conversation chunks, it runs an extraction phase that identifies salient facts and distills them into compact natural language memories. Handles short-term, long-term, and entity memory behind one interface.

# Add a memory
messages = [
    {"role": "user", "content": "I'm a vegetarian and allergic to nuts."},
    {"role": "assistant", "content": "Got it! I'll remember your dietary preferences."},
]
client.add(messages, user_id="user123")

# Search memories
results = client.search(
    "What are my dietary restrictions?",
    user_id="user123",
)
# print(results)

Most major frameworks are building memory in such a way that LangGraph has checkpointing built in (state persists per thread, time travel, crash recovery) and CopilotKit has persistent threads via useThreads.

The gap worth knowing: Mem0 scores 49% on LongMemEval vs Zep's 63.8% - the difference comes from temporal reasoning.

For personalization memory it's the pragmatic choice. For agents that need to reason about how facts changed over time, Graphiti is the better pick.

It has 55k stars on GitHub.

Star Mem0 ⭐️

Alternatives

Graphiti - Zep's temporal knowledge-graph engine. Stores facts with validity windows -- not just what happened, but when and whether it's still true.
Letta (formerly MemGPT) - Memory built into the system prompt at runtime. The agent itself decides what to keep - not retrieved after the fact, baked into how it reasons.
Supermemory - Cross-agent memory API with plugins for OpenCode, OpenClaw, and Claude Code. Say "remember that this project uses Bun" and it saves. Context shows up automatically next session.
Cognee - Deterministic knowledge-graph memory pipelines. Pick for structured memory over large corpora.

16. Testing & Evaluation - DeepEval

Most teams skip evals entirely until something breaks in production. By then, you're debugging blind - no idea if it's the prompt, the retrieval, the model, or all three.

DeepEval brings eval into your test suite. Write assertions the way you write unit tests, run them in CI, and catch regressions before they ship.

50+ metrics covering RAG, agents, tool-use, multi-turn conversations, and safety. The agent-specific ones are the most useful: task completion, argument correctness, tool correctness, step efficiency. Also generates synthetic datasets for edge cases that are hard to collect manually.

It has 15.6k stars on GitHub.

Star DeepEval ⭐️

Alternatives

promptfoo - CLI-first eval and red-teaming in one tool. Pick when adversarial testing matters as much as accuracy metrics.
Phoenix - OTel-native tracing + evals from Arize AI. Auto-instruments LangChain, LlamaIndex, Mastra, Vercel AI SDK. Pick when you're already on OTel and want tracing and evals in one tool.
Opik - Comet's open eval and tracing platform. Pick when you want evals and observability together.
MLflow - Now has full LLM and agent eval features alongside the ML lifecycle tools. Pick when your team already runs MLflow for traditional ML.
garak - NVIDIA's LLM vulnerability scanner. Probes your agent for weaknesses before someone else does.
AI-Infra-Guard - Tencent's red teaming platform. Scans MCP servers, agent skills, and AI infrastructure. Pick when you want to find vulnerabilities before deploying.

17. Monitoring & Observability - Langfuse

Most teams find out their agent is broken when a user reports it. By then, you're debugging a black box - no trace of what the agent did, in what order, or where it went wrong.

Langfuse is the de facto open-source LLM observability stack. Traces, evals, prompt versioning, and cost tracking in one self-hostable package.

The trace shows exactly what the agent did, at what latency, with what cost. Debugging multi-step agents becomes a lot easier.

It has 27.6k stars on GitHub.

Star Langfuse ⭐️

Alternatives

Opik - Comet's tracing, evals, and dashboards. Pick for the Langfuse-equivalent with Comet's ML lineage behind it.
TensorZero - Gateway, observability, and optimization unified in one tool. Pick when you want one box instead of three.
Logfire - OTel-native LLM observability from the Pydantic team. Pick the tightest fit with Python and Pydantic-AI agents.
OpenLLMetry - OTel instrumentation library for LLM apps. Pick when you want agent traces feeding into an existing Grafana or Datadog backend.

A brief note on agent protocols

Three protocols run the modern agent stack. Worth knowing the difference:

MCP (Model Context Protocol) - agent to tools. Anthropic's standard, now under the Linux Foundation, adopted by OpenAI, Google, and Microsoft. GitHub
A2A (Agent-to-Agent) - agent to agent. Google's protocol for inter-agent communication. GitHub
AG-UI - agent to user. The open event protocol for agent-user interaction originated by CopilotKit and is now adopted by Google, AWS, Microsoft, LangChain, and Mastra. GitHub

Extra Resources

1) 12-Factor Agents - Dex Horthy's principles for shipping LLM software that actually works in production. The core insight: most successful AI products aren't autonomous agents -- they're mostly deterministic code with LLM steps placed at exactly the right points. One of my favorite repos in open source.

2) Generative Agents - The Stanford simulacra paper code. 25 agents in a Sims-like sandbox that wake up, form relationships, plan their days, and remember past interactions. The canonical reference for how memory, reflection, and planning work in multi-agent simulations.

3) AI Agents for Beginners (Microsoft) - 12-lesson structured course from Microsoft. Pick this if you want a guided path rather than piecing it together yourself.

4) HuggingFace Agents Course - Stronger on the model and tool-use fundamentals. Better for ML practitioners coming into agents from the model side.

5) Roadmap.sh AI Agents - Visual map of everything you need to know to build agents in 2026. Good starting point if you want to see the full picture before diving in.

Whew! This took a really long time to write, but I enjoyed every bit of it.

This list is based on my opinions and what I've actually seen get adopted in the open source community. If I missed something you think deserves a spot, drop it in the comments.

And on that note, here's what multi-agent collaboration actually looks like in 2026 😅

Have a great day! Until next time :)

Connect with me on GitHub, Twitter and LinkedIn.

thanks for reading!

Chat to build and schedule your own personal Hacker News email digest! 🎉

Anmol Baranwal — Sat, 16 May 2026 13:15:06 +0000

I have been working on a side project for the past few weeks and finally got it polished enough to share.

Hacker News is one of my favorite places to keep up with tech and come across good discussions. But there's so much happening there that I kept missing the posts I actually cared about.

Even checking it twice a day, the front page rarely lines up with my interests. Tried a few existing Hacker News newsletters, but they were too generic -- everyone gets the same top stories.

So I figured, why not let myself and others build their own email digest from live data and schedule it whenever they want? That's why I built HN Digest.

You can chat to build your own personalized Hacker News newsletter, preview it live with real HN data, and schedule it to your inbox -- daily, weekly, or monthly, 70+ timezones, no password, all free.

Here is the GitHub Repository and you can try it live at hndigest.vercel.app.

Here's what it does and how I built it, along with the architecture.

What it does

You just have to describe whatever you want in the chat editor: security news, ask HN, trending posts, discussions, show hn - it adds them to your newsletter.

There are 10 section types with per-section filters for keyword, timeframe (24h to 30d), minimum upvotes and story count.

So you can say "security news from the last 48 hours plus recent Show HN" -- it picks the right sections and applies the filters. Preview is a live React Email render so what you see is the actual email.

You can also just ask the chat:

to show all the sections available
delete or reorder
change the theme (light/dark) of the newsletter

Clicking a story's title takes you to the original site. Clicking comments takes you to the HN post.

Once you are happy, just activate using a magic link. You will need a Resend API key -- their free tier is 3000 emails/month, you will use about 30 even if you send daily.

Sends go through your Resend key (and your quota, not mine). You can pause or delete anytime from the dashboard.

Yes, it lands in my inbox every day but I often forget about it. I definitely need to change the time to evening! 🤣

I have included a self-host guide in the readme docs as well.

Here is the stack:

Resend + React Email for delivery
CopilotKit + Next.js for the chat editor
Neon Postgres for the database
Upstash QStash for scheduling
Firebase Hacker News API + Algolia for live data
AES-256-GCM for key encryption, JWT for auth

Fun fact: I have tried different ways to solve this problem before (web scraping, GitHub Actions) & I believe this is the most practical solution out of all the rest.

How it works

Here's the high-level flow.

Browser (editor + chat + live preview)
 ├→ /api/copilotkit  → LLM returns tool calls
 ├→ /api/preview     → React Email render → iframe
 └→ /api/activate    → encrypt key → Neon → create QStash schedule
                                           ↓ (HTTP cron callback)
                                    /api/send → Neon (fetch schedule)
                                             → User's Resend (send)

The chat doesn't directly write to the database. The LLM's tool calls only update the browser state. Nothing gets persisted to Neon until you hit "Activate". That's the only path that writes to the DB and creates a QStash schedule.

QStash is the only thing that triggers /api/send. The endpoint validates a CRON_SECRET header on every request, magic links and sessions are signed JWTs, and the Resend key stays AES-256-GCM encrypted at rest. Nothing sensitive is reachable from the browser.

I have added dedicated architecture docs to the repo, if you want to explore yourself.

The editor and live preview

Every change in the editor is a typed function call, not a blob of JSON. CopilotKit's useFrontendTool hook takes a zod schema and a handler -- the LLM chooses a tool, fills the params and the handler enforces what's allowed.

useFrontendTool({
  name: "add_section",
  parameters: z.object({
    type: z.enum(["hn-stories", "topic", "recent-gems", "high-signal", ...]),
    count: z.number().min(1).max(30).optional(),
    query: z.string().optional(),
    hours: z.number().optional(),
    minPoints: z.number().optional(),
  }),
  handler: async ({ type, count, query, hours, minPoints }) => {
    // Block duplicates for singleton types, strip invalid params, etc.
    const section = {
      id: `${type}-${Date.now()}`,
      type,
      props: { count, query, hours, minPoints },
    };
    setConfig(prev => ({ ...prev, sections: [...prev.sections, section] }));
    return `Added ${type} section`;
  },
});

The LLM never writes HTML or emits JSON that I have to validate. Guardrails live in the handler -- no duplicate singleton sections, and standard sections silently strip hours/minPoints they don't support.

Every config change hits /api/preview, which renders the React Email template server-side and returns HTML. The editor caches by config hash so repeated edits don't re-fetch.

const cacheKey = `${PREVIEW_CACHE_VERSION}:${JSON.stringify(config)}`;
const cached = previewCacheRef.current.get(cacheKey);
if (cached) { setPreviewHtml(cached); return; }

const { html } = await (await fetch("/api/preview", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ config }),
})).json();

previewCacheRef.current.set(cacheKey, html);
setPreviewHtml(html);

Activation and key encryption

You enter a Resend API key, add a recipient, and hit activate. The key gets encrypted with AES-256-GCM before it ever touches the database. It's decrypted only in memory at send time, never logged, never returned to the browser.

export function encrypt(text: string): string {
  const iv = crypto.randomBytes(16);
  const cipher = crypto.createCipheriv("aes-256-gcm", getKey(), iv);
  let enc = cipher.update(text, "utf8", "hex") + cipher.final("hex");
  return `${iv.toString("hex")}:${cipher.getAuthTag().toString("hex")}:${enc}`;
}

A magic link (signed JWT, 30-minute expiry) is sent through your Resend key. Click it and you're in your dashboard.

Per-user scheduling with QStash

I first tried this with Vercel Hobby cron but it runs once a day for the whole project, which doesn't work when each user wants their own time and timezone. QStash lets every activation create its own schedule so I ended up using this.

const cron = toCron(frequency, time, day);  // "0 9 * * 1" for Mon 9am
await client.schedules.create({
  destination: `${baseUrl}/api/send?id=${scheduleId}`,
  cron,
  headers: { "x-cron-secret": process.env.CRON_SECRET },
});

Timezone conversion happens client-side -- pick Asia/Tokyo 8am, the editor converts to 23:00 UTC, that's what gets stored and scheduled.

The send endpoint

On cron, it runs: decrypt → fetch HN → render → send. Protected by a shared header secret so only QStash can hit it.

// /api/send
if (req.headers.get("x-cron-secret") !== process.env.CRON_SECRET) return 401;

const record = await getScheduleById(id);
const data = await fetchNewsletterData(record);
const html = await render(<NewsletterEmail config={record} data={data} ... />);

const resend = new Resend(decrypt(record.encryptedResendKey));
await resend.emails.send({
  from: "HN Digest <onboarding@resend.dev>",
  to: record.recipients,
  subject,
  html,
});

Every send fetches fresh Hacker News data. No pre-rendering, no staleness. Sends go through your Resend key (and your quota, not mine), with a one-click unsubscribe that cancels the schedule. You can also pause it from the dashboard.

The pattern itself transfers easily. Swap the Hacker News API for Product Hunt, GitHub trending, Reddit, or any RSS feed, and you've got a personalized digest builder for that source. Same encryption, same chat editor, same per-user schedule.

This one was fun to build. A couple of my friends and I have been using it for the past month.

Would really appreciate some honest feedback :)

Try it here: hndigest.vercel.app
GitHub: github.com/Anmol-Baranwal/hndigest

Connect with me on GitHub, Twitter and LinkedIn.

thanks for reading!

Generative UI Project Ideas with Complete Source Code

Anmol Baranwal — Mon, 20 Apr 2026 12:31:15 +0000

Someone reverse-engineered Claude's Generative UI and rebuilt it for a macOS terminal agent. Native Swift window, interactive widgets streaming in under 50ms.

When Claude added Generative UI to their chat interface, people actually noticed. And developers started building wild projects using this concept.

Agents returning real components instead of text, drawing diagrams live, opening Excalidraw from chat.

Here are some impressive projects with complete source code. Demo videos are embedded so you can watch without leaving the page.

Let's jump in.

What is Generative UI?

The UI era as we knew it is dead. At least the assumption that every screen has to be built in advance.

Chat is great for a lot of things, but if you ask an AI to help you book a flight, you probably don't want to read a paragraph about it. You want to see the flights, pick one, and confirm right there in chat.

That's Generative UI. Which lets agents produce interfaces based on what the task actually needs. Here's a simple example!

Four distinct patterns exist right now, each making a different bet on how much the agent should control.

If you are interested in reading more about it, I have already covered everything in The Developer's Guide to Generative UI in 2026.

In some of the examples below, I have uploaded demo videos (from the repository) as unlisted YouTube videos and embedded them here, so you can watch them directly without opening any links.

1. OpenGenerativeUI - visualize anything, live in chat.

This is my personal favorite on the list. It's the open source version of what Claude brought to their chat interface.

Just ask the agent to visualize anything and it figures out what kind of output makes the most sense, then generates it step-by-step as live HTML inside a sandboxed iframe.

It knows when to use a flowchart vs a structural diagram vs an interactive simulation vs just plain text. LangGraph Deep Agents is used for controlling that decision.

The outputs are genuinely impressive. Algorithm visualizations like binary search and BFS vs DFS, 3D WebGL animations, charts, network diagrams, math plotters, sortable tables, interactive simulations.

Here is the demo video!

You can check the GitHub Repository & live demo.

It also ships a standalone MCP server for Claude Desktop and Cursor.

2. Shadify - generate and export Shadcn components.

You describe the UI in simple language and it generates real ShadCN components back, streamed live to your browser. When it looks right, just export it as clean code.

It can also search the web for context. For example, if you ask it to recreate Linear's pricing table -- it will pull the actual page, read it, and build with that context.

You will be surprised to know that Tyler and Mike from the CopilotKit team built this in 4 hours on the Web Dev Challenge. The full episode is on YouTube if you want to see the story behind it & a bunch of other demos.

It looks simple, but it's built using:

CopilotKit as agentic chat
Hashbrown for component schema
LangGraph on the backend
Tavily for search

You can check the GitHub Repository & live demo.

3. Pi Generative UI - reverse-engineered Claude's generative UI for a terminal agent.

Someone went deep on how Claude's generative UI actually works, exported conversations as JSON, pulled out the design guidelines verbatim from the tool call payloads, and rebuilt the whole thing for pi, a macOS terminal coding agent.

Ask it to show you how compound interest works and a live interactive widget opens in a native macOS window in under 50ms. No Electron, just a tiny Swift binary called Glimpse wrapping a WKWebView.

Widgets update token by token via morphdom diffing and can send data back to the agent via window.glimpse.send(). Chart.js, D3, and Three.js are all available via CDN.

It's macOS only and requires the Swift toolchain.

You can check the GitHub Repository.

4. Open MCP Apps Builder - build MCP apps from a prompt.

MCP is how agents connect to external tools like GitHub, Notion, Figma. MCP Apps extends that so those tools can respond with an actual interactive UI, live inside the chat.

Building that from scratch is still pretty involved. This project lets you describe what you want and a Mastra agent spins up a full MCP app in a live E2B sandbox.

You can also connect any existing MCP server by URL and test its tools directly from the sidebar.

You can check the source code.

5. Excalidraw Studio - describe anything, get a live editable diagram.

This is a complete diagramming app where you describe anything you want and watch it draw itself in chat, element by element.

When it's done, you can open it as a full local Excalidraw canvas, drag things around, add your own stuff, then bring it back to chat and keep editing with AI. That loop, going back and forth between chat and canvas, is what makes it different.

This is built on top of the official Excalidraw MCP server but heavily modified to add persistent workspaces, checkpoint storage, blank canvas creation, five diagram modes, toolbar and the "Edit with AI" feature. Everything is saved to disk by default.

You can check the GitHub Repository.

6. Voice Agent Generative UI - real forms inside voice conversations.

Voice agents are genuinely bad at collecting structured data. Spelling out an email address over a call is nobody's idea of a good experience.

This extends the ElevenLabs Conversational AI widget so the agent can pop a real form mid-conversation. User fills it in, validated data goes straight back to the agent. Add it via a script tag from unpkg, drop a prompt snippet into your agent, register three tools in your ElevenLabs dashboard.

Supports text, email, number, date, select, textarea, and more with real-time validation.

You can check the GitHub Repository and live demo.

7. Generative UI Playground - all four patterns in one place.

If you want to see how all the patterns of generative UI actually work under the hood, this one shows exactly that. All four patterns are running in the same chat interface so you can compare them live.

AG-UI protocol runs underneath all of them, with a Python LangGraph and LiteLLM agent on the backend for the declarative pattern.

You can check the GitHub Repository and live demo.

8. HN Digest - chat to build your own Hacker News newsletter.

Instead of subscribing to someone else's Hacker News digest, this project lets you build your own by just describing what you want. The agent configures it and the preview updates live as you chat.

Add sections for top stories, Show HN, Ask HN, hiring, open source, trending, and more. There's also a topic section where you can filter by keyword and timeframe, like "security news from the last 48 hours".

Once you're happy, send a test email, activate with a magic link and schedule it.

Supports daily, weekly or monthly across 70+ timezones.

You can check the GitHub Repository and live demo.

9. Generative UI MCP - design guidelines, loaded only when needed.

Not a Generative UI project itself, but one of the most useful things you can pair with one.

Generating good HTML consistently is the real challenge and putting thousands of tokens of design rules into every prompt solves it badly.

This MCP server takes a different approach. A ~300-token resource at generative-ui://system-prompt covers the basics without the model ever calling a tool. When it needs more, it calls load_ui_guidelines with a specific module and gets only what's relevant.

Diagram guidelines alone are 59KB so lazy loading actually matters here. Five modules: chart, diagram, interactive, mockup, art. Works with Claude Code, Cursor, Windsurf, and any MCP client via npx generative-ui-mcp.

You can check the GitHub Repository.

These projects are all open source. Clone them, break them, build on top of them. The source code alone can teach you a lot.

If you build something with any of these, I'd love to see it. Hope this list gives you some ideas :)

Have a great day! Until next time.

You can connect me on GitHub, Twitter and LinkedIn.

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join the Discord community.

CopilotKit Follow

React UI + elegant infrastructure for AI Copilots, in-app AI agents, AI chatbots, and AI-powered Textareas 🪁

AIMock: One Mock Server For Your Entire AI Stack

Anmol Baranwal — Wed, 08 Apr 2026 17:14:01 +0000

TL;DR Our CI was flaky, our tests hit live APIs, and every run burned tokens unnecessarily. So the CopilotKit team built AIMock - a mock server that covers your entire agentic stack so your tests are fast, free, and actually reliable. Let's dive in!

Why we built AIMock

It started with LLM mocking. We built LLMock, open-sourced it, and thought we were done.

But a single agent request in 2026 can touch six or seven services before it returns a response: the LLM, an MCP tool server, a vector database, a reranker, a web search API, a moderation layer, and a sub-agent over A2A.

Most teams mock one of them. The other six are live, non-deterministic, and quietly making your test suite a lie. That is why LLMock became AIMock.

npm install @copilotkit/aimock

AI apps in 2026 call a lot more than just an LLM
Introducing AIMock: mock your entire agentic stack
LLMock: full streaming, tool calls, reasoning across every major provider
MCPMock: a mock MCP server for testing tool integrations
A2AMock: test multi-agent workflows without live agents
VectorMock: deterministic retrieval for RAG pipelines
Services: search, rerank, and moderation
Drift Detection: catch provider changes before your users do
Record and Replay: stop hand-writing fixtures
Chaos Testing: prove your app handles failure
What else is included
AG-UI uses AIMock in production
Migrating to AIMock

AIMock is open source and free.

AI apps in 2026 call a lot more than just an LLM

A realistic agent request looks like this:

User message
→ LLM decides to use a tool
→ Tool call via MCP (file system, database, calendar)
→ RAG retrieval from Pinecone or Qdrant
→ Web search via Tavily
→ Cohere reranker to sort results
→ Back to LLM with full context

Each of these is a live network call in your test environment. Everyone can fail, return something slightly different, or cost you tokens.

We looked at every tool out there. Some handled LLM mocking. Some handled one protocol. None covered the full picture. You would have to stitch together three or four libraries, each with its own config format, and still have gaps.

Here is how AIMock compares to other options out there.

Introducing AIMock: mock your entire agentic stack

It mocks everything your AI app talks to. Here is what is inside:

LLMock : 11 providers, full streaming, tool calls, reasoning models
MCPMock : local MCP server with full JSON-RPC 2.0, session management, tools, resources, prompts
A2AMock : agent card discovery, message routing, SSE streaming between agents
VectorMock : mock vector database for deterministic RAG retrieval
Services : search, rerank, and moderation. The APIs everyone forgets to mock.

On top of that, AIMock does three things no other mocking tool does:

Drift Detection : runs daily against real provider APIs and catches response format changes within 24 hours, before your users do
Record and Replay : proxies real API calls, saves them as fixtures, replays them forever in CI without touching live APIs again
Chaos Testing : inject 500s, malformed JSON, and mid-stream disconnects to prove your app handles failure

Run all of them on one port with a single config file:

{
  "llm": { "fixtures": "./fixtures/llm", "providers": ["openai", "claude", "gemini"] },
  "mcp": { "tools": "./fixtures/mcp/tools.json" },
  "a2a": { "agents": "./fixtures/a2a/agents.json" },
  "vector": { "path": "/vector", "collections": [] }
}

Get started with the AIMock CLI:

npx aimock --config aimock.json --port 4010

Let's go through each one in brief.

LLMock: full streaming, tool calls, reasoning across every major provider

LLMock runs a real HTTP server on a real port, not in-process patching. Any process on the machine can reach it: your Next.js app, agent workers, LangGraph processes, anything that speaks HTTP.

It has native support for 10 providers: OpenAI, Claude, Gemini, Bedrock, Azure, Vertex AI, Ollama, Cohere, OpenRouter, and Anthropic Azure. Reasoning models are supported across all of them.

Any OpenAI-compatible endpoint, such as Mistral, Groq, Together AI, or vLLM, works out of the box, too. Full streaming, tool calls, structured outputs, extended thinking, multi-turn conversations, and WebSocket APIs are all built in.

AIMock handles the translation internally, so one fixture format works across all providers.

Using the programmatic API with vitest, register a fixture and assert on the response. beforeAll starts the mock server once for the suite, afterAll tears it down, and mock.on() registers a fixture that maps a user message to a deterministic response.

import { LLMock } from "@copilotkit/aimock";
import { describe, it, expect, beforeAll, afterAll } from "vitest";

let mock: LLMock;

beforeAll(async () => {
  mock = new LLMock();
  await mock.start();
});

afterAll(async () => {
  await mock.stop();
});

it("non-streaming text response", async () => {
  mock.on({ userMessage: "hello" }, { content: "Hello! How can I help?" });

  const res = await fetch(`${mock.url}/v1/chat/completions`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "gpt-4",
      messages: [{ role: "user", content: "hello" }],
      stream: false,
    }),
  });

  const body = await res.json();
  expect(body.choices[0].message.content).toBe("Hello! How can I help?");
  expect(body.object).toBe("chat.completion");
  expect(body.id).toMatch(/^chatcmpl-/);
});

Point your existing OpenAI client at the mock URL. Nothing else in your code changes. No API keys, no network calls, the same response every time. Full provider docs here.

MCPMock: a mock MCP server for testing tool integrations

MCP is how agents call tools. If your agent uses MCP, every tool call in your test suite hits a live server with real latency and no control over the response.

MCPMock gives you a local server that speaks the full MCP protocol over JSON-RPC 2.0. Your agent connects to it exactly like a real MCP server. You control what comes back.

import { MCPMock } from "@copilotkit/aimock/mcp";

const mcp = new MCPMock();

mcp.addTool({
  name: "search",
  description: "Search the web",
  inputSchema: { type: "object", properties: { query: { type: "string" } } },
});

mcp.onToolCall("search", (args) => {
  const { query } = args as { query: string };
  return `Found 3 results for "${query}"`;
});

const url = await mcp.start();
// point your MCP client at `url`

In a real agent test, your LLM and MCP tool server run together. Mount MCPMock onto LLMock so they share one port:

import { LLMock, MCPMock } from "@copilotkit/aimock";

const llm = new LLMock({ port: 5555 });
const mcp = new MCPMock();

mcp.addTool({ name: "calc", description: "Calculator" });
mcp.onToolCall("calc", (args) => "42");

llm.mount("/mcp", mcp);
await llm.start();
// MCP available at http://127.0.0.1:5555/mcp

It supports:

Tools - register handlers that return strings or rich content
Resources - static files and data your agent can read via URI
Prompts - reusable prompt templates with arguments
Session management - full Mcp-Session-Id lifecycle per the Streamable HTTP spec

Full docs here.

A2AMock: test multi-agent workflows without live agents

A2A (Agent2Agent) is how agents discover and talk to each other. It handles agent cards, message routing, and streaming responses between agents.

Testing a multi-agent system is hard when every agent is live. One agent goes down, your whole test suite breaks.

A2AMock gives you a local A2A server with full agent card discovery, message routing, task management, and SSE streaming. Register your agents, define how they respond, and test your multi-agent workflows end-to-end without anything actually running.

import { A2AMock } from "@copilotkit/aimock/a2a";

const a2a = new A2AMock();

a2a.registerAgent({
  name: "translator",
  description: "Translates text between languages",
  skills: [{ id: "translate", name: "Translate" }],
});

a2a.onMessage("translator", "translate", [{ text: "Translated text" }]);

const url = await a2a.start();
// Agent card at: ${url}/.well-known/agent-card.json
// JSON-RPC at: ${url}/

Here is the pattern for long-running tasks with incremental updates:

a2a.onStreamingTask("agent", "long-task", [
  { type: "status", state: "TASK_STATE_WORKING" },
  { type: "artifact", parts: [{ text: "partial result" }], name: "output" },
  { type: "artifact", parts: [{ text: "final result" }], lastChunk: true, name: "output" },
], 50); // 50ms delay between events

For task management, agent cards, and JSON-RPC methods, see the full docs here.

VectorMock: deterministic retrieval for RAG pipelines

If your app does retrieval-augmented generation, your tests depend on what is in your vector database right now. Your dev index is messy, your staging index does not match prod, and results shift every time someone upserts new vectors.

VectorMock is a mock vector database server that supports Pinecone, Qdrant, and ChromaDB API formats with collection management, upsert, query, and delete operations.

import { VectorMock } from "@copilotkit/aimock/vector";

const vector = new VectorMock();

vector.addCollection("docs", { dimension: 1536 });
vector.onQuery("docs", [
  { id: "doc-1", score: 0.95, metadata: { title: "Getting Started" } },
  { id: "doc-2", score: 0.87, metadata: { title: "API Reference" } },
]);

const url = await vector.start();
// point your vector DB client at `url`

If you need results to vary based on the query, use a dynamic handler instead:

vector.onQuery("docs", (query) => {
  const topK = query.topK ?? 10;
  return Array.from({ length: topK }, (_, i) => ({
    id: `result-${i}`,
    score: 1 - i * 0.1,
  }));
});

Here are compatible endpoints for Pinecone, Qdrant, and ChromaDB APIs.

Consistent retrieval results. Every test run. Full docs here.

Services: search, rerank, and moderation

These are the APIs most people forget to mock, and the ones that quietly make your test suite non-deterministic.

Services within AIMock provides built-in mocks for web search, reranking, and content moderation. Register fixture patterns on your LLMock instance and requests are matched by query/input text. No separate server needed.

Tavily search - mock web search results by query pattern at POST /search
Cohere reranking - mock reranked document lists at POST /v2/rerank
OpenAI moderation - mock moderation decisions at POST /v1/moderations. Unmatched requests return unflagged by default.

import { LLMock } from "@copilotkit/aimock";

const mock = new LLMock();

// String pattern — case-insensitive substring match
mock.onSearch("weather", [
  { title: "Weather Report", url: "https://example.com/weather", content: "Sunny today" },
]);

// RegExp pattern
mock.onSearch(/stock\s+price/i, [
  { title: "ACME Stock", url: "https://example.com/stocks", content: "$42.00", score: 0.95 },
]);

// Catch-all — empty results for unmatched queries
mock.onSearch(/.*/, []);

mock.onRerank("machine learning", [
  { index: 0, relevance_score: 0.99 },
  { index: 2, relevance_score: 0.85 },
]);

mock.onModerate("violent", {
  flagged: true,
  categories: { violence: true, hate: false },
  category_scores: { violence: 0.95, hate: 0.01 },
});

If you just need catch-all responses to stop live requests, enable all three in config:

{
  "services": {
    "search": true,
    "rerank": true,
    "moderate": true
  }
}

String patterns use case-insensitive substring matching. RegExp patterns do full regex testing. First match wins. All service requests are recorded in the journal, so you can inspect exactly what got called. Full docs here.

Drift Detection: catch provider changes before your users do

This is one of the things no other mocking tool does. A mock that does not match reality is worse than no mock.

Here is the problem: mocks are a snapshot of how an API behaved when you wrote the fixture. OpenAI adds a field. Claude changes a default. Gemini tweaks its streaming format. Your mocks still pass. CI is green. Then your app breaks in production.

Drift Detection runs a three-way comparison in CI every day:

SDK types - what TypeScript type definitions say the shape should be
Real API responses - actual live requests to OpenAI, Anthropic, Gemini
AIMock output - what the mock returns for the same request

If any of those three disagree, you know within 24 hours. Not when a user files a bug.

Here is what the output looks like when drift is detected:

$ pnpm test:drift

[critical] AIMOCK DRIFT — field in SDK + real API but missing from mock
Path:    choices[].message.refusal
SDK:     null    Real: null    Mock: <absent>

[critical] TYPE MISMATCH — real API and mock disagree on type
Path:    content[].input
SDK:     object    Real: object    Mock: string

[warning] PROVIDER ADDED FIELD — in real API but not in SDK or mock
Path:    choices[].message.annotations
SDK:     <absent>    Real: array    Mock: <absent>

✓ 2 critical (test fails) · 1 warning (logged) · detected before any user reported it

Severity levels:

Critical : mock mismatches the real API, test fails immediately
Warning : provider added a new field that neither the SDK nor the mock knows about yet, logged as early warning
Ok : all three sources agree, nothing to do

Here is how the three-way comparison works under the hood:

import { extractShape, triangulate, formatDriftReport, shouldFail } from "./schema";

// 1. Get the SDK shape (what TypeScript says)
const sdkShape = openaiChatCompletionShape();

// 2. Call the real API and the mock in parallel
const [realRes, mockRes] = await Promise.all([
  openaiChatNonStreaming(config, [{ role: "user", content: "Say hello" }]),
  httpPost(`${instance.url}/v1/chat/completions`, { /* ... */ }),
]);

// 3. Extract response shapes
const realShape = extractShape(realRes.body);
const mockShape = extractShape(JSON.parse(mockRes.body));

// 4. Three-way comparison
const diffs = triangulate(sdkShape, realShape, mockShape);
const report = formatDriftReport("OpenAI Chat (non-streaming text)", diffs);

// 5. Critical diffs fail the test
if (shouldFail(diffs)) {
  expect.soft([], report).toEqual(
    diffs.filter(d => d.severity === "critical")
  );
}

Run it yourself against live endpoints:

# Run drift checks against live endpoints
pnpm vitest --config vitest.config.drift.ts

MSW, VidaiMock, Mokksy - none of them do this. Your mocks should never silently go stale. Full docs here.

Record and Replay: stop hand-writing fixtures

Writing fixtures by hand is fine for simple cases. It gets painful fast when you are dealing with multi-turn agent conversations with tool calls, streaming responses, and branching logic. MSW, VidaiMock, mock-llm, piyook - none of the LLM mocking tools solve this.

Record and Replay works like a VCR. Here is exactly what happens:

Client sends a request to AIMock
AIMock attempts fixture matching as usual
On miss: the request is forwarded to the configured upstream provider
The upstream response is relayed back to the client immediately
The response is collapsed if streaming and saved as a fixture to disk and memory
Subsequent identical requests match the newly recorded fixture

Quick start using CLI:

$ npx aimock --fixtures ./fixtures \
  --record \
  --provider-openai https://api.openai.com \
  --provider-anthropic https://api.anthropic.com

Recorded fixtures are saved automatically and look like this:

{
  "fixtures": [
    {
      "match": { "userMessage": "What is the weather?" },
      "response": { "content": "I don't have real-time weather data..." }
    }
  ]
}

AIMock handles stream collapsing automatically across six formats: OpenAI SSE, Anthropic SSE, Gemini SSE, Cohere SSE, Ollama NDJSON, and Bedrock EventStream. Auth headers are forwarded to the upstream provider but never saved in fixtures.

It also supports a programmatic API with enableRecording() and requestTransform for normalizing dynamic data like timestamps.

In CI, add --strict so that any unmatched request returns a 503 and fails immediately, rather than slipping through silently. Without it, missing fixtures return 404 responses, and your test suite may never tell you that a live API call was attempted.

Full docs here.

Chaos Testing: prove your app handles failure

Your app will eventually get a 500 from OpenAI. A malformed JSON response from a tool. A mid-stream disconnect from a vector database. The question is whether you find out in testing or in production.

Chaos Testing lets you inject failures at configurable probabilities at three levels: server-wide, per fixture, or per individual request.

Three failure modes:

drop - returns HTTP 500 with {"error":{"message":"Chaos: request dropped","code":"chaos_drop"}}
malformed - returns HTTP 200 with invalid JSON
disconnect - destroys the TCP connection immediately with no response

Server-level chaos applies to all requests:

import { LLMock } from "@copilotkit/aimock";

const mock = new LLMock();
mock.setChaos({
  dropRate: 0.1,        // 10% of requests return 500
  malformedRate: 0.05,  // 5% return broken JSON
  disconnectRate: 0.02, // 2% drop the connection
});

// remove all chaos later
mock.clearChaos();

Fixture-level chaos targets specific responses only:

{
  "fixtures": [
    {
      "match": { "userMessage": "unstable" },
      "response": { "content": "This might fail!" },
      "chaos": {
        "dropRate": 0.3,
        "malformedRate": 0.2,
        "disconnectRate": 0.1
      }
    }
  ]
}

Per-request headers override everything else - useful for forcing a specific failure in a single test:

// Force 100% disconnect on this specific request
await fetch(`${mock.url}/v1/chat/completions`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-aimock-chaos-disconnect": "1.0",
  },
  body: JSON.stringify({ model: "gpt-4", messages: [{ role: "user", content: "hello" }] }),
});

All chaos events are tracked in the journal with a chaosAction field and counted in Prometheus metrics. Full docs here.

What else is included

Beyond the core modules, AIMock offers a lot more worth knowing.

Embeddings - POST /v1/embeddings is fully supported. Return explicit vectors via fixtures or let AIMock generate them deterministically from a hash of the input text. Same input always produces the same vector, default dimensions are 1536.

WebSocket APIs - OpenAI Realtime, OpenAI Responses over WebSocket, and Gemini Live are all supported using raw RFC 6455 framing. If your app uses voice or real-time streaming agents, these are covered.

Sequential Responses - return different responses for the same prompt on each successive call. Useful for testing retry logic and multi-turn workflows where the same message should behave differently over time.

[
  { "match": { "userMessage": "retry", "sequenceIndex": 0 }, "response": { "content": "First attempt" } },
  { "match": { "userMessage": "retry", "sequenceIndex": 1 }, "response": { "content": "Second attempt" } },
  { "match": { "userMessage": "retry" }, "response": { "content": "Fallback" } }
]

Streaming Physics - configure ttft (time to first token), tps (tokens per second), and jitter to simulate realistic timing. Pre-built profiles cover fast models, reasoning models, and overloaded systems.

Prometheus Metrics - request counts, latency histograms, and current fixture count at /metrics. Enable with --metrics flag.

Docker and Helm - the official Docker image is available at ghcr.io/copilotkit/aimock (GitHub Container Registry) for CI/CD. It runs as a plain HTTP server, so any language can use it. No Node.js needed on the test runner side.

AG-UI uses AIMock in production

AG-UI is the open protocol that connects AI agents to frontend applications - adopted by LangGraph, CrewAI, Mastra, Google ADK, AWS Bedrock AgentCore, and more.

It uses AIMock for its end-to-end test suite, verifying AI agent behavior across LLM providers with fixture-driven responses. AG-UI is a protocol used in production. If you want to see what a real AIMock setup looks like at scale, that is a good place to start.

Migrating to AIMock

The docs include step-by-step migration guides for MSW, VidaiMock, mock-llm, Python mocks and Mokksy. You will find a side-by-side comparison and a breakdown of what you gain and keep.

If you are on MSW, you do not have to replace everything: you can keep MSW for general REST and GraphQL mocking and use AIMock only for AI endpoints.

If you are already using @copilotkit/llmock, the upgrade is a find-and-replace:

pnpm remove @copilotkit/llmock
pnpm add @copilotkit/aimock

The LLMock class, all fixture formats, and the programmatic API are unchanged. Your existing tests will work as-is.

Get started

npm install @copilotkit/aimock

Docs: aimock.copilotkit.dev
GitHub: github.com/CopilotKit/aimock
npm: @copilotkit/aimock
CLI: AIMock CLI

Your test suite should be as complete as your stack. That is what AIMock is for.

You can connect me on GitHub, Twitter and LinkedIn.

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join the Discord community.

CopilotKit Follow

React UI + elegant infrastructure for AI Copilots, in-app AI agents, AI chatbots, and AI-powered Textareas 🪁

Google Maps for Codebases: Paste a GitHub URL, Ask Anything

Anmol Baranwal — Mon, 06 Apr 2026 13:09:38 +0000

Navigating a large codebase for the first time is painful. You clone the repo, realize there are 300 files, and have no idea where anything lives.

You can ask an AI assistant, but it burns through context fast and you never know if it hallucinated a file path that does not even exist.

Codebase Navigator solves this. Paste a URL, ask anything in plain English, and watch a real dependency graph built from actual import statements appear in real time.

It is built using CopilotKit, Zenflow, GitHub API and React Flow. You can run it completely free using Ollama locally.

In this blog, we will go through the architecture, the key patterns and how everything works end-to-end.

What are we building?

Codebase Navigator lets you paste any public GitHub repo URL and ask questions about it in plain English. Instead of getting a wall of text back, you get four panels that all update at once.

Panel	What it does
Graph Canvas	Live dependency graph built from real import statements
Code Viewer	File content fetched live from GitHub, relevant lines highlighted
Repo Explorer	Full file tree, click any file to open it
Chat	Follow-up questions and plain English explanations

The graph morphs to show every relevant file connected by real imports, the code viewer opens the file, and the chat explains what each piece does. No context switching.

You can run it completely free using Ollama locally without any API key.

Here is the full request → response flow of what happens when you ask a question:

User types "how does auth work?"
     ↓
CopilotChat → POST /api/copilotkit
     ↓
LLM receives repo context (file paths, current selection, system rules)
     ↓
LLM calls analyzeRepository tool
     ↓
Tool fetches relevant files via /api/github/file
     ↓
Extracts import/require statements → resolves paths → builds dependency graph
     ↓
Zustand store updates
     ↓
All four panels re-render live: graph, file tree, code viewer, chat response

Tech Stack & Tools

At a high level, this project is built by combining:

Next.js 16 - frontend and API routes
CopilotKit - agent-UI state sync and chat interface. provides built-in hooks & components like useAgentContext, useFrontendTool, CopilotChat
Zenflow - workflow tool that planned, tested and orchestrated the build
React Flow & dagre - interactive dependency graphs with automatic layout
Octokit - GitHub API proxy for fetching repo trees and file contents
Zustand - shared state across all four panels
Tailwind CSS - styling
Ollama / OpenAI - local or cloud LLM backend switchable from the UI

The CopilotKit runtime is self-hosted inside a Next.js API route, which lets you plug into any OpenAI-compatible backend, including Ollama, for completely free local inference.

It connects your UI, agents, and tools into a single interaction loop.

What is Zenflow?

Zenflow is an AI development tool that treats building software as a structured engineering process rather than just autocompleting code.

This project was planned, built, tested, reviewed and deployed using Zenflow (Zencoder's workflow engine) in a single session.

It ran a six-phase process from scratch:

Architecture and spec
Scaffolding and foundation
Data layer (GitHub API integration)
AI layer (CopilotKit actions and context)
Visual layer (React Flow graphs)
Final assembly and wiring

Each phase was verified with lint, type checks and tests before moving to the next. After the build, it performed a code review, found 14 issues and fixed 11 systematically:

API keys exposed in headers → moved to httpOnly cookies
Fake star-shaped graphs → replaced with real import-based dependency resolution
Duplicate file-fetching logic → consolidated into a single cached utility

The .zenflow/ folder in the repo contains the task plan, spec, and report files it generated along the way.

That's how we built it with Zenflow. Here's the architecture that came out of it.

Architecture & Project Structure

The app is split into three layers: the browser handles all UI and AI tool logic, Next.js API routes act as a secure proxy to external services, and GitHub API, plus the LLM backend sits at the bottom. Nothing in the browser talks to GitHub or the LLM directly.

Below is a high-level overview of all the layers and how they are organized.

┌─────────────────────────────────────────────────────────┐
│                        BROWSER                          │
│                                                         │
│  ┌─────────┐  ┌──────────┐  ┌──────────┐  ┌────────┐    │
│  │  Chat   │  │  Graph   │  │  Code    │  │  File  │    │
│  │  Panel  │  │  Canvas  │  │  Viewer  │  │  Tree  │    │
│  └────┬────┘  └────┬─────┘  └────┬─────┘  └───┬────┘    │
│       └────────────┴─────────────┴────────────┘         │
│                          │                              │
│                   ┌──────▼──────┐                       │
│                   │   Zustand   │  ← shared state       │
│                   └──────┬──────┘                       │
│                          │                              │
│              ┌───────────▼────────────┐                 │
│              │      CopilotKit        │                 │
│              │   frontend tools       │                 │
│              │  analyzeRepository     │                 │
│              │  fetchFileContent      │                 │
│              │  highlightCode         │                 │
│              └───────────┬────────────┘                 │
└──────────────────────────┼──────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│                    NEXT.JS API ROUTES                   │
│                                                         │
│   /api/copilotkit     /api/github/*     /api/settings   │
│   (LLM runtime)       (proxy)          (httpOnly cookie)│
└──────────┬────────────────┬─────────────────────────────┘
           │                │
     ┌─────▼─────┐   ┌──────▼──────┐
     │  Ollama   │   │  GitHub API │
     │  / OpenAI │   │  (Octokit)  │
     └───────────┘   └─────────────┘

Here is a simplified view of how the project is structured.

src/
├── app/api/
│   ├── copilotkit/route.ts    → CopilotKit runtime endpoint
│   ├── github/
│   │   ├── tree/route.ts      → fetch repo tree
│   │   ├── file/route.ts      → fetch + base64-decode file
│   │   └── search/route.ts    → code search
│   └── settings/route.ts      → LLM config (httpOnly cookie)
├── components/
│   ├── panels/     → 5 UI panels (chat, graph, code, analysis, repo)
│   └── flow/       → custom React Flow node types
├── hooks/          → AI tool registration, agent context sync, repo loading
├── lib/            → import extraction, dagre layout, Octokit client
├── store/          → Zustand (AppState + SettingsState)
└── .zenflow/       → Zenflow task plan, spec and report

Let's get into how everything works behind the scenes. This will help you understand the key patterns.

How everything works under the hood

Now that you have the big picture, let's go through each piece in detail.

A lot is happening under the surface, so we will break it down into ten parts, starting from when you first load a repo all the way to how state flows across the panels.

1. Loading a repository

When you paste a GitHub URL and click Explore, useRepository.loadRepository() fires (from useRepository.ts hook).

It calls /api/github/tree on the server, which uses Octokit to:

Parse the repo URL into { owner, repo }
Fetch the default branch
Get the full recursive file tree from the GitHub Trees API
Convert the flat array into a nested TreeNode structure

Here is what the src/api/github/tree/route.ts route looks like:

export async function GET(request: NextRequest) {
  const { owner, repo } = parseRepoUrl(repoUrl);
  const resolvedBranch = branch || (await getDefaultBranch(owner, repo));
  const tree = await getRepoTree(owner, repo, resolvedBranch);

  return NextResponse.json({ owner, repo, branch: resolvedBranch, tree });
}

That nested tree powers the sidebar file explorer and becomes the starting point for everything else.

All GitHub calls are proxied through Next.js API routes. The browser never talks to GitHub directly to keep tokens secure.

2. From architecture view to dependency graph

This is the visual moment that makes the app feel alive. The graph you see on load (the architecture view) and the graph you see after asking a question (the dependency graph) are two completely different things, built two different ways.

Both live in src/lib/analyzer.ts - buildOverviewGraph for the architecture view and buildDependencyNodes for the dependency graph.

When the repo loads, useRepository.ts calls buildOverviewGraph right after the tree response comes back and pushes the result straight to the graph:

const overview = buildOverviewGraph(data.tree);
setVisualization(overview.nodes, overview.edges, "architecture");

buildOverviewGraph never fetches any files. It just walks the tree structure and builds a folder map: root node at the top, top-level directories as children, one level of sub-folders below that:

export function buildOverviewGraph(tree: TreeNode) {
  const rootId = `node-${nodeId++}`;
  nodes.push({ id: rootId, type: "module", label: tree.path || "root" });

  const topDirs = (tree.children || []).filter((c) => c.type === "directory");

  for (const dir of topDirs) {
    const fileCount = flattenTree(dir).length;
    nodes.push({ id, label: `${dir.name} (${fileCount})`, type: categorizeDirType(dir.path) });
    edges.push({ source: rootId, target: id, type: "flow" });
  }
}

The edges here mean nothing beyond "this folder is inside that folder."

Once you ask a question, useCopilotActions.ts fetches the actual file content and calls buildDependencyNodes instead:

const graph = buildDependencyNodes(fileData);
setVisualization(graph.nodes, graph.edges, "dependency");

buildDependencyNodes creates one node per file and one edge per real import statement:

export function buildDependencyNodes(files: { path: string; imports: string[] }[]) {
  const nodes = files.map((file) => ({
    id: file.path,
    type: "file",
    label: file.path.split("/").pop() || file.path,
    metadata: { fullPath: file.path },
  }));

  for (const file of files) {
    for (const imp of file.imports) {
      const resolved = resolveImportPath(imp, file.path, filePathSet);
      if (resolved) {
        edges.push({ source: file.path, target: resolved, type: "import" });
      }
    }
  }

  return { nodes, edges };
}

Same setVisualization call, different data. React Flow re-renders and the graph morphs from a folder map into a real dependency graph focused on exactly the files relevant to your question.

3. The CopilotKit runtime

The /api/copilotkit route is where LLM requests actually land. The API key and provider config are stored in an httpOnly cookie, meaning they live on the server and never get sent to the browser.

The route reads that config, creates an OpenAI-compatible client, and hands the request to the CopilotKit runtime:

export const POST = async (req: NextRequest) => {
  const { baseURL, apiKey, model } = await getLLMConfig();

  const openai = new OpenAI({ baseURL, apiKey });
  const serviceAdapter = new OpenAIAdapter({ openai, model });
  const runtime = new CopilotRuntime();

  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter,
    endpoint: "/api/copilotkit",
  });

  return handleRequest(req);
};

The defaults point to Ollama on localhost:11434/v1 with qwen2.5.

Because the OpenAI SDK accepts a custom baseURL, swapping to OpenAI, Groq or any other provider is just a config change with no code changes needed.

4. Feeding the LLM context

Before the LLM can answer anything useful, it needs to know what repo is loaded and what files exist. That is the job of the useCopilotContext.ts hook.

It calls useAgentContext, a CopilotKit hook that attaches structured data to every message you send. You can call it multiple times to attach different pieces of context.

Here it is called twice: once for the file list and once for the system instructions that tell the LLM how to behave:

useAgentContext({
  description: "File paths in the repository (max 500), one per line",
  value: fileList,  // flattened from the TreeNode structure
});

useAgentContext({
  description: "System instructions",
  value: `You are a Codebase Navigator assistant. You MUST use tool calls to answer questions.
CRITICAL RULES:
1. For ANY question about the repository call the "analyzeRepository" tool.
2. To show a file, call "fetchFileContent" with the exact file path.
3. NEVER respond with only text. ALWAYS call a tool first.
4. Use ONLY file paths from the file list above.`
});

The file list is capped at 500 paths to stay within token limits.

Because this hook re-runs every time the Zustand store changes, the LLM always sees the current repo, selected file and analysis state rather than a stale snapshot from when the page loaded.

Note: If you are running a local model with a large context window, you can raise this limit in useCopilotContext.ts.

5. The four frontend tools

This is the core of how the app works. Instead of just replying with text, the LLM can call tools. Each tool has a name, a set of parameters it accepts and a handler, which is just a function that runs when the LLM calls it.

CopilotKit lets you register these tools directly in the browser using the useFrontendTool hook, which is defined in useCopilotActions.ts.

When a tool's handler runs, it updates Zustand state directly, which is why all four panels, the graph, code viewer, analysis panel, and chat, react at the same time without any extra wiring.

analyzeRepository is the main tool the LLM calls for almost every question. Here is what it does step by step:

useFrontendTool({
  name: "analyzeRepository",
  parameters: z.object({
    query: z.string(),
    explanation: z.string(),
  }),
  handler: async ({ query, explanation }) => {
    // 1. Find relevant files using keyword matching
    const matchedPaths = findFilesByQuery(repo.tree, query);

    // 2. Cap at 15 files to keep things fast
    const capped = matchedPaths.slice(0, 15);

    // 3. Fetch each file content (cached)
    const fileData = await Promise.all(
      capped.map(async (p) => ({
        path: p,
        imports: extractImports(await fetchFile(owner, repo, p, branch)),
      }))
    );

    // 4. Build real dependency graph from actual import statements
    const graph = buildDependencyNodes(fileData);

    // 5. Categorize each node by file type
    graph.nodes.forEach(node => {
      node.type = categorizeFileType(node.metadata.fullPath);
    });

    // 6. Push to Zustand - all four panels react
    setAnalysisResult({ explanation, relevantFiles, flowDiagram: graph });
    setVisualization(graph.nodes, graph.edges, "dependency");
  },
});

fetchFileContent opens a specific file in the code viewer. The LLM calls this when you ask it to show you a file, it just fetches the content and calls setCodeViewer:

useFrontendTool({
  name: "fetchFileContent",
  parameters: z.object({
    filePath: z.string(),
  }),
  handler: async ({ filePath }) => {
    const content = await fetchFile(repo.repoInfo.owner, repo.repoInfo.repo, filePath, repo.repoInfo.branch);
    setCodeViewer(filePath, content);
  },
});

generateFlowDiagram is similar to analyzeRepository but more focused. Instead of searching the whole repo, you give it a specific list of file paths and it builds a graph for just those files. Useful when you want to visualize a slice of the codebase like just the API layer:

useFrontendTool({
  name: "generateFlowDiagram",
  parameters: z.object({
    files: z.array(z.string()),
    diagramType: z.enum(["dependency", "flow", "architecture"]),
  }),
  handler: async ({ files, diagramType }) => {
    const fileData = await Promise.all(files.slice(0, 20).map(async (f) => ({
      path: f,
      imports: extractImports(await fetchFile(repo.repoInfo.owner, repo.repoInfo.repo, f, repo.repoInfo.branch)),
    })));
    const graph = buildDependencyNodes(fileData);
    setVisualization(graph.nodes, graph.edges, diagramType);
  },
});

highlightCode fetches a file and highlights specific line numbers with an explanation. The LLM uses this when it wants to point at exactly where something happens in the code:

useFrontendTool({
  name: "highlightCode",
  parameters: z.object({
    filePath: z.string(),
    lines: z.array(z.number()),
    explanation: z.string(),
  }),
  handler: async ({ filePath, lines, explanation }) => {
    const content = await fetchFile(repo.repoInfo.owner, repo.repoInfo.repo, filePath, repo.repoInfo.branch);
    setCodeViewer(filePath, content, lines, explanation);
  },
});

Notice that none of these tools returns text to the chat. They all just update the state. The LLM is not describing what it found, it is directly controlling what you see on screen.

For instance, I asked about the architecture of v2 packages.

6. Finding relevant files

When analyzeRepository gets a query like "how does authentication work?", it needs to figure out which files to actually look at. That is handled by findFilesByQuery in src/lib/analyzer.ts.

It checks if the query matches one of six predefined categories, each with a set of keywords and file path patterns:

const ANALYSIS_CATEGORIES = [
  {
    name: "authentication",
    keywords: ["auth", "login", "logout", "token", "jwt", "session", "oauth"],
    filePatterns: [/auth/i, /login/i, /session/i, /oauth/i, /jwt/i],
  },
  {
    name: "database",
    keywords: ["database", "db", "model", "schema", "prisma", "mongoose"],
    filePatterns: [/model/i, /schema/i, /migration/i, /prisma/i, /db/i],
  },
  // ... api, configuration, testing, styling
]

If the query matches a category, it filters all file paths using that category's regex patterns.

If nothing matches, it falls back to splitting the query into individual terms and checking if any file path contains them. So even a vague query like "where is the config" will still find files with "config" in the name.

This is intentionally simple. No embeddings, no vector search, just regex on file names. It works well because file names in well-structured projects are usually descriptive enough that matching on them gets you the right files.

7. File caching

Every time a tool fetches a file, it goes through src/lib/fetch-file.ts which maintains an in-memory cache with a 5-minute TTL:

const cache = new Map<string, { content: string; timestamp: number }>();
const CACHE_TTL = 5 * 60 * 1000; // 5 minutes

export async function fetchFile(owner, repo, path, ref) {
  const key = `${owner}/${repo}/${ref}/${path}`;
  const cached = cache.get(key);

  if (cached && Date.now() - cached.timestamp < CACHE_TTL) {
    return cached.content; // serve from cache
  }

  const res = await fetch(`/api/github/file?repo=...&path=...&ref=...`);
  const data = await res.json();
  cache.set(key, { content: data.content, timestamp: Date.now() });

  // LRU eviction: if over 200 items, drop the oldest 50
  if (cache.size > 200) {
    const oldest = [...cache.entries()].sort((a, b) => a[1].timestamp - b[1].timestamp);
    for (let i = 0; i < 50; i++) cache.delete(oldest[i][0]);
  }

  return data.content;
}

This matters because analyzeRepository fetches up to 15 files at once. A follow-up question about the same files hits the cache instead of making 15 more API requests.

Without a GitHub token, you are limited to 60 requests per hour, so caching is what makes repeated questions fast and free.

The cache also evicts the oldest entries once it crosses 200 items, so it never grows unbounded in long sessions.

8. Parsing imports and resolving paths

Once analyzeRepository has the relevant file paths, it fetches each file and parses it for import statements. Both functions that handle this live in src/lib/analyzer.ts.

extractImports runs on the raw file content and pulls out every import, handling both ES6 and CommonJS syntax:

export function extractImports(content: string): string[] {
  const imports: string[] = [];

  // ES6: import X from 'y' and import 'y'
  const esImports = content.matchAll(
    /(?:import\s+.*?\s+from\s+['"](.+?)['"]|import\s+['"](.+?)['"])/g
  );
  for (const match of esImports) imports.push(match[1] || match[2]);

  // CommonJS: require('y')
  const requires = content.matchAll(/require\s*\(\s*['"](.+?)['"]\s*\)/g);
  for (const match of requires) imports.push(match[1]);

  return imports;
}

But raw import strings like "./utils" or "@/lib/github" are not file paths yet.

buildDependencyNodes calls resolveImportPath on each one to turn them into actual paths that exist in the repo:

function resolveImportPath(importPath, fromFile, filePathSet) {
  // Skip npm packages (no ./ or @/)
  if (!importPath.startsWith(".") && !importPath.startsWith("@/")) return null;

  // Resolve @/ alias to src/
  if (importPath.startsWith("@/")) {
    resolved = "src/" + importPath.slice(2);
  }

  // Try with common extensions: .ts, .tsx, .js, /index.ts, etc.
  const extensions = ["", ".ts", ".tsx", ".js", ".jsx", "/index.ts", "/index.tsx"];
  for (const ext of extensions) {
    if (filePathSet.has(resolved + ext)) return resolved + ext;
  }
  return null;
}

npm packages like react or zod are skipped immediately since they do not start with ./ or @/. For local imports, it tries common extensions like .ts, .tsx, and /index.ts, so it handles both direct file imports and folders that export through an index file.

If the resolved path does not exist in the repo, it returns null and no edge is created. That is what keeps the graph honest.

9. Graph layout with Dagre

Once the nodes and edges are built, they need actual positions on screen before React Flow can render them. That is what applyDagreLayout does in src/lib/graph-layout.ts.

It uses Dagre, a JavaScript library that automatically calculates x and y coordinates for every node in a directed graph so nothing overlaps, all on the client side with no server needed:

export function applyDagreLayout(nodes, edges, options = {}) {
  const g = new dagre.graphlib.Graph();
  g.setGraph({ rankdir: "TB", ranksep: 80, nodesep: 40 });

  // Tell dagre the size of each node
  for (const node of nodes) {
    g.setNode(node.id, { width: 200, height: 60 });
  }
  for (const edge of edges) {
    g.setEdge(edge.source, edge.target);
  }

  dagre.layout(g); // dagre calculates x,y for every node

  return nodes.map((node) => {
    const pos = g.node(node.id);
    return { ...node, position: { x: pos.x - 100, y: pos.y - 30 } };
  });
}

The direction can be toggled between top-to-bottom (TB) and left-to-right (LR) from the UI.

Each node type maps to a React Flow component based on the file type categorizeFileType assigned earlier:

module / service → ModuleNode (indigo)
function → FunctionNode (teal)
file → FileNode (gray)

10. State management with Zustand

All four panels stay in sync because they all read from the same Zustand store in src/store/index.ts. The store has four slices, one for each major concern:

interface AppState {
  repo: {
    repoInfo: RepoInfo | null;   // owner, repo, branch
    tree: TreeNode | null;        // full file tree
    selectedFile: string | null;  // active file path
    loading: boolean;
    error: string | null;
  };
  analysis: {
    result: AnalysisResult | null; // explanation + relevant files + graph
    loading: boolean;
    error: string | null;
  };
  visualization: {
    nodes: FlowNode[];
    edges: FlowEdge[];
    graphType: "dependency" | "flow" | "architecture" | null;
  };
  codeViewer: {
    filePath: string | null;
    content: string | null;
    highlightedLines: number[];
    explanation: string | null;
  };
}

Each panel subscribes to only the slice it needs using a selector like useAppStore((s) => s.visualization).

So when a tool calls setVisualization, only the graph panel re-renders. When setCodeViewer fires, only the code viewer updates. Nothing else moves.

Settings like LLM provider, API key and model live in a separate SettingsState store. It syncs to two places:

localStorage so the browser remembers your config across reloads
an httpOnly cookie so the server-side /api/copilotkit route can read them without the API key ever appearing in request headers

This is how everything works behind the scenes. It's time to run it locally.

Running it locally

Clone the GitHub repository and install the dependencies.

git clone <repo-url>
cd codebase-navigator
npm install
npm run dev

Open http://localhost:3000 to view it live in the browser. Here is how to set up each provider.

Using OpenAI

You will need an OpenAI API Key. Then just click the Settings icon in the top right, switch the provider to OpenAI, paste your API key and pick a model like gpt-5.2. That's it.

Using Ollama (free, runs locally)

You can download it from ollama.com or install it via these commands based on the operating system you use.

brew install ollama #macOS
curl -fsSL https://ollama.com/install.sh | sh #linux

Then pull the model and start the server:

ollama serve
ollama pull qwen2.5 # in another terminal

The app points to Ollama by default so nothing else needs to change. Just open http://localhost:3000 and it works.

For GitHub, public repos work without a token (the only catch is that it's limited to 60 requests per hour). Add a GITHUB_TOKEN to a .env.local file for higher limits:

GITHUB_TOKEN=your_token_here

I tried it with the CopilotKit GitHub repo and it immediately maps out the entire repo structure.

I asked it, "tell me about architecture" and it gave me:

a graph of every relevant file connected by real imports
a plain english breakdown of what each file does
raw code fetched live from GitHub
a detailed chat response I could keep following up on

I tried difficult and vague questions and it handled all of them.

The bigger picture

This project started with wiring three ideas together and seeing what happened.

GitHub is the data source, not just a host. Every file the AI references gets fetched live, parsed, and reasoned about on the spot. No hallucination, no guessing from memory.

CopilotKit changes what the AI can actually do in the browser. Instead of replying with text, it calls tools that update the state directly. The graph changes. The code viewer opens. The AI is not describing what it found, it is showing you.

The dependency edges are real. A lot of tools draw graphs that look impressive but mean nothing. Every edge here exists because one file has an actual import statement pointing to another. That is it.

Wire those three together and you get something that actually understands the codebase instead of just searching through it.

There is a lot you could extend from here: PR diff exploration, multi-repo comparison, security audit mode. But as a starting point for understanding any codebase, it already does exactly what it needs to.

Let me know what you think in the comments!

You can connect me on GitHub, Twitter and LinkedIn.

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join the Discord community.

CopilotKit Follow

React UI + elegant infrastructure for AI Copilots, in-app AI agents, AI chatbots, and AI-powered Textareas 🪁

Deploying AG-UI Agents to Production with Amazon Bedrock AgentCore

Anmol Baranwal — Tue, 24 Mar 2026 17:00:06 +0000

The Agent-User Interaction Protocol (AG-UI) standardizes how AI agents communicate with user applications.

But deploying those agents means the same manual work: configuring SSE endpoints, writing auth middleware, managing session isolation, and setting up auto-scaling. None of that is your agent logic. All of it had to be built before you could ship.

Amazon recently announced native AG-UI support in Amazon Bedrock AgentCore Runtime across 14 AWS regions. You bring your agent code in a container, set one protocol flag, and AgentCore handles the rest.

So I deployed Strands Agents to learn how it actually works end-to-end. This post covers everything I picked up, including what AgentCore is and the full deployment flow from a local server to a live UI.

Amazon Bedrock AgentCore Runtime

Amazon Bedrock AgentCore is an agentic platform for building, deploying, and operating agents securely at scale. AgentCore Runtime is a fully managed hosting environment.

You bring your agent code in a container, and AgentCore checks who's allowed in (auth), keeps each user's session separate (Session isolation), adds capacity when traffic spikes (auto-scaling) and logs everything to CloudWatch (observability). You don't write any of this code.

It's also framework-agnostic: agents built with Strands, LangGraph, CrewAI, or any other framework can deploy to AgentCore. Think of it as the production layer between your agent and your users.

It supports four protocols:

HTTP: default agent invocation
MCP: agent ↔ tools
A2A: agent ↔ agent
AG-UI: agent ↔ user (new)

You can read more about the differences in message format, discovery and authentication in their docs.

You configure with --protocol AGUI and the protocol layer is taken care of. Your job is just the agent logic.

AG-UI: the interaction protocol

AG-UI (Agent-User Interaction Protocol) is an open, event-based protocol that standardises how AI agents communicate with user-facing applications.

Instead of a custom WebSocket or polling layer, AG-UI defines a shared event stream between your agent backend and your frontend UI. It covers:

Streaming text responses as they generate
Tool call lifecycle (start to result)
State updates and progress
User interactions sent back into the agent mid-execution

In practice, when your agent responds, the frontend receives a sequence of typed events:

data: {"type": "RUN_STARTED", "threadId": "test-123", "runId": "run-456"}
data: {"type": "TEXT_MESSAGE_START", "messageId": "msg-abc", "role": "assistant"}
data: {"type": "TEXT_MESSAGE_CONTENT", "messageId": "msg-abc", "delta": "Hello!"}
data: {"type": "TEXT_MESSAGE_END", "messageId": "msg-abc"}
data: {"type": "RUN_FINISHED", "threadId": "test-123", "runId": "run-456"}

Any AG-UI-compatible frontend, including CopilotKit, can consume that stream directly. In the next section, we will see how it all connects through an architecture diagram.

How It All Connects

CopilotKit is an AG-UI-compatible React client. It connects to an AG-UI endpoint, subscribes to the event stream and renders streaming responses, tool progress, and state updates inside your app.

Once your agent is deployed to AgentCore, HttpAgent points to the AgentCore invocation URL with a Cognito Bearer token in the headers, and user messages flow back over AG-UI into the running agent.

Here is what the full stack looks like when deployed:

Your agent framework  [Strands · LangGraph · CrewAI]
│
│  deployed to
▼
Amazon Bedrock AgentCore Runtime  [Bearer token via Cognito OAuth 2.0]
│  auth · session isolation · auto-scaling · observability
│
│  invokes container
▼
Agent container  [ARM64 · port 8080]
│  FastAPI · /invocations · /ping
│
│  calls model
▼
Amazon Bedrock model
│
│  streams back via
▼
AG-UI protocol  [SSE event stream]
│
│  consumed by
▼
CopilotKit  [HttpAgent · @ag-ui/client]
│
├─  renders streaming responses + tool progress
├─  syncs state updates in real time
└─  forwards user messages → agent via AG-UI
▼
Your application UI

If you want a complete full-stack starting point, AWS also released the Fullstack AgentCore Solution Template - a ready-made stack with AgentCore, LangGraph, CopilotKit and AG-UI all wired together, deployed via CDK. It's the fastest way to go from zero to a production-ready agentic app on AWS.

Deploying your AG-UI agent with Strands

This example uses AWS Strands as the agent framework but the same pattern works with LangGraph, CrewAI and any other AG-UI-compatible framework.

Before deploying, make sure you have:

Python 3.12+
AWS account with credentials configured (aws configure)
IAM permissions for AgentCore Runtime
Docker, Finch, or Podman for building container images
Amazon Cognito user pool configured (Setup guide here)

Start by installing the required packages:

pip install fastapi uvicorn ag_ui_strands strands-agents boto3

ag_ui_strands wraps your Strands agent with AG-UI protocol support, handling event encoding automatically.
strands-agents is the AWS agent framework used in this example.
boto3 is the AWS SDK, used internally by Strands to call Bedrock models.

Step 1: Build your AG-UI server

Create my_agui_server.py. This defines the model, wraps the agent with AG-UI support, and exposes the endpoints AgentCore needs.

import uvicorn
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse, JSONResponse
from ag_ui_strands import StrandsAgent
from ag_ui.core import RunAgentInput
from ag_ui.encoder import EventEncoder
from strands import Agent
from strands.models.bedrock import BedrockModel

model = BedrockModel(
    model_id="us.anthropic.claude-3-5-sonnet-20241022-v2:0",
    region_name="us-west-2",
)

strands_agent = Agent(
    model=model,
    system_prompt="You are a helpful assistant.",
)

agui_agent = StrandsAgent(
    agent=strands_agent,
    name="my_agent",
    description="A helpful assistant",
)

app = FastAPI()

@app.post("/invocations")
async def invocations(input_data: dict, request: Request):
    accept_header = request.headers.get("accept")
    encoder = EventEncoder(accept=accept_header)

    async def event_generator():
        run_input = RunAgentInput(**input_data)
        async for event in agui_agent.run(run_input):
            yield encoder.encode(event)

    return StreamingResponse(
        event_generator(),
        media_type=encoder.get_content_type()
    )

@app.get("/ping")
async def ping():
    return JSONResponse({"status": "Healthy"})

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8080)

Here, StrandsAgent wraps your Strands agent and handles all AG-UI event encoding so you don't write any protocol logic yourself.

AgentCore requires three things from your container:

POST /invocations - handles AG-UI requests, returns the SSE event stream.
GET /ping - health check, must return HTTP 200.
Port 8080 - required, do not change.

For LangGraph, CrewAI, and other frameworks, see the AG-UI framework integrations.

Step 2: Test locally

Start your server.

python my_agui_server.py

The server starts on http://localhost:8080. Send a test request to confirm the SSE stream is working before deploying.

curl -N -X POST http://localhost:8080/invocations \
  -H "Content-Type: application/json" \
  -d '{
    "thread_id": "test-123",
    "run_id": "run-456",
    "state": {},
    "messages": [{"role": "user", "content": "Hello!", "id": "msg-1"}],
    "tools": [], "context": [], "forwardedProps": {}
  }'

You should see the SSE events stream back:

data: {"type": "RUN_STARTED", "threadId": "test-123", "runId": "run-456"}
data: {"type": "TEXT_MESSAGE_START", "messageId": "msg-abc", "role": "assistant"}
data: {"type": "TEXT_MESSAGE_CONTENT", "messageId": "msg-abc", "delta": "Hello!"}
data: {"type": "TEXT_MESSAGE_END", "messageId": "msg-abc"}
data: {"type": "RUN_FINISHED", "threadId": "test-123", "runId": "run-456"}

Step 3: Deploy to AgentCore

The AgentCore starter toolkit handles the entire deployment process for you.

It builds an ARM64 container image, pushes it to Amazon ECR, and creates the AgentCore runtime with protocol flag serverProtocol: "AGUI". You don't write a Dockerfile or configure any infrastructure manually.

Install it first:

pip install bedrock-agentcore-starter-toolkit

Create a requirements.txt so the toolkit knows what to install in the container:

fastapi 
uvicorn 
ag_ui_strands

Now configure and deploy. The configure command sets the protocol to AG-UI and links your Cognito auth setup. The deploy command does the rest:

# Configure with AG-UI protocol + OAuth authentication
agentcore configure -e my_agui_server.py --protocol AGUI

# Deploy to AWS
agentcore deploy

After a successful deploy, you receive an agent runtime ARN. You will need this to invoke the agent and connect your frontend.

arn:aws:bedrock-agentcore:us-west-2:ACCOUNT_ID:runtime/my_agui_server-xyz123

Step 4: Invoke your deployed agent

With the agent running on AgentCore, you can now invoke it and stream AG-UI events over the wire. Set your environment variables first:

export BEARER_TOKEN="<your-cognito-bearer-token>"
export AGENT_ARN="arn:aws:bedrock-agentcore:us-west-2:ACCOUNT_ID:runtime/my_agui_server-xyz123"

Install the HTTP client libraries for making SSE requests:

pip install httpx httpx-sse

Then invoke the agent:

import asyncio
import json
import os
from urllib.parse import quote
from uuid import uuid4

import httpx
from httpx_sse import aconnect_sse

async def invoke_agui_agent(message: str):
    agent_arn = os.environ.get('AGENT_ARN')
    bearer_token = os.environ.get('BEARER_TOKEN')
    escaped_arn = quote(agent_arn, safe='')

    url = f"https://bedrock-agentcore.us-west-2.amazonaws.com/runtimes/{escaped_arn}/invocations?qualifier=DEFAULT"
    headers = {
        "Authorization": f"Bearer {bearer_token}",
        "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": str(uuid4()),
    }
    payload = {
        "threadId": str(uuid4()),
        "runId": str(uuid4()),
        "messages": [{"id": str(uuid4()), "role": "user", "content": message}],
        "state": {},
        "tools": [],
        "context": [],
        "forwardedProps": {},
    }

    async with httpx.AsyncClient(timeout=300) as client:
        async with aconnect_sse(client, "POST", url, headers=headers, json=payload) as sse:
            async for event in sse.aiter_sse():
                data = json.loads(event.data)
                event_type = data.get("type")
                if event_type == "TEXT_MESSAGE_CONTENT":
                    print(data.get("delta", ""), end="", flush=True)
                elif event_type == "RUN_ERROR":
                    print(f"Error: {data.get('code')} - {data.get('message')}")

asyncio.run(invoke_agui_agent("Hello from production!"))

You should see the agent's response stream token by token.

The same AG-UI events as local testing, now served from AgentCore with managed authentication, session isolation, and auto-scaling.

Step 5: Connect your frontend

With the agent deployed, point HttpAgent from @ag-ui/client at the AgentCore invocation URL instead of localhost. The rest of the client code stays the same.

This is the same AG-UI client used locally - you are just pointing it at the AgentCore URL instead of localhost.

import { HttpAgent } from "@ag-ui/client";

const agent = new HttpAgent({
  url: `https://bedrock-agentcore.us-west-2.amazonaws.com/runtimes/${encodedArn}/invocations?qualifier=DEFAULT`,
  headers: {
    "Authorization": `Bearer ${token}`,
    "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id": sessionId,
  }
});

For SigV4 authentication, sign requests using the AWS SDK. For OAuth 2.0, obtain a token from your configured Cognito user pool. Authentication configuration is the same across all AgentCore protocols. See the AgentCore auth docs for setup.

AG-UI support in AgentCore is available today across 14 AWS regions.

Resources

You can start building right away.

Explore the AgentCore AG-UI docs
Explore the AG-UI Protocol Docs
Check out the AWS Announcement

If you want a complete full-stack starting point, AWS also released the Fullstack AgentCore Solution Template - AgentCore, LangGraph, CopilotKit, and AG-UI all wired together and deployed via CDK.

What This Means

AG-UI works with LangGraph, CrewAI, AWS Strands, Google ADK, Mastra, Microsoft Agent Framework, LlamaIndex, Agno, and more. The protocol itself isn't tied to any of them - you pick the framework, AG-UI handles the interaction layer.

What was missing was a managed way to run these agents in production. That's what this support adds.

Beyond AG-UI hosting, AgentCore also covers persistent memory across sessions, tool connectivity via MCP Gateway, and production observability (though I didn't explore all of these in-depth).

thanks for reading!

You can connect me on GitHub, Twitter and LinkedIn.

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join the Discord community.

CopilotKit Follow

React UI + elegant infrastructure for AI Copilots, in-app AI agents, AI chatbots, and AI-powered Textareas 🪁

How to add AI to your App in 5 minutes

Anmol Baranwal — Wed, 11 Mar 2026 15:10:31 +0000

Most AI integrations are overengineered.

Even basic tutorials on adding an AI assistant get mixed up with LangGraph workflows and FastAPI servers. You don't need any of that.

Today, we will build one without any external agent framework or orchestration layers. Just an AI that knows what's in your app and can actually do things.

You will also learn how to implement the Generative UI pattern, which lets you render actual components rather than just text responses.

Let's jump in.

Why AI Integrations Get Complicated

Adding AI to an app sounds simple until you actually try it.

The moment you go beyond a basic chat interface, you run into a fundamental problem: LLMs on their own can't do anything. They can only generate text.

To make an AI that actually takes actions: reads your app's state, updates data, calls APIs - you need to build the plumbing yourself.

That plumbing is what an AI agent is. At its core, every agent runs the same loop:

Observe - take in context (user input, tool results, memory)
Reason - decide what to do next
Act - call a tool, write to a file, hit an API, whatever's needed

Credit: Anthropic

Simple enough in theory. But in practice, you end up writing the orchestration layer that manages this loop, tool registries that expose your app's functions to the LLM, state management to track what the agent knows and error handling for when the model goes sideways.

context = [initial_event]
while True:
  next_step = await llm.determine_next_step(context)
  context.append(next_step)
  if next_step.intent == "done":
    return next_step.final_answer
  result = await execute_step(next_step)
  context.append(result)

In a full agent setup, the LLM sits at the center - orchestrating across data sources, tools, models, and external services.

Credit: Medium blog

That's why tutorials teach for LangGraph and FastAPI servers. And for complex automation pipelines, that complexity is justified.

But if your goal is just to add an AI assistant inside your product - one that understands your UI, reads your data, and can take actions on behalf of the user, you are building a lot of infrastructure to solve a much smaller problem.

CopilotKit handles the loop, the context, the streaming and the frontend integration so you don't have to wire any of it yourself.

For this tutorial, we will use CopilotKit without any external agent framework, just hooks and your LLM of choice.

What CopilotKit Actually Does

CopilotKit is an open source framework to add AI agents to your app that abstracts away everything you just read about.

Instead of building your own agent loop, tool registry and streaming layer, you get a set of hooks and pre-built components that plug directly into your existing frontend.

The mental model is simple, you provide the AI with two things:

Your app's state - so it understands what's on screen and what the user is working with
A set of tools - so it can actually do something useful, not just respond with text

In practice, this maps to two hooks:

useAgentContext - share app state with the AI. Whatever you pass in becomes part of the LLM's context: current user, selected items, loaded data - the AI knows what your app knows.
useFrontendTool - define actions the AI can trigger. You give it a name, a description, and a handler. The LLM decides when to call it based on what the user asks.

The result is that adding an AI assistant to your app becomes a frontend problem, not an infrastructure problem.

Your UI, agent, tools are in one single interaction loop.

How to wire everything (1 minute)

You can follow along with the official docs or just stick with this. I will walk through exactly what each piece does and why it's there.

I'm using Next.js with TypeScript, but this works with any React/Angular framework out there!

// creates a nextjs app  
npx create-next-app@latest .

Install the three CopilotKit packages you need:

npm install @copilotkit/react-core @copilotkit/react-ui @copilotkit/runtime

@copilotkit/react-core/v2 : hooks and built-in chat components
@copilotkit/react-ui/v2 : styles
@copilotkit/runtime : backend runtime and LLM adapters

Create the API route at app/api/copilotkit/route.ts.

It's the single endpoint that receives messages from your UI, runs them through the agent loop and streams responses back.

BuiltInAgent is what runs the observe → reason → act loop
CopilotRuntime manages the session, streaming, and tool execution

That's your entire backend.

import { CopilotRuntime, copilotRuntimeNextJSAppRouterEndpoint } from "@copilotkit/runtime";
import { BuiltInAgent } from "@copilotkit/runtime/v2";
import { NextRequest } from "next/server";

const builtInAgent = new BuiltInAgent({
  model: "openai:gpt-5",
  // apiKey: process.env.OPENAI_API_KEY,
});

const runtime = new CopilotRuntime({
  agents: { default: builtInAgent },
});

export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    endpoint: "/api/copilotkit",
  });
  return handleRequest(req);
};

Create .env.local in the root and add your OpenAI API key.

OPENAI_API_KEY=sk-proj-...

If you want to switch to any other LLM Provider, you just need to pass the modified string to BuiltInAgent in your API route. Everything else remains the same.

// Anthropic
const builtInAgent = new BuiltInAgent({ model: "anthropic:claude-sonnet-4-5" });

// Google
const builtInAgent = new BuiltInAgent({ model: "google:gemini-2.0-flash" });

Add the matching key to .env.local and you are done. For custom models like Azure OpenAI, AWS Bedrock or Ollama, check the model selection docs.

Now, connect the frontend to that backend by wrapping your app in app/layout.tsx.

import { CopilotKit } from "@copilotkit/react-core";
import "@copilotkit/react-ui/v2/styles.css";

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        <CopilotKit runtimeUrl="/api/copilotkit">
          {children}
        </CopilotKit>
      </body>
    </html>
  );
}

CopilotKit is the context provider that every hook below it depends on. runtimeUrl points to the route you just created.

There are many in-built components like CopilotSidebar, CopilotChat and CopilotPopup. For this, I will be adding the chat sidebar.

import { CopilotSidebar } from "@copilotkit/react-core/v2";

export default function Page() {
  return (
    <main>
      <h1>Your App</h1>
      <CopilotSidebar />
    </main>
  );
}

Run npm run dev and you now have a working AI sidebar assistant! 🎉

It can respond to anything but it has no idea what's actually in your app. It doesn't know your data, your UI state or what the user is looking at. The next two steps fix that.

Give the AI Context

By default, the AI knows nothing about what's in your app. You can ask it, "how much did I spend on food?" and it has no idea - it only sees the conversation, not your app's state.

useAgentContext solves that. It pushes your React state into the agent's context window on every turn, so the AI always has a current snapshot of what's in your UI.

Let's build a simple expense tracker to see how this works.

"use client";
import { useState } from "react";
import { useAgentContext, CopilotSidebar } from "@copilotkit/react-core/v2";

type Expense = {
  id: number;
  description: string;
  amount: number;
  category: string;
};

const initialExpenses: Expense[] = [
  { id: 1, description: "Groceries", amount: 85, category: "Food" },
  { id: 2, description: "Netflix", amount: 15, category: "Entertainment" },
  { id: 3, description: "Uber", amount: 22, category: "Transport" },
];

export default function Page() {
  const [expenses, setExpenses] = useState<Expense[]>(initialExpenses);

  useAgentContext({
    description: "The user's current expense list. Each item has an id, description, amount in dollars, and category.",
    value: expenses,
  });

  return (
    <main className="p-8">
      <h1 className="text-2xl font-bold mb-4">My Expenses</h1>
      <ul className="space-y-2">
        {expenses.map((e) => (
          <li key={e.id} className="flex justify-between border-b py-2">
            <span>
              {e.description}{" "}
              <span className="text-gray-400 text-sm">({e.category})</span>
            </span>
            <span>${e.amount}</span>
          </li>
        ))}
      </ul>
      <CopilotSidebar />
    </main>
  );
}

expenses is your normal React state. useAgentContext takes it and injects it into the LLM's context on every turn.

Now, if the user asks "what's my biggest expense?", the AI is looking at the same list your UI renders.

Let the AI Do Things

Awareness is one thing. Being able to act is what makes an agent actually useful.

useFrontendTool lets you hand the AI a set of actions it can trigger. You define a tool with a name, a description, and a handler - the LLM decides when to call it based on what the user asks.

The LLM reads the description fields to understand what each tool does and what each parameter expects, so write them clearly and the model will fill them accurately even from casual conversational input.

Add useFrontendTool inside the same page.

parameters takes a Zod schema (a TypeScript-first validation library). If you haven't used it before, the idea is simple: you define the shape of your data with z.object({...}) and each field gets a .describe() call that tells the LLM what it means.

Install it with npm install zod.

"use client";
import { useState } from "react";
import { useAgentContext, useFrontendTool, CopilotSidebar } from "@copilotkit/react-core/v2";
import { z } from "zod";

// ... type and initialExpenses stay the same

export default function Page() {
  const [expenses, setExpenses] = useState<Expense[]>(initialExpenses);

  // ... useAgentContext

  useFrontendTool({
  name: "addExpense",
  description:
    "Add a new expense when the user mentions spending money on something.",
  parameters: z.object({
    description: z
      .string()
      .describe("What the expense was for, e.g. Lunch, Taxi, Coffee"),
    amount: z.number().describe("How much was spent in dollars"),
    category: z
      .string()
      .describe("Category: Food, Transport, Entertainment, Health, or Other"),
  }),
  handler: async ({ description, amount, category }) => {
    setExpenses((prev) => [
      ...prev,
      { id: Date.now(), description, amount, category },
    ]);
  },
});

  return (
    // ... remains the same
  );
}

You can send a sample query like "I spent $40 on dinner last night." The agent calls addExpense with { description: "Dinner", amount: 40, category: "Food" }, your handler updates expenses and the new item appears in the list instantly.

This is the observe → reason → act loop running end to end.

The agent observed your message and your current expenses via useAgentContext, reasoned that addExpense is the right call, and acted by invoking your handler with the right parameters. CopilotKit handled everything in between.

Bonus: Generative UI

The AI can read your data and update it. But so far, it only responds in text.

Generative UI is a new idea: instead of the agent describing results, it renders actual UI. For example, if someone asks about their spending, the app would display a breakdown card.

CopilotKit supports this through the render property on useFrontendTool. Instead of a text reply, the tool returns a React component - rendered inline in the chat, using your own design system.

Let's add a summary tool inside the same component.

import { ToolCallStatus, useFrontendTool } from "@copilotkit/react-core/v2";

useFrontendTool({
  name: "showSpendingSummary",
  description:
    "Call this when the user asks for a summary or overview of their expenses.",
  parameters: z.object({}),
  handler: async () => {
    const summary = expenses.reduce(
      (acc, e) => {
        acc[e.category] = (acc[e.category] ?? 0) + e.amount;
        return acc;
      },
      {} as Record<string, number>,
    );
    const total = expenses.reduce((sum, e) => sum + e.amount, 0);
    return JSON.stringify({ summary, total });
  },
  render: ({ result, status }) => {
    return (
      <div className="rounded-lg border p-4 mt-2 space-y-3">
        <p className="font-semibold text-sm">
          {status === ToolCallStatus.InProgress ? "Calculating..." : "Spending Breakdown"}
        </p>
        {status === ToolCallStatus.Complete && result && (
          <>
            {Object.entries(
              (JSON.parse(result) as { summary: Record<string, number>; total: number }).summary
            ).map(([category, amount]) => (
              <div key={category} className="flex justify-between text-sm">
                <span className="text-gray-600">{category}</span>
                <span className="font-medium">${amount}</span>
              </div>
            ))}
            <div className="flex justify-between text-sm font-semibold border-t pt-2">
              <span>Total</span>
              <span>${(JSON.parse(result) as { summary: Record<string, number>; total: number }).total}</span>
            </div>
          </>
        )}
      </div>
    );
  },
});

What's happening here:

parameters: z.object({}) is empty. The LLM doesn't need to pass anything, it just needs to decide when to call the tool
handler does the actual work, calculates category totals from the expenses state and returns them
ToolCallStatus gives you the exact lifecycle states of the tool call
result is the JSON string returned by your handler, parsed back into an object inside render

When you ask, "summarize my spending." LLM reads the query, looks at the context, recognizes that showSpendingSummary is the right tool to call and triggers it.

That's the pattern: the LLM decides when to act, your code does the work, your components display the result.

The working implementation of this minimal expense tracker is available in this GitHub repo in the example/basic branch.

The main branch takes the same pattern further with a full kanban board:

Move tasks between columns by asking the AI
Add, delete and reassign tasks from natural language
Get a visual board summary with generative UI

Here is the demo!

Most AI features that feel magical aren't. They are just an app that knows what's on screen and has a few actions wired up. That's it.

The hard part was always the infrastructure. Turns out it doesn't have to be.

You can check my work at anmolbaranwal.com. Thank you for reading! 🥰

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join the Discord community.

CopilotKit Follow

React UI + elegant infrastructure for AI Copilots, in-app AI agents, AI chatbots, and AI-powered Textareas 🪁

Give Your AG2 Agents a UI with AG‑UI and CopilotKit

Anmol Baranwal — Thu, 26 Feb 2026 15:01:48 +0000

Agent frameworks have become increasingly capable at reasoning, orchestration and tool execution. But most of them still operate as isolated systems.

AG2 (formerly AutoGen) changes that. It provides a shared runtime where specialized agents with defined roles, tools and conversation patterns can collaborate as one coordinated team across models and even frameworks.

But orchestration alone isn’t the same as shipping an interactive experience. Every application ends up building custom protocols and UI wiring to observe runs, show tool progress and capture user actions during execution.

AG-UI (Agent-User Interaction Protocol) defines that interaction contract. It standardizes how agent activity is streamed, so any frontend can stay synchronized in real time.

Today, AG2 integrates with AG‑UI and CopilotKit to bring agent orchestration into real applications.

You can refer to the docs and try the interactive AG-UI Dojo if you want to explore it yourself.

Let’s break down how these layers fit together, the end-to-end integration flow and what this unlocks for agent builders.

AG2: Runtime for Agent Teams

AG2 (formerly AutoGen) is an open-source programming framework built around multi-agent collaboration. You define specialized agents, wire tools and run structured conversations to solve tasks.

Instead of a single monolithic loop, AG2 treats agents as composable actors that collaborate through shared context.

At its core is ConversableAgent, the primitive that defines an agent’s role, reasoning model and tool access while handling message exchange.

from autogen import ConversableAgent, LLMConfig
from dotenv import load_dotenv
import os
load_dotenv()

# Create LLM configuration first
llm_config = LLMConfig({
  "api_type": "openai",
  "model": "gpt-5-mini",
  "api_key": os.getenv("OPENAI_API_KEY"),
})

# Create the agent
lesson_planner = ConversableAgent(
    name="planner_agent",
    system_message="You are a lesson planner for a fourth grade class.",
    description="Creates lesson plans.",
    llm_config=llm_config,
)

Agents can be organized into coordinated workflows using built-in patterns such as sequential flows, group conversations, nested calls and human-in-the-loop checkpoints.

In practice, you might model a planner, executor and reviewer as distinct agents: each with its own role and tools and let AG2 manage how they interact over multiple turns.

auto_selection = AutoPattern(
    agents=[teacher, lesson_planner, lesson_reviewer],
    initial_agent=lesson_planner,
    group_manager_args={"llm_config": llm_config},
)

response = run_group_chat(
    pattern=auto_selection,
    messages="Introduce students to the solar system.",
    max_rounds=20,
)

response.process()

This allows backend behavior to evolve by assembling agents differently, rather than rewriting control logic for every new workflow.

AG-UI integration currently wraps a single ConversableAgent. Multi-agent patterns (group chat, delegation, nested flows) still run inside AG2, but are exposed through a unified interaction surface.

Staged or multi-agent UX can be modeled using context variables. The pattern is covered in the integration section.

Read all the basic concepts involved in the docs and check the example repo, which includes many examples.

The Gap: Orchestration Without an Application Contract

Even if multi-agent logic is portable, it does not automatically become app-portable.

In a production application, the UI needs a stable runtime contract for:

Streaming messages and partial results.
Tool lifecycle (start → progress → finish/fail).
State snapshots and updates while the run is still executing.
Human-in-the-loop actions that can steer or interrupt a run

AG2 manages delegation, shared context and orchestration internally. But it does not define a standard event/state interface for how that execution is exposed to a frontend.

Without that contract, every AG2 app ends up inventing its own WebSocket/SSE payload shapes and UI mapping rules. Integration becomes tightly coupled glue code instead of a reusable interaction layer.

This is where AG-UI Protocol fits.

AG-UI: The Execution & Interaction Layer

AG-UI (Agent–User Interaction Protocol) is an event-based protocol that standardizes the live interaction stream between an agent runtime and an application.

During a run, the backend emits a stream of structured event types. The frontend consumes that stream and can send compatible interaction events back.

This matters because AG2 already manages multi-agent coordination internally. What AG-UI defines is how that coordination becomes observable and interactive at the application boundary.

Instead of every app inventing custom payload formats, AG-UI defines a small set of self-describing JSON event types.

Each event includes a clear type and structured payload. For example:

TEXT_MESSAGE_CONTENT : event streams LLM tokens
TOOL_CALL_START/END : convey function call progress
STATE_DELTA : carries JSON Patch deltas to sync state

In practice, AG‑UI standardizes:

Message streaming
Tool lifecycle events (start → progress → finish / fail)
State snapshots and incremental updates
User interaction events (clicks, form submits, confirmations)

Because the stream is standard and ordered, the frontend can reliably interpret what the backend is doing and render progress in real time.

The result is decoupling. The backend doesn’t need to know how the UI renders progress and the UI doesn’t need to understand internal orchestration logic.

Here, AG-UI uses Server-Sent Events (SSE) over HTTP as its primary transport. The AG2 integration serves a text/event-stream response per run.

In AG2, this protocol is implemented via AGUIStream, which wraps a ConversableAgent and exposes it as an ASGI endpoint that streams AG-UI events.

Explore the full event specification in the AG-UI documentation and implementation details in the AG2 user guide, including best practices for when to adopt this integration.

How CopilotKit wires AG2 to AG-UI

On the backend, AG2 exposes agent runs through an AG-UI-compatible endpoint using AGUIStream. That endpoint streams standardized AG-UI events for the lifecycle of a run.

CopilotKit acts as that client and UI runtime. It connects to the AG‑UI endpoint (via @ag-ui/client), subscribes to the event stream, renders agent activity inside your application and forwards user actions back into the same execution loop.

The wiring is straightforward:

Your backend runs an AG2 agent workflow
AGUIStream wraps the agent and translates execution into AG-UI events
Those events are exposed through an AG-UI endpoint over SSE
CopilotKit connects and subscribes to the live event stream
Agent messages, tool progress and state updates are rendered in the UI
User interactions are sent back over AG-UI into the same AG2 run

The same endpoint can be consumed by any other AG-UI-compatible client. At a high level, the full interaction loop looks like this:

AG2 Runtime (multi-agent orchestration)
│
│ executes ConversableAgents
│ manages delegation + shared state
│ invokes tools
▼
AG2 ↔ AG-UI Integration Layer (AGUIStream)
│
│ wraps ConversableAgent
│ maps AG2 activity → standardized AG-UI events
│ handles SSE transport
▼
AG-UI Endpoint (event + state stream)
│
│ agent ↔ application synchronization
│ TEXT_MESSAGE_CONTENT
│ TOOL_CALL_START / END
│ STATE_DELTA / STATE_SNAPSHOT
│ USER_INTERACTION
▼
CopilotKit (AG-UI client runtime)
│
├─ subscribes to live stream
├─ renders messages + tool lifecycle
├─ synchronizes state updates
└─ forwards user actions → AG2 via AG-UI
▼
Application UI

The backend remains purely AG2. CopilotKit simply consumes the AG-UI stream and renders it as product UI.

Integration Flow: AG2 → AG-UI → CopilotKit

We have seen the conceptual flow. Now let’s look at the actual implementation.

You can check the AG-UI integration examples in the build-with-ag2 repository, which consists of two patterns: weather/ directory shows a single-agent chat with tools & factory/ directory demonstrates a multi-stage pipeline using ContextVariables.

For complete CopilotKit + AG2 integration with full UI implementations, explore the ag2-samples repository.

The integration centers on AGUIStream, which wraps a ConversableAgent and exposes it as an AG‑UI endpoint over SSE.

Define your agent exactly as you normally would, then mount the stream on FastAPI. The only important detail is enabling streaming in the LLM configuration.

from autogen import ConversableAgent, LLMConfig
from autogen.ag_ui import AGUIStream
from fastapi import FastAPI

agent = ConversableAgent(
    name="weather_agent",
    system_message="You are a helpful weather assistant. Use the get_weather tool to look up current conditions for any city.",
    llm_config=LLMConfig({
        "model": "gpt-5-mini",
        "stream": True
    }),
    functions=[get_weather],
)

# wrap the agent and mount it as an ASGI endpoint
stream = AGUIStream(agent)  

app = FastAPI()  
app.mount("/chat", stream.build_asgi())

That’s the entire backend bridge.

AGUIStream automatically maps:

Token streaming → TEXT_MESSAGE_*
Tool execution → TOOL_CALL_START, TOOL_CALL_RESULT
Context updates → STATE_SNAPSHOT / STATE_DELTA
Run lifecycle → RUN_STARTED, RUN_FINISHED, RUN_ERROR

The endpoint serves Server-Sent Events (SSE). When a client sends a RunAgentInput payload with Accept: text/event-stream, the response becomes a live event stream.

Here's a simplified example of what flows back:

data: {"type":"RUN_STARTED", ...}
data: {"type":"TEXT_MESSAGE_CONTENT","delta":"The"}
data: {"type":"TEXT_MESSAGE_CONTENT","delta":" weather"}
data: {"type":"TOOL_CALL_START", ...}
data: {"type":"TOOL_CALL_RESULT", ...}
data: {"type":"RUN_FINISHED", ...}

The frontend does not need to understand AG2 internals. It reacts to event types. At the lowest level, consuming AG-UI is just opening an SSE stream:

const response = await fetch("/chat", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Accept": "text/event-stream"
  },
  body: JSON.stringify(runAgentInput)
});

But in practice, you don’t parse SSE manually.

CopilotKit understands AG-UI natively. You keep the same backend (AGUIStream), and simply point CopilotKit to that endpoint. Proxy AG‑UI traffic through a Next.js API route.

import {
  CopilotRuntime,
  ExperimentalEmptyAdapter,
  copilotRuntimeNextJSAppRouterEndpoint,
} from "@copilotkit/runtime";
import { HttpAgent } from "@ag-ui/client";
import { NextRequest } from "next/server";

// You can use any service adapter here for multi-agent support
// Use ExperimentalEmptyAdapter since AG2 handles the LLM directly
const serviceAdapter = new ExperimentalEmptyAdapter();

const runtime = new CopilotRuntime({
  agents: {
    weather_agent: new HttpAgent({
      url: "http://localhost:8000/chat"
    })
  }
});

export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter,
    endpoint: "/api/copilotkit",
  });

  return handleRequest(req);
};

Finally, wrap your app with CopilotKit and render the chat. CopilotKit subscribes to the AG‑UI stream and renders messages, tool progress and state updates in real time.

import { CopilotKit } from "@copilotkit/react-core";
import { CopilotChat } from "@copilotkit/react-ui";

<CopilotKit runtimeUrl="/api/copilotkit" agent="weather_agent">
  <CopilotChat />
</CopilotKit>

That's the complete loop. The weather example in the build-with-ag2 repository shows this minimal streaming setup.

Multi-Agent Orchestration

The current AG-UI integration wraps a single ConversableAgent. Multi-agent workflows (group chats, delegation, nested conversations) still run inside AG2, but they are exposed through one interaction surface.

To model staged or multi-agent UX, use an orchestrator agent with ContextVariables. It is a shared state that agents use to coordinate, track progress, and make decisions across a conversation.

When a tool updates context_variables, AGUIStream automatically emits a STATE_SNAPSHOT event.

The frontend can consume that snapshot to:

Highlight which agent is active
Show pipeline progress
Render stage-specific UI

Here's the pattern:

from autogen import ConversableAgent
from autogen.ag_ui import AGUIStream

def submit_plan(plan: str, context): # context injected automatically
    """Submit the plan and move to draft stage."""
    context["active_agent"] = "drafter"
    context["stage"] = "drafting"
    context["plan"] = plan
    return {"status": "plan submitted"}

def submit_draft(draft: str, context):
    """Submit the draft and move to review stage."""
    context["active_agent"] = "reviewer"
    context["stage"] = "reviewing"
    context["draft"] = draft
    return {"status": "draft submitted"}

def submit_review(approved: bool, context):
    """Submit review and finalize."""
    context["stage"] = "complete" if approved else "revising"
    return {"status": "review complete"}

orchestrator = ConversableAgent(
    name="orchestrator",
    system_message="You coordinate a multi-stage workflow.",
    llm_config=llm_config,
    functions=[submit_plan, submit_draft, submit_review],
)

stream = AGUIStream(orchestrator)

Each tool call that updates context triggers a STATE_SNAPSHOT. The frontend reads active_agent and stage from the snapshot and updates the UI accordingly.

The factory example shows this pattern with a 3-panel UI: agent pipeline, document preview and conversation log.

Native multi-agent support is actively being developed. This orchestrator pattern is the current workaround.

What This Enables for Agent Builders

With this integration, you can keep AG2 where it’s strong and still ship real UIs without protocol glue. You can:

✅ Build AG2 agent teams and expose them as AG‑UI endpoints with one wrapper (AGUIStream).

✅ Connect those endpoints to CopilotKit (or any AG‑UI client) without custom WebSocket or polling infrastructure.

✅ Reuse AG2 agent logic across products while keeping a single frontend contract (AG-UI) instead of inventing new payloads for every application

✅ Stream agent runs, tool progress and state in real time using CopilotKit components, so users see what the system is doing instead of waiting on a single final response.

Getting Started

You can start building with this integration today.

👉 Explore the AG2 documentation

👉 Check out the CopilotKit / AG2 docs to get started

👉 Check out our example repo for a minimal integration setup

👉 Explore the build-with-ag2 AG‑UI examples: weather/ directory shows a single-agent chat with tools, factory/ directory demonstrates a multi-stage pipeline using ContextVariables

👉 Try the AG2 demo in the AG-UI Dojo playground

What's Next

This launch makes AG2 a first‑class backend for AG‑UI‑compatible frontends.

On the CopilotKit side, the focus is on richer AG‑UI rendering components, better devtools for inspecting AG‑UI streams and more starter projects that bundle AG2 + CopilotKit out of the box.

Native multi-agent support in AG‑UI is an active focus. The AG2 and CopilotKit teams are working together to bring first-class multi-agent patterns to the protocol so you won't need the orchestrator workaround.

Over the next few weeks, expect more reference implementations, clearer integration patterns and production-grade examples that show how these layers work together in real applications.

How to Build a Research Assistant using Deep Agents

Anmol Baranwal — Fri, 20 Feb 2026 14:22:05 +0000

LangChain's Deep Agents provide a new way to build structured, multi-agent systems that can plan, delegate and reason across multiple steps.

It comes with planning, a filesystem for context and subagent spawning built in. But connecting that agent to a live frontend and actually showing what’s happening behind the scenes in real time is still surprisingly hard.

Today, we will build a Deep Agents powered research assistant using Tavily and connect it to a live Next.js UI with CopilotKit, so every step the agent takes streams to the frontend in real time.

You will find architecture, the key patterns, how state flows between the UI ↔ agent and the step-by-step guide to building this from scratch.

Let's build it.

What is covered?

In summary, we are covering these topics in detail.

What are Deep Agents?
Core Components
What are we building?
Building Frontend
Building Backend (FastAPI + Deep Agents + AG-UI)
Running Application
Data flow (frontend ↔ Agent)

Here is the GitHub Repository, deployed link and official docs if you want to explore yourself.

1. What are Deep Agents?

Most agents today are just “LLM in a loop + tools”. That works but it tends to be shallow: no explicit plan, weak long-horizon execution and messy state as runs get longer.

Popular agents like Claude Code, Deep Research and Manus get around this by following a common pattern: they plan first, externalize working context (often via files or a shell) and delegate isolated pieces of work to sub-agents.

Deep Agents package those primitives into a reusable agent runtime.

Instead of designing your own agent loop from scratch, you call create_deep_agent(...) and get a pre-wired execution graph that already knows how to plan, delegate and manage state across many steps.

Credit: LangChain

At a practical level, a Deep Agent created via create_deep_agent is just a LangGraph graph. There’s no separate runtime or hidden orchestration layer.

The "context management" in deep agents is also very practical -- they offload large tool payloads to the filesystem and only fall back to summarization when token usage approaches the model’s context window. You can read more on Context Management for Deep Agents blog by LangChain.

The mental model (how it runs)

Conceptually, the execution flow looks like this:

User goal
  ↓
Deep Agent (LangGraph StateGraph)
  ├─ Plan: write_todos → updates "todos" in state
  ├─ Delegate: task(...) → runs a subagent with its own tool loop
  ├─ Context: ls/read_file/write_file/edit_file → persists working notes/artifacts
  ↓
Final answer

That gives you a usable structure for “plan → do work → store intermediate artifacts → continue” without inventing your own plan format, memory layer or delegation protocol.

You can check official docs.

Where CopilotKit Fits

Deep Agents push key parts into explicit state (e.g. todos + files + messages), which makes runs easier to inspect. That explicit state is also what makes Copilotkit integration possible.

CopilotKit is a frontend runtime that keeps UI state in sync with agent execution by streaming agent events and state updates in real time (using AG-UI under the hood).

This middleware (CopilotKitMiddleware) is what allows the frontend to stay in lock-step with the agent as it runs. You can read the docs at docs.copilotkit.ai/langgraph/deep-agents.

agent = create_deep_agent(
    model="openai:gpt-4o",
    tools=[get_weather],
    middleware=[CopilotKitMiddleware()], # for frontend tools and context
    system_prompt="You are a helpful research assistant."
)

2. Core Components

Here are the core components that we will be using later on:

1) Planning Tools (built-in via Deep Agents) - built-in planning/to‑do behavior so the agent can break the workflow into steps without you writing a separate planning tool.

# Conceptual example (not required in codebase)
@tool
def todo_write(tasks: List[str]) -> str:
    formatted = "\n".join([f"- {task}" for task in tasks])
    return f"Todo list created:\n{formatted}"

2) Subagents - let the main agent delegate focused tasks into isolated execution loops. Each sub-agent has its own prompt, tools and context.

subagents = [
    {
        "name": "job-search-agent",
        "description": "Finds relevant jobs and outputs structured job candidates.",
        "system_prompt": JOB_SEARCH_PROMPT,
        "tools": [internet_search],
    }
]

3) Tools - this is how the agent actually does things. Here, finalize() signals completion.

@tool
def finalize() -> dict:
    """Signal that the agent is done."""
    return {"status": "done"}

How Deep Agents are implemented (Middleware)

If you are wondering how create_deep_agent() actually injects planning, files and subagents into a normal LangGraph agent, the answer is middleware.

Each feature is implemented as a separate middleware. By default, three are attached:

To-do list middleware - adds the write_todos tool and instructions that push the agent to explicitly plan and update a live todo list during multi-step tasks.
Filesystem middleware - adds file tools (ls, read_file, write_file, edit_file) so the agent can externalize notes and artifacts instead of stuffing everything into chat history.
Subagent middleware - adds the task tool, allowing the main agent to delegate work to subagents with isolated context and their own prompts/tools.

This is what makes Deep Agents feel “pre-wired” without introducing a new runtime. If you want to go deeper, the middleware docs show the exact implementation details.

3. What are we building?

Let's create an agent that:

Accepts a research question from the user
Uses Deep Agents to plan multi-step and orchestrate sub-agents
Searches the web using Tavily
Writes intermediate research artifacts using the filesystem middleware
Streams tool results back to the UI via CopilotKit (AG-UI)

The interface is a two-panel app where the left side is a CopilotKit chat UI and the right side is a live workspace that shows the agent’s plan, generated files and sources as the agent works.

Here's a simplified call request → response flow of what will happen:

[User asks research question]
        ↓
Next.js Frontend (CopilotChat + Workspace)
        ↓
CopilotKit Runtime → LangGraphHttpAgent
        ↓
Python Backend (FastAPI + AG-UI)
        ↓
Deep Agent (research_assistant)
    ├── write_todos        (planning, built-in)
    ├── write_file         (filesystem, built-in)
    ├── read_file          (filesystem, built-in)
    └── research(query)
            └── internal Deep Agent [thread-isolated]
                    └── internet_search (Tavily)

We will see the concepts in action as we build the agent.

4. Frontend: wiring the agent to the UI

Let's first build the frontend part. This is how our directory will look.

The src directory hosts the Next.js frontend, including the UI, shared components and the CopilotKit API route (/api/copilotkit) used for agent communication.

.
├── src/                             ← Next.js frontend
│   ├── app/
│   │   ├── page.tsx          
│   │   ├── layout.tsx               ← CopilotKit provider
│   │   └── api/
│   │       └── copilotkit/route.ts  ← CopilotKit AG-UI runtime
│   ├── components/
│   │   ├── FileViewerModal.tsx      ← Markdown file viewer
│   │   ├── WorkSpace.tsx            ← Research progress display
│   │   └── ToolCard.tsx             ← Tool call visualizer
├── lib/
│   └── types.ts
├── package.json                     
├── next.config.ts                   
└── README.md

If you don’t have a frontend, you can create a new Next.js app with TypeScript.

// creates a nextjs app  
npx create-next-app@latest .

Step 1: CopilotKit Provider & Layout

Install the necessary CopilotKit packages.

npm install @copilotkit/react-core @copilotkit/react-ui @copilotkit/runtime

@copilotkit/react-core provides the core React hooks and context that connect your UI to an AG-UI compatible agent backend.
@copilotkit/react-ui offers ready-made UI components like <CopilotChat /> to build AI chat or assistant interfaces quickly.
@copilotkit/runtime is the server-side runtime that exposes an API endpoint and bridges the frontend with an AG-UI compatible backend (e.g., a LangGraph HTTP agent).

The <CopilotKit> component must wrap the Copilot-aware parts of your application. In most cases, it's best to place it around the entire app, like in layout.tsx.

import type { Metadata } from "next";

import { CopilotKit } from "@copilotkit/react-core";
import "./globals.css";
import "@copilotkit/react-ui/styles.css";

export const metadata: Metadata = {
  title: "Deep Research Assistant | CopilotKit Deep Agents Demo",
  description: "A research assistant powered by Deep Agents and CopilotKit - demonstrating planning, memory, subagents, and generative UI",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body className="antialiased">
        <CopilotKit runtimeUrl="/api/copilotkit" agent="research_assistant">
          {children}
        </CopilotKit>
      </body>
    </html>
  );
}

Here, runtimeUrl="/api/copilotkit" points to the Next.js API route CopilotKit uses to talk to the agent backend.

Each page is wrapped in this context so UI components know which agent to invoke and where to send requests.

Step 2: Next.js API Route: Proxy to FastAPI

This Next.js API route acts as a thin proxy between the browser and the Deep Agents. It:

Accepts CopilotKit requests from the UI
Forwards them to the agent over AG-UI
Streams agent state and events back to the frontend

Instead of letting the frontend talk to the FastAPI agent directly, all requests go through a single endpoint /api/copilotkit.

import {
  CopilotRuntime,
  ExperimentalEmptyAdapter,
  copilotRuntimeNextJSAppRouterEndpoint,
} from "@copilotkit/runtime";
import { LangGraphHttpAgent } from "@copilotkit/runtime/langgraph";
import { NextRequest } from "next/server";

// Empty adapter since the LLM is handled by the remote agent
const serviceAdapter = new ExperimentalEmptyAdapter();

// Configure CopilotKit runtime with the Deep Agents backend
const runtime = new CopilotRuntime({
  agents: {
    research_assistant: new LangGraphHttpAgent({
      url: process.env.LANGGRAPH_DEPLOYMENT_URL || "http://localhost:8123",
    }),
  },
});

export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter,
    endpoint: "/api/copilotkit",
  });

  return handleRequest(req);
};

Here's a simple explanation of the above code:

The code above registers the research_assistant agent.
LangGraphHttpAgent : defines a remote LangGraph agent endpoint. It points to the Deep Agents backend running on FastAPI.
ExperimentalEmptyAdapter : simple no-op adapter used when the agent backend handles its own LLM calls and orchestration
copilotRuntimeNextJSAppRouterEndpoint : small helper that adapts the Copilot runtime to a Next.js App Router API route and returns a handleRequest function

Step 3: Types (Research State)

Before building components, let's define shared state for todos, files, and sources in lib/types/research.ts. These are the contracts between the tool results from the agent and the local React state.

// Uses local state + useDefaultTool instead of CoAgent (avoids type issues with Python FilesystemMiddleware)

export interface Todo {
  id: string;
  content: string;
  status: "pending" | "in_progress" | "completed";
}

export interface ResearchFile {
  path: string;
  content: string;
  createdAt: string;
}

// Sources found via internet_search (includes content)
export interface Source {
  url: string;
  title: string;
  content?: string;
  status: "found" | "scraped" | "failed";
}

export interface ResearchState {
  todos: Todo[];
  files: ResearchFile[];
  sources: Source[];
}

export const INITIAL_STATE: ResearchState = {
  todos: [],
  files: [],
  sources: [],
};

Instead of dumping raw tool JSON into chat, each tool result routes into a dedicated state slot - write_todos updates todos, write_file appends to files and research appends to sources. This becomes the foundation of the Workspace panel.

Step 4: Building Key Components

I'm only covering the core logic behind each component since the overall code is huge. You can find all the components in the repository at src/components.

✅ ToolCard Component

This is the client component that renders every tool call inline inside chat. It has two modes:

SpecializedToolCard for known tools (write_todos, research, write_file, read_file) with icons, status indicators and result previews
DefaultToolCard for unknown tools that fall back to expandable JSON.

"use client";

import { useState } from "react";
import { Pencil, ClipboardList, Search, Save, BookOpen, Check, ChevronDown } from "lucide-react";

const TOOL_CONFIG = {
  write_todos: {
    icon: Pencil,
    getDisplayText: () => "Updating research plan...",
    getResultSummary: (result, args) => {
      const todos = (args as { todos?: unknown[] })?.todos;
      if (Array.isArray(todos)) {
        return `${todos.length} todo${todos.length !== 1 ? "s" : ""} updated`;
      }
      return null;
    },
  },
  research: {
    icon: Search,
    getDisplayText: (args) =>
      `Researching: ${((args.query as string) || "...").slice(0, 50)}${(args.query as string)?.length > 50 ? "..." : ""}`,
    getResultSummary: (result) => {
      if (result && typeof result === "object" && "sources" in result) {
        const { sources } = result as { summary: string; sources: unknown[] };
        return `Found ${sources.length} source${sources.length !== 1 ? "s" : ""}`;
      }
      return "Research complete";
    },
  },
  write_file: {
    icon: Save,
    getDisplayText: (args) => {
      const path = args.path as string | undefined;
      const filename =
        path?.split("/").pop() || (args.filename as string | undefined);
      return `Writing: ${filename || "file"}`;
    },
    getResultSummary: (_result, args) => {
      const content = args.content as string | undefined;
      if (content) {
        const firstLine = content.split("\n")[0].slice(0, 50);
        return firstLine + (content.length > 50 ? "..." : "");
      }
      return "File written";
    },
  },
  // read_todos, read_file follow the same pattern
};

export function ToolCard({ name, status, args, result }: ToolCardProps) {
  const config = TOOL_CONFIG[name];
  if (config) {
    return (
      <SpecializedToolCard
        name={name}
        status={status}
        args={args}
        result={result}
        config={config}
      />
    );
  }
  return (
    <DefaultToolCard name={name} status={status} args={args} result={result} />
  );
}

Here's a brief explanation:

research and write_todos are expandable, clicking reveals the full query + findings or the live todo checklist with pending / inprogress / completed states.
resultSummary appears as a small green line below the display text so you can glance at the output without expanding.
DefaultToolCard handles any tool not in TOOL_CONFIG, showing collapsible args and result JSON.

In this component, ExpandedDetails handles the per-tool expanded view. research and write_todos get structured layouts; everything else falls back to a JSON pre block.

function ExpandedDetails({ name, result, args }) {
  if (name === "research") {
    const summary = typeof result === "object" && result && "summary" in result
      ? (result as any).summary
      : "";
    return (
      <div>
        <p>Query: {args.query as string}</p>
        <p>{summary}</p>
      </div>
    );
  }

  if (name === "write_todos") {
    const todos = (args as any)?.todos;
    return (
      <div>
        {todos?.map((todo: any, i: number) => (
          <div key={todo.id || i}>
            <span>{todo.status === "completed" ? "✓" : todo.status === "inprogress" ? "●" : "○"}</span>
            <span>{todo.content}</span>
          </div>
        ))}
      </div>
    );
  }

  // Fallback
  return (
    <pre>
      {typeof result === "string" ? result : JSON.stringify(result, null, 2)}
    </pre>
  );
}

Check out the complete code at src/components/ToolCard.tsx.

✅ Workspace Component

This is the right-side panel that displays research progress as it happens. It has three collapsible sections: Research Plan, Files and Sources - each with a live badge count and an empty state when nothing has arrived yet.

export function Workspace({ state }: { state: ResearchState }) {
  const { todos, files, sources } = state;
  const fileCount = files.length;
  const todoCount = todos.length;
  const sourceCount = sources.length;
  
  // State for file viewer modal
  const [selectedFile, setSelectedFile] = useState<ResearchFile | null>(null);

  return (
    <div className="workspace-panel p-6">
      <div className="mb-6">
        <h2 className="text-xl font-bold">Workspace</h2>
        <p className="text-sm">Research progress and artifacts</p>
      </div>
      <Section title="Research Plan" icon={ListTodo} badge={todos.length}>
        <TodoList todos={todos} />
      </Section>
      <Section title="Files" icon={FileText} badge={files.length}>
        <FileList files={files} onFileClick={setSelectedFile} />
      </Section>
      <Section title="Sources" icon={Globe} badge={sources.length}>
        <SourceList sources={sources} />
      </Section>
      <FileViewerModal
        file={selectedFile}
        onClose={() => setSelectedFile(null)}
      />
    </div>
  );
}

function TodoList({ todos }: { todos: Todo[] }) {
  if (todos.length === 0) return <div>...</div>; // empty state

  return (
    <div className="space-y-1">
      {todos.map((todo) => (
        <div
          key={todo.id}
          className={`todo-item ${
            todo.status === "completed"
              ? "todo-item-completed"
              : todo.status === "in_progress"
                ? "todo-item-inprogress"
                : "todo-item-pending"
          }`}
        >
          <span>{/* Check / CircleDot / Circle icon based on status */}</span>
          <span className="text-sm">{todo.content}</span>
        </div>
      ))}
    </div>
  );
}

// FileList — same pattern, items are clickable (onFileClick)
// each row has a download button with e.stopPropagation()
function FileList({
  files,
  onFileClick,
}: {
  files: ResearchFile[];
  onFileClick: (file: ResearchFile) => void;
}) {
  if (files.length === 0) return <div>...</div>; // empty state

  return (
    <div className="space-y-2">
      {files.map((file, i) => (
        <div
          key={`${file.path}-${i}`}
          className="file-item"
          onClick={() => onFileClick(file)}
        >
          <div className="flex items-center gap-3">
            {/* FileText icon */}
            <div>
              <p className="text-sm font-medium">
                {file.path.split("/").pop()}
              </p>
              <p className="text-xs">{file.path}</p>
            </div>
          </div>
          <button
            onClick={(e) => {
              e.stopPropagation();
              downloadFile(file);
            }}
          >
            {/* Download icon */}
          </button>
        </div>
      ))}
    </div>
  );
}

// SourceList — same pattern, colour-codes by source.status 
// (scraped → green check, failed → red X, found → grey circle) 
// each source links to source.url
function SourceList({ sources }: { sources: Source[] }) {
  if (sources.length === 0) return <div>...</div>; // empty state

  return (
    <div className="space-y-2">
      {sources.map((source, i) => (
        <div
          key={`${source.url}-${i}`}
          className={`file-item ${source.status === "failed" ? "source-failed" : ""}`}
        >
          <span>{/* Check / X / Circle based on source.status */}</span>
          <div className="flex-1 min-w-0">
            <p className="text-sm font-medium truncate">
              {source.title || new URL(source.url).hostname}
            </p>
            <a
              href={source.url}
              target="_blank"
              rel="noopener noreferrer"
              className="text-xs truncate block"
            >
              {source.url}
            </a>
          </div>
        </div>
      ))}
    </div>
  );
}

A brief explanation:

TodoList renders each todo with a pending / inprogress / completed icon. Completed items get a strikethrough.
FileList items are clickable: they open FileViewerModal with full Markdown rendering and a download button.
SourceList colour-codes sources: green check for scraped, red X for failed, grey circle for found-but-not-yet-scraped.

Check out the complete code at src/components/WorkSpace.tsx.

✅ FileViewerModal Component

This component renders a modal that displays the contents of a research file written by the Deep Agent. It renders file content as formatted Markdown using the react-markdown package.

Install it using the following command.

npm install react-markdown

Here is the full core implementation:

export function FileViewerModal({ file, onClose }: FileViewerModalProps) {
  const handleKeyDown = useCallback(
    (e: KeyboardEvent) => { if (e.key === "Escape") onClose(); },
    [onClose]
  );

  useEffect(() => {
    if (file) {
      document.addEventListener("keydown", handleKeyDown);
      document.body.style.overflow = "hidden";
    }
    return () => {
      document.removeEventListener("keydown", handleKeyDown);
      document.body.style.overflow = "";
    };
  }, [file, handleKeyDown]);

  if (!file) return null;

  const filename = file.path.split("/").pop() || file.path;

  const handleDownload = () => {
    const blob = new Blob([file.content], { type: "text/markdown" });
    const url = URL.createObjectURL(blob);
    const a = document.createElement("a");
    a.href = url;
    a.download = filename;
    document.body.appendChild(a);
    a.click();
    document.body.removeChild(a);
    URL.revokeObjectURL(url);
  };

  return (
    <div className="fixed inset-0 z-50 flex items-center justify-center p-4">
      <div className="absolute inset-0 bg-black/30 backdrop-blur-sm" onClick={onClose} aria-hidden="true" />
      <div className="relative max-w-3xl w-full max-h-[85vh] flex flex-col" role="dialog" aria-modal="true">
        {/* Header -- filename + download + close */}
        <div className="flex items-center justify-between border-b ...">
          <div className="flex items-center gap-3">
            {/* FileText icon */}
            <h2 className="truncate max-w-md">{filename}</h2>
          </div>
          <div className="flex items-center gap-2">
            <button onClick={handleDownload}>...</button>
            <button onClick={onClose}>...</button>
          </div>
        </div>

        {/* Scrollable Markdown body */}
        <div className="flex-1 overflow-y-auto p-8">
          <div className="prose prose-sm prose-slate max-w-none">
            <ReactMarkdown>{file.content}</ReactMarkdown>
          </div>
        </div>

        {/* Footer -- full file path */}
        <div className="border-t ...">
          <code>{file.path}</code>
        </div>
      </div>
    </div>
  );
}

Here's a brief explanation:

useEffect registers the Escape listener and locks body scroll only when a file is open, and cleans both up on close.
handleDownload creates a temporary anchor element to trigger a browser download without any server round-trip.

Check out the complete code at src/components/FileViewerModal.tsx.

Step 5: Connecting the Chat UI to the Agent

At this point, all the pieces are in place. This page owns ResearchState, passes it to Workspace, uses CopilotChat component for the conversational UI, runs useDefaultTool to intercept every tool call and routes results into state.

research → appends to sources
write_todos → replaces todos
write_file → appends to files

Here's the code.

"use client";

import { useState, useRef } from "react";
import { CopilotChat } from "@copilotkit/react-ui";
import { useDefaultTool } from "@copilotkit/react-core";
import { Workspace } from "@/components/Workspace";
import { ResearchState, INITIAL_STATE, Todo } from "@/types/research";
import { ToolCard } from "@/components/ToolCard";

export default function Page() {
  const [state, setState] = useState<ResearchState>(INITIAL_STATE);
  const processedKeysRef = useRef<Set<string>>(new Set());

  useDefaultTool({
    render: (props) => {
      const { name, status, args, result } = props;

      // Deduplicate on re-renders
      if (status === "complete") {
        const resultStr = result ? JSON.stringify(result) : "";
        const resultHash = resultStr
          ? `${resultStr.length}-${resultStr.slice(0, 100)}`
          : "";
        const key = `${name}-${JSON.stringify(args)}-${resultHash}`;
        if (processedKeysRef.current.has(key)) return <ToolCard {...props} />;
        processedKeysRef.current.add(key);
      }

      if (name === "research" && status === "complete" && result) {
        const researchResult = result as {
          summary: string;
          sources: Array<{
            url: string;
            title: string;
            content?: string;
            status: "found" | "scraped" | "failed";
          }>;
        };
        if (researchResult.sources && researchResult.sources.length > 0) {
          queueMicrotask(() =>
            setState((prev) => ({
              ...prev,
              sources: [...prev.sources, ...researchResult.sources],
            })),
          );
        }
      }

      if (name === "write_todos" && status === "complete" && args?.todos) {
        const todosWithIds = (
          args.todos as Array<{ id?: string; content: string; status: string }>
        ).map((todo, index) => ({
          ...todo,
          id: todo.id || `todo-${Date.now()}-${index}`,
        }));
        queueMicrotask(() =>
          setState((prev) => ({ ...prev, todos: todosWithIds as Todo[] })),
        );
      }

      if (name === "write_file" && status === "complete" && args?.file_path) {
        queueMicrotask(() =>
          setState((prev) => ({
            ...prev,
            files: [
              ...prev.files,
              {
                path: args.file_path as string,
                content: args.content as string,
                createdAt: new Date().toISOString(),
              },
            ],
          })),
        );
      }

      return <ToolCard {...props} />;
    },
  });

  return (
    <div className="relative min-h-screen">
      <main className="relative z-10 h-screen flex overflow-hidden">
        {/* Chat panel -- left side */}
        <div className="w-[38%] h-full border-r flex flex-col">
          {/* Header */}
          <div style={{ flex: 1, minHeight: 0, overflow: "hidden" }}>
            <CopilotChat
              className="h-full"
              labels={{
                title: "Deep Research Assistant",
                initial: "What topic would you like me to research?",
                placeholder: "Ask me to research any topic...",
              }}
            />
          </div>
        </div>

        {/* Workspace panel -- right side */}
        <div className="w-[62%] h-full overflow-hidden">
          <Workspace state={state} />
        </div>
      </main>
    </div>
  );
}

A few things worth noting:

useDefaultTool intercepts every tool call and renders a ToolCard inline inside CopilotChat.
queueMicrotask defers setState so it never fires mid-render.
processedKeysRef deduplicates results since the render callback fires multiple times as status updates stream in.

4. Backend: Building the Agent Service (FastAPI + Deep Agents + Tavily)

We will now build the FastAPI backend that hosts our Deep Agent.

Under the /agent directory lives a FastAPI server that runs the research agent. Here's the project structure of the backend.

agent/
├── main.py           ← FastAPI server + AG-UI
├── agent.py          ← Deep Agents graph
├── tools.py          ← Tavily search tools
├── pyproject.toml    ← Python deps (uv)
├── uv.lock
└── .env

At a high level, the backend:

Builds a Deep Agent graph
Uses Tavily for real-time web search
Wraps research in an isolated sub-agent
Exposes a CopilotKit-compatible AG-UI endpoint
Streams tool calls to the frontend

The backend uses uv for dependency management. Install it if you don't have it:

pip install uv

Initialize a new uv project. This will generate a fresh pyproject.toml.

cd agent
uv init
uv python pin 3.12

Then install the dependencies:

uv add copilotkit deepagents fastapi langchain langchain-openai python-dotenv tavily-python "uvicorn[standard]" ag-ui-langgraph

copilotkit : connects agents to a frontend with streaming, tools and shared state.
deepagents : planning-first agent framework for multi-step execution
fastapi : web framework that exposes the agent API
langchain : agent and tool orchestration layer.
langchain-openai : OpenAI model integration for LangChain.
tavily-python : web search for real-time research
ag-ui-langgraph : AG-UI protocol adapter for LangGraph
uvicorn[standard] : ASGI server to run FastAPI

Now run the following command to generate a uv.lock file pinned with exact versions.

uv sync

Add necessary API Keys

Create a .env file under the agent directory and add your OpenAI API Key and Tavily API Key to the file. I have attached the docs link so it's easy to follow.

OPENAI_API_KEY=sk-proj-...
TAVILY_API_KEY=tvly-dev-...
OPENAI_MODEL=gpt-5.2

OpenAI API Key

Tavily API Key

Step 1: Implement Research Tools

Let's define two tools (tools.py):

internet_search is the low-level Tavily wrapper. It runs a search and returns up to max_results formatted results.
research wraps an internal Deep Agent that runs in a separate thread to prevent its streaming events from leaking to the frontend via LangChain callback propagation.

Here is the code.

import os
from typing import Any
from concurrent.futures import ThreadPoolExecutor
from langchain_core.tools import tool
from langchain_core.messages import HumanMessage
from tavily import TavilyClient


def _do_internet_search(query: str, max_results: int = 5) -> list[dict[str, Any]]:
    tavily_key = os.environ.get("TAVILY_API_KEY")
    if not tavily_key:
        raise RuntimeError("TAVILY_API_KEY not set")

    try:
        client = TavilyClient(api_key=tavily_key)
        results = client.search(
            query=query,
            max_results=max_results,
            include_raw_content=False,
            topic="general",
        )
        return [
            {
                "url": r.get("url", ""),
                "title": r.get("title", ""),
                "content": (r.get("content") or "")[:3000],
            }
            for r in results.get("results", [])
        ]
    except Exception as e:
        return [{"error": str(e)}]


@tool
def internet_search(query: str, max_results: int = 5) -> list[dict[str, Any]]:
    """Search the web and return results with content."""
    return _do_internet_search(query, max_results)


@tool
def research(query: str) -> dict:
    """Research a topic using web search. Returns structured data with sources."""

    from deepagents import create_deep_agent
    from langchain_openai import ChatOpenAI

    def _run_research_isolated():
        search_results = []

        def internet_search_tracked(query: str, max_results: int = 5):
            """Search the web and return results with content."""
            results = _do_internet_search(query, max_results)
            search_results.extend(results)
            return results

        llm = ChatOpenAI(
            model=os.environ.get("OPENAI_MODEL", "gpt-5.2"),
            temperature=0.7,
            api_key=os.environ.get("OPENAI_API_KEY"),
        )

        research_agent = create_deep_agent(
            model=llm,
            system_prompt="""You are a Research Specialist.
Use internet_search to find information. Return a prose summary of findings.
Rules:
- Call internet_search ONCE with a focused query
- Return a brief summary (2-3 sentences) of key findings
- No JSON, no code blocks, just prose""",
            tools=[internet_search_tracked],
        )

        result = research_agent.invoke({"messages": [HumanMessage(content=query)]})

        summary = result["messages"][-1].content
        sources = [
            {
                "url": r["url"],
                "title": r.get("title", ""),
                "content": r.get("content", "")[:3000],
                "status": "found",
            }
            for r in search_results
            if "url" in r and not r.get("error")
        ]

        return {"summary": summary, "sources": sources}

    with ThreadPoolExecutor(max_workers=1) as executor:
        future = executor.submit(_run_research_isolated)
        return future.result()

Here's what's happening:

_do_internet_search caps each result at 3000 chars to stay within token limits and returns an error dict instead of crashing.
internet_search_tracked extends search_results on every call, building a clean source list to return alongside the final summary.
research_agent is constrained to call internet_search once -- prevents looping and keeps latency predictable.
ThreadPoolExecutor isolates the subagent in a separate thread so its stream events don't leak into the frontend chat.

Step 2: Define the Agent Behavior

This is the brain of the system. The agent is a Deep Agents graph that:

Creates a research plan using write_todos
Calls research() for each research question
Writes a final report to /reports/final_report.md

Here is the main system prompt that enforces this workflow:

MAIN_SYSTEM_PROMPT = """You are a Deep Research Assistant, an expert at planning and
executing comprehensive research on any topic.

Hard rules (ALWAYS follow):
- NEVER output raw JSON, data structures, or code blocks in your messages
- Communicate with the user only in natural, readable prose
- When you receive data from research, synthesize it into insights

Your workflow:
1. PLAN: Create a research plan using write_todos with clear, actionable steps
2. RESEARCH: Use research(query) tool to investigate each topic
3. SYNTHESIZE: Write a final report to /reports/final_report.md using write_file

Important guidelines:
- Always start by creating a research plan with write_todos
- Call research() for each distinct research question
- The research tool returns prose summaries of findings
- You write all files - compile findings into a comprehensive report
- Update todos as you complete each step

Example workflow:
1. write_todos(["Research topic A", "Research topic B", "Synthesize findings"])
2. research("Find information about topic A") -> receives prose summary
3. research("Find information about topic B") -> receives prose summary
4. write_file("/reports/final_report.md", "# Research Report\n\n...")

Always maintain a professional, comprehensive research style."""

Here is the core agent graph (build_agent):

import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from deepagents import create_deep_agent
from langgraph.checkpoint.memory import MemorySaver
from copilotkit import CopilotKitMiddleware
from tools import research

load_dotenv()

def build_agent():
    api_key = os.environ.get("OPENAI_API_KEY")
    if not api_key:
        raise RuntimeError("Missing OPENAI_API_KEY environment variable")

    tavily_key = os.environ.get("TAVILY_API_KEY")
    if not tavily_key:
        raise RuntimeError("Missing TAVILY_API_KEY environment variable")

    model_name = os.environ.get("OPENAI_MODEL", "gpt-5.2")
    llm = ChatOpenAI(
        model=model_name,
        temperature=0.7,
        api_key=api_key,
    )

    agent_graph = create_deep_agent(
        model=llm,
        system_prompt=MAIN_SYSTEM_PROMPT,
        tools=[research],
        middleware=[CopilotKitMiddleware()],
        checkpointer=MemorySaver(),
    )

    return agent_graph.with_config({"recursion_limit": 100})

Here's what's happening:

CopilotKitMiddleware() enables AG-UI streaming to the frontend
MemorySaver() enables stateful execution
Recursion limit increased to support multi-step research
research is the only explicit tool since write_todos, read_file, write_file are built into Deep Agents automatically

Step 3: Expose the Agent via FastAPI + AG-UI

The last step is to wire everything together and expose it as a FastAPI app. The main.py builds the agent graph, configures which tool calls to emit to the frontend and registers the AG-UI endpoint.

import os
import uvicorn
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from dotenv import load_dotenv
from ag_ui_langgraph import add_langgraph_fastapi_endpoint
from copilotkit import LangGraphAGUIAgent
from copilotkit.langgraph import copilotkit_customize_config

from agent import build_agent

load_dotenv()

app = FastAPI(title="Deep Research Assistant", version="1.0.0")

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.get("/health")
def health():
    return {"status": "ok", "service": "deep-research-agent"}

agent_graph = build_agent()

# Only emit main agent tools — suppress internal subagent tool calls
agui_config = copilotkit_customize_config(
    emit_tool_calls=["research", "write_todos", "write_file", "read_file", "edit_file"]
)
agui_config["recursion_limit"] = 100

add_langgraph_fastapi_endpoint(
    app=app,
    agent=LangGraphAGUIAgent(
        name="research_assistant",
        description="A deep research assistant that plans, searches, and synthesizes research reports",
        graph=agent_graph,
        config=agui_config,
    ),
    path="/",
)

def main():
    uvicorn.run(
        "main:app",
        host=os.getenv("SERVER_HOST", "0.0.0.0"),
        port=int(os.getenv("SERVER_PORT", "8123")),
        reload=True,
    )

if __name__ == "__main__":
    main()

The emit_tool_calls config controls exactly which tool calls CopilotKit streams to the frontend. Without it, the subagent's internal internet_search calls leak through as raw JSON noise in the chat UI.

5. Running the Application

After completing all the parts of the code, it's time to run it locally. Please make sure you have added the credentials to the agent/.env.

From the project root, navigate to the agent directory and start the FastAPI server:

cd agent
uv run python main.py

The backend will start on http://localhost:8123.

In a new terminal window, start the frontend development server using:

npm run dev

Once both servers are running, open the frontend in your browser at http://localhost:3000/ to view it locally.

CopilotKit also provides the Agent Inspector, which is a live AG-UI runtime view that lets you inspect agent runs, state snapshots, messages and tool calls as they stream from the backend. It's accessible from a copilotkit button overlaid on your app.

Here is the complete demo!

6. Data flow

Now that we have built both the frontend and the agent service, this is how data actually flows between them. This should be easy to follow if you have been building along so far.

[User sends research question]
        ↓
Next.js Frontend (CopilotChat + Workspace)
        ↓
POST /api/copilotkit (AG-UI protocol)
        ↓
CopilotKit Runtime → LangGraphHttpAgent (localhost:8123)
        ↓
FastAPI Backend (AG-UI endpoint "/")
        ↓
Deep Agent (research_assistant)
        ↓
Deep Agents orchestration
    ├── write_todos  → todos state   → Workspace (Research Plan)
    ├── research()   → sources state → Workspace (Sources)
    │       └── internal Deep Agent [thread-isolated]
    │               └── internet_search (Tavily API)
    └── write_file   → files state   → Workspace (Files)
        ↓
AG-UI streaming (SSE)
        ↓
useDefaultTool intercepts tool calls
        ↓
ToolCard renders in chat + Workspace updates live

That's it! 🎉

You now have a fully working Deep Agents research assistant that plans, searches, and writes reports, with every step visible in the frontend as it happens.

The real win here is visibility. CopilotKit's AG-UI layer turns what would otherwise be a black-box agent into something users can actually follow and trust.

You can check my work at anmolbaranwal.com. Thank you for reading! 🥰

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join the Discord community.

CopilotKit Follow

React UI + elegant infrastructure for AI Copilots, in-app AI agents, AI chatbots, and AI-powered Textareas 🪁

How to Build a LangGraph Research Agent that Embeds Dynamic Charts via MCP Apps (CopilotKit & Tako)

Anmol Baranwal — Tue, 03 Feb 2026 15:49:30 +0000

AI research assistants are getting smarter, but turning that reasoning into something usable is still hard.

Real research involves more than text: searching the web, exploring structured data, comparing sources, and gradually assembling a report. Doing that well requires state, tools, and UI -- not just a chat box.

Today, we will build a chat + canvas research assistant that searches the web, queries structured data via MCP, and embeds interactive charts directly into the report, while streaming progress to the frontend in real time.

We will use CopilotKit (Next.js) for the frontend, LangGraph to orchestrate the agent workflow, and MCP (Model Context Protocol) to connect external tools like Tako, which returns embeddable chart UI.

Check out CopilotKit's GitHub ⭐️

You will find architecture, key concepts, how state flows between the UI and the agent, how MCP Apps fit into the system, and a step-by-step guide to building this from scratch.

Let’s build it.

What is covered?

In summary, we will cover these topics in detail.

What are we building?
Tech Stack and Project Structure
Building the Frontend (CopilotKit)
Agent Orchestration (LangGraph)
Backend: MCP Integration & Tools
Running the Application
Complete Data Flow between UI ↔ agent

Here’s the GitHub Repo if you are interested in exploring it yourself.

1. What are we building?

We are building an AI research assistant with a live chat + canvas UI that searches, collects sources, and assembles a report while streaming progress to the frontend in real time.

It will:

Turn a user question into a research plan (research question + data sub-questions)
Run parallel retrieval: Tavily for web search + Tako via MCP apps for charts
Deduplicate results into a clean resources list
Generate a report with [CHART:title] placeholders and then embed the real charts inline
Stream intermediate state (logs/resources/report/charts) so the UI updates as the agent works.

We will see all of these concepts in action as we build the agent.

Core Components

A few days ago, the Anthropic team shipped MCP Apps as an official MCP extension, meaning tools can return interactive UI that the host can render, not just JSON or text.

That’s the pattern we will use here: our agent will call Tako (over MCP) to fetch charts as embeddable UI, so interactive visualizations can drop straight into the report/canvas instead of us building chart components.

At a high level, this project is built by combining:

CopilotKit for the agent chat + canvas UI
AG‑UI for streaming agent ↔ UI events (protocol)
LangGraph for stateful agent orchestration
MCP Apps for external tools
Tako for charts (search engine)
Tavily for web search

For context: AG-UI (Agent–User Interaction Protocol) is an event-based protocol that standardizes this “agent ↔ frontend” real-time communication.

Because it’s transport-agnostic (SSE, WebSockets, etc.), the same frontend can work across different agent backends without custom wiring. CopilotKit uses AG-UI as its synchronization layer.

Here's a simplified call request → response flow of what will happen:

User question
   ↓
CopilotKit UI (chat + canvas)
   ↓
LangGraph agent workflow
   ├─ chat: interpret intent + plan
   ├─ search: parallel retrieval (Tavily + Tako via MCP Apps)
   └─ report: narrative + [CHART:*] markers
   ↓
Streaming state updates (AG‑UI events)
   ↓
Canvas renders report + embedded charts

2. Tech Stack and Architecture

At the core, we are going to use this stack for building the agent:

Next.js 15 : frontend framework with TypeScript
CopilotKit 1.50 : agent UI components + streaming runtime (@copilotkit/react-core, @copilotkit/react-ui, @copilotkit/runtime)
LangGraph: stateful agent orchestration (StateGraph nodes, conditional routing)
FastAPI & Uvicorn : HTTP server for the agent
Tavily: web search API (tavily-python)
Model Context Protocol (MCP): connects the agent to Tako (mcp, httpx)
CopilotKit LangGraph SDK (Python): streams LangGraph state to CopilotKit (copilotkit)
OpenAI: LLM provider (openai)

There are also other libraries used, like react-markdown & remark-gfm for report rendering, langgraph-checkpoint-sqlite for state persistence, and Shadcn UI for components.

See package.json and agents/python/pyproject.toml in the repo for the complete dependency list.

Project structure

This is how our directory will look.

The agents/python/ directory hosts the Python LangGraph agent, exposed via FastAPI, which orchestrates the workflow (chat, search, report), streams state updates, and calls external tools (Tavily, Tako via MCP).

The src/ directory hosts the Next.js frontend, including the UI components, shared types, and the CopilotKit API route (/api/copilotkit/route.ts) that bridges the frontend to the agent backend.

.
├── src/                           ← Next.js frontend (TypeScript)
│   ├── app/
│   │   ├── page.tsx               ← CopilotKit provider & model selector
│   │   ├── Main.tsx               ← Chat + ResearchCanvas split layout
│   │   └── api/
│   │       └── copilotkit/
│   │           └── route.ts      ← CopilotRuntime bridge to agent
│   ├── components/
│   │   ├── ResearchCanvas.tsx    ← Main canvas (orchestrates report + resources)
│   │   ├── Resources.tsx         ← Displays Tavily + Tako resources list
│   │   ├── MarkdownRenderer.tsx  ← Renders report + embeds charts
│   │   └── ui/                   ← Reusable Shadcn UI components
│   └── lib/
│       ├── types.ts                     ← AgentState type
│       ├── utils.ts                     ← Utility functions
│       └── model-selector-provider.tsx  ← Model selection context
│
├── agents/
│   ├── python/                        ← Python LangGraph agent (primary)
│   │   ├── src/
│   │   │   ├── agent.py               ← StateGraph definition & compile
│   │   │   └── lib/
│   │   │       ├── state.py           ← AgentState (Pydantic)
│   │   │       ├── model.py       ← LLM factory (OpenAI / Anthropic / Gemini)
│   │   │       ├── chat.py            ← Chat node & tool definitions
│   │   │       ├── search.py          ← Parallel Tavily + Tako search
│   │   │       ├── mcp_integration.py ← MCP client & iframe helpers
│   │   │       ├── download.py        ← Download node
│   │   │       └── delete.py          ← Delete node
│   │   ├── main.py                    ← FastAPI/Uvicorn entrypoint
│   │   ├── requirements.txt
│   │   └── pyproject.toml
│   │
│   └── src/                       ← TypeScript agent (optional, local dev)
│       └── server.ts              ← Express + CopilotRuntime
│
├── package.json                   ← Frontend deps & scripts
├── .env.example
├── .env.local
└── README.md

Here's the GitHub repository and deployed live at tako-copilotkit.vercel.app if you want to explore yourself. I will cover the implementation and all key concepts in the following sections.

The easiest way to follow along is to clone the repo.

git clone https://github.com/TakoData/tako-copilotkit.git
cd tako-copilotkit

Add necessary API Keys

You can copy the environment template (.env.example) by using this command to create a .env.local in the root directory.

cp .env.example .env.local

Add your OpenAI API Key, Tavily API Key and Tako API Key to the file. I have attached the docs link so it's easy to follow.

OPENAI_API_KEY=sk-proj-...
TAVILY_API_KEY=tvly-dev-...
TAKO_API_TOKEN=your_api_token_here
TAKO_MCP_URL=https://mcp.tako.com  # URL of Tako's MCP server
TAKO_URL=https://tako.com  # URL of Tako's main API

The Tako/MCP integration is an optional data source for the research agent. If you want to query charts or structured datasets via a Tako MCP server, provide TAKO_API_TOKEN and the related URLs.

Otherwise, you can leave these unset & the agent will continue to work normally. For this tutorial, we will be using it.

OpenAI API Key

Tavily API Key

Tako API Key

3. Frontend: wiring the agent to the UI

Let's first build the frontend part.

The src/ directory hosts the Next.js frontend, including the UI components, shared types, and the CopilotKit API route (/api/copilotkit/route.ts) that bridges the frontend to the Python agent backend.

At a high level, the frontend is responsible for:

Sending user queries to the agent backend via CopilotChat
Receiving and rendering real-time state updates from the agent
Embedding Tako chart iframes in the rendered report
Managing the CopilotKit runtime bridge between UI and Python agent

If you are building this from scratch, the easiest approach is to copy the existing package.json from the repo.

It already includes all the required dependencies for CopilotKit, LangGraph integration, UI components, and local dev tooling, so you don’t have to assemble them manually.

I'm only covering the core frontend dependencies you actually need to understand.

Step 1: CopilotKit Provider & Layout

Install the necessary packages:

npm install @copilotkit/react-core @copilotkit/react-ui @copilotkit/runtime

@copilotkit/react-core: Core React hooks and context that connect your UI to the agent backend via AG-UI protocol
@copilotkit/react-ui: Ready-made UI components like <CopilotChat /> for building AI chat interfaces
@copilotkit/runtime: Server-side runtime that exposes an API endpoint and bridges the frontend with the LangGraph agent using HTTP and SSE

Everything else in the repo (radix, tailwind, react-split, etc.) is there to support layout, styling, and developer experience -- not the core agent wiring.

The <CopilotKit> component must wrap the agent-aware parts of your application and point to the runtime endpoint so it can communicate with the agent backend.

This is the main entry point (page.tsx):

// page.tsx
"use client";

import { CopilotKit } from "@copilotkit/react-core";
import Main from "./Main";
import {
  ModelSelectorProvider,
  useModelSelectorContext,
} from "@/lib/model-selector-provider";

export default function ModelSelectorWrapper() {
  return (
    <ModelSelectorProvider>
      <Home />
    </ModelSelectorProvider>
  );
}

function Home() {
  const { agent, lgcDeploymentUrl } = useModelSelectorContext();

  // This logic is implemented to demonstrate multi-agent frameworks in this demo project.
  // There are cleaner ways to handle this in a production environment.
  const runtimeUrl = lgcDeploymentUrl
    ? `/api/copilotkit?lgcDeploymentUrl=${lgcDeploymentUrl}`
    : `/api/copilotkit${
        agent.includes("crewai") ? "?coAgentsModel=crewai" : ""
      }`;

  return (
    <div style={{ height: "100vh", overflow: "hidden" }}>
      <CopilotKit runtimeUrl={runtimeUrl} showDevConsole={false} agent={agent}>
        <Main />
      </CopilotKit>
    </div>
  );
}

Here, ModelSelectorProvider is a convenience for switching agents/models in development.

Step 2: Chat + Canvas Layout

The Main.tsx component sets up the core UI. It sets up:

the chat interface (CopilotChat)
the research canvas (where results & charts appear)
state binding to the agent

// app/Main.tsx
import { useCoAgent } from "@copilotkit/react-core";
import { CopilotChat } from "@copilotkit/react-ui";
import Split from "react-split";
import { ResearchCanvas } from "@/components/ResearchCanvas";
import { ChatInputWithModelSelector } from "@/components/ChatInputWithModelSelector";
import { AgentState } from "@/lib/types";
import { useModelSelectorContext } from "@/lib/model-selector-provider";

export default function Main() {
  const { model, agent } = useModelSelectorContext();

  const { state, setState } = useCoAgent<AgentState>({
    name: agent,
    initialState: {
      model,
      research_question: "",
      resources: [],
      report: "",
      logs: [],
    },
  });

  return (
    <div style={{ height: "100%", display: "flex", flexDirection: "column" }}>
      <h1>Research Helper</h1>

      <Split sizes={[30, 70]} style={{ flex: 1, display: "flex" }}>
        {/* Chat Panel */}
        <CopilotChat
          Input={ChatInputWithModelSelector}
          onSubmitMessage={async () => {
            setState({ ...state, logs: [] });
            await new Promise((r) => setTimeout(r, 30));
          }}
        />

        {/* Canvas Panel */}
        <ResearchCanvas />
      </Split>
    </div>
  );
}

Here's what's happening:

useCoAgent<AgentState>: bi-directional state synchronization between UI and agent. When the agent emits state updates via copilotkit_emit_state(), this hook automatically picks them up.
CopilotChat: drop-in chat UI that sends messages to the agent and renders tool calls inline
ResearchCanvas: custom component that renders the streaming report, resources list, and embedded charts
Split: provides a resizable split-pane layout

Step 3: Building Key Components

I'm only covering the core logic behind the main components since the overall code is huge. You can find all the components in the repository at src\components.

These components use CopilotKit hooks (like useCoAgentStateRender) to tie everything together.

✅ Research Canvas Component

ResearchCanvas.tsx is where the agent’s current state is rendered:

the accumulating report text
linked resources
embedded charts (iframes from MCP Apps like Tako)

This component listens to the agent state and displays elements as they arrive. It translates [CHART:title] markers into real embedded charts. This pattern is part of CopilotKit's support for generative UI.

// core logic
import { useCoAgent, useCoAgentStateRender, useCopilotAction } from "@copilotkit/react-core";
import { MarkdownRenderer } from "./MarkdownRenderer";
import { AgentState } from "@/lib/types";
import { useRef, useEffect } from "react";

export function ResearchCanvas() {
  const { state, setState } = useCoAgent<AgentState>({
    name: "research_agent",
  });

  // Use refs to prevent flicker during streaming updates
  const lastReportRef = useRef<string>("");
  const lastResourcesRef = useRef<Resource[]>([]);

  if (state.report) lastReportRef.current = state.report;
  if (state.resources?.length) lastResourcesRef.current = state.resources;

  const report = state.report || lastReportRef.current;
  const resources = state.resources || lastResourcesRef.current;

  // Render progress logs during execution
  useCoAgentStateRender({
    name: "research_agent",
    render: ({ state, status }) => {
      if (state.logs?.length) return <Progress logs={state.logs} />;
      return null;
    },
  });

  // Generative UI: Agent requests deletion confirmation from user
  useCopilotAction({
    name: "DeleteResources",
    description: "Prompt user for resource delete confirmation",
    available: "remote",
    parameters: [{ name: "urls", type: "string[]" }],
    renderAndWait: ({ args, handler }) => (
      <div>
        <h3>Delete these resources?</h3>
        <Resources resources={resources.filter(r => args.urls.includes(r.url))} />
        <button onClick={() => handler("YES")}>Delete</button>
        <button onClick={() => handler("NO")}>Cancel</button>
      </div>
    ),
  });

  return (
    <div>
      <h2>Research Question</h2>
      <div>{state.research_question || "Agent will identify your question..."}</div>

      {/* Resources Panel */}
      <Resources resources={resources} />

      {/* Report Panel with Embedded Charts */}
      <MarkdownRenderer content={report} />
    </div>
  );
}

The full component also includes collapsible sections for Tako charts vs web resources, and resource management dialogs. Check src/components/ResearchCanvas.tsx.

✅ Chart Embedding Pattern

The MCP Apps-style embedding happens in MarkdownRenderer.tsx. It embeds Tako charts without flickering during streaming updates. Instead of simple regex replacement, it uses a stable iframe pattern.

// core logic
import React, { useEffect, useRef, useMemo } from "react";
import ReactMarkdown from "react-markdown";

// Global registry: iframe src never changes for a given ID
const iframeRegistry = new Map<string, string>();

export function MarkdownRenderer({ content }: { content: string }) {
  // Split content into text and iframe segments
  const segments = useMemo(() => {
    const embedPattern = /<!doctype html>[\s\S]*?<\/html>/gi;
    const parts = [];
    let lastIndex = 0;

    content.replace(embedPattern, (match, offset) => {
      // Add text before iframe
      if (offset > lastIndex) {
        parts.push({ type: 'text', content: content.slice(lastIndex, offset) });
      }

      // Extract iframe src and create stable ID
      const src = match.match(/src=["']([^"']+)["']/)?.;
      if (src) {
        const id = `embed-${src.replace(/[^a-zA-Z0-9]/g, '-').slice(0, 50)}`;
        iframeRegistry.set(id, src);
        parts.push({ type: 'iframe', id, src });
      }

      lastIndex = offset + match.length;
      return match;
    });

    // Add remaining text
    if (lastIndex < content.length) {
      parts.push({ type: 'text', content: content.slice(lastIndex) });
    }

    return parts;
  }, [content]);

  return (
    <div>
      {segments.map((segment, i) => 
        segment.type === 'text' 
          ? <ReactMarkdown key={i}>{segment.content}</ReactMarkdown>
          : <StableIframe key={segment.id} src={segment.src} />
      )}
    </div>
  );
}

// Memoized iframe - never re-renders once mounted
const StableIframe = React.memo(({ src }: { src: string }) => {
  const iframeRef = useRef<HTMLIFrameElement>(null);

  // Handle Tako resize messages
  useEffect(() => {
    const handleResize = (event: MessageEvent) => {
      if (event.data?.type === "tako::resize" && 
          iframeRef.current?.contentWindow === event.source) {
        iframeRef.current.style.height = `${event.data.height}px`;
      }
    };
    window.addEventListener("message", handleResize);
    return () => window.removeEventListener("message", handleResize);
  }, []);

  return <iframe ref={iframeRef} src={src} style={{ height: "400px" }} />;
});

Tako returns fully-formed chart UI via MCP Apps so the frontend only needs to embed the iframe. Here's what's happening:

Iframes are rendered as stable siblings (outside ReactMarkdown), preventing remounts during streaming updates
Each embed uses a deterministic ID derived from its src, ensuring consistent DOM identity
Resize events from Tako (tako::resize) are handled via postMessage to keep charts responsive
rehypeRaw: Allows ReactMarkdown to render raw HTML (the iframe elements)

Check out the complete code at src/components/MarkdownRenderer.tsx.

✅ Model Selector Provider

The ModelSelectorProvider (src\lib\model-selector-provider.tsx) is a convenience wrapper for switching between different agent configurations during development. It reads URL query parameters to determine which agent/model to use.

"use client";

import React from "react";
import { createContext, useContext, useState, ReactNode } from "react";

type ModelSelectorContextType = {
  model: string;
  setModel: (model: string) => void;
  agent: string;
  lgcDeploymentUrl?: string | null;
  hidden: boolean;
  setHidden: (hidden: boolean) => void;
};

const ModelSelectorContext = createContext<ModelSelectorContextType | undefined>(undefined);

export const ModelSelectorProvider = ({ children }: { children: ReactNode }) => {
  // Read model from URL query params (e.g., ?coAgentsModel=google_genai)
  const model =
    globalThis.window === undefined
      ? "openai"
      : new URL(window.location.href).searchParams.get("coAgentsModel") ?? "openai";

  const [hidden, setHidden] = useState<boolean>(false);

  const setModel = (model: string) => {
    const url = new URL(window.location.href);
    url.searchParams.set("coAgentsModel", model);
    window.location.href = url.toString();
  };

  // Optional: LangGraph Cloud deployment URL for production
  const lgcDeploymentUrl =
    globalThis.window === undefined
      ? null
      : new URL(window.location.href).searchParams.get("lgcDeploymentUrl");

  // Map model to agent name
  let agent = "research_agent";
  if (model === "google_genai") {
    agent = "research_agent_google_genai";
  } else if (model === "crewai") {
    agent = "research_agent_crewai";
  }

  return (
    <ModelSelectorContext.Provider
      value={{ model, agent, lgcDeploymentUrl, hidden, setModel, setHidden }}
    >
      {children}
    </ModelSelectorContext.Provider>
  );
};

export const useModelSelectorContext = () => {
  const context = useContext(ModelSelectorContext);
  if (context === undefined) {
    throw new Error("useModelSelectorContext must be used within a ModelSelectorProvider");
  }
  return context;
};

The repo also includes UI components like Resources.tsx (resource cards), Progress.tsx (logs display), EditResourceDialog.tsx and AddResourceDialog.tsx for resource management, plus ChatInputWithModelSelector for model switching during development.

Step 4: Runtime Bridge (API Route)

This Next.js API route acts as a thin proxy between the browser and the Python LangGraph agent. It:

Accepts CopilotKit requests from the UI
Forwards them to the LangGraph agent via HTTP
Streams agent state and events back to the frontend via SSE

Instead of letting the frontend talk to the FastAPI agent directly, all requests go through a single endpoint /api/copilotkit.

import {
  CopilotRuntime,
  copilotRuntimeNextJSAppRouterEndpoint,
  EmptyAdapter,
} from "@copilotkit/runtime";
import {
  LangGraphHttpAgent,
  LangGraphAgent,
} from "@copilotkit/runtime/langgraph";
import { NextRequest } from "next/server";

// const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
// const llmAdapter = new OpenAIAdapter({ openai } as any);
const llmAdapter = new EmptyAdapter();
const langsmithApiKey = process.env.LANGSMITH_API_KEY as string;

export const POST = async (req: NextRequest) => {
  const searchParams = req.nextUrl.searchParams;
  const deploymentUrl =
    searchParams.get("lgcDeploymentUrl") || process.env.LGC_DEPLOYMENT_URL;

  const baseUrl =
    process.env.REMOTE_ACTION_URL || "http://localhost:2024/copilotkit";

  let runtime = new CopilotRuntime({
    agents: {
      research_agent: new LangGraphHttpAgent({
        url: `${baseUrl}/agents/research_agent`,
      }),
      research_agent_google_genai: new LangGraphHttpAgent({
        url: `${baseUrl}/agents/research_agent_google_genai`,
      }),
    },
  });

  if (deploymentUrl) {
    runtime = new CopilotRuntime({
      agents: {
        research_agent: new LangGraphAgent({
          deploymentUrl,
          langsmithApiKey,
          graphId: "research_agent",
        }),
        research_agent_google_genai: new LangGraphAgent({
          deploymentUrl,
          langsmithApiKey,
          graphId: "research_agent_google_genai",
        }),
      },
    });
  }

  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter: llmAdapter,
    endpoint: "/api/copilotkit",
  });

  return handleRequest(req);
};

The frontend is fully reactive: as the agent works, the UI updates in real time.

4. Agent Orchestration (LangGraph)

Now that the frontend can communicate with an agent, we need to define how the agent thinks and operates. The backend agent lives in agents/python/src/.

It uses LangGraph to orchestrate a multi-step research workflow (chat → search → download → report...) through well-defined steps (nodes), each responsible for a specific task.

Each step:

receives the current AgentState
performs one responsibility
returns partial state updates
decides what node runs next

Step 1: Define the StateGraph (agent.py)

The workflow is defined using LangGraph's StateGraph in agent.py. It models the research assistant as a stateful workflow, where each step (node) operates on shared state and decides which step to run next.

At a high level:

AgentState is the shared memory across the entire agent (messages, report, resources, logs, ...)
Each node is an async function that:
- reads from the current state
- updates part of that state
- returns control-flow instructions
Edges control execution flow between nodes (download → chat → search → download loop)

One important detail is the interrupt_after hook, which allows the agent to pause execution for user confirmation before deleting a resource (delete_node).

# agents/python/src/agent.py
import os

from langgraph.graph import StateGraph

from src.lib.chat import chat_node
from src.lib.delete import delete_node, perform_delete_node
from src.lib.download import download_node
from src.lib.search import search_node
from src.lib.state import AgentState

workflow = StateGraph(AgentState)
workflow.add_node("download", download_node)
workflow.add_node("chat_node", chat_node)
workflow.add_node("search_node", search_node)
workflow.add_node("delete_node", delete_node)
workflow.add_node("perform_delete_node", perform_delete_node)


workflow.set_entry_point("download")
workflow.add_edge("download", "chat_node")
workflow.add_edge("delete_node", "perform_delete_node")
workflow.add_edge("perform_delete_node", "chat_node")
workflow.add_edge("search_node", "download")

# Compile with interrupt support for deletion confirmation
compile_kwargs = {"interrupt_after": ["delete_node"]}

# Check if we're running in LangGraph API mode
if os.environ.get("LANGGRAPH_FASTAPI", "false").lower() == "false":
    # When running in LangGraph API, don't use a custom checkpointer
    graph = workflow.compile(**compile_kwargs)
else:
    # For CopilotKit and other contexts, use MemorySaver
    from langgraph.checkpoint.memory import MemorySaver

    memory = MemorySaver()
    compile_kwargs["checkpointer"] = memory
    graph = workflow.compile(**compile_kwargs)

Why this structure works well:

The download → chat → search → download loop naturally supports iterative research
Optional checkpointing (MemorySaver) allows the same graph to run locally or on LangGraph Cloud (state persistence)

Step 2: Chat Node & Tools (chat.py)

The chat node (chat_node) is the brain of the agent. It orchestrates the LLM, defines tools, manages state streaming, and handles chart injection. It:

Define LLM-callable tools (Search, WriteReport, WriteResearchQuestion, GenerateDataQuestions)
Configure intermediate state emission (stream research question and data questions as they are generated)
Build prompts with available resources and Tako chart context
Inject Tako chart iframes into the report after generation
Route to next node based on tool calls

Here are the core tools (agents/python/src/lib/chat.py).

@tool
def Search(queries: List[str]):
    """A list of one or more search queries to find good resources."""

@tool
def WriteReport(report: str):
    """Write the research report."""

@tool
def WriteResearchQuestion(research_question: str):
    """Write the research question."""

@tool
def GenerateDataQuestions(questions: List[DataQuestion]):
    """
    Generate 3-6 ATOMIC data-focused questions for Tako search.

    CRITICAL: Each query should ask for ONE metric/dimension only.
    BAD: "Compare US GDP and inflation" → compound query
    GOOD: "US GDP 2020-2024", "US inflation rate" → separate atomic queries
    """

Here is the chat node logic.

async def chat_node(state: AgentState, config: RunnableConfig):
    # Configure intermediate state emission
    config = copilotkit_customize_config(
        config,
        emit_intermediate_state=[
            {"state_key": "research_question", "tool": "WriteResearchQuestion", "tool_argument": "research_question"},
            {"state_key": "data_questions", "tool": "GenerateDataQuestions", "tool_argument": "questions"},
        ],
    )

    # Build Tako charts context for LLM
    tako_charts_map = {}
    available_tako_charts = []
    for resource in state["resources"]:
        if resource.get("resource_type") == "tako_chart":
            title = resource.get("title", "")
            card_id = resource.get("card_id")
            description = resource.get("description", "")
            tako_charts_map[title] = {"card_id": card_id, "embed_url": resource.get("embed_url")}
            available_tako_charts.append(f"  - **{title}**\n    Description: {description}")

    # Build prompt with workflow instructions
    response = await model.bind_tools([Search, WriteReport, WriteResearchQuestion, GenerateDataQuestions]).ainvoke([
        SystemMessage(content=f"""
        You are a research assistant. 

        WORKFLOW:
        1. Use WriteResearchQuestion to extract the core research question
        2. Use GenerateDataQuestions to create 3-6 ATOMIC data questions (one metric per query)
        3. Use Search for web resources
        4. Write a comprehensive report using WriteReport

        AVAILABLE CHARTS ({len(tako_charts_map)}):
        {available_tako_charts_str}

        WRITING GUIDELINES:
        - Write a detailed analysis with substantial narrative text
        - For EACH chart, write 1-2 paragraphs discussing insights and trends
        - DO NOT include chart markers or image syntax - charts will be inserted automatically
        - Reference specific data points from chart descriptions
        """),
        *state["messages"],
    ], config)

    # Handle tool calls
    if response.tool_calls:
        if response.tool_calls[0]["name"] == "WriteReport":
            report = response.tool_calls[0]["args"]["report"]

            # Clean up any markdown images LLM incorrectly added
            report = re.sub(r'!\[[^\]]*\]\([^)]+\)', '', report)

            # Inject charts using second LLM pass
            if tako_charts_map:
                chart_list = "\n".join([f"- {title}" for title in tako_charts_map.keys()])

                inject_response = await model.ainvoke([
                    SystemMessage(content=f"""Insert [CHART:exact_title] markers at appropriate positions.

                    AVAILABLE CHARTS:
                    {chart_list}

                    RULES:
                    - Place markers AFTER relevant paragraphs
                    - NEVER place more than two charts consecutively
                    - Each chart is used exactly once
                    """),
                    HumanMessage(content=f"Insert chart markers:\n\n{report}")
                ])

                report_with_markers = inject_response.content

                # Replace markers with actual iframe HTML
                async def replace_marker(match):
                    chart_title = match.group(1).strip()
                    chart_info = tako_charts_map.get(chart_title)
                    if chart_info:
                        iframe_html = await get_visualization_iframe(
                            item_id=chart_info.get("card_id"),
                            embed_url=chart_info.get("embed_url")
                        )
                        return "\n" + iframe_html + "\n" if iframe_html else ""
                    return ""

                # Apply replacements
                markers = list(re.finditer(r'\[CHART:([^\]]+)\]', report_with_markers))
                processed_report = report_with_markers
                for match in reversed(markers):
                    replacement = await replace_marker(match)
                    processed_report = processed_report[:match.start()] + replacement + processed_report[match.end():]

            return Command(goto="chat_node", update={"report": processed_report})

        elif response.tool_calls[0]["name"] == "GenerateDataQuestions":
            data_questions = response.tool_calls[0]["args"]["questions"]
            return Command(goto="search_node", update={"data_questions": data_questions})

    # Route based on tool call
    goto = "__end__"
    if response.tool_calls:
        tool_name = response.tool_calls[0]["name"]
        if tool_name == "Search":
            goto = "search_node"
        elif tool_name == "DeleteResources":
            goto = "delete_node"

    return Command(goto=goto, update={"messages": response})

How this node works in practice:

The LLM never writes free-form text directly. Every meaningful action goes through an explicit tool (WriteResearchQuestion, GenerateDataQuestions, Search, WriteReport). This keeps state changes deterministic and easy to reason about.
Intermediate state (research question and data questions) is streamed immediately using copilotkit_customize_config. The report itself is intentionally not streamed to avoid UI flicker while charts are being injected later.
Chart handling is deliberately two-phase: the model first writes a clean narrative report, then a second pass inserts [CHART:title] markers, which are finally replaced with real Tako iframes. To keep the generation stable and layout-safe.
Control flow is explicit. Each tool call results in a Command that routes execution to the next node (search_node, delete_node, or back to chat_node), making the agent’s behavior predictable and easier to debug.

Step 3: Search Node (search.py)

The search node (search_node) runs web search (Tavily) and structured chart data search (Tako via MCP) in parallel, streams progress to the UI, deduplicates results, and extracts the most relevant resources.

Tavily and Tako searches run concurrently using asyncio.gather, so web pages and structured datasets are fetched at the same time
TavilyClient is synchronous (uses requests library), so we wrap it with loop.run_in_executor() to avoid blocking the async event loop
As each search completes, copilotkit_emit_state is called to update state.logs, allowing the frontend to show real-time progress
Tako chart results are deduplicated using the title to avoid embedding the same visualization multiple times
Web results and chart results are passed through an LLM tool (ExtractResources) to select only the most relevant resources
Each resource is tagged as web or tako_chart, which the frontend uses to decide whether to render a link or an embedded chart

Here is the code for Tavily async wrapper (bridging sync client with async workflow):

# agents/python/src/lib/search.py

from tavily import TavilyClient
import asyncio

_tavily_client = None

def get_tavily_client():
    """Lazy initialization to avoid repeated client creation"""
    global _tavily_client
    if _tavily_client is None:
        _tavily_client = TavilyClient(api_key=os.getenv("TAVILY_API_KEY"))
    return _tavily_client

async def async_tavily_search(query: str) -> Dict[str, Any]:
    """
    Async wrapper for synchronous Tavily client.
    TavilyClient.search() is blocking, so we run it in a thread pool.
    """
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(
        None,
        lambda: get_tavily_client().search(
            query=query,
            search_depth="advanced",
            include_answer=True,
            max_results=5,
        ),
    )

Search node implementation (parallel execution and resource extraction):

# agents/python/src/lib/search.py (core search pattern)
async def search_node(state: AgentState, config: RunnableConfig):
    """
    Perform Tavily web search and Tako MCP chart search in parallel.
    Stream progress, dedupe results, extract top resources.
    """
    # Extract queries from tool_calls or fall back to research_question
    # (Actual code handles dual routing: Search tool OR GenerateDataQuestions)
    queries = state.get("search_queries", [])
    data_questions = state.get("data_questions", [])

    # PHASE 1: Build parallel tasks
    tavily_tasks = [async_tavily_search(query) for query in queries]
    tako_tasks = [
        search_knowledge_base(q["question"], search_effort="fast")
        for q in data_questions
    ]
    all_tasks = tavily_tasks + tako_tasks

    # Initialize logs for progress tracking
    for query in queries:
        state["logs"].append({"message": f"Web search: {query}", "done": False})
    for q in data_questions:
        state["logs"].append({"message": f"Tako search: {q['question']}", "done": False})

    if all_tasks:
        await copilotkit_emit_state(config, state)

    # PHASE 2: Run all searches concurrently
    all_results = await asyncio.gather(*all_tasks, return_exceptions=True)

    # Split results back into Tavily and Tako
    num_tavily = len(tavily_tasks)
    tavily_results = all_results[:num_tavily]
    tako_results_raw = all_results[num_tavily:]

    # PHASE 3: Stream completion and process results
    search_results = []
    tako_results = []

    # Process Tavily results
    for i, result in enumerate(tavily_results):
        if isinstance(result, Exception):
            search_results.append({"error": str(result)})
        else:
            search_results.append(result)
        state["logs"][i]["done"] = True
        await copilotkit_emit_state(config, state)

    # Process Tako results
    tako_log_offset = num_tavily
    for i, result in enumerate(tako_results_raw):
        if isinstance(result, Exception):
            tako_results.append({"error": str(result)})
        elif result:
            tako_results.extend(result)
        state["logs"][tako_log_offset + i]["done"] = True
        await copilotkit_emit_state(config, state)

    # PHASE 4: Deduplicate Tako charts by title
    # (Same chart may appear in multiple searches)
    seen_titles = set()
    unique_tako = []
    for chart in tako_results:
        title = chart.get("title", "")
        if title and title not in seen_titles:
            seen_titles.add(title)
            unique_tako.append(chart)
        elif not title:
            unique_tako.append(chart)

    # PHASE 5: Use LLM to extract the most relevant resources
    model = get_model(state)

    # Prepare search message with both web and Tako results
    search_message = f"Web search results: {search_results}"
    if unique_tako:
        search_message += f"\n\nTako chart results: {unique_tako}"

    # Build extraction prompt
    extract_messages = [
        SystemMessage(content="""
        Extract the 3-5 most relevant resources from the search results.
        Prioritize Tako charts when they match the research topic.
        """),
        *state["messages"],
        SystemMessage(content=f"Search results:\n{search_message}")
    ]

    # Call LLM with ExtractResources tool
    response = await model.bind_tools(
        [ExtractResources], 
        tool_choice="ExtractResources"
    ).ainvoke(extract_messages, config)

    # PHASE 6: Tag resources with type and attach content
    resources = response.tool_calls[0]["args"]["resources"]

    for resource in resources:
        # Check if this is a Tako chart by matching URL or title
        is_tako = False
        for tako_result in unique_tako:
            if (resource.get("url") == tako_result.get("url") or 
                resource.get("title") == tako_result.get("title")):
                is_tako = True
                resource["resource_type"] = "tako_chart"
                resource["source"] = "Tako"
                resource["card_id"] = tako_result.get("id")
                resource["embed_url"] = tako_result.get("embed_url")
                resource["content"] = tako_result.get("description", "")
                break

        if not is_tako:
            resource["resource_type"] = "web"
            resource["source"] = "Tavily Web Search"
            # Find matching Tavily result for content summary
            for search_result in search_results:
                if isinstance(search_result, dict) and "results" in search_result:
                    for tavily_item in search_result["results"]:
                        if tavily_item.get("url") == resource.get("url"):
                            resource["content"] = tavily_item.get("content", "")
                            break

    # Deduplicate and add to state resources
    # (Production code enforces MAX_TOTAL_RESOURCES limit and generates iframe HTML)
    state["resources"].extend(resources)

    return state

This simplified version shows the core search pattern.

The production code adds fallback searches when Tako returns no results, generates iframe HTML for chart embedding via get_visualization_iframe(), enforces resource limits, and handles dual routing for different tool paths.

Check out the complete implementation at agents/python/src/lib/search.py.

5. Backend: MCP Integration & Tools

At this point, the UI is wired, the agent graph is defined, and the execution flow is clear. What’s left is the backend work that actually fetches data, talks to external systems, and turns results into something the agent can reason about.

The most interesting part here is the MCP integration for Tako charts. The rest of the backend nodes are intentionally simple.

MCP Apps Pattern for Tako Charts

Tako is a data visualization platform with searchable charts on economics, markets, demographics, and more.

What Tako provides:

Searchable chart catalog via knowledge_search tool
Chart search with configurable effort: "fast" (quick lookups), "medium", or "deep" (thorough searches)
Ready-to-embed iframe HTML via open_chart_ui tool
Auto-resizing charts via postMessage protocol

This is the core idea behind MCP Apps: tools return UI components, not just data.

This module mcp_integration.py is where the agent connects to Tako via the Model Context Protocol (MCP). I have broken the core parts of this integration, so it's easier to follow along.

The MCP Client: Session Management

The SimpleMCPClient class handles the Model Context Protocol lifecycle:

SSE connection establishes and maintains the session ID
HTTP messages send JSON-RPC tool calls to the server
Automatic reconnection on session expiry (404/410 errors)

# core client logic -- agents/python/src/lib/mcp_integration.py

class SimpleMCPClient:
    """
    Minimal MCP client following the Model Context Protocol spec.
    Handles connection lifecycle, session management, and message passing.
    """

    async def connect(self):
        """Connect to MCP server via SSE to get session ID."""
        self._sse_task = asyncio.create_task(self._sse_reader())

        # Wait for session_id establishment
        for _ in range(50):
            if self.session_id:
                return True
            await asyncio.sleep(0.1)
        return False

    async def reconnect(self):
        """Reconnect with new session on expiry."""
        if self._sse_task:
            self._sse_task.cancel()

        # Clear state and reconnect
        self.session_id = None
        self._responses.clear()
        await self.connect()
        await self.initialize()

    async def _send(self, method: str, params: dict = None, _retry: bool = True):
        """
        Send JSON-RPC message to server via HTTP.
        Automatically reconnects and retries once on session expiration (410/404).
        """
        msg = {"jsonrpc": "2.0", "id": self.message_id, "method": method}
        if params:
            msg["params"] = params

        future = asyncio.get_event_loop().create_future()
        self._responses[self.message_id] = future

        try:
            resp = await self._client.post(
                f"{self.base_url}/messages/?session_id={self.session_id}",
                json=msg,
            )

            # Handle session expiration
            if resp.status_code in (404, 410):
                if _retry:
                    await self.reconnect()
                    return await self._send(method, params, _retry=False)
                raise SessionExpiredException("Session expired")

        except httpx.HTTPStatusError as e:
            if e.response.status_code in (404, 410) and _retry:
                await self.reconnect()
                return await self._send(method, params, _retry=False)
            raise

        return await asyncio.wait_for(future, timeout=120.0)

# Global client instance (reused across all calls)
_mcp_client: Optional[SimpleMCPClient] = None

A single global client instance maintains the session across all tool calls. If a session expires mid-conversation, the client automatically reconnects and retries once.

Searching Tako's Knowledge Base

The search_knowledge_base function wraps the MCP knowledge_search tool to find relevant charts.

Each result becomes a Resource with resource_type = "tako_chart". The search_node runs these queries in parallel with Tavily web searches using asyncio.gather() for speed (as we discussed in the previous section).

async def search_knowledge_base(
    query: str,
    count: int = 5,
    search_effort: str = "fast"
) -> List[Dict[str, Any]]:
    """
    Search Tako knowledge base via MCP 'knowledge_search' tool.

    Returns a list of charts with metadata (id, title, description, embed_url).
    """
    result = await _call_mcp_tool("knowledge_search", {
        "query": query,
        "api_token": TAKO_API_TOKEN,
        "count": count,
        "search_effort": search_effort
    })

    if result and "results" in result:
        formatted_results = []
        for item in result["results"]:
            item_id = item.get("card_id") or item.get("id")
            title = item.get("title", "")
            description = item.get("description", "")
            embed_url = f"{DATA_SOURCE_URL}/embed/{item_id}/?theme=dark" if item_id else None

            formatted_results.append({
                "type": "data_visualization",
                "id": item_id,
                "title": title,
                "description": description,
                "embed_url": embed_url,
                "url": f"{DATA_SOURCE_URL}/card/{item_id}"
            })

        return formatted_results

    return []

Fetching Embeddable Chart UI

The get_visualization_iframe function is where MCP Apps shines. Instead of returning chart data, it returns complete iframe HTML.

There are two paths involved here:

MCP UI (open_chart_ui): Returns complete iframe HTML with interactive controls and resize logic
Fallback embed URL: Generates basic iframe HTML manually if the MCP call fails

The returned HTML is injected directly into the markdown report. The frontend's MarkdownRenderer extracts these iframe blocks and renders them as siblings to the markdown content (preventing flickering during streaming updates).

async def get_visualization_iframe(item_id: str = None, embed_url: str = None) -> Optional[str]:
    """
    Get embeddable iframe HTML for a Tako chart.
    Uses the MCP 'open_chart_ui' tool with automatic session reconnection.

    Returns ready-to-embed iframe HTML with resize script.
    """
    if item_id:
        result = await _call_mcp_tool("open_chart_ui", {
            "pub_id": item_id,
            "dark_mode": True,
            "width": 900,
            "height": 600
        })

        # Extract HTML from MCP response
        if result and result.get("type") == "resource":
            resource = result.get("resource", {})
            html_content = resource.get("htmlString") or resource.get("text")
            if html_content:
                return html_content

    # Fallback: Generate iframe HTML with embed_url
    if embed_url:
        return f'''<iframe
  width="100%"
  height="600"
  src="{embed_url}"
  scrolling="no"
  frameborder="0"
></iframe>

<script type="text/javascript">
window.addEventListener("message", function(e) {{
  if (e.data.type !== "tako::resize") return;

  for (let iframe of document.querySelectorAll("iframe")) {{
    if (iframe.contentWindow === e.source) {{
      iframe.style.height = e.data.height + "px";
    }}
  }}
}});
</script>'''

    return None

This is how MCP Apps actually works. The agent writes [CHART:title] markers in the report, and the backend replaces them with full iframe HTML from Tako.

The frontend never needs to know how to render Tako charts; it just embeds the iframe. Check out the complete code at agents/python/src/lib/mcp_integration.py.

Other backend nodes (briefly)

The repository includes a few more helper nodes, but they follow standard patterns and don’t introduce new concepts, so I have left them out to avoid unnecessary length.

download_node: Downloads web resource content and caches it (truncated to 3000 chars to reduce context bloat)
delete_node: handles resource deletion with user confirmation (interrupt pattern for generative UI)
state.py: defines TypedDict schemas for AgentState, Resource, DataQuestion, and Log
model.py: function to get LLM instance (OpenAI/Anthropic/Google) based on env config

6. Running the Application

After completing all the parts of the code, it's time to run it locally. Please make sure you have added the credentials to the .env in the root directory.

Install all dependencies (frontend + Python agent) by running this command from the root. This installs the Next.js frontend packages and also runs uv sync in agents/python to install Python dependencies.

npm install

This repository runs the frontend and backend together in development mode. Just run the following command from the root:

npm run dev

Behind the scenes, this is wired up using concurrently in package.json:

"dev": "concurrently -k -n ui,agent -c blue,red \"PORT=3000 npm run dev:ui\" \"PORT=2024 npm run dev:agent\"",
"dev:ui": "next dev",
"dev:agent": "cd agents/python && uv run main.py"

dev:ui → runs Next.js frontend on port 3000
dev:agent → runs Python FastAPI agent backend on port 2024 (using uv run)
concurrently runs both in parallel with colored output (blue for UI, red for agent)

You can navigate to http://localhost:3000 in your browser to view it locally.

Here is the demo!

6. Data flow

Now that we've built the frontend (Next.js + CopilotKit), the agent orchestration (LangGraph) and the backend integrations (MCP + Tavily), this is how data actually flows between them.

This should be easy to follow if you have been building along so far.

[User enters research question]
      ↓
Next.js UI (CopilotKit Provider + CopilotChat + ResearchCanvas)
        ↓
POST /api/copilotkit (CopilotKit runtime endpoint)
        ↓
CopilotRuntime → Python Agent (LangGraph)
        ↓
LangGraph StateGraph orchestration
   ├─ chat_node: interpret intent, decide tools 
   │   └─ (generates report with [CHART:title] markers, then replaces with iframe HTML)
   ├─ search_node: parallel gather + dedupe/normalize resources
   │      ├─ Tavily search (web results)
   │      └─ Tako MCP search (returns chart metadata)
   └─ download_node: cache web resource content
        ↓
copilotkit_emit_state(): stream intermediate AgentState with report containing iframe HTML (SSE)
        ↓
Frontend receives state updates (resources/report with embedded iframes)
        ↓
MarkdownRenderer extracts <!doctype html>...</html> iframe blocks and renders as siblings to markdown
        ↓
User sees: streaming report + resources list + interactive embedded charts

That’s it! 🎉

You now have a research assistant that streams agent state into a chat + canvas UI, blends web search with MCP-powered charts, and produces an interactive report.

What makes this work is the structure: clear agent state, explicit tool boundaries, and a UI that renders progress instead of hiding it. Once those pieces are in place, adding new MCP tools or constraints becomes an implementation detail.

I hope you learned something valuable. Have a great day!

The Developer's Guide to Generative UI in 2026

Anmol Baranwal — Thu, 29 Jan 2026 14:41:12 +0000

AI agents have become much better at reasoning and planning. The UI layer has mostly stayed the same, and it is holding back the experience.

Most agent experiences still rely on chat, even when the task clearly needs forms, previews, controls, or step-by-step feedback.

Generative UI is the idea that allows agents to influence the interface at runtime, so the UI can change as context changes. This is usually done through UI specs like A2UI, Open-JSON-UI, or MCP Apps.

Check out CopilotKit's GitHub ⭐️

Today, we will break down Generative UI, the three practical patterns, and how CopilotKit supports them (using AG-UI protocol under the hood).

Let's jump in.

What is covered?

In summary, we will cover these topics in detail.

What is Generative UI, and why does it matter?
Three Patterns of Generative UI
- Static Generative UI (AG-UI)
- Declarative Generative UI (A2UI and Open-JSON-UI)
- Open-ended Generative UI (MCP Apps)
How AG-UI enables Generative UI at runtime

Here’s the GitHub Repo for the live playground that implements all three approaches.

1. What is Generative UI?

Generative UI (sometimes called GenUI) is a pattern in which parts of the user interface are generated, selected, or controlled by an AI agent at runtime rather than being fully predefined by developers.

Instead of hardcoding every button, for,m or panel upfront, the UI is created dynamically based on the user's needs and context. The agent can decide what UI to show, what inputs it needs from the user, and how the state should update as the task progresses.

Instead of only generating text, the agent can:

request structured inputs
render task-specific UI
update the interface as tools run and state changes

This turns the UI into an active part of the agent’s execution rather than a static wrapper around a chat box. The frontend no longer has to infer intent from text.

Generative UI = LLM output → live, interactive UI

Examples:

A travel query → itinerary cards + map + expandable sections
A comparison query → sortable tables

In the CopilotKit ecosystem, Generative UI is not a single abstraction. It is a set of patterns built on top of agentic UI specifications (such as A2UI and MCP Apps) that define how agents communicate UI updates to applications. You can read more on the website and refer to the docs.

Google describes Generative UI as a way to give models a rich, custom, and interactive user experience that adapts to any prompt. If you are interested in reading more, you can check out their research blog.

A high-level system overview of the generative UI implementation (from Google Research).

Source: Google Research

Why does it matter?

Most agent experiences today still look like chat apps. When agents move beyond chatting and start doing things, text alone becomes a bottleneck.

Common problems with text-only agents:

tool execution and progress are hidden behind chat messages
user inputs are vague, easy to misinterpret, and hard to validate
multi-step flows feel opaque, so users do not trust the outcome

With Generative UI:

render task-specific UI exactly when it is needed
collect structured, validated input
show progress and intermediate results as real UI
adapt the interface as plans evolve

This makes agent systems easier to understand and guide. Instead of interacting with a black box, users interact with a system that exposes its state through UI.

2. Three Patterns of Generative UI

Most Generative UI implementations fit into three patterns. What separates them is how much control stays in the frontend versus how much freedom is given to the agent.

Here are the three types (along with example specs):

1) Static Generative UI (high control, low freedom) - streamed via AG-UI

The frontend owns the UI. The agent only selects which predefined component to show and fills it with data.

2) Declarative Generative UI (shared control) - using A2UI, Open-JSON-UI

The agent returns a structured UI spec (cards, lists, forms). The frontend renders it with its own constraints and styling.

3) Open-ended Generative UI (low control, high freedom) - enabled by MCP Apps

The agent returns a full UI surface (often embedded or free-form). The frontend mainly hosts it, so you trade consistency and safety for flexibility.

If you want to see these patterns side by side, there’s a live playground showing all three, along with a code implementation:

Live playground: go.copilotkit.ai/gen-ui-demo
Code: github.com/CopilotKit/generative-ui-playground

Here is a short demo!

You can check the docs and pick the integration you are using to see detailed implementation guides and examples.

Now, let's cover each pattern in depth and see how to implement them.

✅ Static Generative UI (AG-UI)

Static Generative UI is the most constrained pattern.

Developers pre-build UI components and the agent’s role is limited to deciding when a component appears and what data it receives. The layout and interactions are fully owned by the application.

In CopilotKit, this pattern is implemented using the useFrontendTool hook, which binds predefined UI components to the lifecycle of an action.

// Weather tool - a callable tool that displays weather data in a styled card
useFrontendTool({
  name: "get_weather",
  description: "Get current weather information for a location",
  parameters: z.object({ location: z.string().describe("The city or location to get weather for") }),
  handler: async ({ location }) => {
    await new Promise((r) => setTimeout(r, 500));
    return getMockWeather(location);
  },
  render: ({ status, args, result }) => {
    if (status === "inProgress" || status === "executing") {
      return <WeatherLoadingState location={args?.location} />;
    }
    if (status === "complete" && result) {
      const data = JSON.parse(result) as WeatherData;
      return (
        <WeatherCard
          location={data.location}
          temperature={data.temperature}
          conditions={data.conditions}
          humidity={data.humidity}
          windSpeed={data.windSpeed}
        />
      );
    }
    return <></>;
  },
});

Here, the useFrontendTool hook lets the application register the get_weather tool and attach a predefined React render to each phase of its lifecycle.

Output:

The agent can invoke the tool and stream arguments, but it never controls layout or invents UI.

You can check the complete implementation at github.com/CopilotKit/generative-ui-playground.

✅ Declarative Generative UI (A2UI and Open-JSON-UI)

Declarative Generative UI sits between static and fully open-ended approaches.

Instead of selecting predefined components, the agent returns a structured UI description. This description defines things like cards, lists, forms, or widgets, and the frontend renders them.

Two common declarative specifications used here are:

1) A2UI → declarative Generative UI spec from Google, described as JSONL-based, and agents can use it to return UI widgets as part of their responses

2) Open‑JSON‑UI → open standardization of OpenAI’s internal declarative Generative UI schema

Both define structured, platform-agnostic ways for agents to describe UI in JSON.

Let's first understand the basic flow for implementing A2UI.

Instead of writing A2UI JSON by hand, you can use the A2UI Composer to generate the spec for you. Copy the output and paste it into your agent’s prompt as a reference template.

In your prompt_builder.py, pick one representative A2UI template (e.g., a simple form or list). This is what the agent learns to emit:

UI_EXAMPLES = """
---BEGIN FORM_EXAMPLE---
[
  { "beginRendering": { "surfaceId": "form-surface", "root": "form-column",
                        "styles": { "primaryColor": "#9B8AFF", "font": "Plus Jakarta Sans" } } },
  { "surfaceUpdate": {
      "surfaceId": "form-surface",
      "components": [
        { "id": "form-column", "component": { "Column": {
            "children": { "explicitList": ["form-title", "name-field", "submit-button"] }
        } } },
        { "id": "form-title", "component": { "Text": {
            "usageHint": "h2", "text": { "literalString": "Contact Us" }
        } } },
        { "id": "name-field", "component": { "TextField": {
            "label": { "literalString": "Your Name" },
            "text": { "path": "name" },
            "textFieldType": "shortText"
        } } },
        { "id": "submit-button", "component": { "Button": {
            "child": "submit-text", "primary": true,
            "action": { "name": "submit_form", "context": [
              { "key": "name", "value": { "path": "name" } }
            ] }
        } } },
        { "id": "submit-text", "component": { "Text": {
            "text": { "literalString": "Send Message" }
        } } }
      ]
    }
  },
  { "dataModelUpdate": {
      "surfaceId": "form-surface", "path": "/", "contents": [
        { "key": "name", "valueString": "" }
      ]
    }
  }
]
---END FORM_EXAMPLE---
"""

This single JSONL‑formatted block defines a “Contact Us” form in A2UI. You feed it to the agent so it learns the three message envelopes A2UI expects: surfaceUpdate (components), dataModelUpdate (state), then beginRendering (render signal).

Then add UI_EXAMPLES into the system prompt that the agent uses:

if use_ui:
    instruction = AGENT_INSTRUCTION + get_ui_prompt(self.base_url, UI_EXAMPLES)
else:
    instruction = get_text_prompt()

return LlmAgent(
    model=LiteLlm(model=LITELLM_MODEL),
    name="ui_generator_agent",
    description="Generates dynamic UI via A2UI declarative JSON.",
    instruction=instruction,
    tools=[],
)

By appending UI_EXAMPLES (and the A2UI schema) to your agent’s instruction, you ensure it streams back valid JSON based on your templates whenever the user asks for a UI.

The final step is to use createA2UIMessageRenderer, which takes the agent’s A2UI JSON and your theme, and then renderActivityMessages tells CopilotKit to use it when rendering chat messages.

import { CopilotKitProvider, CopilotSidebar } from "@copilotkitnext/react";
import { createA2UIMessageRenderer } from "@copilotkit/a2ui-renderer";
import { a2uiTheme } from "../theme";

// Instantiate the A2UI renderer once with your theme
const A2UIRenderer = createA2UIMessageRenderer({ theme: a2uiTheme });

export function A2UIPage({ children }: { children: React.ReactNode }) {
  return (
    <CopilotKitProvider
      runtimeUrl="/api/copilotkit-a2ui"
      renderActivityMessages={[A2UIRenderer]}   // ← hook in the A2UI renderer
      showDevConsole={false}
    >
      {children}
      <CopilotSidebar defaultOpen labels={{ modalHeaderTitle: "A2UI Assistant" }} />
    </CopilotKitProvider>
  );
}

Output:

The playground repo is wired for A2UI, but the pattern is the same for Open‑JSON‑UI.

An agent can respond with a JSON payload describing a UI “card,” and the frontend renders it.

// Example: Agent returns Open-JSON-UI specification
{
  type: "open-json-ui",
  spec: {
    components: [
      {
        type: "card",
        properties: {
          title: "Data Visualization",
          content: {...}
        }
      }
    ]
  }
}

Output:

You can check the specs overview at docs.copilotkit.ai/learn/generative-ui/specs/open-json-ui to understand how CopilotKit's AG-UI protocol natively supports Open-JSON-UI.

This pattern is a good fit if you need more flexibility than static UI, while still keeping clear boundaries and predictable rendering.

✅ Open-ended Generative UI (MCP Apps)

Open-ended Generative UI is the most flexible and powerful.

In this pattern, the agent returns an entire UI surface. This can be HTML, an iframe, or other free-form content. The frontend mainly acts as a container that displays whatever the agent provides.

This approach is commonly used with MCP Apps, where external applications expose their own UI that can be embedded into an agent experience.

Traditional MCP tools return text, images, resources or structured data that the host displays as part of the conversation. MCP Apps extend this pattern by allowing tools to declare a reference to an interactive UI in their tool description that the host renders in place.

The trade-offs are real. Rendering arbitrary UI introduces security and performance concerns, inconsistent styling, and harder portability outside the web.

But for complex tools and rich applications, this pattern unlocks capabilities that are impossible with static or declarative UI alone.

In CopilotKit, MCP Apps support is enabled by attaching MCPAppsMiddleware to your agent, which allows the runtime to connect to one or more MCP Apps servers.

// app/api/copilotkit/route.ts
import {
  CopilotRuntime,
  ExperimentalEmptyAdapter,
  copilotRuntimeNextJSAppRouterEndpoint,
} from "@copilotkit/runtime";
import { BuiltInAgent } from "@copilotkit/runtime/v2";
import { NextRequest } from "next/server";
import { MCPAppsMiddleware } from "@ag-ui/mcp-apps-middleware";

// 1. Create your agent and add the MCP Apps middleware
const agent = new BuiltInAgent({
  model: "openai/gpt-4o",
  prompt: "You are a helpful assistant.",
}).use(
   new MCPAppsMiddleware({
    mcpServers: [
      {
        type: "http",
        url: "http://localhost:3108/mcp",
        serverId: "my-server" // Recommended: stable identifier
      },
    ],
  }),
)

// 2. Create a service adapter, empty if not relevant
const serviceAdapter = new ExperimentalEmptyAdapter();

// 3. Create the runtime and add the agent
const runtime = new CopilotRuntime({
  agents: {
    default: agent,
  },
});

// 4. Create the API route
export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter,
    endpoint: "/api/copilotkit",
  });

  return handleRequest(req);
};

Here is the GitHub Repo for the complete code implementation, and you can check the complete integration flow in the blog.

3. How AG-UI enables Generative UI at runtime

Across all three patterns, AG-UI (Agent-User Interaction Protocol) serves as the underlying runtime interaction layer in CopilotKit. It provides a bidirectional connection between the agent and the application that makes Generative UI work.

It is not a UI specification. It is an event/state protocol that defines how UI updates, agent state, and user interactions flow while the agent is working.

At a high level, AG-UI handles:

Tool lifecycles (started → streaming → finished / failed)
User interactions (clicks, form submissions, selections)
Agent state updates (progress, partial results, next steps)
Real-time coordination between agent ↔ UI ↔ application

For instance, here is the role of AG-UI protocol across Generative UI patterns:

Static → signals tool lifecycle → frontend renders predefined components
Declarative → carries UI specs + state → frontend renders structured UI
Open-ended → coordinates tools + UI references → app embeds external UI

Because AG-UI sits below specs like A2UI, Open-JSON-UI, and MCP Apps, CopilotKit can support multiple Generative UI approaches without locking you into a single approach.

In many cases, the real bottleneck is the interface that sits between the agent and the user.

Generative UI provides a practical way forward, and once you build one of these patterns yourself, it becomes clear which approach fits which problem.

I hope you learned something valuable. Have a great day!

You can check my work at anmolbaranwal.com. Thank you for reading! 🥰

Don't forget to follow CopilotKit on Twitter and join the Discord community of agent builders.

Happy building!

CopilotKit Follow

React UI + elegant infrastructure for AI Copilots, in-app AI agents, AI chatbots, and AI-powered Textareas 🪁

Bring MCP Apps into your OWN app with CopilotKit & AG-UI

Anmol Baranwal — Thu, 22 Jan 2026 15:06:34 +0000

Agent frameworks have become very good at calling tools and returning structured data.

The gap appears when an agent has to present information (charts, dashboards) or collect user input (forms, approvals). If the tool only returns JSON, every host ends up rebuilding the same UI logic for the same tools.

MCP Apps was proposed to solve this exact problem: a standardized way for MCP servers to ship interactive HTML UIs alongside tool outputs.

Today, we are excited to announce CopilotKit’s support for MCP Apps. Now, MCP servers can finally ship an interactive UI that works out of the box in real agent applications.

But showing UI is only step one. The hard part is keeping everything coordinated while the user is interacting:

Tool lifecycle (started → streaming → finished / failed)
User interactions (clicks, form submits, selections)
Agent state updates (progress, partial results, next steps)
Real-time back-and-forth between agent ↔ UI ↔ app

That’s why this integration is powered by AG‑UI: an event/state protocol that ships structured events back and forth (tool progress, UI actions, state updates) so the agent, the UI and your app stay synchronized while MCP Apps run.

The easiest way to get started is using the CopilotKit CLI:

npx copilotkit create -f mcp-apps

You can refer to the official docs or try the live demo if you want to explore it yourself. Now, let's understand the technical details of what is happening under the hood.

1. MCP-UI: where the idea first worked

This proposed standard is inspired by MCP-UI and OpenAI's Apps SDK.

MCP-UI was the community’s first proof that UI can be a first-class MCP resource (not just text/JSON). Tools could ship HTML rendered in sandboxed iframes and hosts could display rich, interactive content instead of rebuilding the UI for every tool.

MCP Apps take the core idea behind MCP-UI and formalize it into a shared, interoperable spec.

SEP-1865 was authored by MCP core maintainers at OpenAI and Anthropic, together with MCP‑UI creators and lead maintainers of the MCP-UI Community Working Group.

You can review the full SEP at modelcontextprotocol/ext-apps.

2. MCP Apps: standardizing UI within ecosystem

MCP Apps doesn’t just say “tools can return UI.” It standardizes how a server exposes UI in a way that hosts can reliably discover, render and interact with.

Credit: MCP Apps post

At a high level, it introduces a few core pieces:

MCP server declares UI resources (using the ui://... URI scheme).
Tools reference UI resources through the _meta field so the host knows what to render.
Hosts can prefetch and review templates before running tools
UI runs in sandboxed iframes for security
All UI ↔ host communication happens using the standard MCP JSON-RPC over postMessage.

A simplified example from a tool response might look like this:

{
  "result": {
    "data": {
      "revenue": [280, 400, 560]
    }
  },
  "_meta": {
    "ui/resourceUri": "ui://charts/revenue"
  }
}

On the server side, that UI is declared explicitly:

{
  "ui://charts/revenue": {
    "type": "html",
    "entrypoint": "/ui/revenue-chart.html"
  }
}

The point is: a server can ship a UI once and multiple hosts can render it consistently because the “resource + link + message channel” pattern is standardized.

What's still missing?

MCP Apps defines the UI surface and a standard way for the UI to talk to the host, but it doesn’t tell you how to keep agent state, tool lifecycle and UI events synchronized in real-time.

That orchestration layer is where AG‑UI protocol fits in.

3. AG‑UI: the sync layer that makes it usable

You still need a consistent way to keep agent ↔ UI ↔ app aligned while the user interacts.

AG-UI (Agent–User Interaction Protocol) is an event-based protocol that standardizes this “agent ↔ frontend” real-time communication.

Because it’s transport-agnostic (SSE, WebSockets, etc.), the same frontend can work across different agent backends without custom wiring.

CopilotKit uses AG-UI as its synchronization layer, which makes it a natural fit for running interactive MCP Apps.

What it looks like (events on the wire)

When an MCP App UI is on screen, AG‑UI is the stream CopilotKit (or any compatible agent framework) uses to keep tool progress and shared state synced while the user interacts.

When the agent starts a tool call, you will see lifecycle events like:

{ "type": "TOOL_CALL_START", "toolCallId": "tool_1", "toolCallName": "get_revenue_data" }

As the agent/frontend state changes, you will see state snapshots like:

{ "type": "STATE_SNAPSHOT", "snapshot": { "status": "loading" } }

Followed by incremental updates (such as JSON Patch deltas) as the workflow progresses:

{
  "type": "STATE_DELTA",
  "delta": [{ "op": "replace", "path": "/status", "value": "ready" }]
}

STATE_DELTA uses JSON Patch so the UI can update without re-sending the entire state each time.

This is how it works in practice.

4. How CopilotKit wires MCP Apps ↔ AG‑UI

CopilotKit acts as the runtime that binds MCP Apps and AG-UI together inside a real application. Here is the simple flow:

The agent calls an MCP tool and receives structured output
The tool response includes a ui://... reference, indicating which UI to render
CopilotKit renders that UI inside your app using a sandboxed iframe
UI sends interactions back and CopilotKit forwards them into the agent loop
AG-UI streams state/progress updates back to the UI while the agent runs.

Agent  
│  
│ calls tool  
▼  
MCP Tool ──────► returns data + ui:// reference  
│  
▼  
CopilotKit (runtime)  
│  
├─ renders MCP App UI (iframe)  
│  
├─ forwards UI messages → agent  
│  
└─ participates in AG-UI event stream  
│ (state / progress updates)  
▼  
MCP App UI

MCP Apps provide the UI surface, AG-UI provides the sync contract and CopilotKit runs the loop.

5. Integration flow: MCP Apps with CopilotKit

The fastest way to get started is to use the CopilotKit CLI:

npx copilotkit create -f mcp-apps

It creates a starter repo, which shows an end-to-end MCP Apps integration with CopilotKit and uses the official Three.js MCP Server.

If you are integrating into an existing Next.js app, here's how you can set things up manually.

To run MCP Apps inside a real agent, you need a runtime layer that can fetch UI resources, sandbox them and keep UI state in sync with the agent.

That’s what the MCPAppsMiddleware provides. Install it along with the required CopilotKit packages:

npm install @copilotkit/react-ui @copilotkit/react-core @copilotkit/runtime @ag-ui/mcp-apps-middleware

Next, configure the CopilotKit API route. This is where the agent is created, the MCP Apps middleware is attached and the CopilotKit runtime is exposed.

BuiltInAgent(...) : defines the agent’s LLM (model) and behavior (prompt)
mcpServers : the list of MCP servers to connect to (you can add multiple)
serverId : stable name for that server so CopilotKit can reload the same MCP App UI later, even if the server URL changes (local, staging, prod)

// app/api/copilotkit/route.ts
import {
  CopilotRuntime,
  ExperimentalEmptyAdapter,
  copilotRuntimeNextJSAppRouterEndpoint,
} from "@copilotkit/runtime";
import { BuiltInAgent } from "@copilotkit/runtime/v2";
import { NextRequest } from "next/server";
import { MCPAppsMiddleware } from "@ag-ui/mcp-apps-middleware";

// 1. Create your agent and add the MCP Apps middleware
const agent = new BuiltInAgent({
  model: "openai/gpt-4o",
  prompt: "You are a helpful assistant.",
}).use(
   new MCPAppsMiddleware({
    mcpServers: [
      {
        type: "http",
        url: "http://localhost:3108/mcp",
        serverId: "my-server" // Recommended: stable identifier
      },
    ],
  }),
)

// 2. Create a service adapter, empty if not relevant
const serviceAdapter = new ExperimentalEmptyAdapter();

// 3. Create the runtime and add the agent
const runtime = new CopilotRuntime({
  agents: {
    default: agent,
  },
});

// 4. Create the API route
export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter,
    endpoint: "/api/copilotkit",
  });

  return handleRequest(req);
};

This will allow the agent to understand ui:// references returned by MCP tools and resolve them into rendered UI.

Create a .env.local file in your frontend directory and add your API key. The example uses OpenAI’s GPT-4o by default but you can also configure other language models.

OPENAI_API_KEY=your_openai_api_key

Wrap your application with the CopilotKit provider so the frontend can connect to the runtime (layout.tsx).

import { CopilotKit } from "@copilotkit/react-core";
import "@copilotkit/react-ui/styles.css";

// ...

export default function RootLayout({ children }: {children: React.ReactNode}) {
    return (
        <html lang="en">
            <body>
                <CopilotKit runtimeUrl="/api/copilotkit">
                    {children}
                </CopilotKit>
            </body>
        </html>
    );
}

Now add the CopilotSidebar chat component to your page.

// app/page.tsx
"use client";

import { CopilotSidebar } from "@copilotkit/react-ui";

export default function Page() {
    return (
        <main>
            <h1>Your App</h1>
            <CopilotSidebar />
        </main>
    );
}

That's it! MCP Apps will now render automatically whenever the agent uses tools that reference UI resources.

The middleware supports both HTTP and SSE transports. You can find the full config options in the official docs.

You can try some of the open source MCP Apps servers from the official repo at github.com/modelcontextprotocol/ext-apps.

What this enables for Agent Builders

This brings interactive UIs into MCP in a way that works across hosts. You can:

Ship interactive tool UIs without rebuilding them per host
Keep UI + agent state in sync without custom wiring
Go from prototype to production with fewer rewrites

You can even collect structured input and include human-in-the-loop steps inside your app.

5. Getting started

To start building:

👉 Start with CopilotKit & MCP Apps: npx copilotkit create -f mcp-apps

👉 Check out the CopilotKit / MCP Apps docs to get started

👉 Check out our example repo for a minimal integration setup

👉 Try the live CopilotKit demo and interact with an MCP App in real time

What's Next

AG-UI is shaping up to be the bridge. Over the next few weeks, expect more examples, tighter primitives and clearer patterns for this integration.

If you want to follow along (or help shape what ships next), join the CopilotKit or AG-UI communities.

There’s a lot more coming soon.

Vibe Coding Forem: Anmol Baranwal

Open Source Toolkit for Building AI Agents in 2026

Categories

1. Frontend & UI Layer - CopilotKit

Alternatives

2. Skills & Plugins - agent-skills

Alternatives

3. Computer Use - UI-TARS Desktop

Alternatives

4. Agent Orchestration - LangGraph

Alternatives

5. Coding Agent Harness - Deep Agents

Alternatives

6. Open-Source Coding Agents - OpenCode

Alternatives

7. Browser Automation - Browser Use

Alternatives

8. Web Scraping & Ingestion - Firecrawl

Alternatives

9. Multi-Agent Frameworks - CrewAI

Alternatives

10. Document Processing - Docling

Alternatives

11. Voice Agents - Pipecat

Alternatives

12. Visual Builders - Langflow

Alternatives

13. MCP & Tool Integration - Composio

Alternatives

14. Sandboxing & Code Execution - E2B

Alternatives

15. Agent Memory - mem0

Alternatives

16. Testing & Evaluation - DeepEval

Alternatives

17. Monitoring & Observability - Langfuse

Alternatives

A brief note on agent protocols

Extra Resources

Chat to build and schedule your own personal Hacker News email digest! 🎉

What it does

How it works

The editor and live preview

Activation and key encryption

Per-user scheduling with QStash

The send endpoint

Generative UI Project Ideas with Complete Source Code

What is Generative UI?

1. OpenGenerativeUI - visualize anything, live in chat.

2. Shadify - generate and export Shadcn components.

3. Pi Generative UI - reverse-engineered Claude's generative UI for a terminal agent.

4. Open MCP Apps Builder - build MCP apps from a prompt.

5. Excalidraw Studio - describe anything, get a live editable diagram.

6. Voice Agent Generative UI - real forms inside voice conversations.

7. Generative UI Playground - all four patterns in one place.

8. HN Digest - chat to build your own Hacker News newsletter.

9. Generative UI MCP - design guidelines, loaded only when needed.

CopilotKit Follow

AIMock: One Mock Server For Your Entire AI Stack

Why we built AIMock

Table of Contents

AI apps in 2026 call a lot more than just an LLM

Introducing AIMock: mock your entire agentic stack

LLMock: full streaming, tool calls, reasoning across every major provider

MCPMock: a mock MCP server for testing tool integrations

A2AMock: test multi-agent workflows without live agents

VectorMock: deterministic retrieval for RAG pipelines

Services: search, rerank, and moderation

Drift Detection: catch provider changes before your users do

Record and Replay: stop hand-writing fixtures

Chaos Testing: prove your app handles failure

What else is included

AG-UI uses AIMock in production

Migrating to AIMock

Get started

CopilotKit Follow

Google Maps for Codebases: Paste a GitHub URL, Ask Anything

What are we building?

Tech Stack & Tools

What is Zenflow?