Build AI Apps with Bun: Streaming, Tools, and RAG

Bun practices for streaming, tool calls, RAG, safety, cost, and production observability

Where Bun fits

Bun can run official JavaScript SDKs, standard fetch, streaming Response bodies, and server-side tools. OpenAI's official JavaScript SDK explicitly supports Bun, and Vercel's AI SDK provides common streaming and tool-call interfaces. Select an SDK based on existing project dependencies, providers, and the deployment target; do not rewrite a working boundary merely for uniformity.

Minimal streaming endpoint

This example uses AI SDK with Bun's native HTTP server. The model comes from configuration so source code does not freeze a fast-changing model name:

bun add ai @ai-sdk/openai zod

src/server.ts

import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';
import { z } from 'zod';

const inputSchema = z.object({
  prompt: z.string().trim().min(1).max(8_000),
});

Bun.serve({
  idleTimeout: 30,
  routes: {
    '/api/generate': {
      POST: async (request) => {
        const parsed = inputSchema.safeParse(await request.json());
        if (!parsed.success) {
          return Response.json({ error: 'invalid input' }, { status: 400 });
        }

        const model = Bun.env.OPENAI_MODEL;
        if (!model) throw new Error('OPENAI_MODEL is required');

        const result = streamText({
          model: openai(model),
          prompt: parsed.data.prompt,
          abortSignal: request.signal,
          timeout: { totalMs: 60_000, chunkMs: 15_000 },
          maxRetries: 2,
        });

        return result.toTextStreamResponse();
      },
    },
  },
  fetch: () => new Response('Not found', { status: 404 }),
});

Check all four timeout layers

The model, SDK, Bun.serve, and cloud platform each impose timeouts. The values above are visible starting points, not universal recommendations; derive them from the real SLA, model latency, platform limits, and load tests.

Security model for tool calls

Treat the model as a component that proposes structured actions, not as a principal that owns authority:

Narrow tool set: expose only tools required for the task, never a universal shell by default.
Validate input: enforce types, lengths, IDs, and enums with schemas; model arguments remain untrusted.
Authorize server-side: derive tenant and user from the authenticated session, never from an owner ID supplied by the model.
Classify side effects: reads may run automatically; messages, payments, deletion, and production writes need confirmation or policy approval.
Bound the budget: cap steps, concurrency, tokens, tool calls, and money; return an explicit error at the limit.
Constrain networking: defend URL fetchers from SSRF, allow approved schemes/domains, and block metadata and internal networks.
Handle output safely: escape, parameterize, or sandbox generated HTML, SQL, shell, and Markdown before rendering or execution.

RAG and data boundaries

Preserve headings, URL, version, locale, and update time in chunks so every answer can link back to evidence.
A bilingual semantic index can be shared, but store locale, prefer the user's language, and keep duplicate translations from crowding out top-k results.
Start with a small human-curated question set, including cases that should be refused, before tuning chunk size, embeddings, or reranking.
Retrieved web and repository text is untrusted data; it cannot override policy or tool authority.
For regulated data, verify provider retention, region, and training policies. Minimize raw text in logs by default.

Production observability and evaluation

Area	Record at minimum	Avoid
Request	request ID, tenant, route, outcome	secrets, access tokens, raw prompts by default
Model	provider, model, latency, tokens/cost	averages without p95/p99
Tools	tool name, duration, result class, approval state	unbounded recursion and invisible side effects
Quality	versioned eval set, pass rate, regression cases	demos as the only release gate

Record provider request IDs for support investigations. Apply sampling, redaction, access control, and retention limits to prompts and outputs. Run representative evaluations whenever the model, prompt, tool schema, or retrieval chain changes.

Official references: OpenAI JavaScript quickstart, AI SDK streamText, AI SDK tools, and Bun HTTP server.

Build AI apps with Bun