RAG

RAG, or Retrieval Augmented Generation, lets your chatbot retrieve relevant information from your own documents before generating an answer.

react-ai-chat provides the tooling to create an embedding index and retrieve relevant chunks during a chat request.

How it works

The RAG pipeline has two stages.

Indexing

Your documents are processed and converted into embeddings:

Markdown / MDX files
        ↓
Clean document content
        ↓
Split into chunks
        ↓
Generate embeddings in batches
        ↓
Save embedding index

Retrieval

When a user asks a question:

User question
      ↓
Generate query embedding
      ↓
Compare against index
      ↓
Rank by cosine similarity
      ↓
Select topK chunks
      ↓
Add context to system prompt
      ↓
AI model

The model can then use the retrieved content when answering the question.

1. Prepare your documents

Place your Markdown or MDX documents in a directory.

For example:

content/
├── getting-started/
│   ├── installation.md
│   └── quick-start.md
├── guides/
│   ├── authentication.mdx
│   └── deployment.mdx
└── faq.md

The CLI reads Markdown and MDX content from the selected source directory.

2. Generate the embedding index

Run:

npx react-ai-chat embed

The CLI walks you through the embedding configuration.

You select:

1. Embedding provider
2. Embedding model
3. Output dimensions
4. Optional fallback model
5. Source directory
6. Output path

See CLI for the complete command documentation.

Chunking

Documents are split before embeddings are generated.

The current chunking configuration is:

The overlap helps preserve context when information spans two neighboring chunks.

For example:

Document
──────────────────────────────────────────────>

[ Chunk 1: 1000 chars ]
                     [ 200 char overlap ]
                              [ Chunk 2: 1000 chars ]
                                                   [ ... ]

Batched embeddings

During index generation, document chunks are grouped into batches before being sent to the embedding provider.

Each embedding provider has a configured maximum batch size. react-ai-chat uses that limit when sending chunks for embedding.

For example, if a provider supports a maximum batch size of 32 and your documents produce 70 chunks, the chunks are processed as:

70 document chunks
        ↓
Batch 1: chunks 1-32
        ↓
Batch 2: chunks 33-64
        ↓
Batch 3: chunks 65-70

Batches are processed sequentially. This reduces unnecessary API requests and helps avoid putting unnecessary pressure on provider rate limits.

During index generation, the CLI logs the progress of each batch:

Embedding batch 1/3: chunks 1-32/70
Embedding batch 2/3: chunks 33-64/70
Embedding batch 3/3: chunks 65-70/70

The exact number of chunks in each batch depends on the provider's configured maxBatchSize.

Embedding providers

The package supports:

• Google
• OpenAI
• Voyage AI
• Cohere
• Jina AI
• Hugging Face

Each provider has its own models and API credentials.

Embedding providers also have different batch limits. The indexer uses each provider's configured maximum batch size when generating the embedding index.

See Providers for the current provider configuration and model catalogue.

Embedding dimensions

Every vector in an embedding index must have the same number of dimensions.

For example:

Document chunk → [768 values]
Document chunk → [768 values]
Document chunk → [768 values]

The query embedding must also have the same dimension.

A mismatch causes retrieval to fail.

This is why the CLI filters compatible fallback models according to the selected dimensions.

Fallback models

You can configure a fallback model during index generation.

The fallback is useful when the primary embedding provider encounters an error or reaches its rate limit while processing a batch.

The generation process can then continue using the fallback configuration for that batch.

Primary model
     ↓
Batch embedding
     ↓
Error?
   ↙     ↘
 No       Yes
 ↓         ↓
Continue  Fallback model
             ↓
       Process the batch

Fallback handling is applied during batch processing, so a failed batch can be retried using the configured fallback provider.

The generated index records the embedding model used for each chunk.

Connect the index to your chatbot

After generating your embedding index, import the generated embeddings.json file into your server route.

The embedding provider should use the same model recorded in the generated index.

For example, if you generated the index with Google:

import { google } from "@ai-sdk/google";
import { GoogleGenAI } from "@google/genai";
import { createChatRoute, googleEmbedding } from "react-ai-chat/server";
import embeddings from "@/chatbot/embeddings.json";

const client = new GoogleGenAI({
  apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY,
});

const provider = googleEmbedding(client, {
  model: embeddings.model,
});

export const POST = createChatRoute({
  model: google("gemini-3.5-flash"),
  rag: {
    index: embeddings,
    provider,
    topK: 3,
  },
});

What's happening here?

The generated embeddings.json contains metadata about the embedding configuration used when the index was created.

const provider = googleEmbedding(client, {
  model: embeddings.model,
});

Using embeddings.model keeps the retrieval provider aligned with the model stored in the index.

The language model used to generate the chatbot response is configured separately:

model: google("gemini-3.5-flash"),

The embedding model and chat model serve different purposes.

The embedding model converts text into vectors for retrieval. The chat model generates the final response.

Configure topK

topK controls how many matching document chunks are retrieved for each user question.

rag: {
  index: embeddings,
  provider,
  topK: 5,
},

The default is 3.

A higher value gives the language model more retrieved context. Choose a value that fits the amount of context your application needs.

Provider-specific setup

The same pattern applies to the other supported embedding providers.

The provider client and embedding function change according to the provider used to generate the index.

See Providers for the provider-specific configuration examples.

What gets retrieved?

The latest user message is used as the retrieval query.

For example:

User:
"How do I configure authentication?"

The question is converted into an embedding and compared against every indexed chunk.

The chunks with the highest cosine similarity are selected.

The selected content is then provided to the language model as relevant context.

Retrieval errors

RAG retrieval is handled independently from model generation.

If retrieval fails, react-ai-chat logs the retrieval error and continues the chat request without the retrieved context.

This means a temporary embedding provider problem does not automatically prevent the language model from responding.

Updating your knowledge base

Regenerate the index whenever your source documents change:

npx react-ai-chat embed

You should also regenerate it when you change:

• Embedding provider
• Embedding model
• Embedding dimensions
• Source documents

The generated index represents the embedding configuration used when it was created.

Embedding index

The generated JSON index contains the information required for retrieval, including:

• Index metadata
• Embedding provider
• Embedding model
• Embedding dimensions
• Document chunks
• Chunk embeddings
• Per-chunk embedding metadata

The retrieval system validates the index before performing a search.

Example architecture

After running the embedding command, your project can look like this:

project/
├── app/
│   └── api/
│       └── chat/
│           └── route.ts
├── chatbot/
│   └── embeddings.json
├── content/
│   ├── docs/
│   └── faq/
└── package.json

The embeddings.json file is generated by the CLI and contains the embedding index and its metadata.

Your API route can import that file directly:

import { google } from "@ai-sdk/google";
import { GoogleGenAI } from "@google/genai";
import { createChatRoute, googleEmbedding } from "react-ai-chat/server";
import embeddings from "@/chatbot/embeddings.json";

const client = new GoogleGenAI({
  apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY,
});

const provider = googleEmbedding(client, {
  model: embeddings.model,
});

export const POST = createChatRoute({
  model: google("gemini-3.5-flash"),
  rag: {
    index: embeddings,
    provider,
    topK: 3,
  },
});

The resulting request flow is:

Source documents
      ↓
npx react-ai-chat embed
      ↓
Split into chunks
      ↓
Generate embeddings in batches
      ↓
chatbot/embeddings.json
      ↓
createChatRoute()
      ↓
Retrieve relevant chunks
      ↓
Add context to the system prompt
      ↓
Chat model
      ↓
Response

This gives the chatbot access to your project's own documentation instead of relying only on the model's training data.

Next steps