Knowledge

Knowledge bases provide context to your agent’s AI models. They allow your agent to answer questions based on documents, websites, or other data sources.

Creating a knowledge base

Create a knowledge source in src/knowledge/:

import { Knowledge, DataSource } from "@botpress/runtime";

export default new Knowledge({
  name: "documentation",
  description: "Product documentation",
  sources: [
    // Add knowledge sources
  ],
});

Data sources

You can add knowledge from several different data sources:

Website source

There are multiple ways to index websites:

From a sitemap

Index a website using a sitemap XML file:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromSitemap(
  "https://example.com/sitemap.xml",
  {
    filter: ({ url }) => !url.includes("/admin"),
    maxPages: 1000,
    maxDepth: 10,
  }
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From a base URL

Crawl a website starting from a base URL (requires the Browser integration):

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromWebsite(
  "https://example.com",
  {
    filter: ({ url }) => !url.includes("/admin"),
    maxPages: 500,
    maxDepth: 5,
  }
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From llms.txt

Index pages referenced in an llms.txt file:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromLlmsTxt(
  "https://example.com/llms.txt"
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From specific URLs

Index a specific list of URLs:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromUrls([
  "https://example.com/page1",
  "https://example.com/page2",
  "https://example.com/page3",
]);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

Website source options

Option	Type	Description
id	string	Optional unique identifier for the source
filter	(ctx) => boolean	Filter function receiving { url, lastmod?, changefreq?, priority? }
fetch	string | function	Fetch strategy: 'node:fetch' (default), 'integration:browser', or custom function
maxPages	number	Maximum pages to index (1-50000, default: 50000)
maxDepth	number	Maximum sitemap depth (1-20, default: 20)

File source

Index files from a local directory (development only):

const FileSource = DataSource.Directory.fromPath("./docs", {
  filter: (filePath) => filePath.endsWith(".md") || filePath.endsWith(".txt"),
});

export default new Knowledge({
  name: "local-docs",
  sources: [FileSource],
});

Directory sources only work during development (adk dev). For production, use website sources or upload files to Botpress Cloud.

Directory source options

Option	Type	Description
id	string	Optional unique identifier for the source
filter	(filePath: string) => boolean	Filter function to include/exclude files

Table source

Index data from a table:

import { Knowledge, DataSource } from "@botpress/runtime";
import OrderTable from "../tables/OrderTable";

const TableSource = DataSource.Table.fromTable(OrderTable, {
  id: "orders",
  transform: ({ row }) => {
    return `Order #${row.id}\nCustomer: ${row.userId}\nTotal: $${row.total}\nStatus: ${row.status}`;
  },
});

export default new Knowledge({
  name: "orders-kb",
  sources: [TableSource],
});

Using knowledge in conversations

Provide knowledge bases to your conversation handlers:

import { WebsiteKB } from "../knowledge/docs";

export default new Conversation({
  channel: "*",
  handler: async ({ execute }) => {
    await execute({
      instructions: "You are a helpful assistant.",
      knowledge: [WebsiteKB],
    });
  },
});

Refreshing knowledge

Refresh knowledge bases to update their content:

import { Workflow } from "@botpress/runtime";
import WebsiteKB from "../knowledge/docs"

export default new Workflow({
  name: "refresh-knowledge",
  schedule: "0 0 * * *", // Daily
  handler: async () => {
    await WebsiteKB.refresh();
  },
});

Force refresh

Force re-indexing of all content even if unchanged:

await WebsiteKB.refresh({ force: true });

Refresh a specific source

Refresh a single data source within a knowledge base:

await WebsiteKB.refreshSource("my-source-id", { force: true });

Reference

Knowledge props

Show Knowledge props

name · string · required

Unique name for the knowledge base.

description · string

Optional description of what the knowledge base contains.

sources · DataSource[] · required

Array of data sources to index. Can include Website, Directory, or Table sources.