> ## Documentation Index
> Fetch the complete documentation index at: https://botpress.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Apify (Advanced Website Crawler)

> Add a bot to Apify using the official integration.

[Open in Hub](https://studio.botpress.cloud/home?exploreHub=1\&hubItemId=intver_01KH6R8P2ZYNACH8KQED02E9YC)
*v1.0.4*

The official Apify integration connects your bot to Apify, an advanced website crawling platform. It allows your bot to manage your Apify crawler runs.

This integration is useful if you need more powerful crawling than Studio's [default web crawler](https://botpress.com/docs/studio/concepts/knowledge-base/add-sources.md#websites) provides.

## Setup

> [!NOTE]
>
> You will need:
>
> * A [published bot](https://botpress.com/docs/get-started/quick-start.md)
> * An [Apify account](https://console.apify.com/sign-up)

**1. Install the integration in Botpress**

First, install the integration to your bot:

1. In Botpress Studio, select \*\* Explore Hub\*\* in the upper-right corner.
2. Search for the **Apify** integration. In the Hub, it's listed as **Advanced Website Crawler**.
3. Select **Install Integration**.

In the **Configuration** menu, you should see two fields:

* **API Token**
* **Webhook Secret** (Optional)

Leave these empty for now—you'll come back to them after completing the setup in your Apify account.

**2. Get your Apify API token**

Next, get your Apify API token:

1. [Log in to your Apify account](https://console.apify.com/sign-in).
2. Select **Settings** from the bottom-left corner.
3. Go to **API & Integrations**.
4. Under **Personal API tokens**, copy the API token—it should start with `apify_api_`.

**3. Configure the integration in Botpress**

Now, you can configure the integration in Botpress:

1. In the integration's **API Token** field, paste your Apify API token.
2. Select **Save Configuration**.
3. Copy the integration's webhook URL (starting with `https://webhook.botpress.cloud`).

**4. Configure your Apify webhook**

Finally, create a Webhook integration:

1. Go to the [Website Content Crawler](https://apify.com/apify/website-content-crawler) in the Apify Store.
2. Select **Try for Free**, then **Save & Start**. This opens the Actor's configuration page.
3. Select the **Integrations** tab (on the Actor's page, not the left sidebar).
4. Scroll down to **Generic integrations**, then select **HTTP webhook**.
5. Under **Start when**, select the following options:

   * Run succeeded
   * Run created
   * Run failed
   * Run timed out
   * Run resurrected
   * Run aborted
6. Paste your Botpress webhook URL into the **URL** field.
7. Select **Save**.

> [!TIP]
>
> Your Apify integration is ready. You can use the integration's [Cards](#cards) and [Triggers](#triggers) to manage your Apify crawler runs.

## Cards

{/* vale off */}

Here's a reference for all [Cards](https://botpress.com/docs/studio/concepts/cards/introduction.md) available with the integration:

### Get Run Status

Check the status of a crawler run (useful for monitoring)

**input** `object`

**runId** `string` *required*

The run ID to check status for

**output** `object`

**success** `boolean` *required*

Whether the operation completed successfully

**message** `string` *required*

Status message describing the result

**data** `object` *required*

Run details including run ID and current status

**runId** `string` *required*

**status** `string` *required*

### Start Crawler Run

Start a crawler run asynchronously. Use with webhooks for production crawling. You can either use individual parameters for simple cases, or provide rawInputJsonOverride for full control.

**input** `object`

**startUrls** `array` *required*

URLs to start crawling from

**excludeUrlGlobs** `array`

URL patterns to exclude from crawling

**includeUrlGlobs** `array`

URL patterns to include in crawling

**maxCrawlPages** `number` *required*

Maximum number of pages to crawl

**saveMarkdown** `boolean` *required*

Save content as Markdown format

**htmlTransformer** `enum\<string\>` *required*

HTML processing method

Available options: `readableTextIfPossible`, `readableText`, `extractus`, `none`

**removeElementsCssSelector** `string`

CSS selectors for elements to remove

**crawlerType** `enum\<string\>` *required*

Browser type for crawling

Available options: `playwright:adaptive`, `playwright:firefox`, `cheerio`, `jsdom`, `playwright:chrome`

**expandClickableElements** `boolean` *required*

Expand clickable elements for better content extraction

**headers** `string`

Custom HTTP headers for authentication/requests

**rawInputJsonOverride** `string`

JSON string to override any crawler parameters, please refer to [https://console.apify.com/actors/\<actor-id>/input](https://console.apify.com/actors/\<actor-id>/input) and select JSON format for the available parameters

**kbId** `string` *required*

Knowledge Base ID to save the crawled content to

**output** `object`

**success** `boolean` *required*

Whether the operation completed successfully

**message** `string` *required*

Status message describing the result

**data** `object` *required*

Run details including run ID and current status

**runId** `string` *required*

**status** `string` *required*

### Sync Run Results

Get the results from a completed crawler run and sync to Botpress KB

**input** `object`

**runId** `string` *required*

The run ID to get results for

**kbId** `string` *required*

Knowledge Base ID to tag the crawled content with

**output** `object`

**success** `boolean` *required*

Whether the operation completed successfully

**message** `string` *required*

Status message describing the result

**data** `object` *required*

Details about the synced run including run ID and dataset ID

**runId** `string` *required*

**datasetId** `string` *required*

## Triggers

{/* vale off */}

Here's a reference for all [Triggers](https://botpress.com/docs/studio/concepts/triggers.md) available with the integration:

> [!TIP]
>
> You can access data returned from any of these Triggers by reading `event.payload` after the Trigger fires.

### Crawler Completed

Triggered when an Apify crawler run completes successfully

**payload** `object`

**actorId** `string` *required*

ID of the triggering Actor

**actorTaskId** `string`

If task was used, its ID

**actorRunId** `string` *required*

ID of the triggering Actor run

**eventType** `string` *required*

Type of webhook event (e.g., ACTOR.RUN.SUCCEEDED)

**runId** `string` *required*

Alias for actorRunId for easier access

**itemsCount** `number`

Number of items crawled

**filesCreated** `number`

Number of files created in Botpress

**syncTargetPath** `string`

Path where results were synced

**hasMore** `boolean`

Whether there are more items to sync in subsequent batches

### Crawler Failed

Triggered when an Apify crawler run fails, times out, or is aborted

**payload** `object`

**actorId** `string` *required*

ID of the triggering Actor

**actorRunId** `string` *required*

ID of the triggering Actor run

**runId** `string` *required*

Alias for actorRunId for easier access

**eventType** `string` *required*

Type of webhook event (e.g., ACTOR.RUN.FAILED, ACTOR.RUN.TIMED\_OUT, ACTOR.RUN.ABORTED)

**reason** `string` *required*

Reason for failure (FAILED, TIMED\_OUT, or ABORTED)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.