Apify (Advanced Website Crawler)

Add a bot to Apify using the official integration.

The official Apify integration connects your bot to Apify, an advanced website crawling platform. It allows your bot to manage your Apify crawler runs.

This integration is useful if you need more powerful crawling than Studio’s default web crawler provides.

Setup

You will need:

• A published bot
• An Apify account

1. Install the integration in Botpress

   First, install the integration to your bot:

   1. In Botpress Studio, select Explore Hub in the upper-right corner.
   2. Search for the Apify integration. In the Hub, it’s listed as Advanced Website Crawler.
   3. Select Install Integration.

   In the Configuration menu, you should see two fields:

   • API Token
   • Webhook Secret (Optional)

   Leave these empty for now—you’ll come back to them after completing the setup in your Apify account.

2. Get your Apify API token

   Next, get your Apify API token:

   1. Log in to your Apify account.
   2. Select Settings from the bottom-left corner.
   3. Go to API & Integrations.
   4. Under Personal API tokens, copy the API token—it should start with apify_api_.
3. Configure the integration in Botpress

   Now, you can configure the integration in Botpress:

   1. In the integration’s API Token field, paste your Apify API token.
   2. Select Save Configuration.
   3. Copy the integration’s webhook URL (starting with https://webhook.botpress.cloud).
4. Configure your Apify webhook

   Finally, create a Webhook integration:

   1. Go to the Website Content Crawler in the Apify Store.

   2. Select Try for Free, then Save & Start. This opens the Actor’s configuration page.

   3. Select the Integrations tab (on the Actor’s page, not the left sidebar).

   4. Scroll down to Generic integrations, then select HTTP webhook.

   5. Under Start when, select the following options:

      • Run succeeded
      • Run created
      • Run failed
      • Run timed out
      • Run resurrected
      • Run aborted

   6. Paste your Botpress webhook URL into the URL field.

   7. Select Save.

Your Apify integration is ready. You can use the integration’s Cards and Triggers to manage your Apify crawler runs.

Cards

Here’s a reference for all Cards available with the integration:

Get Run Status

output · object

Show child attributes

data · object · required

Show child attributes

runId · string · required

status · string · required

Start Crawler Run

input · object

Show child attributes

htmlTransformer · enum<string> · required

Available options: readableTextIfPossible, readableText, extractus, none

crawlerType · enum<string> · required

Available options: playwright:adaptive, playwright:firefox, cheerio, jsdom, playwright:chrome

output · object

Show child attributes

data · object · required

Show child attributes

runId · string · required

status · string · required

Sync Run Results

output · object

Show child attributes

data · object · required

Show child attributes

runId · string · required

datasetId · string · required

Triggers

Here’s a reference for all Triggers available with the integration:

You can access data returned from any of these Triggers by reading event.payload after the Trigger fires.

Crawler Completed

Crawler Failed