Crawl

Overview

Use the Crawl MCP server from your ATXP-powered agent to search and extract information from the web. The Crawl MCP server can be used to:

crawl up to a specificied maximum number of pages
extract information from websites

Example prompts

“Scrape https://docs.atxp.ai and give me the text from the page.”
“Crawl https://www.baseball-reference.com/teams/NYM/ and give me the details on the Mets.”

Cloudflare pay per crawl

The Crawl MCP server is compatible with Cloudflare’s pay per crawl scheme. If you instruct the service to crawl a website with pay per crawl enabled, the cost of the tool call will include the added fee imposed by the content provider.

Tools

crawl_scrape

Scrape a website and return the text content. It is useful if you need a single page of text from a website.

Arguments

Accepts a JSON object with the following properties:

url

string

required

The URL of the website to scrape.

Response

A JSON object with the following properties:

status

string

The status of the scrape operation. The status key will have the value “success” when the scrape is complete and HTML content was found. If the scrape fails to find any HTML content, the status key will have a value of “error”.

html

string

The HTML content scraped from the specified URL.

crawl_crawl

Crawl a website and return the text content. It is useful if you need to crawl a website and get all the text content.

Arguments

Accepts a JSON object with the following properties:

url

string

required

The URL of the website to crawl.

maxPages

number

The maximum number of pages to crawl. The default value is 10.

Response

Returns a JSON object with the following properties:

status

string

The status of the crawl operation. The status key will have the value “success” when the crawl is complete.

text

string

The text content crawled from the specified URL.

executionId

string

The ID of the crawl task.

etaSeconds

number

The estimated time in seconds until the crawl is complete.

Usage

Define the Crawl service

Create a reusable service configuration that points to the MCP server and standardizes how you pass arguments and read results. This lets your agent easily interact with the Crawl tools in a consistent manner.

const crawlService = {
    mcpServer: 'https://crawl.mcp.atxp.ai/',
    scrapeToolName: 'crawl_scrape',
    description: 'ATXP Crawl MCP server',
    getArguments: (url: string) => ({ url }),
    getResult: (result: any) => {
      const jsonResult = result.content[0].text
      return JSON.parse(jsonResult);
    }
  };

Create an ATXP client

Using an ATXP account
Using a Base account
Using a Solana account
Using a Worldchain account
Using a Polygon account

Create a client using an ATXP account by importing the ATXP client SDK and other dependencies.

// Import the ATXP client SDK
import { atxpClient, ATXPAccount } from '@atxp/client';

// Read the ATXP account details from environment variables
const atxpConnectionString = process.env.ATXP_CONNECTION;

// Create a client using the `atxpClient` function
const client = await atxpClient({
  mcpServer: crawlService.mcpServer,
  account: new ATXPAccount(atxpConnectionString),
});

Create a client using a Base account by importing the ATXP client SDK and other dependencies.

// Import the ATXP client SDK and Base account
import { atxpClient } from '@atxp/client';
import { BaseAccount } from '@atxp/base';

// Read the Base account details from the environment variables
const baseRpcUrl = process.env.BASE_RPC_URL;
const basePrivateKey = process.env.BASE_PRIVATE_KEY;

// Create a client using the `atxpClient` function
const client = await atxpClient({
  mcpServer: crawlService.mcpServer,
  account: new BaseAccount(baseRpcUrl, basePrivateKey),
});

Create a client using a Solana account by importing the ATXP client SDK and other dependencies.

// Import the ATXP client SDK and Solana account
import { atxpClient } from '@atxp/client';
import { SolanaAccount } from '@atxp/solana';

// Read the Solana account details from the environment variables
const solanaRpcUrl = process.env.SOLANA_RPC_URL;
const solanaPrivateKey = process.env.SOLANA_PRIVATE_KEY;

// Create a client using the `atxpClient` function
const client = await atxpClient({
  mcpServer: crawlService.mcpServer,
  account: new SolanaAccount(solanaRpcUrl, solanaPrivateKey),
});

Create a client using a Worldchain account with MiniKit integration.

// Import the ATXP client SDK and Worldchain account creator
import { atxpClient } from '@atxp/client';
import { createMiniKitWorldchainAccount } from '@atxp/worldchain';
import { MiniKit } from '@worldcoin/minikit-js';

// Create a Worldchain account using MiniKit
const account = await createMiniKitWorldchainAccount({
  walletAddress: '0x1234...', // User's wallet address
  miniKit: MiniKit
});

// Create a client using the `atxpClient` function
const client = await atxpClient({
  mcpServer: crawlService.mcpServer,
  account,
});

Browser applications:

// Import the ATXP client SDK and Polygon browser account
import { atxpClient } from '@atxp/client';
import { PolygonBrowserAccount } from '@atxp/polygon';

// Initialize the Polygon browser account with wallet provider
const account = await PolygonBrowserAccount.initialize({
  provider: window.ethereum, // or any EIP-1193 provider
  walletAddress: '0x1234...', // User's wallet address
});

// Create a client using the `atxpClient` function
const client = await atxpClient({
  mcpServer: crawlService.mcpServer,
  account,
});

Server/CLI applications:

// Import the ATXP client SDK and Polygon server account
import { atxpClient } from '@atxp/client';
import { PolygonServerAccount } from '@atxp/polygon';

// Read the Polygon account details from the environment variables
const polygonRpcUrl = process.env.POLYGON_RPC_URL;
const polygonPrivateKey = process.env.POLYGON_PRIVATE_KEY;

// Create a Polygon server account
const account = new PolygonServerAccount(
  polygonRpcUrl,
  polygonPrivateKey,
  137 // Chain ID: 137 = Polygon mainnet, 80002 = Amoy testnet
);

// Create a client using the `atxpClient` function
const client = await atxpClient({
  mcpServer: crawlService.mcpServer,
  account,
});

Use the Crawl service in your agent

Call the Crawl tool by passing your natural‑language instruction as the argument the getArguments method.Read the response using the getResult method.

const url = "https://docs.atxp.ai";

try {
  const result = await client.callTool({
      name: crawlService.scrapeToolName,
      arguments: crawlService.getArguments(url),
  });
  const result = crawlService.getResult(result);
  console.log('Status:', result.status);
  console.log('HTML:', result.html);
} catch (error) {
  console.error(`Error with ${crawlService.description}:`, error);
  process.exit(1);
}

You should see the content of the crawled pages printed in your console.

Overview

Web and Data

Storage

Creative

Development

Other

Overview

Example prompts

Cloudflare pay per crawl

Tools

Arguments

Response

Arguments

Response

Usage

Overview

Web and Data

Storage

Creative

Development

Other

Documentation Index

​Overview

​Example prompts

​Cloudflare pay per crawl

​Tools

​Arguments

​Response

​Arguments

​Response

​Usage

Overview

Example prompts

Cloudflare pay per crawl

Tools

Arguments

Response

Arguments

Response

Usage