Skip to main content

​
Overview

Use the Crawl MCP server from your ATXP-powered agent to search and extract information from the web. The Crawl MCP server can be used to:
  • crawl up to a specificied maximum number of pages
  • extract information from websites

​
Example prompts

​
Cloudflare pay per crawl

The Crawl MCP server is compatible with Cloudflare’s pay per crawl scheme. If you instruct the service to crawl a website with pay per crawl enabled, the cost of the tool call will include the added fee imposed by the content provider.

​
Tools

Scrape a website and return the text content. It is useful if you need a single page of text from a website.

​
Arguments

Accepts a JSON object with the following properties:
​
url
string
required
The URL of the website to scrape.

​
Response

A JSON object with the following properties:
​
status
string
The status of the scrape operation. The status key will have the value “success” when the scrape is complete and HTML content was found. If the scrape fails to find any HTML content, the status key will have a value of “error”.
​
html
string
The HTML content scraped from the specified URL.
Crawl a website and return the text content. It is useful if you need to crawl a website and get all the text content.

​
Arguments

Accepts a JSON object with the following properties:
​
url
string
required
The URL of the website to crawl.
​
maxPages
number
The maximum number of pages to crawl. The default value is 10.

​
Response

Returns a JSON object with the following properties:
​
status
string
The status of the crawl operation. The status key will have the value “success” when the crawl is complete.
​
text
string
The text content crawled from the specified URL.
​
executionId
string
The ID of the crawl task.
​
etaSeconds
number
The estimated time in seconds until the crawl is complete.

​
Usage

1

Define the Crawl service

Create a reusable service configuration that points to the MCP server and standardizes how you pass arguments and read results. This lets your agent easily interact with the Crawl tools in a consistent manner.
const crawlService = {
    mcpServer: 'https://crawl.mcp.atxp.ai/',
    scrapeToolName: 'crawl_scrape',
    description: 'ATXP Crawl MCP server',
    getArguments: (url: string) => ({ url }),
    getResult: (result: any) => {
      const jsonResult = result.content[0].text
      return JSON.parse(jsonResult);
    }
  };
2

Create an ATXP client

  • Using an ATXP account
  • Using a Base account
  • Using a Solana account
  • Using a Worldchain account
  • Using a Polygon account
Create a client using an ATXP account by importing the ATXP client SDK and other dependencies.
// Import the ATXP client SDK
import { atxpClient, ATXPAccount } from '@atxp/client';

// Read the ATXP account details from environment variables
const atxpConnectionString = process.env.ATXP_CONNECTION;

// Create a client using the `atxpClient` function
const client = await atxpClient({
  mcpServer: crawlService.mcpServer,
  account: new ATXPAccount(atxpConnectionString),
});
3

Use the Crawl service in your agent

Call the Crawl tool by passing your natural‑language instruction as the argument the getArguments method.Read the response using the getResult method.
const url = "https://docs.atxp.ai";

try {
  const result = await client.callTool({
      name: crawlService.scrapeToolName,
      arguments: crawlService.getArguments(url),
  });
  const result = crawlService.getResult(result);
  console.log('Status:', result.status);
  console.log('HTML:', result.html);
} catch (error) {
  console.error(`Error with ${crawlService.description}:`, error);
  process.exit(1);
}
You should see the content of the crawled pages printed in your console.
⌘I