> ## Documentation Index
> Fetch the complete documentation index at: https://docs.atxp.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Crawl the web

## Overview

Use the Crawl MCP server from your ATXP-powered agent to search and extract information from the web. The Crawl MCP server can be used to:

* crawl up to a specificied maximum number of pages
* extract information from websites

## Example prompts

* "Scrape [https://docs.atxp.ai](https://docs.atxp.ai) and give me the text from the page."
* "Crawl [https://www.baseball-reference.com/teams/NYM/](https://www.baseball-reference.com/teams/NYM/) and give me the details on the Mets."

## Cloudflare pay per crawl

The Crawl MCP server is compatible with Cloudflare's [pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) scheme. If you instruct the service to crawl a website with pay per crawl enabled, the cost of the tool call will include the added fee imposed by the content provider.

## Tools

<AccordionGroup>
  <Accordion title="crawl_scrape">
    Scrape a website and return the text content. It is useful if you need a single page of text from a website.

    ### Arguments

    Accepts a JSON object with the following properties:

    <ParamField body="url" type="string" required>
      The URL of the website to scrape.
    </ParamField>

    ### Response

    A JSON object with the following properties:

    <ResponseField name="status" type="string">
      The status of the scrape operation. The `status` key will have the value "success" when the scrape is complete and HTML content was found. If the scrape fails to find any HTML content, the `status` key will have a value of "error".
    </ResponseField>

    <ResponseField name="html" type="string">
      The HTML content scraped from the specified URL.
    </ResponseField>
  </Accordion>

  <Accordion title="crawl_crawl">
    Crawl a website and return the text content. It is useful if you need to crawl a website and get all the text content.

    ### Arguments

    Accepts a JSON object with the following properties:

    <ParamField body="url" type="string" required>
      The URL of the website to crawl.
    </ParamField>

    <ParamField body="maxPages" type="number">
      The maximum number of pages to crawl. The default value is 10.
    </ParamField>

    ### Response

    Returns a JSON object with the following properties:

    <ResponseField name="status" type="string">
      The status of the crawl operation. The `status` key will have the value "success" when the crawl is complete.
    </ResponseField>

    <ResponseField name="text" type="string">
      The text content crawled from the specified URL.
    </ResponseField>

    <ResponseField name="executionId" type="string">
      The ID of the crawl task.
    </ResponseField>

    <ResponseField name="etaSeconds" type="number">
      The estimated time in seconds until the crawl is complete.
    </ResponseField>
  </Accordion>
</AccordionGroup>

## Usage

<Steps>
  <Step title="Define the Crawl service">
    Create a reusable service configuration that points to the MCP server and standardizes how you pass arguments and read results. This lets your agent easily interact with the Crawl tools in a consistent manner.

    ```typescript theme={null}
    const crawlService = {
        mcpServer: 'https://crawl.mcp.atxp.ai/',
        scrapeToolName: 'crawl_scrape',
        description: 'ATXP Crawl MCP server',
        getArguments: (url: string) => ({ url }),
        getResult: (result: any) => {
          const jsonResult = result.content[0].text
          return JSON.parse(jsonResult);
        }
      };
    ```
  </Step>

  <Step title="Create an ATXP client">
    <Tabs>
      <Tab title="Using an ATXP account">
        Create a client using an <a href="/developers/build-agents/create-account" target="_blank">ATXP account</a> by importing the ATXP client SDK and other dependencies.

        ```typescript theme={null}
        // Import the ATXP client SDK
        import { atxpClient, ATXPAccount } from '@atxp/client';

        // Read the ATXP account details from environment variables
        const atxpConnectionString = process.env.ATXP_CONNECTION;

        // Create a client using the `atxpClient` function
        const client = await atxpClient({
          mcpServer: crawlService.mcpServer,
          account: new ATXPAccount(atxpConnectionString),
        });
        ```
      </Tab>

      <Tab title="Using a Base account">
        Create a client using a Base account by importing the ATXP client SDK and other dependencies.

        ```typescript theme={null}
        // Import the ATXP client SDK and Base account
        import { atxpClient } from '@atxp/client';
        import { BaseAccount } from '@atxp/base';

        // Read the Base account details from the environment variables
        const baseRpcUrl = process.env.BASE_RPC_URL;
        const basePrivateKey = process.env.BASE_PRIVATE_KEY;

        // Create a client using the `atxpClient` function
        const client = await atxpClient({
          mcpServer: crawlService.mcpServer,
          account: new BaseAccount(baseRpcUrl, basePrivateKey),
        });
        ```
      </Tab>

      <Tab title="Using a Solana account">
        Create a client using a Solana account by importing the ATXP client SDK and other dependencies.

        ```typescript theme={null}
        // Import the ATXP client SDK and Solana account
        import { atxpClient } from '@atxp/client';
        import { SolanaAccount } from '@atxp/solana';

        // Read the Solana account details from the environment variables
        const solanaRpcUrl = process.env.SOLANA_RPC_URL;
        const solanaPrivateKey = process.env.SOLANA_PRIVATE_KEY;

        // Create a client using the `atxpClient` function
        const client = await atxpClient({
          mcpServer: crawlService.mcpServer,
          account: new SolanaAccount(solanaRpcUrl, solanaPrivateKey),
        });
        ```
      </Tab>

      <Tab title="Using a Worldchain account">
        Create a client using a Worldchain account with MiniKit integration.

        ```typescript theme={null}
        // Import the ATXP client SDK and Worldchain account creator
        import { atxpClient } from '@atxp/client';
        import { createMiniKitWorldchainAccount } from '@atxp/worldchain';
        import { MiniKit } from '@worldcoin/minikit-js';

        // Create a Worldchain account using MiniKit
        const account = await createMiniKitWorldchainAccount({
          walletAddress: '0x1234...', // User's wallet address
          miniKit: MiniKit
        });

        // Create a client using the `atxpClient` function
        const client = await atxpClient({
          mcpServer: crawlService.mcpServer,
          account,
        });
        ```
      </Tab>

      <Tab title="Using a Polygon account">
        **Browser applications:**

        ```typescript theme={null}
        // Import the ATXP client SDK and Polygon browser account
        import { atxpClient } from '@atxp/client';
        import { PolygonBrowserAccount } from '@atxp/polygon';

        // Initialize the Polygon browser account with wallet provider
        const account = await PolygonBrowserAccount.initialize({
          provider: window.ethereum, // or any EIP-1193 provider
          walletAddress: '0x1234...', // User's wallet address
        });

        // Create a client using the `atxpClient` function
        const client = await atxpClient({
          mcpServer: crawlService.mcpServer,
          account,
        });
        ```

        **Server/CLI applications:**

        ```typescript theme={null}
        // Import the ATXP client SDK and Polygon server account
        import { atxpClient } from '@atxp/client';
        import { PolygonServerAccount } from '@atxp/polygon';

        // Read the Polygon account details from the environment variables
        const polygonRpcUrl = process.env.POLYGON_RPC_URL;
        const polygonPrivateKey = process.env.POLYGON_PRIVATE_KEY;

        // Create a Polygon server account
        const account = new PolygonServerAccount(
          polygonRpcUrl,
          polygonPrivateKey,
          137 // Chain ID: 137 = Polygon mainnet, 80002 = Amoy testnet
        );

        // Create a client using the `atxpClient` function
        const client = await atxpClient({
          mcpServer: crawlService.mcpServer,
          account,
        });
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Use the Crawl service in your agent">
    Call the Crawl tool by passing your natural‑language instruction as the argument the `getArguments` method.

    Read the response using the `getResult` method.

    ```typescript theme={null}
    const url = "https://docs.atxp.ai";

    try {
      const result = await client.callTool({
          name: crawlService.scrapeToolName,
          arguments: crawlService.getArguments(url),
      });
      const result = crawlService.getResult(result);
      console.log('Status:', result.status);
      console.log('HTML:', result.html);
    } catch (error) {
      console.error(`Error with ${crawlService.description}:`, error);
      process.exit(1);
    }
    ```

    <Check>
      You should see the content of the crawled pages printed in your console.
    </Check>
  </Step>
</Steps>