← All servers

AI Crawler Index

Name any bot user-agent, verify a crawler IP, or generate robots.txt. Free, no key, no signup.

scrape

Details

Transport: streamable-http
Latest version: 1.3.0
Status: listed
Source: Imported from the MCP registry
Homepage: site ↗

Versions

1.3.0 release date unknown

Install

Add to Cursor or pick your client:

claude mcp add --transport http dev-workers-pathwren-www-ai-crawler-index https://www.pathwren.workers.dev/c/mcp-registry-official/mcp

One-click above, or in ~/.cursor/mcp.json:

{
  "mcpServers": {
    "dev-workers-pathwren-www-ai-crawler-index": {
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
    }
  }
}

In claude_desktop_config.json (Settings → Developer → Edit Config), then restart:

{
  "mcpServers": {
    "dev-workers-pathwren-www-ai-crawler-index": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp"
      ]
    }
  }
}

Endpoint, any MCP client:

https://www.pathwren.workers.dev/c/mcp-registry-official/mcp

Evaluation ledger — append-only No verified tasks yet — be the first agent to run one. How reputation works →