Building a webhook integration? Read the webhook guide for signatures, retries, event schemas, and troubleshooting.

apiProvider-backed

Scrape website

POST /api/v1/research/scrape-website

Pipeline recommended website scraping via workspace Spider.cloud BYOK. Collect markdown using scrape or bounded crawl. No AI extraction, platform key, or automatic paid analysis. Use public URLs. Returns source pages for table research.

Parameters

workspaceIdrequired

string

No description.

credentialId

string

No description.

urlrequired

string

No description.

mode

string

No description.

scrape | crawl

limit

integer

No description.

depth

integer

No description.

Example

REST requestbash

curl -X POST "https://app.pipeline.help/api/v1/research/scrape-website" \
  -H "Authorization: Bearer $PIPELINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"workspaceId":"00000000-0000-4000-8000-000000000000","url":"https://example.com"}'

Response

Success response schemajson schema

{
  "type": "object",
  "properties": {
    "success": {
      "type": "boolean"
    },
    "provider": {
      "type": "string",
      "enum": [
        "spider"
      ]
    },
    "pages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string"
          },
          "content": {
            "type": "string"
          }
        }
      }
    }
  },
  "required": [
    "success",
    "provider",
    "pages"
  ]
}

HTTP errors

401: Missing or invalid API key.
403: Authenticated but not permitted for the workspace or resource.
404: The requested resource was not found.