sdkProvider-backed

Scrape website

pipeline.scrapeWebsite

Pipeline recommended website scraping via workspace Spider.cloud BYOK. Collect markdown using scrape or bounded crawl. No AI extraction, platform key, or automatic paid analysis. Use public URLs. Returns source pages for table research.

Parameters

workspaceIdrequired

string

No description.

credentialId

string

No description.

urlrequired

string

No description.

mode

string

No description.

scrape | crawl

limit

integer

No description.

depth

integer

No description.

Example

TypeScripttypescript

await pipeline.scrapeWebsite({
  "workspaceId": "00000000-0000-4000-8000-000000000000",
  "url": "https://example.com"
})

Response

Success response schemajson schema

{
  "type": "object",
  "properties": {
    "success": {
      "type": "boolean"
    },
    "provider": {
      "type": "string",
      "enum": [
        "spider"
      ]
    },
    "pages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string"
          },
          "content": {
            "type": "string"
          }
        }
      }
    }
  },
  "required": [
    "success",
    "provider",
    "pages"
  ]
}

Exceptions

PipelineError: Thrown for validation, authentication, JSON-RPC, HTTP, and network failures. Inspect code and data when present.