Module cloudflare_crawl 

Cloudflare Browser Rendering crawl adapter (feature = “cloudflare-crawl”) Cloudflare Browser Rendering crawl adapter

Delegates whole-site crawling to Cloudflare’s /crawl endpoint (open beta). Useful when managed, infrastructure-free crawling is preferred over running a local Chrome pool — trades stealth/anti-detection capability for zero operational overhead on the scraping infrastructure.

Feature flag

Gated behind cloudflare-crawl. Enable in Cargo.toml:

[dependencies]
stygian-graph = { version = "...", features = ["cloudflare-crawl"] }

Example

use stygian_graph::adapters::cloudflare_crawl::CloudflareCrawlAdapter;
use stygian_graph::ports::{ScrapingService, ServiceInput};
use serde_json::json;

let adapter = CloudflareCrawlAdapter::new().unwrap();
let input = ServiceInput {
    url: "https://docs.example.com".to_string(),
    params: json!({
        "account_id": "abc123",
        "api_token":  "my-cf-token",
        "output_format": "markdown",
        "max_depth": 3,
        "max_pages": 50,
    }),
};
// let output = adapter.execute(input).await.unwrap();

Structs

CloudflareCrawlAdapter

Cloudflare Browser Rendering crawl adapter.

CloudflareCrawlConfig

Configuration for the Cloudflare crawl adapter.

Functions

extract_cf_error

Extract a human-readable error message from a Cloudflare API response body.