Proxy Rotation Overview

stygian-proxy is a high-performance, resilient proxy rotation library for the Stygian scraping ecosystem. It manages a pool of proxy endpoints, tracks per-proxy health and latency metrics, learns per-domain effectiveness online, and integrates directly with both stygian-graph HTTP adapters and stygian-browser page contexts.

Feature summary

Quick start

Add the dependency:

[dependencies]
stygian-proxy = { version = "0.14", features = ["graph"] }

Build a pool and make a request:

use std::sync::Arc;
use stygian_proxy::{MemoryProxyStore, ProxyConfig, ProxyManager};
use stygian_proxy::types::{Proxy, ProxyType, IpClass, TargetVendorCompatibility};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let storage = Arc::new(MemoryProxyStore::default());
    let manager = Arc::new(
        ProxyManager::with_round_robin(storage, ProxyConfig::default())?
    );

    // Register a mobile-carrier residential proxy with vendor metadata.
    manager.add_proxy(Proxy {
        url: "http://proxy1.example.com:8080".into(),
        proxy_type: ProxyType::Http,
        username: None,
        password: None,
        weight: 1,
        tags: vec!["us-east".into()],
        ip_class: IpClass::Residential,
        target_compatibility: TargetVendorCompatibility::preferred(),
        ..Default::default()
    }).await?;

    // Start background health checks.
    let (cancel, _task) = manager.start();

    // Acquire a proxy for a request.
    let handle = manager.acquire_proxy().await?;
    println!("using proxy: {}", handle.proxy_url);

    // Signal success — omitting this counts as a failure toward the circuit breaker.
    handle.mark_success();

    cancel.cancel(); // stop health checker
    Ok(())
}

Architecture

┌──────────────────────────────────────────────┐
│                ProxyManager                  │
│                                              │
│  ┌───────────────┐  ┌─────────────────────┐  │
│  │ HealthChecker │  │   CircuitBreakers   │  │
│  │   + jitter    │  │     (per proxy)     │  │
│  └───────────────┘  └─────────────────────┘  │
│                                              │
│  ┌────────────────────────────────────────┐  │
│  │           RotationStrategy             │  │
│  │  RoundRobin / Random / Weighted /      │  │
│  │  LeastUsed / ThompsonSampling          │  │
│  └────────────────────────────────────────┘  │
│                                              │
│  ┌────────────────────────────────────────┐  │
│  │      CapabilityRequirement filter      │  │
│  │  tls_profile / cdn_edge / ip_class /   │  │
│  │  vendor / asn / city / postal_code     │  │
│  └────────────────────────────────────────┘  │
│                                              │
│  ┌────────────────────────────────────────┐  │
│  │    CoherenceValidator (optional)       │  │
│  │    WebRTC + DNS + tz + locale + lang   │  │
│  └────────────────────────────────────────┘  │
│                                              │
│  ┌────────────────────────────────────────┐  │
│  │  VendorStickinessMap (optional)        │  │
│  │  per-vendor sticky / fresh-per-domain  │  │
│  └────────────────────────────────────────┘  │
│                                              │
│  ┌────────────────────────────────────────┐  │
│  │          ProxyStoragePort              │  │
│  │     MemoryProxyStore (built-in)        │  │
│  └────────────────────────────────────────┘  │
└──────────────────────────────────────────────┘
         │                       │
         ▼                       ▼
   stygian-graph           stygian-browser
   HTTP adapters           page contexts

ProxyManager is the main entry point. It composes a storage backend, a rotation strategy, a background health checker, a map of per-proxy circuit breakers, and — when the corresponding cargo features are enabled — an optional coherence validator and a per-vendor stickiness map. Callers interact primarily via acquire_proxy() → ProxyHandle → mark_success().

Cargo features

ProxyConfig defaults

Capability-aware acquisition

Beyond the legacy tags-based filter, CapabilityRequirement is the primary way to declare what a request needs from a proxy. The struct carries one Option field per constraint; all fields are independently #[serde(default, skip_serializing_if = "Option::is_none")] so a serialised requirement with no filters round-trips cleanly.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use stygian_proxy::{MemoryProxyStore, ProxyConfig, ProxyManager};
use stygian_proxy::types::{
    CapabilityRequirement, IpClassRequirement, ProxyType, VendorId,
};

let storage = Arc::new(MemoryProxyStore::default());
let mgr = ProxyManager::with_round_robin(storage, ProxyConfig::default())?;

let req = CapabilityRequirement {
    target_vendor: Some(VendorId::Akamai),
    require_ip_class: Some(IpClassRequirement::at_least(stygian_proxy::types::IpClass::Residential)),
    require_https_connect: Some(true),
    require_cdn_edge: None,
    require_tls_profile: Some("chrome_131".into()),
    require_asn: Some(20_940),              // Akamai
    require_city: Some("Cambridge".into()),
    require_postal_code: None,
    require_geo_country: None,
    require_socks5_udp: None,
    require_http3_tunnel: None,
};
let handle = mgr.acquire_with_capabilities(&req).await?;
}

If no candidate in the pool satisfies the requirement, acquire_with_capabilities returns ProxyError::NoCompatibleProxy (distinct from ProxyError::AllProxiesUnhealthy, which signals every proxy is currently circuit-open rather than structurally incompatible).

Geo metadata & ingest validation

ProxyCapabilities carries three optional geo fields that acquisition can filter on: asn: Option<u32>, city: Option<String>, postal_code: Option<String>. Proxy carries the same fields so operators can curate metadata at ingest time without going through ProxyCapabilities.

#![allow(unused)]
fn main() {
use stygian_proxy::types::well_known;
use std::sync::Arc;
use stygian_proxy::{MemoryProxyStore, ProxyConfig, ProxyManager};

let storage = Arc::new(MemoryProxyStore::default());
let mgr = ProxyManager::with_round_robin(storage, ProxyConfig::default())?;

mgr.add_proxy_with_metadata(
    "http://user:pass@proxy.example.com:8080".into(),
    well_known::KNOWN_ASN_AKAMAI,        // 20_940
    "Cambridge".into(),
    "02142".into(),
).await?;
}

Ingest validates the metadata at the call site:

• validate_asn(u32) -> Result<(), ProxyError> — rejects 0 and u32::MAX.
• validate_city(&str) -> Result<(), ProxyError> — rejects empty strings.
• validate_postal_code(&str) -> Result<(), ProxyError> — rejects empty strings.

The well_known submodule exposes canonical ASNs for the most common anti-bot vendor networks and CDN providers as pub const u32 values, with an ALL_KNOWN_ASNS: &[u32] companion slice for "is this ASN one we recognise" checks:

Vendor-specific URL quirks

Provider-specific URL formats silently break scrapers in characteristic ways. The Crawlera and Zyte :8011 plain-HTTP endpoints, for example, sit behind a TLS terminator and crash with BoringSSL WRONG_VERSION_NUMBER if the proxy adapter forwards them without TLS — failing late, deep in the TLS handshake.

vendor_quirks::check runs at ingest time and returns a Vec<QuirkMatch> describing any quirks that apply to a parsed ProxyUrl. Error-severity quirks are rejected outright (the proxy never enters the pool); warning-severity quirks are accepted and logged.

#![allow(unused)]
fn main() {
use stygian_proxy::vendor_quirks::{check, ProxyUrl, Scheme, QuirkSeverity};

let url = ProxyUrl::parse("http://proxy.crawlera.com:8011").unwrap();
for m in check(&url) {
    match m.severity {
        QuirkSeverity::Error  => eprintln!("rejected: {}", m.description),
        QuirkSeverity::Warning => tracing::warn!(?m, "vendor-quirk"),
        QuirkSeverity::Info    => tracing::info!(?m, "vendor-quirk"),
    }
}
}

The built-in VENDOR_QUIRKS table seeds four entries:

Quirk descriptions are static &'static str literals that do not echo any credential component.

Network-identity coherence

A clean fingerprint that disagrees with the egress network identity (WebRTC leaking the real IP, Accept-Language not matching the proxy country, DNS resolver living in a different jurisdiction) is the most commonly overlooked leak cited by the 2026 scraping guide. CoherencePort + DefaultCoherenceValidator checks the five-vector match at the orchestration layer before any request is sent.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use stygian_proxy::{
    CoherenceContext, CoherencePolicy, CoherenceValidator, MismatchField,
    MemoryProxyStore, ProxyConfig, ProxyManager,
};

let storage = Arc::new(MemoryProxyStore::default());
let mgr = ProxyManager::with_round_robin(storage, ProxyConfig::default())?
    .with_coherence_validator(Arc::new(CoherenceValidator::default()))?;

let ctx = CoherenceContext::from_browser_page(&page).await?;
let policy = CoherencePolicy::hard_fail_on(MismatchField::WebRtcPublicIp);
let handle = mgr.acquire_proxy_with_coherence(&ctx, &policy).await?;
}

CoherencePolicy is an advisory-or-hard-fail severity model: by default the validator logs mismatches at tracing::warn! and still issues the proxy; hard_fail_on(field) upgrades the chosen field to a hard reject. The MismatchField enum lists the five vectors: ProxyGeoVsDns, WebRtcPublicIp, Timezone, Locale, AcceptLanguage.

The validator runs in-process with zero allocation per call once the CoherenceContext is built — it is safe to run on the hot acquisition path.

Thompson-sampling Bayesian rotation

Round-robin and weighted strategies pick blindly. On protected-target workloads, that wastes requests on proxies the target has already flagged. ThompsonStrategy learns per-proxy health online and concentrates traffic on proxies whose posterior Beta(α, β) distribution favours success.

Cites 76% success vs 36% round-robin on identical proxies in the internal ProxyOps benchmark (549,114 requests over 7 days). Per-proxy counters use AtomicU64; the hot-path acquire stays sub-microsecond.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use std::time::Duration;
use stygian_proxy::{MemoryProxyStore, ProxyConfig, ProxyManager};

let storage = Arc::new(MemoryProxyStore::default());
let mgr = ProxyManager::with_thompson_sampling(
    storage,
    ProxyConfig::default(),
    Duration::from_secs(300), // decay_interval — 5 min default
)?;

// Optional: seed the bandit from a known-good feed so cold-start
// traffic is already informed.
mgr.strategy_warmup_observe(proxy_id_a, true).await;
mgr.strategy_warmup_observe(proxy_id_b, false).await;
}

When the strategy is enabled, ProxyHandle::mark_success and the implicit "failure on drop" path both feed the bandit — there is no separate observer step required at the call site. The decay_interval (default 5 min) and decay_factor (default 0.95) knobs control how quickly stale observations age out of the posterior. A prior-bias seam lets the strategy weight proxies whose TargetVendorCompatibility is higher for the target vendor.

See ThompsonStrategy for the full type.