Module pipeline_parser 

Advanced TOML pipeline definitions: parsing, template expansion, DAG validation, DOT/Mermaid export Advanced TOML pipeline definition parser & validator

Supports [[nodes]] and [[services]] TOML blocks that describe scraping Directed Acyclic Graphs (DAGs). Includes:

• Layered config loading: TOML file → environment overrides
• Template variable expansion: ${env:VAR}, ${node:NAME.field}
• DAG cycle detection via DFS
• Service reference validation against the registry
• Graph visualization export (Graphviz DOT and Mermaid)

TOML format

[[services]]
name = "fetcher"
kind = "http"

[[services]]
name = "extractor"
kind = "claude"
api_key = "${env:ANTHROPIC_API_KEY}"

[[nodes]]
name = "fetch"
service = "fetcher"
url = "https://example.com"

[[nodes]]
name = "extract"
service = "extractor"
depends_on = ["fetch"]

Example

use stygian_graph::application::pipeline_parser::{PipelineParser, PipelineDefinition};

let toml = r#"
[[nodes]]
name = "a"
service = "http"
url = "https://example.com"

[[nodes]]
name = "b"
depends_on = ["a"]
service = "http"
url = "https://example.com"
"#;

let def = PipelineParser::from_str(toml).unwrap();
assert_eq!(def.nodes.len(), 2);
assert!(def.validate().is_ok());

Structs

NodeDecl

A single node in the pipeline DAG

PipelineDefinition

Top-level pipeline definition

PipelineParser

TOML pipeline parser

PipelineWatcher

Hot-reload watcher for pipeline definition files.

ServiceDecl

A service adapter declaration in the TOML config

Enums

PipelineError

Errors produced during pipeline parsing and validation