Redshank

Redshank is an autonomous recursive language-model investigation agent written in Rust. It ingests heterogeneous public datasets — campaign finance, lobbying disclosures, federal contracts, corporate registries, sanctions lists, court records, individual-person OSINT, and media intelligence — resolves entities across all of them, and surfaces non-obvious connections through evidence-backed analysis written into a live knowledge-graph wiki.

Redshank is a from-scratch Rust rewrite of OpenPlanter, replacing the Python runtime with a compiled binary that has zero Python or Node.js dependency.

Key capabilities

• 90+ fetcher modules across government, corporate, sanctions, courts, and OSINT sources
• Recursive agent engine with subtask delegation and context condensation
• Knowledge-graph wiki with petgraph DAG and fuzzy entity resolution
• Interactive ratatui TUI — session sidebar, chat log, and wiki-graph canvas
• Multi-provider LLM support — Anthropic, OpenAI, OpenRouter, Cerebras, Ollama
• Security-first architecture — fail-secure RBAC, typed AuthContext, chmod 600 credential storage
• Single compiled binary — no Python, Node.js, or runtime dependencies

Where to start

• Installation — build from source or install with cargo install redshank-cli --locked
• Quickstart — run your first investigation in five minutes
• CLI Reference — full command reference
• Architecture Overview — hexagonal DDD, CQRS, security model

Installation

From crates.io

cargo install redshank-cli --locked

From source

Requires Rust 1.94+ stable.

git clone https://github.com/greysquirr3l/redshank.git
cd redshank
cargo build --release

The binary lands at target/release/redshank.

Optional features

Build with a feature:

cargo build --release --features stygian

Quickstart

1. Set credentials

At minimum, set one LLM provider key. Redshank resolves credentials from environment variables, a .env file in the working directory, or ~/.redshank/credentials.json.

See Configuration for the full resolution order and available keys.

export ANTHROPIC_API_KEY="sk-ant-..."

2. Launch the TUI

redshank tui

The TUI opens with a session sidebar on the left, chat log in the center, and a wiki-graph canvas on the right. Type your investigation objective at the prompt and press Enter.

3. Or run a one-shot investigation

redshank run "Who are the top donors to PACs linked to defense contractors with active SAM.gov registrations?"

The agent runs autonomously, calling fetchers and writing findings to wiki/ in the working directory.

4. Inspect the wiki

ls wiki/
cat wiki/index.md

Each entity gets its own Markdown file. Cross-references between entities are tracked in the petgraph DAG and rendered in the TUI canvas.

Next steps

• Configuration — set default models, reasoning effort, and workspace options
• CLI Reference — full flag reference for redshank run, redshank fetch, and more
• Running Investigations — strategies for complex multi-source investigations

Configuration

Credential resolution order

Redshank merges credentials from all sources, with earlier sources taking priority:

1. Process environment variables (e.g. ANTHROPIC_API_KEY)
2. .env file in the current working directory
3. <workspace>/.redshank/credentials.json
4. ~/.redshank/credentials.json (user-level fallback)

Credential files are written chmod 600. Keys never appear in log output at any level.

Environment variables

Each credential is resolved with the following priority (first match wins):

1. REDSHANK_<KEY> — app-namespaced; use when running multiple different agents side-by-side
2. OPENPLANTER_<KEY> — legacy backward compatibility with the OpenPlanter predecessor
3. <KEY> — bare/global env var; sufficient for most users

Copy .env.example from the repo root and fill in the keys you need:

cp .env.example .env
chmod 600 .env

credentials.json

For persistent storage, copy credentials.example.json to .redshank/credentials.json:

mkdir -p .redshank
cp credentials.example.json .redshank/credentials.json
chmod 600 .redshank/credentials.json

The file format maps directly to the environment variable names (snake_case):

{
  "anthropic_api_key": "sk-ant-...",
  "openai_api_key": "sk-...",
  "openrouter_api_key": "sk-or-...",
  "cerebras_api_key": "...",
  "ollama_base_url": "http://localhost:11434",
  "exa_api_key": "...",
  "voyage_api_key": "...",
  "hibp_api_key": "...",
  "github_token": "ghp_...",
  "fec_api_key": "...",
  "opencorporates_api_key": "..."
}

All fields are optional. Unknown keys are silently ignored.

settings.json

Persistent settings live in <workspace>/.redshank/settings.json. Copy settings.example.json to get started:

cp settings.example.json .redshank/settings.json

{
  "default_model": "claude-sonnet-4-20250514",
  "default_reasoning_effort": "medium",
  "default_model_anthropic": "claude-sonnet-4-20250514",
  "default_model_openai": "gpt-4o",
  "default_model_openrouter": "anthropic/claude-sonnet-4",
  "default_model_cerebras": "llama-3.3-70b",
  "default_model_ollama": "llama3.2"
}

default_reasoning_effort accepts low, medium, or high. Per-provider model names override the global default_model fallback for that provider only.

Stygian fallback probe

When the stygian feature is enabled (built with --features redshank-fetchers/stygian), the CLI probes the stygian-mcp server at TUI startup using detect_stygian_availability. The probe uses hardcoded defaults:

Configuration of the probe endpoint via settings.json is planned for a future release.

The current stygian availability is reflected in the TUI footer: ▲ (green) = available, ▼ (red) = down, ? (gray) = probe not yet run.

See Stygian Fallback for setup instructions, troubleshooting, and licensing-boundary rationale.

CLI Reference

redshank run

Run a one-shot investigation.

redshank run [OPTIONS] <OBJECTIVE>

redshank tui

Launch the interactive TUI.

redshank tui [OPTIONS]

redshank fetch

Run a standalone data fetcher.

redshank fetch uk_corporate_intelligence --query <QUERY> [--output <DIR>]
redshank fetch uk-corporate-intelligence --query <QUERY> [--output <DIR>]

Currently the CLI fetch dispatcher wires the UK corporate intelligence fetcher only.

redshank session

Manage saved sessions.

redshank session list
redshank session resume <ID>
redshank session delete <ID>

redshank configure

Launch the interactive credential setup wizard.

redshank configure
redshank configure credentials
redshank setup

redshank configure credentials and redshank setup are compatibility aliases for the same interactive flow.

redshank version

Print the version and build info.

TUI Guide

Launch the TUI with redshank tui.

Layout

┌─────────────┬──────────────────────────────┬──────────────┐
│  Sessions   │          Chat Log            │ Wiki Graph   │
│             │                              │              │
│ > Session 1 │  [user] Investigate...       │  Entity A    │
│   Session 2 │  [agent] I'll start with...  │    │         │
│             │  [tool] fec_filings result   │  Entity B    │
│             │                              │              │
├─────────────┴──────────────────────────────┴──────────────┤
│ > _                                                        │
└────────────────────────────────────────────────────────────┘

• Sessions (left) — sidebar listing saved sessions. Navigate with ↑/↓, select with Enter.
• Chat Log (center) — scrolling log of the agent conversation. Scroll with ↑/↓ or PgUp/PgDn.
• Wiki Graph (right) — character-cell canvas rendering the petgraph entity DAG. Nodes are entity names; edges are relationships discovered during the investigation.
• Input (bottom) — type an objective or slash command and press Enter.

Slash commands

Running Investigations

Writing a good objective

The objective should be specific enough for the agent to know when it’s done, but open enough to allow autonomous tool selection.

Too vague:

Tell me about this company.

Too prescriptive:

Look up Acme Corp in GLEIF, then OpenCorporates, then FinCEN BOI, then cross-reference with OFAC.

Just right:

Identify all beneficial owners of Acme Corp and any related entities that appear on OFAC, UN, or EU sanctions lists.

Fetcher chaining

The agent decides which fetchers to call based on the objective. Common patterns:

Corporate investigation: GLEIF (LEI) → OpenCorporates (subsidiaries) → FinCEN BOI (beneficial owners) → OFAC/UN/EU (sanctions check) → SEC EDGAR (filings) → state SOS portals

Political finance: FEC (contributions) → Senate/House lobbying disclosures → USASpending (contracts) → SAM.gov (registrations) → FPDS (awards)

Individual OSINT: HIBP (breach exposure) → GitHub profile → WHOIS/RDAP → Wayback Machine → voter rolls → USPTO inventors

Multi-depth investigations

Use --max-depth to allow the agent to spin up child agents for subtasks:

redshank run --max-depth 3 "Map the full ownership network behind the top 10 SAM.gov contractors in the defense sector"

Each child invocation gets its own step budget and writes its findings back to the shared wiki.

Reviewing results

All findings are written to wiki/ in the working directory:

• wiki/index.md — master index of all entities discovered
• wiki/<entity-slug>.md — per-entity pages with sourced claims and cross-references

The wiki is plain Markdown and can be committed to a repository, rendered with any static site generator, or read directly.

Data Sources Overview

Redshank ships 90+ fetcher modules across campaign finance, contracts, corporate registries, sanctions, courts, OSINT, environmental, and media categories.

The redshank fetch CLI dispatcher currently exposes UK corporate intelligence only; additional fetchers are available as crate modules and are being expanded into the dispatcher over time.

Categories

Running a fetcher directly

redshank fetch uk_corporate_intelligence --query "Acme Holdings" --output ./out
redshank fetch uk-corporate-intelligence --query "Acme Holdings" --output ./out

The CLI writes NDJSON files to the chosen output directory (one JSON object per line), suitable for ingestion with tools such as jq.

Rate limits and credentials

Each fetcher documents its rate limits and required credentials in its own page. All fetchers apply exponential backoff on 429 responses.

Campaign Finance

Note: redshank fetch CLI dispatch currently exposes uk_corporate_intelligence only. The command snippets on this page document fetcher IDs and expected query shapes as dispatcher targets are expanded.

FEC

Fetches campaign contributions, expenditures, and committee filings from the FEC bulk data API.

Credential: FEC_API_KEY

redshank fetch fec --name "ACME CORP" --type committee
redshank fetch fec --candidate "John Smith" --cycle 2024

Senate Lobbying Disclosures

Fetches lobbying registrations and activity reports from the Senate LDA system.

redshank fetch senate_lobbying --registrant "Acme Lobbying LLC"

House Lobbying Disclosures

Fetches House of Representatives lobbying disclosure forms from the House Clerk system.

redshank fetch house_lobbying --registrant "Acme Lobbying LLC"

Government Contracts

Note: redshank fetch CLI dispatch currently exposes uk_corporate_intelligence only. The command snippets on this page document fetcher IDs and expected query shapes as dispatcher targets are expanded.

USASpending

Federal contract and grant awards from USASpending.gov.

redshank fetch usaspending --recipient "Acme Corp" --award-type contract

SAM.gov

Entity registrations and exclusions from SAM.gov.

redshank fetch sam_gov --name "Acme Corp"

FPDS

Contract awards from the Federal Procurement Data System.

redshank fetch fpds --vendor "Acme Corp" --naics 541511

Federal Audit Clearinghouse

Single audit findings from the Federal Audit Clearinghouse.

redshank fetch federal_audit --ein "12-3456789"

Corporate Registries

Note: redshank fetch CLI dispatch currently exposes uk_corporate_intelligence only. The command snippets on this page document fetcher IDs and expected query shapes as dispatcher targets are expanded.

GLEIF

Global LEI (Legal Entity Identifier) lookups via the GLEIF API.

redshank fetch gleif --lei "529900T8BM49AURSDO55"
redshank fetch gleif --name "Acme Corp"

OpenCorporates

Company search and officer/subsidiary data via OpenCorporates.

Credential: OPENCORPORATES_API_KEY

redshank fetch opencorporates --name "Acme Corp" --jurisdiction us_de

Licence — ODbL 1.0 (attribution required)

OpenCorporates data is published under the Open Database Licence (ODbL 1.0). Every report, wiki entry, or UI element that surfaces OpenCorporates data must include a visible hyperlink:

from OpenCorporates — or the canonical entity URL returned in the API response

Attribution rules from opencorporates.com/terms-of-use-2:

• The link text must read “from OpenCorporates” and must resolve to the OpenCorporates homepage or the specific entity page (prefer the entity URL when available — it is returned in every API response).
• The link must be at least 70 % the size of the largest font used for the related information, and never smaller than 7 px, whichever of the two is larger.
• If OpenCorporates data forms the substantial part of a web page, add <link rel="canonical" href="{entity_url}"> so search engines treat the OpenCorporates page as the authoritative source.
• If you expose this data through your own API, your downstream consumers inherit the same obligations.

How attribution propagates in redshank

FetchOutput for the opencorporates fetcher carries a populated attribution field:

#![allow(unused)]
fn main() {
pub struct Attribution {
    pub source: String,         // "OpenCorporates"
    pub text: String,           // "from OpenCorporates"
    pub url: String,            // "https://opencorporates.com"
    pub min_font_size_px: u8,   // 7
    pub licence: String,        // "ODbL-1.0"
}
}

The report and wiki-graph layers read FetchOutput::attribution and must render the hyperlink before writing any entity page that includes OpenCorporates fields. If the url field in a specific company record differs (i.e. the canonical entity URL was returned by the API), use that URL instead of the homepage.

FinCEN BOI

Beneficial Ownership Information filings from FinCEN.

redshank fetch fincen_boi --name "Acme Corp"

State SOS Portals

Secretary of State business registry searches for Delaware, Florida, Nevada, and Wyoming. Pipelines configured in redshank-fetchers/pipelines/state_sos/.

redshank fetch state_sos --state DE --name "Acme Corp"

UK Corporate Intelligence

Merged UK company enrichment combining Companies House with OpenCorporates.

Credentials: UK_COMPANIES_HOUSE_API_KEY, optional OPENCORPORATES_API_KEY

redshank fetch uk_corporate_intelligence --query "Acme Holdings"

When OpenCorporates contributes data, the same ODbL attribution rules above apply to downstream reports and UI.

SEC EDGAR

Company filings, ownership reports, and insider transactions via EDGAR.

redshank fetch sec_edgar --name "Acme Corp"
redshank fetch sec_edgar --cik 0001234567 --form 13F

FDIC

Bank and institution data from the FDIC BankFind Suite.

redshank fetch fdic --name "First National Bank"

ProPublica Nonprofit 990

IRS Form 990 filings via ProPublica Nonprofit Explorer.

redshank fetch propublica_990 --ein "12-3456789"

Sanctions

Note: redshank fetch CLI dispatch currently exposes uk_corporate_intelligence only. The command snippets on this page document fetcher IDs and expected query shapes as dispatcher targets are expanded.

OFAC SDN

Office of Foreign Assets Control Specially Designated Nationals list.

redshank fetch ofac_sdn --name "Ivan Petrov"

UN Consolidated Sanctions

United Nations Security Council consolidated sanctions list.

redshank fetch un_sanctions --name "Ivan Petrov"

EU Sanctions

European Union consolidated sanctions list.

redshank fetch eu_sanctions --name "Petrov"

World Bank Debarred Firms

Firms and individuals debarred from World Bank Group projects.

redshank fetch world_bank_debarred --name "Acme Corp"

Courts & Leaks

Note: redshank fetch CLI dispatch currently exposes uk_corporate_intelligence only. The command snippets on this page document fetcher IDs and expected query shapes as dispatcher targets are expanded.

CourtListener (RECAP)

Federal court dockets and documents via the CourtListener RECAP API.

redshank fetch courtlistener --party "Acme Corp"
redshank fetch courtlistener --docket 1234567

ICIJ Offshore Leaks

Entities from the Panama Papers, Pandora Papers, and other ICIJ leak databases via the ICIJ Offshore Leaks API.

redshank fetch icij_leaks --name "Acme Corp"

Individual OSINT

Note: The following individual-person OSINT fetchers are available via the CLI and TUI. Use the command snippets below to invoke each fetcher, or browse the Sources tab in the TUI for real-time availability.

HIBP — Have I Been Pwned

Breach exposure lookup via the HIBP API.

Credential: HIBP_API_KEY

redshank fetch hibp --query "user@example.com"

Alias source ID (same paid API flow):

redshank fetch haveibeenpwnd --query "user@example.com"

GitHub Profiles

Public profile, repositories, and contribution history via the GitHub API.

redshank fetch github_profile --username "octocat"

GitLab Profiles

Public GitLab profile search via the GitLab Users API.

redshank fetch gitlab_profile --query "jane investigator"

Stack Exchange Profiles

Public Stack Overflow/Stack Exchange profile lookup via the Stack Exchange API.

redshank fetch stackexchange_profile --query "Jane Investigator"

Wayback Machine

Historical snapshots of a domain or URL via the Wayback CDX API.

redshank fetch wayback --url "example.com"

WHOIS / RDAP

Domain registration and RDAP lookups.

redshank fetch whois_rdap --domain "example.com"

Voter Registration

Voter roll data from state portals. Currently supports North Carolina.

redshank fetch voter_reg --state NC --last-name "Smith" --first-name "John"

USPTO Patent & Trademark

Patent and trademark filings with inventor/applicant search via the USPTO API.

redshank fetch uspto --inventor-last "Smith" --inventor-first "John"

Username Enumeration

Check username availability across 37+ platforms. Platforms configured in redshank-fetchers/pipelines/username_enum/platforms.toml.

redshank fetch username_enum --username "jsmith"

Social Profiles

Social media profile enumeration across configured platforms. Pipelines in redshank-fetchers/pipelines/social_profiles/.

redshank fetch social_profiles --username "jsmith"

Reverse Phone (Basic)

Best-effort phone normalization and country-code metadata hints (no paid identity lookup).

redshank fetch reverse_phone_basic --query "+1 415 555 2671"

Reverse Phone (Twilio)

Carrier and line-type enrichment via the Twilio Lookup API.

Credentials: TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN

redshank fetch reverse_phone_twilio --query "+1 415 555 2671"

Reverse Phone (Truecaller)

Paid subscriber enrichment (name/address/carrier metadata) via the Truecaller API.

Credential: TRUECALLER_API_KEY

redshank fetch reverse_phone_truecaller --query "+1 415 555 2671"

Reverse Address (Public)

Public address normalization/geocoding via the U.S. Census geocoder.

redshank fetch reverse_address_public --query "1600 Pennsylvania Ave NW, Washington, DC"

Environmental & Reference

Note: redshank fetch CLI dispatch currently exposes uk_corporate_intelligence only. The command snippets on this page document fetcher IDs and expected query shapes as dispatcher targets are expanded.

EPA ECHO

Compliance and enforcement data from the EPA ECHO system.

redshank fetch epa_echo --name "Acme Manufacturing"

OSHA Inspections

Workplace inspection records from the OSHA API.

redshank fetch osha --establishment "Acme Corp"

Census ACS

American Community Survey demographic data via the Census API.

redshank fetch census_acs --fips 12086 --variables B01001_001E,B19013_001E

Wikidata

Entity facts and relationships via the Wikidata SPARQL endpoint.

redshank fetch wikidata --entity "Q312"
redshank fetch wikidata --sparql "SELECT ?item WHERE { ?item wdt:P31 wd:Q6256 }"

GDELT

Global media event and tone analysis via the GDELT 2.0 API.

redshank fetch gdelt --query "Acme Corp" --timespan 1month

Architecture Overview

Redshank is structured as a Cargo workspace of four crates:

Internal layout (redshank-core)

The internal layout follows hexagonal DDD with explicit CQRS:

src/
  domain/          # Pure types — zero I/O, zero async
  ports/           # Trait interfaces (inbound + outbound)
  application/
    commands/      # Mutating CQRS handlers + IdempotencyKey
    queries/       # Read-only CQRS handlers
    services/      # Orchestration (agent engine, condensation)
  adapters/
    providers/     # LLM provider impls (Anthropic, OpenAI-compat)
    tools/         # WorkspaceTools (filesystem, shell, web, patching)
    persistence/   # SQLite session store

Dependency rule

No domain type may reference an adapter or application type. The compiler enforces this: redshank-core/Cargo.toml has zero I/O crates (tokio, reqwest, sqlx) as non-optional direct dependencies.

Further reading

• Hexagonal DDD
• CQRS & Domain Events
• Security Model
• Agent Engine
• Wiki Graph

Hexagonal DDD

Redshank uses a DDD-Lite hexagonal architecture. The key invariant is the dependency rule: inner rings know nothing about outer rings.

          ┌─────────────────────────────────────┐
          │           Adapters (outer)           │
          │  ┌───────────────────────────────┐  │
          │  │      Application layer        │  │
          │  │  ┌─────────────────────────┐  │  │
          │  │  │    Domain (inner)        │  │  │
          │  │  │  Types, events, rules    │  │  │
          │  │  └─────────────────────────┘  │  │
          │  │  Commands / Queries / Services │  │
          │  └───────────────────────────────┘  │
          │  Providers / Tools / Persistence     │
          └─────────────────────────────────────┘

Domain layer (src/domain/)

Pure Rust structs and enums. No async, no tokio, no I/O of any kind.

• agent.rs — AgentSession aggregate root, AgentConfig, ProviderKind
• auth.rs — AuthContext, Role, Permission, SecurityPolicy
• credentials.rs — CredentialBundle, resolution order
• errors.rs — DomainError enum (all thiserror)
• events.rs — DomainEvent variants
• session.rs — session value objects
• settings.rs — PersistentSettings
• wiki.rs — WikiEntry, WikiEntryId

Ports layer (src/ports/)

Trait interfaces that the domain and application layers call. Adapters implement these traits; the domain never sees the implementations.

• ModelProvider — LLM completions with streaming
• SessionStore — session persistence (CQRS-aware)
• WikiStore — wiki entry persistence
• ToolDispatcher — tool invocation
• ReplayLog — JSONL delta-encoded call log
• FetcherPort — data fetcher abstraction

CQRS & Domain Events

Command handlers

Every mutating operation is a Command struct carrying an IdempotencyKey (newtype Uuid v4), handled by a CommandHandler in application/commands/.

#![allow(unused)]
fn main() {
pub struct RunInvestigationCommand {
    pub idempotency_key: IdempotencyKey,
    pub objective: String,
    pub config: AgentConfig,
    pub auth: AuthContext,
}
}

Handlers check the idempotency_keys table before executing. Duplicate commands return the cached result without re-running.

Query handlers

Every read operation is a Query struct handled by a QueryHandler in application/queries/. Queries never mutate state.

Domain events

Every significant state transition emits a typed DomainEvent variant:

Events are immutable value types. Aggregate methods append them to a pending_events: Vec<DomainEvent>; the session store persists them via append_event.

Security Model

Default deny

Every port method that accesses or mutates keyed data accepts auth: &AuthContext and calls SecurityPolicy::check before any data access. The default is Err(SecurityError::AccessDenied) unless the policy explicitly grants the required Permission.

Roles and permissions

AuthContext

#![allow(unused)]
fn main() {
pub struct AuthContext {
    pub user_id: UserId,
    pub role: Role,
}
}

AuthContext is created at the CLI/TUI boundary and propagated through every command and query handler. It never crosses crate boundaries as a raw struct — it’s always validated at the port layer.

Credential security

• Credentials are stored in ~/.redshank/credentials.json (user level) or <workspace>/.redshank/credentials.json (workspace level), both written chmod 600.
• Keys never appear in log output at any level (enforced by Debug impls that redact sensitive fields).
• No credential is ever written to disk with broader permissions than 0o600.

Security rules location

All policy logic lives in redshank-core/src/domain/auth.rs as pure functions — no I/O, no async. This makes the security model independently testable without any infrastructure.

Agent Engine

The core agent loop lives in redshank-core/src/application/services/engine.rs.

Loop structure

RunInvestigationCommand
  └─► RunInvestigationHandler
        └─► RLMEngine::solve(objective)
              └─► solve_recursive(objective, depth=0)
                    ├─ model.complete(messages, tools)
                    ├─ process_tool_calls(...)
                    │    ├─ tool_dispatcher.dispatch(tool_name, args)
                    │    └─ solve_recursive(subtask, depth+1)  [for delegate_task]
                    └─ condense_tool_outputs() [if token budget > 75%]

Key parameters

Context condensation

When token usage exceeds 75% of the model’s context window, condense_tool_outputs truncates older tool results in place, keeping the original objective and recent messages intact.

Acceptance criteria

An optional judge model evaluates the final answer against the original objective. The judge uses a lightweight model (configurable) to return pass or fail with a rationale. On fail, the engine attempts one more solve pass with the judge’s feedback injected.

Cancellation

The engine checks a CancellationToken at the start of each step (when built with the runtime feature). Cancellation is clean — the engine returns the best answer accumulated so far.

Wiki Graph

The wiki graph tracks entities and their relationships across all findings written during an investigation.

Storage

Each entity is a Markdown file in wiki/. The master index is wiki/index.md. Cross-references between entities use bold entity names (**Entity Name**) which the graph parser extracts.

Graph structure

Redshank uses petgraph::DiGraph to represent the entity graph:

• Nodes — WikiEntry structs (entity name, slug, last-updated timestamp)
• Edges — directed relationships extracted from cross-reference mentions

Fuzzy matching (Levenshtein distance) resolves name variants across different data sources (e.g. “ACME CORPORATION” and “Acme Corp” merge into one node).

Rendering in the TUI

The TUI’s right-hand pane renders the graph as a character-cell canvas using a simple force-directed layout. Nodes are entity name labels; edges are drawn with ─, │, ╭, ╰ box-drawing characters.

Exporting

redshank wiki export --format dot > graph.dot
dot -Tsvg graph.dot > graph.svg

The DOT export is compatible with Graphviz for publication-quality rendering.

Stygian MCP Fallback

Redshank fetches from two broad categories of data sources:

• JSON/REST sources — respond to plain HTTP requests; fetched with reqwest.
• JS-heavy sources — government portals, SOS registries, and social platforms that require a real browser to render state before content is extractable.

For JS-heavy sources, redshank uses an optional stygian fallback: when the stygian-mcp server is running and healthy, fetch operations are delegated to it. When it is unavailable, the fetcher fails-soft (logs a warning and returns an empty result set) instead of hard-crashing the investigation.

Decision flow

fetch(source)
  ├── is_js_heavy? ──NO──▶ NativeHttp (reqwest)
  └── YES
       ├── stygian reachable? ──YES──▶ StygianMcpFallback
       └── NO──▶ FailSoft (warn + return empty)

This logic lives in redshank-fetchers/src/fallback.rs::select_execution_mode. Each JS-heavy fetcher module exposes a thin wrapper:

All three delegate immediately to select_execution_mode. The policy stays in one place; the per-source wrapper communicates domain intent (whether that source is JS-heavy).

Feature flag

Stygian is an opt-in feature. The redshank-fetchers crate compiles without it by default:

# Without stygian (default) — JS-heavy fetchers always fail-soft
cargo build --workspace

# With stygian
cargo build --workspace --features redshank-fetchers/stygian

When the stygian feature is absent at compile time, detect_stygian_availability short-circuits and returns StygianAvailability::Unavailable(FeatureDisabled) without attempting any network connection. No stale connection errors; no silent DNS hits.

Runtime probing

At TUI startup, the CLI calls detect_stygian_availability(&StygianProbeConfig) and emits AppEvent::FetcherHealthChanged before the command loop begins. The probe config drives the health check:

The probe accepts any HTTP 2xx response whose body contains one of: "ok", "healthy", status:ok, or the bare string ok.

The result is broadcast as AppEvent::FetcherHealthChanged(FetcherHealth) so the TUI can update the footer indicator without the probe code knowing anything about ratatui.

TUI health indicator

The footer of the interactive TUI shows the current stygian health:

Setup

Local development

1. Install stygian-mcp:

   cargo install stygian-mcp --locked
   

2. Start the server on the default port:

   stygian-mcp --port 8787
   

3. Build redshank with stygian enabled:

   cargo build --workspace --features redshank-fetchers/stygian
   

4. Confirm the TUI footer shows stygian: ▲.

Production / headless

For server deployments, run stygian-mcp as a managed service (systemd example):

[Unit]
Description=stygian-mcp browser automation server
After=network.target

[Service]
ExecStart=/usr/local/bin/stygian-mcp --port 8787
Restart=on-failure
RestartSec=5s
User=stygian

[Install]
WantedBy=multi-user.target

Redshank currently probes http://127.0.0.1:8787/health by default. Run stygian-mcp on that host/port, or build a custom binary that sets a different StygianProbeConfig. Environment-variable and settings.json overrides for the probe endpoint are planned for a future release.

Troubleshooting

Licensing boundary

Stygian is used via MCP server rather than as a direct Cargo crate dependency. This boundary is intentional even though both redshank and stygian share the same author.

Rationale:

• stygian-browser requires a running Chrome/Chromium instance accessed via CDP. Linking it directly into redshank would force every user to install Chrome regardless of whether they need JS-heavy fetchers. The MCP boundary keeps the Chrome dependency in a separate process that operators can opt into.
• Process isolation: a crash or OOM in the browser automation process cannot corrupt the investigation agent’s in-memory state or SQLite session store.
• Independent restarts: the stygian-mcp server can be restarted, updated, or swapped out without recompiling or restarting redshank.

When to use MCP indirection (general criteria):

1. The dependency requires a native runtime or process that should be optional (Chrome, GPU driver, external daemon).
2. Process-boundary isolation is operationally valuable (crash containment, independent restarts, separate resource limits).
3. The capability is genuinely optional — callers should work without it.

When direct Cargo crate linking is fine:

1. Pure Rust or C library with no separate runtime process.
2. No binary-size or isolation concern.
3. All downstream users can reasonably be expected to have its native dependencies.

Contributing

Prerequisites

• Rust 1.94+ stable (rustup toolchain install 1.94)
• cargo-deny (cargo install cargo-deny)

Setup

git clone https://github.com/greysquirr3l/redshank.git
cd redshank
cargo build --workspace

Workflow

1. Create a branch: git checkout -b feat/my-feature
2. Write a failing test first (TDD strategy)
3. Implement until the test passes
4. Run the full suite: cargo test --workspace
5. Check lint: cargo clippy --workspace -- -D warnings
6. Check deps: cargo deny check
7. Commit with a conventional commit message
8. Open a pull request

Commit conventions

Focus commit messages on user impact, not file counts or line changes.

Testing

Running tests

# Full workspace
cargo test --workspace

# Single crate
cargo test -p redshank-core
cargo test -p redshank-fetchers

Test organisation

TDD policy

Write a failing test before implementation code. Every public function needs at least one test. All tests must pass before a task is marked complete.

Integration test

redshank-core/tests/integration.rs runs a full agent loop using a scripted MockModelProvider that returns deterministic tool calls and a final answer. No network access required.

Adding a Fetcher

This guide walks through adding a new data fetcher to redshank-fetchers.

1. Add the module

Create redshank-fetchers/src/fetchers/my_source.rs.

2. Implement the struct

use crate::{client::FetcherClient, domain::FetcherError};
use serde::{Deserialize, Serialize};

/// Fetches data from My Source.
pub struct MySourceFetcher {
    client: FetcherClient,
}

impl MySourceFetcher {
    /// Creates a new fetcher with the given HTTP client.
    #[must_use]
    pub const fn new(client: FetcherClient) -> Self {
        Self { client }
    }

    /// Fetch records matching `query`.
    ///
    /// # Errors
    ///
    /// Returns [`FetcherError`] on HTTP or parse failure.
    pub async fn fetch(&self, query: &str) -> Result<Vec<MySourceRecord>, FetcherError> {
        // ...
    }
}

#[derive(Debug, Serialize, Deserialize)]
pub struct MySourceRecord {
    pub id: String,
    pub name: String,
}

3. Write a test first

#[cfg(test)]
mod tests {
    use super::*;
    use wiremock::{MockServer, Mock, ResponseTemplate};
    use wiremock::matchers::{method, path};

    #[tokio::test]
    async fn test_fetch_returns_records() {
        let server = MockServer::start().await;
        Mock::given(method("GET"))
            .and(path("/api/search"))
            .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!([
                {"id": "1", "name": "Test Entity"}
            ])))
            .mount(&server)
            .await;

        let client = FetcherClient::with_base_url(server.uri());
        let fetcher = MySourceFetcher::new(client);
        let results = fetcher.fetch("Test Entity").await.unwrap();
        assert_eq!(results.len(), 1);
        assert_eq!(results[0].name, "Test Entity");
    }
}

4. Register the module

In redshank-fetchers/src/fetchers/mod.rs:

pub mod my_source;

5. Add a CLI subcommand

In redshank-cli/src/main.rs, add a variant to the FetchCommand enum and wire it to MySourceFetcher::fetch.

6. Document it

Add an entry to Data Sources Overview and create a page in docs/src/data-sources/.