Enum RobotsPolicy 

pub enum RobotsPolicy {
    Obey,
    IgnoreWithAudit,
    IgnoreSilently,
}

How a pipeline treats robots.txt across recon and production.

The default is RobotsPolicy::Obey — honouring robots.txt is the only choice that’s safe to ship without a written authorisation.

Example

use stygian_graph::domain::policy::RobotsPolicy;
use std::str::FromStr;

let p: RobotsPolicy = "obey".parse().unwrap();
assert_eq!(p, RobotsPolicy::Obey);
assert_eq!(p.to_string(), "obey");

Variants

Obey

Honour robots.txt. Refuse to fetch URLs the guard forbids.

The canonical choice — anything else requires explicit operator consent and audit.

IgnoreWithAudit

Ignore robots.txt but record every ignored URL with the reason. Surfaced through the pipeline run report.

IgnoreSilently

Ignore robots.txt without record. Escape hatch only.

Trait Implementations

impl Clone for RobotsPolicy

fn clone(&self) -> RobotsPolicy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for RobotsPolicy

impl Debug for RobotsPolicy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RobotsPolicy

fn default() -> RobotsPolicy

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for RobotsPolicy

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for RobotsPolicy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for RobotsPolicy

impl FromStr for RobotsPolicy

type Err = StygianError

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<Self, Self::Err>

Parses a string s to return a value of this type.

impl Hash for RobotsPolicy

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for RobotsPolicy

fn eq(&self, other: &RobotsPolicy) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for RobotsPolicy

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for RobotsPolicy