Semantic Pipelines

Semantic pipelines describe how contracted data moves through Guida. They join a source or extractor to validation gates, mappers, and destinations, then make the proposed work inspectable before anything is executed.

A pipeline is not a workflow folder, worker pool, or unattended scheduler. Workflows activate operational scope; workers consume queues; the Workflow Ledger records durable run and item truth. A pipeline connects those surfaces with explicit provenance.

Stage model

Stage	Responsibility
Source	Declares the incoming type or schema
Extractor	Runs a named extractor contract
Validator	Applies a schema and bounded item rules
Mapper	Transforms one declared contract into another
Destination	Names the queue or other supported target

Validation gates, mappers, and destinations are catalogs of their own. Reusing a named component keeps policy and data-shape decisions out of ad hoc scripts.

{
  "id": "careers.example.discovery-to-fetch",
  "workflowName": "careers-crawl",
  "status": "draft",
  "stages": [
    {
      "id": "discover",
      "kind": "extractor",
      "extractorId": "careers.example.jobs"
    },
    {
      "id": "validate",
      "kind": "validator",
      "schema": "schemas/career-job-output.schema.json",
      "validation": {
        "itemRules": {
          "requiredFields": ["url"],
          "urlFields": ["url"],
          "dedupeKey": "url"
        }
      }
    },
    {
      "id": "map",
      "kind": "mapper",
      "mapperId": "careers.example.jobs.fetchPayload"
    },
    {
      "id": "enqueue",
      "kind": "queue",
      "destinationId": "careers.fetch.queue"
    }
  ]
}

Authored, templated, and generated

You can author a pipeline directly or define a reusable template. A generated rule selects descriptors and binds template values explicitly:

{
  "template": "careers.discovery-to-fetch",
  "selector": {
    "domain": "careers",
    "source": { "kind": "web" },
    "certifiable": true
  },
  "idPattern": "${extractor.id}.discovery-to-fetch",
  "bindings": {
    "extractorId": "$extractor.id",
    "companyName": "$extractor.input.defaultCompanyName",
    "sourcePortalUrl": "$extractor.source.url"
  }
}

Bindings are data, not guesses. Guida does not derive a company, destination, or contract from an ID, filename, or directory name.

Safe path to execution

1. Draft and diff resolve the effective pipeline and show proposed file changes.
2. Validate checks structure, component references, contracts, and graph rules.
3. Preflight checks current workspace and operational prerequisites.
4. Plan reports stages, edges, expected mutations, required capabilities, policy and credential requirements, provenance, recovery information, blockers, and warnings.
5. Preview returns a hash for the exact proposal and states whether it is ready to execute.
6. Start accepts bounded input and queue limits. High-impact operations keep their confirmation and policy gates.

An upsert or delete changes workspace files and is never an implicit side effect of opening an editor. Starting a run is likewise separate from authoring.

Execution, provenance, and recovery

Pipeline execution creates an explicit workflow-run relationship. Queue destinations declare whether to create an isolated run or use an existing one; they do not attach work through name-based inference. Guida records stage state, dependencies, output references, deduplication decisions, and operational outcomes.

workspace.pipelines.execution.lifecycle.get returns the pipeline-owned run state and stage progression. Downstream preview shows the queues and mutations that would follow; downstream start and control remain separate gated actions. Queue and ledger results are reconciled explicitly, so an empty queue alone does not prove that a run succeeded.

Recovery uses durable queue and ledger facts. Preview a recovery or control action, review the affected run and items, then execute the bounded proposal.

Remote operation

The Remote Ops API provides list, get, draft, diff, validate, preflight, plan, preview, start, result, evidence, downstream, and control contracts. Guida Control builds its pipeline workspace on those contracts and never starts a pipeline merely because an operator saved a draft.