The knowledge ingestion pipeline

When a document is uploaded, whether to the operator Global Library or to an organization's Policy Hub, it runs through a multi-stage pipeline that ends with embedded, searchable chunks. The pipeline lives in @repo/ingestion, is driven by @repo/jobs, and is stored in the ingestionJob and kbChunk tables.

Upload and registration

1. The client requests a presigned upload URL and uploads the file directly to object storage.
2. The server registers a knowledgeDoc (status pending) recording the storage key, MIME type, size, scope, and category.
3. An ingestionJob is created in the queued stage and processing begins.

Stages

The job's stage column is the state machine. The happy path runs through:

There are also two terminal failure stages:

• failed: processing exhausted its retries.
• dead_letter: an unrecoverable condition, such as the document being deleted mid-flight or its metadata being invalid.

The document's own status is a simpler view of this. It reads pending while in flight, indexed on success, and failed once the job gives up.

Supported file types

An unsupported MIME type fails the parse stage with a clear error instead of producing empty content.

Embedding

Embeddings are generated through @repo/ai, which resolves a provider and model. By default this is OpenRouter with a 1536-dimension model. In development only, you can use a local Ollama model by setting EMBEDDING_PROVIDER=ollama while NODE_ENV is not production.

Chunks are embedded in batches with bounded concurrency. Safety guards cap the chunk count and total bytes per document so one oversized upload can't run away. The embedding run is recorded in aiUsageLog for cost tracking. The produced vector dimension is checked against the configured dimension; a mismatch fails the job rather than corrupting the index.

Failure handling and retries

When a stage throws, the worker:

1. Increments attempts and records lastError.
2. If attempts is below maxAttempts (default 5), schedules a retry with exponential back-off and jitter, capped at roughly 15 minutes, via nextRunAt. Failures early in the pipeline retry from queued; failures during embedding or indexing retry from embedding.
3. Once retries are exhausted, moves the job to failed, or to dead_letter for unrecoverable conditions, and sets the document's status to failed.

Some conditions skip retries and dead-letter immediately, for example a missing document, a missing storage key, or an embedding-dimension mismatch.

Health and observability

The pipeline exposes a health summary used by the operator Ingestion Health panel and the GP Knowledge settings:

An operator can retry failed jobs from the Command Center. See ingestion troubleshooting.