Storage & indexes
Qdrant global retrieval, incremental synchronization, verified external purge and collection cleanup are implemented. Production-scale acceptance and backup/restore erasure guarantees remain to be demonstrated.
Code and tests
- apps/kesita-app/platform/db/lineage-policies.sql
- services/product-api/internal/workspace/lineage_integration_test.go
- services/product-worker/internal/capabilities/documents_integration_test.go
- services/legal-core-api/internal/vectorindex/projector.go
- services/legal-core-api/internal/vectorindex/sync.go
- services/legal-core-api/internal/vectorindex/lifecycle.go
- services/legal-core-api/internal/vectorindex/lifecycle_integration_test.go
- services/legal-core-api/internal/vectorindex/client_test.go
- services/legal-core-api/internal/search/retrieval_integration_test.go
- services/legal-core-api/schema/target.sql
This page owns publication and erasure behavior. Product and legal storage both separate source content from derived search data, but their cleanup implementations do not yet provide every target retention guarantee.
Legal search storage
Section titled “Legal search storage”Implemented boundary: the Go API uses Qdrant for shared-corpus semantic/hybrid candidates, with PostgreSQL as canonical authority and hydration store. PostgreSQL FTS remains the explicit lexical mode; pgvector and optional pg_search remain a differential-test backend and vector rebuild source, not an automatic serving fallback. Private-document storage is outside this change. Existing deployed/reference collections are untouched.
| Store | Responsibility |
|---|---|
| Legal PostgreSQL | Resources, external-ID mappings, original-text snapshots, sections, legal validity, complete enrichments, parties/roles, citations, classification assignments, profiles, publication manifests and jobs |
| Qdrant | Dense and sparse vectors, IDs needed to hydrate a hit, publication/revision markers and the allowlisted filter projection below |
| Blob storage | Verified original bytes and retained artifacts; not retrieval payloads |
An embedding input need not be stored alongside its vector. Exact summary/article/act-profile input, input recipe and hash remain in PostgreSQL. Fetch the candidate summaries, outcome descriptions and normative texts in batches after retrieval, from the pinned owners, before reranking or answering. Do not replace faithful reranker inputs with short Qdrant snippets.
Collections and identity
Section titled “Collections and identity”Keep judgment-summary, act-profile and article-text candidate pools separate. Route each pool to a profile-specific collection through a PostgreSQL registry; collection names, vector names, dimensions, distance and sparse recipe belong to that registry, not repeated model descriptions in each payload. A collection contains one immutable compatible dense/sparse recipe pair. Separate experiment arms must not mix vector spaces or lexical statistics.
A global point ID is the global representation ID, not just the resource ID: multiple summaries, snapshots and profiles must coexist without overwriting each other. Later passage collections use the chunk ID and carry snapshot/generation references; retaining global search does not remove chunk embeddings from the design. Classification prototype vectors may have a separate collection when nearest-prototype lookup is implemented; this does not reintroduce clustering. Full prototype metadata and assignments remain in PostgreSQL.
The SQL ER retains representation/owner/profile relationships and vector columns used as rebuild artifacts. search.qdrant_builds selects a collection per representation/profile and records its published watermark. search.projection_clock tracks eligibility changes; search.projection_changes coalesces affected resources. The serving dense/sparse vectors are in Qdrant. SQL vectors are not an implicit second search authority.
Payload allowlist
Section titled “Payload allowlist”Only fields used for hit identity, eligibility, filtering or projection freshness belong in the payload. Do not copy a canonical row wholesale. Fields marked optional are materialized only for an implemented consumer; adding one requires its query semantics, payload index and parity test. Missing values remain missing, not zero, an empty string or an inferred default.
The implemented subset is defined by internal/vectorindex.Payload, build_rows.sql and Client.Create: resource/source/enrichment identity, eligibility/revision, grade/year/date, authority/jurisdiction, parent act and normative validity. Outcome, participant, semantic-classification and restricted-code filters below remain target fields until their consumers are implemented. The examples describe the full allowlist, not a claim that all specialist filters already exist.
| Scope | Fields | Purpose / index contract |
|---|---|---|
| Every point | resource_id, source_version_id | Keyword/UUID identity filters and batched source hydration; version refers to original text, not embedding input |
| Every point | eligible, projection_revision | Boolean publication/access filter and monotonic PostgreSQL projection revision; revision is checked on hydration, not a relevance feature |
| Judgment summary | enrichment_id | Exact retained summary/outcome owner; original text still resolves through source_version_id |
| Judgment filters | grade, year, decision_date, authority_code, jurisdiction_code | Keyword, integer and date indexes; preserve stored grade values; year and decision date remain distinct where the source only provides a year |
| Judgment filters | outcome | Integer index for the reference outcome filter, with explicit mapping of rejected/partial/accepted; unknown is absent, never rejected by default |
| Judgment filters | matter, section, seat | Minimal strings consumed by existing text/exact filters; configure text and/or keyword indexes to preserve each operation’s semantics |
| Judgment participants | party_keys, president_name, reporter_name | Exact keyword-array party matching and existing judge-name text filters; no complete participant objects |
| Judgment classification | semantic_matter_ids, classification_set_id | Keyword arrays/ID for semantic-category inclusion/exclusion in cross-domain retrieval; IDs are scoped to a pinned taxonomy/prototype set |
| Filter coverage | party_keys_ready, classification_ready | Boolean completion markers when those filters are supported; distinguish processed-empty from not processed |
| Acts/articles | normative_version_id, validity_coverage, valid_from, valid_to | Version reference, keyword coverage and indexed half-open legal-time bounds; omit the upper bound only for a documented open-ended interval |
| Articles | act_id | Keyword/UUID parent scope for hierarchical act → article retrieval |
| Normative metadata filters | year, authority_code | Only where exposed by the operation: act year/issuer, not an article amendment year or an inferred judgment court |
| Restricted normative search | code_id | Optional canonical code membership when the restricted-code search consumes it; resolve external editorial codes through PostgreSQL first |
| Future passages | generation_id | Generation eligibility and exact chunk hydration; point ID is already the chunk ID, so do not duplicate it in payload |
Common eligible includes the selected representation within its profile and permitted source access, not “currently in force.” Known historical normative versions can remain indexed for explicit historical searches. Apply valid_from <= as_of < valid_to with the open-ended case handled explicitly; unknown validity cannot silently pass a current-law filter. A cached is_current flag is not the authority for time-sensitive selection.
No complete classification scores, labels, raw classifier output or legacy cluster IDs are copied. Fetch scores from PostgreSQL if the cross-domain algorithm uses them for ranking/grouping. The classification set is independent of the discovery embedding profile. Do not infer that every label in every historical assignment is eligible for the current classification filter.
Participant filter semantics require care: today’s reference judge filter matches president or reporter text, and party filters match normalized keys. Replacing these with person UUIDs alone would change behavior for unresolved people. An eventual ID resolver must preserve that behavior before the name fields can be removed. Full identities, addresses, representation relationships and other personal data stay in PostgreSQL. Count/statistical operations retain their narrower permitted filters; payload availability does not authorize judge/seat filters on every endpoint.
For act/article exact resolution, identifiers, titles, article numbers, annex paths and issuing-authority metadata are read from PostgreSQL. Do not add them to Qdrant merely to display a result. If a future candidate query actually filters on one, extend this allowlist and its indexed projection deliberately.
Example payloads
Section titled “Example payloads”These are synthetic payload objects, not full Qdrant upsert requests. Dense vectors and BM25 sparse vectors occupy named vector slots outside payload; dimensions and names are taken from the collection registry. The point ID is the representation UUID supplied separately. Optional fields absent in the source are omitted.
Judgment-summary payload:
{ "resource_id": "10000000-0000-4000-8000-000000000001", "source_version_id": "20000000-0000-4000-8000-000000000001", "enrichment_id": "30000000-0000-4000-8000-000000000001", "eligible": true, "projection_revision": 7, "grade": "cassazione_civile", "year": 2025, "decision_date": "2025-06-04T00:00:00Z", "jurisdiction_code": "civil", "outcome": 1, "matter": "civile", "section": "III", "party_keys": [], "party_keys_ready": true, "semantic_matter_ids": ["category-example-1"], "classification_set_id": "40000000-0000-4000-8000-000000000001", "classification_ready": true}Article-text payload (the same validity envelope applies to an act-profile point, without act_id):
{ "resource_id": "10000000-0000-4000-8000-000000000002", "source_version_id": "20000000-0000-4000-8000-000000000002", "normative_version_id": "50000000-0000-4000-8000-000000000001", "act_id": "10000000-0000-4000-8000-000000000003", "eligible": true, "projection_revision": 3, "validity_coverage": "open_ended", "valid_from": "2020-01-01T00:00:00Z"}Date-only values above are encoded at UTC midnight solely for ordered date filtering, not as asserted event timestamps. Resolve “today” in the legal timezone before encoding the search date. Optional authority_code, seat and judge names are omitted from the synthetic judgment rather than invented.
What is removed from legacy payloads
Section titled “What is removed from legacy payloads”The reference builders in ComproCausaAPI/api/scripts/embeddings/ currently copy fields for both filtering and hydration. The target deliberately splits those responsibilities:
embed_pipeline_sentenze_canonical.py: moveoggetto,descrizione,descrizione_esito, amounts, legal costs, raw classification and processing/model metadata to PostgreSQL-only hydration. Retain the filter projection, not the whole summary object.embed_pipeline_normattiva.py: remove duplicated canonical/editorial aliases, duplicated titles/numbers, headings/path inventories, Gazette details, quality flags and pipeline metadata from Qdrant. Retrieve act-profile rerank content from its PostgreSQL representation.embed_pipeline_articoli.py: remove article headings, full structural objects, parent-act display metadata, quality scores and processing fingerprints from payload. Keep parent scope and legal-time eligibility; article text and transient rerank chunks come from PostgreSQL.
Filter mapping is grounded in api/semantic/bibliography_search_service.py, bibliography_count_service.py, cross_domain_search_service.py and full_normativa_search_service.py in the reference repository. Collapse legacy aliases such as anno/anno_provvedimento and esito/Esito during projection using a tested precedence rule; do not duplicate them forever. Preserve unknowns, existing exclusions and filter-coverage guards. In particular, semantic_matter_ids_v1 maps to the version-scoped classification fields, not to legacy clustering.
Retrieval and publication contract
Section titled “Retrieval and publication contract”Dense and BM25 branches apply the same eligible/profile/type/time/user-filter scope before their candidate limits and fusion. Sparse vectors are built from the retained recipe input; their vocabulary/model and corpus-statistics policy are versioned with the collection. No full text is needed in the payload to compute sparse similarity. Do not claim identical rankings after changing tokenizer, sparse recipe or fusion semantics without differential tests.
After retrieval, Legal Core batch-loads representation inputs and source metadata by ID from PostgreSQL, verifies profile selection, revision, ownership, access and legal-time eligibility, then runs the existing branch-specific rerank algorithms. Preserve Qdrant candidate order when hydrating. Drop stale/missing rows; bounded refill or an explicit incomplete/degraded result handles a shrinking pool. Never expose stale text or present a partial pool as an exact count. Pool-based reranking restricts retrieval by canonical resource IDs; the caller need not cache vectors or full payloads.
Qdrant is not a relational join engine or a second canonical database. Citations, participant analytics, monetary aggregation and full report data remain SQL operations over the authorized retrieved resource set. Counts distinguish candidate caps, approximate results and validated totals. Existing filters and scoring behavior remain endpoint-specific even when they share collections.
Strict keyword/exact-reference operations are not automatically equivalent to sparse similarity. Retain the necessary PostgreSQL text/exact-lookup indexes for those operations unless a tested replacement preserves their match semantics. This does not justify copying full text into every Qdrant payload.
The target projector records durable work in PostgreSQL, writes idempotent point updates under a single-writer/fenced publication protocol, and marks the projection ready only after acknowledgement and coverage validation. A revision field alone does not prevent an old retry from overwriting a newer point: writer ordering and stale-attempt checks are required. Collection switches are profile-specific and must not invalidate another experiment arm or pinned configuration. Reconciliation detects missing, stale and orphaned points. The PostgreSQL transaction and Qdrant write are not atomic; failures remain pending/retryable, never silently published.
Implemented projection lifecycle
Section titled “Implemented projection lifecycle”internal/vectorindex bootstraps a fresh collection from retained SQL vectors and embedding inputs. Native Qdrant BM25 derives sparse vectors from transient input text; that text is not stored in the payload. Writes must acknowledge completion and pass read-back verification, coverage must match, and publication locks/rechecks the SQL revision before selecting the collection. Failed/incomplete builds remain recorded but are not selected.
corpus-worker project-watch continuously reconciles supported profiles, processes erasure intents and collects retired builds. Its health endpoints distinguish a live process from a successful synchronization cycle. The process owns its goroutine and cancellation; work survives restarts in PostgreSQL, not in memory. Local Compose starts this worker after fixture seeding and waits for its readiness before starting the API.
corpus-worker project-all executes one cycle. corpus-worker project <profile-uuid> <representation-kind> explicitly builds a replacement pool. These commands use the existing worker database capability and the owning service’s LEGAL_QDRANT_URL / LEGAL_QDRANT_API_KEY; deployed environments require HTTPS and credentials. Give the API a read-only Qdrant credential and the projector a separate write credential. To request a manual cycle:
docker compose -f infra/compose.yaml run --rm --no-deps legal-search-projector project-allAfter bootstrap, updates replay only resources whose coalesced revision exceeds the pool’s watermark. Metadata changes, newly selected representations and withdrawn sources update or fence their existing points. Unchanged points keep their original revision. PostgreSQL’s per-point ledger validates the exact revision during hydration; a collection watermark alone is not proof that a hit is current. Act metadata changes also enqueue dependent articles. Query-time legal dates do not require a daily rebuild. Synchronization makes no embedding-provider calls.
SQL advisory coordination prevents overlapping active cycles. Before any external write, registration locks the resource against removal and records every possible copy in search.qdrant_points. Conditional Qdrant writes use an insert-only empty marker followed by an update-only revision guard; verification checks the resulting payload and vector presence. A stale retry cannot overwrite a newer revision or a permanent purge fence. The implementation requires Qdrant’s conditional-upsert protocol, as exercised by internal/vectorindex/lifecycle_integration_test.go.
A relevant committed corpus change conservatively invalidates older serving watermarks until synchronization succeeds. This brief fail-closed interval also applies during concurrent imports; it is not a zero-downtime indexing guarantee. The new watermark publishes only after coverage verification and a SQL compare-and-swap. Interrupted work is replayed idempotently. Missing collections or mismatched eligible counts trigger a fresh build. Same-count out-of-band content corruption is not comprehensively audited: service credentials must prevent external index mutation. Transport errors are never treated as empty search results.
Retired and failed collections are deleted externally before SQL records cleanup. Abandoned builds are retired after the heartbeat timeout defined in internal/vectorindex/lifecycle.go. Collection names are unique per build, so deleting an old build cannot remove the selected replacement. Delayed point updates cannot recreate a deleted collection.
The PostgreSQL and Qdrant integration tests cover dense/hybrid selection, temporal validity, filters, separate model profiles, incremental metadata updates, failed/stale publication, verified erasure, stale-write fencing and retired-collection cleanup. These are functional tests, not a representative-scale latency/quality benchmark or a backup/restore exercise.
Publication
Section titled “Publication”flowchart LR
Source["Immutable source version"] --> Build["Build representation for profile"]
Build --> Check{"Complete and still eligible?"}
Check -->|Yes| Select["Publish and select"]
Check -->|No| Reject["Reject incomplete or stale result"]
Select --> Read["Eligible search or pinned read"]
A provider can return only part of a batch, or a source can change while vectors are generated. Publication checks prevent either result from becoming a complete searchable generation.
Legal publication checks source freshness, vector completeness and the selected embedding profile before a representation becomes searchable. Private document generations apply their own readiness and publication checks. Historical legal versions remain available for authorized pinned reads, while unpublished or withdrawn sources are excluded from ordinary retrieval.
Different profiles may retain ready representations for the same source. Search configurations owns model selection; Workers & recovery owns attempt authority.
Erasure
Section titled “Erasure”Deletion has two phases because a database transaction cannot atomically delete an Azure blob. The product first invalidates access and derived records, then records durable cleanup work. For example, deleting a source document must also invalidate a generated answer that used it, even when the answer omitted a citation.
Input dependencies are recorded so deletion can invalidate derived content even when a source was not cited in the final answer. This depends on callers recording the complete lineage. Access is blocked before asynchronous blob cleanup, so a delay in physical deletion does not keep server-side content readable. A durable purge intent survives source deletion and can be retried if blob removal fails. Withdrawn legal sources are denied across reads, although withdrawal alone does not prove that every retained copy has been erased.
Target retention guarantees
Section titled “Target retention guarantees”Logical denial is committed to PostgreSQL first and immediately enforced during hydration. The projector finds all registered copies, including unfinished writes and non-default profiles. In retained collections it replaces their dense/sparse vectors and filter payloads with an empty-vector permanent fence containing only opaque resource identity, revision and denial flags. These tombstone identifiers are intentionally retained to prevent delayed retries from resurrecting erased content; they are not indexed source text or searchable vectors. Obsolete collections are deleted entirely. External read-back must confirm cleanup before a receipt is stored in ops.qdrant_removal_receipts.
ops.require_qdrant_cleanup refuses completion when registered copies remain unpurged or an intent with registered copies lacks a verified external receipt. The existing SQL/blob cleanup still has its own completion requirements; a Qdrant receipt does not claim that originals or backups were erased. Do not bypass the guard or delete ledger records to claim completion. Restore must replay erasure intents before enabling reads: this operational recovery guarantee still requires a demonstrated restore procedure.
Complete legal-corpus erasure across summaries, prototypes and external caches, backup-expiry guarantees, and a demonstrated database/blob restore followed by deletion-ledger replay are not established by the current tests. These require an implemented recovery procedure and a restore exercise before being claimed. Server cleanup also cannot recall files already downloaded by a user.
Membership removal and content deletion are distinct: losing access should not delete a document still legitimately used by another authorized scope. That decision belongs to resource authorization; Identity & access links its enforcement.