7 min read
Last updated:
Most retrieval-augmented generation tutorials end at “chunk, embed, query, prompt.” That is sufficient for an internal proof of concept. It is not sufficient for a hospital, a bank, or a federal agency. In regulated environments, the question is not whether the model produced a good answer. The question is whether you can prove who saw what data, when, under which authorization, and whether you can erase that data tomorrow if a court order arrives.
This piece is a compliance-first reference architecture for RAG, written for teams shipping into HIPAA, GDPR, PCI-DSS, FedRAMP, and similar regimes. We will treat the retriever, the vector store, the prompt assembler, and the LLM as four separate trust boundaries, each with its own audit obligations.
Why Naive RAG Fails Compliance
The default LangChain or LlamaIndex tutorial assumes a single corpus, a single user class, and no retention policy. That assumption breaks in three predictable ways once a regulator looks at it.
- Authorization leaks at retrieval. Embeddings have no concept of access control. If you index a CFO memo and a junior analyst’s chatbot retrieves the nearest neighbors, the memo will surface. Post-hoc filtering in the prompt is not a control; the data already crossed a boundary.
- No erasure path. When a GDPR Article 17 request arrives, you must delete the data and any derivatives. Embeddings are derivatives. So are cached completions, prompt logs, and fine-tuning datasets that absorbed the document. Most teams cannot enumerate these surfaces, let alone purge them within the 30-day statutory window.
- Unverifiable answers. A model that paraphrases three sources without citation cannot be defended in an audit. Regulators do not accept “the model said so.” They accept “chunk 47 of document FDA-2024-N-0312, retrieved at 14:03:17 UTC, fed verbatim to the prompt at position 4.”
The Four Trust Boundaries
Treat your RAG stack as four discrete services, each logging independently and each enforcing its own policy. The boundaries are: ingestion and indexing, retrieval and authorization, prompt assembly, and inference. A failure at any boundary should fail closed, not degrade silently to an unfiltered response.
Boundary 1: Ingestion and Indexing
Every chunk that enters the vector store must carry metadata that survives every downstream operation: a stable document ID, a chunk hash, the source classification (PHI, PII, public, internal), an access control list, a retention class, and the ingestion timestamp. Treat this metadata as load-bearing. If your vector store cannot filter on it at query time, you have the wrong vector store.
Boundary 2: Retrieval and Authorization
Authorization happens before similarity search, not after. The user’s identity, role, and clearance flow into the query as a metadata filter. Pinecone, Weaviate, and pgvector all support this; the question is whether your retriever code uses it correctly. The pattern is: resolve the caller’s effective ACL, translate it into a filter expression, then execute the vector query with that filter as a hard predicate. Never rely on a re-ranker or the LLM to enforce access.
Boundary 3: Prompt Assembly
The prompt assembler is the last point at which you control what the model sees. Log the full prompt, including system message, retrieved chunks, and user query, with cryptographic hashes that link back to the source documents. This log is your audit evidence. Store it for the longer of (a) your retention policy and (b) the statute of limitations for the relevant regulation. Encrypt at rest with a separate key from your application database.
Boundary 4: Inference
If you are calling a hosted model, your data leaves your perimeter. Read the data processing addendum carefully. OpenAI, Anthropic, Google, and AWS Bedrock all offer zero-retention or BAA-compliant tiers, but the defaults are not always those tiers. Verify, in writing, that prompts are not used for training, are not logged beyond the request lifecycle, and are processed in the geographic region your data residency rules require.
Vector Database Selection
The vector store choice is downstream of your compliance posture, not your latency target. Three pragmatic options, with the tradeoffs that actually matter in regulated work.
Pinecone
Managed, fast, mature metadata filtering. Offers SOC 2 Type 2, HIPAA BAA, and dedicated regional pods. The tradeoff is that you are sending vectors and metadata to a third party, which forces a vendor risk assessment and a sub-processor disclosure. For PHI workloads in the US, this is acceptable with a signed BAA. For data subject to EU sovereignty rules, choose the EU region explicitly and verify the support contract does not allow US-based engineers to access pods during incidents.
Weaviate
Self-hostable, with strong multi-tenancy primitives. The tenant abstraction is the cleanest in the category for organizations that need hard isolation between business units or customers. Operational cost is real: you own the cluster, the upgrades, and the backup verification. Choose Weaviate when your compliance team will not approve any external vector vendor, or when you need per-tenant encryption keys.
pgvector
The pragmatic default for teams that already operate Postgres at scale. You inherit your existing backup, encryption, audit, and access control posture. Performance is adequate up to roughly ten million vectors with HNSW indexes; beyond that, you start fighting Postgres for memory and connection pooling. The compliance argument writes itself: it is the same database your auditors already approved last year.
Deletion Cascades for Right-to-Erasure
GDPR Article 17, CCPA Section 1798.105, and HIPAA’s accounting of disclosures all assume you can find and delete data on demand. RAG systems generate derivatives that traditional deletion scripts miss. Build the cascade explicitly, and test it.
- Source store. Delete the original document and any object storage replicas, including versioned buckets.
- Vector store. Delete every chunk associated with the document ID. Verify with a metadata-filtered count query that returns zero.
- Prompt logs. Identify every logged prompt that included a chunk from the deleted document. Either redact the chunk content or delete the log entry, depending on whether you need to retain the audit trail of the interaction.
- Completion cache. If you cache LLM responses keyed on input hash, invalidate every cached response that referenced the deleted document.
- Fine-tuning corpora. If the document was used in any training set, you cannot unlearn the model. Disclose this in your data processing notice and avoid using user data in fine-tuning unless you have a defensible deletion story.
Run the full cascade as part of CI. A monthly synthetic deletion test, with a canary document inserted and removed end-to-end, will catch the regressions that real deletion requests would otherwise expose at the worst possible time.
Hallucination Guardrails
In regulated work, a hallucination is not a quality issue. It is a misrepresentation, and depending on the domain it is a regulatory violation. The defense is layered.
First, force the model to cite. Use a structured output schema that requires every factual claim to reference a chunk ID from the retrieved set. Reject responses that contain unsourced claims, either via a parser or a second-pass verifier model. Second, ground the prompt aggressively: instruct the model to answer only from the retrieved context and to return a refusal token when the context is insufficient. Third, run a post-generation verifier that re-retrieves based on the answer and checks that each cited chunk actually contains the claim attributed to it. This catches the failure mode where the model invents a citation that points to a real chunk that does not support the assertion.
Evaluation Methods That Hold Up in Audit
Vibes-based evaluation will not survive an external review. Build a labeled evaluation set with at least 200 questions per use case, drawn from real user queries and reviewed by a domain expert. Track four metrics: retrieval recall at k, citation accuracy (does every cited chunk support its claim), refusal correctness (does the model refuse when it should), and end-to-end factuality scored by a held-out human reviewer on a quarterly sample. Report these metrics in the same governance forum that reviews your model risk management framework.
Recommendation
Start with pgvector inside your existing compliance perimeter. Build the four trust boundaries before you optimize anything else. Implement the deletion cascade and test it monthly. Force citations and run a verifier. Only after these are in production should you evaluate Pinecone or Weaviate for scale, and only with a documented sub-processor and a signed BAA or DPA in hand.
When This Applies, and When It Does Not
This architecture applies when your RAG system handles PHI, PII, financial records, classified material, or any data subject to a statutory deletion right. It is overkill for a public documentation chatbot or an internal engineering knowledge base where the worst-case disclosure is embarrassing rather than actionable. For those, the standard tutorial stack is fine. The boundary is whether a regulator, a plaintiff, or a journalist could materially harm the organization by reading the prompt logs. If yes, build the boundaries. If no, ship faster.