Building the RAG system

Inventory is a tool. Policy is a document. Never let the model be the database.

Before any retrieval, split knowledge into two classes and route every message to the right one. This single decision stops the worst failure mode: the LLM inventing a price, a mileage, or an "accident-free" claim a customer could sue over.

Where each kind of knowledge lives

Every RAG chunk also carries mandatory metadata — tenant_id, branch_id, locale, effective_date, expiry_date, access_level, topic_tags — which powers tenant isolation, branch-specific answers, stale-promo filtering, and never leaking internal notes to customers.

The pgvector table for Class B

One table holds the chunk text, its dense vector, a full-text vector for BM25, and the metadata. Two indexes — HNSW for vectors, GIN for text — make hybrid search fast.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE policy_chunks (
    id           bigserial PRIMARY KEY,
    tenant_id    bigint      NOT NULL,
    source_id    bigint      NOT NULL,         -- which document
    doc_version  int         NOT NULL,
    content      text        NOT NULL,         -- the chunk itself
    embedding    halfvec(1536) NOT NULL,       -- Gemini Embedding 001 (Matryoshka-truncated)
    search_tsv   tsvector    NOT NULL,         -- BM25 / lexical (newmm-segmented Thai)
    branch_id    bigint,                       -- NULL = applies to all branches
    locale       text        NOT NULL DEFAULT 'th',
    effective_date date,
    expiry_date  date,                         -- filter dead promos at query time
    access_level text        NOT NULL DEFAULT 'customer',  -- vs 'internal'
    topic_tags   text[]      NOT NULL DEFAULT '{}'
);

-- Dense vector search (cosine). HNSW = fast, graceful updates.
CREATE INDEX ON policy_chunks
    USING hnsw (embedding halfvec_cosine_ops);

-- Lexical search for BM25 / exact-term recall.
CREATE INDEX ON policy_chunks USING gin (search_tsv);

-- Cheap metadata filters applied before ranking.
CREATE INDEX ON policy_chunks (tenant_id, branch_id, locale);

-- HNSW vs IVFFlat — pick HNSW for a dealer-scale KB.
--
-- HNSW   graph index. Best recall/speed for < ~5M vectors,
--        handles inserts/updates gracefully (you re-embed
--        a policy and it just works). Slightly more memory.
--   CREATE INDEX ON policy_chunks
--       USING hnsw (embedding halfvec_cosine_ops)
--       WITH (m = 16, ef_construction = 64);
--
-- IVFFlat  clusters vectors into lists. Smaller, faster to
--          build, but recall degrades as data shifts and it
--          needs periodic REINDEX. Better for huge, static sets.
--   CREATE INDEX ON policy_chunks
--       USING ivfflat (embedding halfvec_cosine_ops)
--       WITH (lists = 100);
--
-- Verdict: a dealer's policy KB is small and edited often
-- -> HNSW. Reach for IVFFlat only past millions of rows.

-- Why halfvec(1536) instead of vector(1536)?
--
-- halfvec stores each dimension as a 16-bit float instead
-- of 32-bit. For RAG retrieval the recall loss is negligible
-- but you get:
--   * ~50% less storage + index memory
--   * faster index build & query
--   * the HNSW dimension ceiling doubles (2000 -> 4000)
--
-- Gemini Embedding 001 is Matryoshka: you can ask for 3072,
-- 1536, or 768 dims. Start at 1536 -> good quality, half the
-- footprint of 3072. Drop to 768 if storage dominates and
-- your Thai eval (lesson 3, section 9) still passes.
--
-- ALTER TABLE ... ALTER COLUMN embedding TYPE halfvec(768);
-- -- then re-embed. Always re-measure recall after a dim change.

Hybrid search — dense + BM25, fused

Dense vectors catch meaning; BM25 catches exact terms embeddings miss (model codes, the word "พรบ."). For Thai, hybrid is non-negotiable. Toggle the knobs and watch the SQL the app would run.

:q = the query embedding (from Gemini Embedding 001) · :text = the newmm-segmented query · 60 in 1/(60+rank) is the RRF damping constant. Inventory search (Class A) is a separate faceted query with price/year filters — this builder is the policy (Class B) path.

The cheapest big quality win

Hybrid search casts a wide net — the top 20 are plausibly relevant. A cross-encoder reranker scores each query–chunk pair jointly and reshuffles them, so only the genuinely-best 3–5 reach the LLM. Vendors cite +20–35% answer accuracy. Click to rerank.

Answer only from the chunks — or don't answer

The LLM is told to use only the reranked chunks for policy facts. And a confidence gate decides: if the top rerank score is weak, the bot asks one clarifying question or escalates to a human. It never guesses. Drag to see the behavior.

Don't use a frontier model for every step

Route by task. A cheap model classifies and extracts; deterministic code does the math (free); the strong model only writes the final Thai reply. Set your volume and watch the monthly bill.

Prices verified Jun 2026: Gemini Flash $0.30/$2.50, Gemini Pro $1.25/$10, Gemini Embedding 001 $0.15/M (per 1M tokens); Cohere Rerank $2/1k searches. Prompt-cache the persona + system prefix to cut Pro input cost ~90%.

Measure retrieval separately from generation

RAG quality = retrieval quality. If the right chunk isn't in the top-k, the LLM can't use it. Build a golden set of real Thai questions labelled with the correct chunk, and track recall@k — before and after rerank. Drag k.

Sequence to good Thai replies, fastest

1. Class A inventory tools + Thai normalization — answers the highest-volume question correctly, hallucination-free.
2. Persona + reply composer + style checker — consistent voice, LINE Flex cards, grounded numbers, one CTA.
3. RAG pipeline (Class B) — ingest → newmm chunk → embed → hybrid → rerank → gate.
4. Dialogue state + slot-filling — multi-turn discovery, returning-customer memory.
5. Financing estimate tool + booking flow — deterministic math, viewing booking.
6. Confidence-gated human handoff.
7. Eval harness + golden set — stood up early enough to gate every later change.

The hardest part isn't the pipeline — it's surviving messy Thai input. That's lesson 3.