The Library That Thinks — RAG, Embeddings & Vector DB

Why LLMs hallucinate, what embeddings measure, and how retrieval gives AI a working memory

The Analogy

A Library With a Brilliant Librarian

Before RAG, asking an LLM about your specific infrastructure was like calling a scholar who has read every book ever written — but has never stepped inside your building. They are brilliant in the abstract but blind to your specific nodes, your playbooks, your drift history. RAG is the mechanism that hands them the relevant pages from your private library before they answer.

The Four-Step Library Process

Without RAG vs. With RAG

The same question. Completely different answer quality.

Comparison — Bare LLM vs. RAG-Augmented LLM

Three Systems, One Pipeline

RAG is the composition of three distinct components. Each does exactly one job.

🔢

Embeddings

Convert any text to a point in high-dimensional space. Texts with similar meaning land close together — even if they share no words in common. The embedding model is the only part that requires a GPU or an inference server.

📍

Vector Database

Stores millions of vectors and answers the question: “what are the nearest points to this query vector?” Uses approximate nearest-neighbour indexes (HNSW, IVFFlat) for millisecond search. kri uses pgvector — no separate service needed.

🧩

RAG

Retrieval-Augmented Generation. At query time: embed the question, search the index, inject the top-k results into the LLM’s context window. The LLM reads your actual data — not stale weights from its training run.

The Technology

What Is an Embedding, Really?

An embedding model reads a string and outputs a fixed-length array of floating-point numbers — a vector. For nomic-embed-text-v1.5 that is 768 numbers. The critical property: texts with similar meaning produce vectors that are geometrically close in that 768-dimensional space. “mm1 went offline” and “node mm1 is unreachable” will have nearly identical vectors, even though they share only the token “mm1”.

Text → Embedding Model → 768-Dimensional Vector

The Vector Space — Similar Meaning Clusters Together

After embedding, every text is a point in 768-dimensional space. Similar texts cluster together. When a query arrives, the embedding model converts it to a vector, and the index returns the geometrically nearest stored vectors — regardless of whether the words match.

2D Projection of 768-Dimensional Embedding Space

Hybrid Search: BM25 + Vector + RRF

Pure vector search can miss exact strings — node hostnames like “mm1”, package names, file paths. Pure keyword search misses synonyms and paraphrases. kri runs both and fuses the two ranked lists using Reciprocal Rank Fusion.

🔍

BM25 — Keyword Ranking

Full-text search via Postgres tsvector + plainto_tsquery. Perfect for exact node names, package versions, error codes. Fast. Fails on paraphrase and synonyms.

📐

Vector — Semantic Ranking

Cosine similarity in 768-dimensional space. Perfect for meaning — “unreachable node” finds “host went down”. Can miss exact identifiers it has never seen in training.

Reciprocal Rank Fusion (RRF):  Each result scores 1 / (60 + rank) from each retrieval method. Scores are summed. A chunk ranked #2 in both BM25 and vector beats a chunk ranked #1 in only one. This naturally promotes results that are lexically and semantically relevant.

In kri

The kri RAG Pipeline

kri uses pgvector directly — no separate vector database service. The fleet_embeddings table has a Vector(768) column and an HNSW index. Three Celery beat tasks keep it fresh. The AI Fleet Assistant queries it on every fleet request.

kri Fleet Assistant — RAG Pipeline End to End

Three Data Sources, One Index

Three Celery beat tasks populate fleet_embeddings. They all check LLM_EMBED_BASE_URL before running — if unset, they skip silently.

🖥

Node Profiles

One chunk per node: hostname, IP, status, group, OS, last-seen. Re-embedded only when content_hash changes. Runs every 5 min.

📋

Playbooks

One chunk per Ansible play: play name, target hosts, task list. Reads every .yml under the configured playbooks directory. Runs every 15 min.

📊

Drift Records

One chunk per drift report from the last 7 days: node, score, missing/extra packages, version mismatches. Runs every 5 min.

Background Ingestion — Keeping the Index Fresh

One Setting Activates Everything

All code is already deployed. The only required step is pointing to an embedding server.

1. Run the embedding server — ./scripts/setup-embed-server.sh downloads llama.cpp and nomic-embed-text-v1.5.Q8_0, then serves POST /v1/embeddings on port 8080.
2. Set LLM_EMBED_BASE_URL in kri Platform Settings to http://<host>:8080.
3. Within 5 minutes the three Celery tasks run and populate fleet_embeddings with vectors for every node, playbook, and drift record.
4. Fleet Assistant queries with intent fleet_query or fleet_command now automatically retrieve top-6 relevant chunks and inject them as ## Retrieved Knowledge in the system prompt before calling the LLM.

Why pgvector and not a dedicated vector DB?
pgvector adds Vector(N) column types and HNSW index operators directly to PostgreSQL. kri already runs PostgreSQL in the cluster — no new service, no consistency gap, no sync delay. At fleet sizes of thousands of nodes the HNSW index returns nearest neighbours in under a millisecond. A dedicated vector database would only be worth the operational cost beyond millions of vectors.

What happens without LLM_EMBED_BASE_URL?
The three Celery tasks return {"skipped": "no embed_base_url configured"} and exit. Fleet Assistant queries still work — they just receive the live node table without semantic retrieval. No errors, no broken UI. RAG activates the moment the setting is saved.