Beyond RAG
Note: This article is a continuation of RAG on vllm-gke. The full codebase is available on GitHub: iheb24/vllm-gke, and every file I mention is linked directly.
In this article of the series I will be exploring other acronyms that are popular on inference stacks. After exploring RAG in depth, I wanted to check the neighbouring methods: TAG, CAG, MAG and KAG.
At first look we can say that all of these methods are augmented generation, but this does not apply to all of them, notably MAG, which stays close to Memory-Augmented Generation in name while behaving a bit differently, as I will explain in its section.
At a first look I also thought that these concepts would replace RAG procedures entirely. They do not. They are used together with RAG, each on a defined use case where it brings extra accuracy and efficiency where it is needed.
RAG serves knowledge to the LLM around the prompt, and these methods behave differently to best serve their own purpose. The acronyms are easy to explore: they usually refer to the source of information.
- TAG: Table
- CAG: Cache
- MAG: Memory
- KAG: Knowledge
KAG is the one that goes the furthest from simple retrieval, since it navigates a knowledge graph instead of one flat source.
I was able to showcase the first three methods on my repository. They are deployable next to the existing RAG stack, one at a time, to demonstrate what each one is capable of. Each method also required its dedicated resources and custom configuration, which I will point out in each section.
TAG
Instead of using a vector DB and parsing unstructured data, this method relies on tables that are available for the LLM to query. In this repository it will rely on SQL, so it is based on structured data more than unstructured data.
It is a perfect fit when dealing with numerical and precise details that we want to give to the LLM at execution time, for example financial reporting, logistics details, or similar use cases. TAG works perfectly there. It will not ingest a document file or a PDF; it relies on strictly structured data with column access.
The retrieval is very similar to RAG in shape, but the system will use the LLM not to embed the request: it will translate it into an SQL query that is then executed on the tables available to the system. The found information is inserted into the prompt together with the user request, for the final LLM call that handles the answer.
On the prompt, the additional information is in consequence structured (columns and rows) and inserted directly into the message going to the LLM.
Testing and deployment on the repo
The demo lives in showcases/tag/ with a synthetic serving-analytics database (thirty days of request and cost metrics plus free-text incident notes, all clearly labelled synthetic). The pipeline is in pipeline.py and the SQL guard in sql_guard.py. One detail I like in this design: the guard is enforced by SQLite itself through a connection authorizer, so writes, unknown tables and oversized results are rejected at the engine level, not by string matching.
For testing, there are 41 local tests that run with no GPU and no cluster, using a scripted fake model. The evaluation harness is covering a 17-question set through eval/run_tag_eval.py: exact analytics, semantic classification, write attempts and out-of-scope questions. Live on the cluster with the 14B model, I measured exact 8/9, all 3 write attempts rejected end-to-end, and all out-of-scope questions correctly produced no SQL. The full numbers and failure analysis are in the live report. The honest takeaway: the safety contract held at 100%, while answer formatting and scope judgment vary with the model.
| Pros | Cons |
|---|---|
| Exact numbers, no vector drift | Only answers what the schema contains |
| SQL is inspectable and auditable | Model-generated SQL needs a strict guard |
| Cheap: one SQLite file | Semantic questions still need a capable model |
Best for: analytics over business or operational tables, where being exactly right beats being approximately relevant.
CAG
CAG operates differently, and it was very showcaseable since we are using vLLM as the inference engine. C is for cache, and here the cache lives exactly within the inference machine.
The huge advantage is that there is no vector DB and no retrieval system at all: everything is present within the GPU's reach. But GPUs are limited, so think of this as a small extension of knowledge accessible with very low latency, not as a cheap storage place.
Concretely, the knowledge will sit as KV state in the KV cache of the GPU, next to the live prompt state. On the access path, we load the whole knowledge once and it is processed into the KV cache, so even before a question or a task comes to the LLM, the knowledge is loaded. When the prompt arrives, it is appended right after the cached prefix, and the model will either use the cache to answer instantly or work on an answer with it. After responding, what stays warm is the preloaded knowledge only, waiting for the next prompt.
Two things I learned the hard way here. First, this reduces the prefill work only: decode time and model startup are untouched. Second, the cache is process-local, so it dies with the pod, and after a scale-to-zero the first request pays the full prefill again. Knowledge usage can be proven by following exactly the KV cache metrics of the GPU, which is what I did.
Testing and deployment on the repo
The demo lives in showcases/cag/, with the prefix builder in corpus.py and the benchmark harness in benchmark.py. The corpus is pinned by a manifest with a SHA-256 version and a token count measured with the served model's tokenizer: 7,628 tokens, capped at 12,000 so the question and the answer still fit the 32K window.
For testing, 29 local tests cover prefix byte-identity, manifest staleness, budget failures and the counter-based verdict logic. Live on the cluster, I measured the time to first token at 7.23s on the first cold-prefix run (fresh pod, counters at zero) against about 0.53s warm. The committed benchmark file holds the counter evidence from the verification run: 85,798 prefix-cache queries, 85,712 hits (99.9%) across 11 requests, scraped from the server's own metrics. The full provenance of both numbers is in the run_history field of eval/reports/2026-10-28-cag-benchmark.md and the quality runs in eval/reports/2026-10-28-cag-live.md. The answer-quality side is honestly weaker with my 14B: content stays grounded, but citation discipline varies from run to run.
| Pros | Cons |
|---|---|
| Zero retrieval infra: no vector DB, no embeddings | Corpus must stay small (context budget) |
| Warm prefill nearly free (13x TTFT here) | Cache dies on pod restart and scale-to-zero |
| Deterministic, versioned knowledge | Latency gain, not a quality guarantee over RAG |
Best for: many questions against one small, stable corpus: handbooks, runbooks, API references.
MAG
So the acronym looks like augmented generation, but after deep diving the concept, it is interesting to say that it is slightly different.
MAG can really shine for LLMs wanting to have persistent state storage. As seen on the AgentCore and ADK articles, an agent may and will need a persistent storage for storing, updating and recalling historical context on a session or across sessions. The difference is exactly here: the knowledge is not preloaded, it is created and saved from historical conversations, contexts, findings and results.
This is perfect for companions learning incrementally as they are used. At first I thought the LLM itself knew the history, but it is really an outside memory block, which could be key-value as well if needed. The best example is when you use GPT or Gemini or another competitor over time: you will notice it remembers old sessions and information about you. This is exactly this pattern at work, and it can also be included with managed frameworks as mentioned, like ADK or AgentCore on AWS.
But for once I was intuitively right on MAG: the writing and retrieval really happen around the LLM during its execution. In my demo I kept the writes explicit (a remember: command or an API call) so that nothing happens silently, and every injected memory ID shows up in the response trace. Frameworks like mem0 or the managed agent services go further and let the model drive memory writes by itself.
Testing and deployment on the repo
The demo lives in showcases/mag/, with the store in memory_store.py: per-user memories with full-text search, bounded session history, and a small PVC so state survives pod restarts.
For testing, 47 local tests cover the lifecycle: remember in session A, recall in session B, correction, deletion, isolation between users, secret-like input rejection, and reopening the database. Live on the cluster, every deterministic check passed (3/5 scenarios on strict substring scoring, both misses were phrasing variance): Alice and Bob stay isolated, cross-user edits return 404, and after I deleted the pod, the new pod mounted the PVC and still recalled the stored preference. The report is in eval/reports/2026-10-28-mag-live.md.
| Pros | Cons |
|---|---|
| Personalization that survives restarts | Explicit writes only; no automatic extraction yet |
| Full user control: correct, delete, delete-all | user_id is a demo identifier, not authentication |
| Small: one SQLite file on a 1Gi PVC | Keyword retrieval, no semantic memory search |
Best for: assistants with continuity: preferences, corrections and project context across sessions.
Finally, MAG is a very useful method. It is not really augmented generation in the strict sense, but it is so valuable that it deserves its place in this list. Some companies have started databases containing high-level knowledge about their product to give to LLMs, and there are also locally available tools for AI companions that do exactly this, like agent memory frameworks or mem0.
KAG
KAG is for Knowledge-Augmented Generation and relies heavily on knowledge graphs. It is perfect for advanced and big knowledge bases, where it will offer multi-hop accuracy and reduce considerably the search effort compared to a traditional vectorial database alone.
KAG resembles RAG in that the knowledge is prepared before the question, but here it is stored on a mapped graph of entities, indexed together with the text. So instead of only looking for precise information with pattern matching, the system will perform multi-hop reasoning on the mapped indexes. For huge knowledge graphs this is very efficient: medical history of a patient for an LLM assistant, STEM research, legal research, fraud detection.
Its biggest flaw is its implementation complexity: on top of the huge amount of information needed, the graph has to be processed and tuned so the agent can hop freely and efficiently between knowledge nodes.
In consequence, this method is not showcased on the repository, but I have made some research on the how-to. To get the knowledge, a system will take the prompt and map its concepts to specific nodes on the graph. These nodes have logical external connections to other nodes, hence the traversal, or the hop. The system will gather the subgraph made of the starting nodes and the logically connected nodes around them. The subgraph is then translated and injected into the prompt for the LLM. One precision that matters: it will not drop matching-based search here, it combines it with the logical connections between the nodes, which makes the result stronger and less prone to hallucinations, and also justifies the complexity of creating good knowledge graphs for the LLM to have the desired results.
On the research side, I drafted a concrete use case for this stack: multi-hop troubleshooting over a graph of components, node pools, secrets and runbooks, with the full architecture, the cost analysis and a go/no-go checklist for a future spike with the official OpenSPG/KAG framework. It is all in docs/kag-architecture.md. It is not implemented, for reasons I will explain at the end of this section: the operational footprint does not fit a one-GPU, cost-audited cluster, and a hand-rolled graph script labelled KAG would be dishonest.
| Pros | Cons |
|---|---|
| Multi-hop reasoning with provenance | Heaviest stack to build and operate |
| Schema keeps the graph clean and queryable | Schema design is ongoing human work |
| Strongest fit for relationship-heavy domains | Several model calls per question: latency and cost |
Best for: large, relation-heavy knowledge bases where the answer is a path, not a passage.
Runbook
Everything is one-at-a-time by policy, enforced by showcases/demo.sh. It will refuse to start a second showcase unless you force it.
Deploy one showcase
Evaluate
Manifests are ClusterIP-only and live in k8s/showcases/. Reports, including the live cluster runs, are committed under eval/reports/. The wider decision matrix is in docs/augmented-generation-comparison.md.
GPU scale-to-zero subtleties worth knowing
A few operational details from this project cost me real time, so I will list them briefly:
- The KEDA HTTP interceptor holds your request while the GPU node scales from zero. On a fresh cluster the first wait includes a ~10GB model download, so the first answer can take close to ten minutes.
- The gateway has its own upstream timeout and can return 504 before a cold vLLM is ready. The scale-up continues regardless; wait for the pod and retry.
- Between calls, the GPU stays warm for
scaledownPeriod(300s here), then everything goes back to zero. - vLLM's prefix cache is process-local. After a scale-down, the counters reset to zero and the CAG prefix must be primed again. I measured exactly that.
- On-demand L4 was out of resources in my zone; the GPU pool now runs spot, which is both easier to get and roughly 60 to 70% cheaper.
- MAG's live run used the CPU tier deliberately (
AG_GATEWAY_MODEL=qwen3-4b-cpu), so the whole memory scenario never woke the GPU.
Closing
The four acronyms are easier to remember by the state they own than by their names: tables, cache, memory, graph. None of them replaces retrieval; each one wins where its state shape matches the question. The repo shows the three I have built honestly, measured, with the fourth left as a documented design. If you pick one, pick by your data shape first, not by the acronym.