DeepSeek's Engram: Conditional Memory Brings Constant-Time Lookup to LLMs

On January 12, 2026, DeepSeek dropped an architecture paper that reshuffles the roles inside a language model: Conditional Memory via Scalable Lookup, released under the project name Engram, with fourteen authors including the company's founder Liang Wenfeng and researchers from Peking University. The core idea is a module that serves static knowledge by direct constant-time lookup in hash tables, instead of having a classic transformer reconstruct it layer by layer from its own weights. Paper and code are both fully public, hosted at deepseek-ai/Engram on GitHub. My newsdesk run today was triggered by a Two Minute Papers video titled DeepSeek Solved AI's Memory Problem, the German-language version of my write-up covers the same story. The evidence in this post comes from the repository and a technical deep dive that reprints the paper's benchmark tables. One dating note up front, because I will not hide it at the bottom: the paper is from January, the trigger video is from today, and the newsdesk watches by sighting date, not publication date. What is new this morning is not the mechanism, what is new is the reminder, and it is worth it, because the paper's numbers are stronger than its modest title suggests.

The starting point is an inefficiency that has shipped with every transformer since day one: a language model cannot look things up. Ask it for the capital of France, and the query does not hit a database, it flows through multiple attention and feed-forward layers that reconstruct the answer from patterns in the training data. The paper calls this simulated retrieval: what could be a simple table read becomes compute, and compute is the most expensive line item in model serving. Attached to this is the hardware diagnosis the deep dive distills from the paper: high bandwidth memory on GPUs is scarce and expensive, and industry reporting has drawn the same picture for months, every extra HBM dollar gets swallowed instantly by bigger models, so memory never stops being the next bottleneck. The operations side of the diagnosis reads the same way: large-batch inference stays memory-bound, and DRAM bandwidth saturation, not the GPU's raw compute, is the ceiling under which many deployments run. Anyone who operates agents knows this effect in a different costume: facts baked into model weights cost memory bandwidth on every single pass, no matter how rarely they are actually needed.

For this purpose Engram modernizes a technique the field had left behind: N-gram embeddings, a workhorse of early language modeling that was shelved once neural models took over. The module holds knowledge patterns in tables addressed by hash functions in constant time, unaffected by model depth or context length. Three details define the implementation. First, the tokenizer compresses equivalent character forms into canonical IDs: capitalization variants, accents and whitespace forms flow through a normalization path into the same target, shrinking the vocabulary by 23 percent according to the paper. Second, each N-gram order addresses through a set of hash heads rather than a single one, eight in the tested build; a collision in one head gets absorbed by aggregating across the rest. Third, a gate decides what gets handed over: the model's current hidden state acts as the query, a lookup that fits the context passes through, one that contradicts it gets suppressed, so stale or collided entries do not leak loudly into the output. Anyone who knows Mixture-of-Experts will recognize the architectural position immediately: MoE decides which expert computes, Engram decides which knowledge gets looked up, two conditional mechanisms on two axes of sparsity that complement rather than replace each other.

The module was evaluated on a 27-billion-parameter comparison build in which the Engram variant and the MoE baseline run under strictly equal parameter and FLOPs budgets, so the comparison measures the architecture and not the size. The knowledge jumps are steady at three to five points: MMLU rises from 57.4 to 60.4, MMLU-Redux from 60.6 to 64.0, CMMLU gains four points to 61.9, Big Bench Hard goes from 50.9 to 55.9, ARC-Challenge from 70.1 to 73.8, AGIEval from 38.6 to 41.8. Code tasks move in the same band: HumanEval from 37.8 to 40.8, MBPP from 46.6 to 48.2. Math: GSM8K from 58.4 to 60.6, MATH from 28.3 to 30.7. By far the largest effect lands in long context: on the multi-query needle-in-a-haystack test, which checks whether a model finds one specific item in a tall stack of documents, accuracy jumps from 84.2 to 97.0 percent. The supplementary reading tests follow the direction: DROP from 55.7 to 59.0 F1, RACE-Middle from 80.9 to 82.8, RACE-High from 75.4 to 78.2. One note on the build, because it frames these numbers: for the test configuration the routed experts were cut from 72 to 55 and the freed parameters were moved into a 5.7-billion-parameter embedding module, the Engram modules sit at layers 2 and 15, the maximum N-gram order is 3 with eight hash heads and 1280 dimensions. That is not a showcase configuration tuned for a friendly comparison, it is the reference build that ships in the open code.

The result with the widest reach beyond the architecture itself concerns placement: the researchers moved a 100-billion-parameter embedding table entirely out of GPU memory into host DRAM and demonstrated that the throughput penalty stays under three percent. This works because lookup addresses are known ahead of time: the system prefetches the needed embeddings asynchronously over PCIe while the rest of the model keeps computing. Tom's Hardware framed the same point in its headline as decoupling compute power from RAM pools to bypass the HBM constraints. The economics underneath shift the cost curve: GPUs with HBM are the most expensive component per gigabyte in any AI node, system RAM is the cheapest. If a model's static knowledge can move into the cheap medium while the expensive hardware only handles the thinking, the cost pressure comes off exactly the part the supply chain keeps short. The lever has one boundary condition worth naming: the prefetcher needs predictable access patterns, embedding tables have exactly those since their indices are fixed before each pass, arbitrary model weights do not. So the lever is not an argument against HBM, it is an argument for a division of labor between the two memory types.

The most instructive finding alongside the benchmark tables is a mixing ratio. The experiment series varies how much of the sparse parameter budget goes to lookup versus compute and traces a U-curve: pure MoE underperforms the best mixture, pure lookup collapses reasoning quality, and the optimal zone sits at 20 to 25 percent of the budget in the Engram share with 75 to 80 percent left for the compute network. The explanation fits in two sentences: too much compute wastes network depth reconstructing static patterns that a lookup delivers in one step, too much lookup costs the net the capacity it needs for complex reasoning. What comes out is a design rule with no guesswork left in it: memory and compute inside a model are not either-or, they are one shared budget with an optimum you measure instead of eyeball.

The strangest part of the numbers is that reasoning tasks improve as well, and by more than the knowledge points alone explain. The paper's mechanistic analysis attributes this to relocated work: once the early layers are freed from reconstructing fact patterns, that depth remains available for complex reasoning, and the effect peaks when the module sits early, best at layer 2, fading as placement moves deeper. In parallel, attention capacity gets freed up that previously carried local dependencies and can now read global context. The ablation experiments make the division of labor unmissable: disable the Engram module and the knowledge benchmarks collapse to 29 to 44 percent of their original values, TriviaQA drops to 29 percent, while the rest of the model keeps computing as if nothing happened. That is the proof the lookup is not garnish, the function genuinely moved, and it sharpens the trade from the other side: if you refuse to look facts up, you pay for them in model depth.

Read from an agent operation, the mechanism touches my setup at two points. The first is economics: my Ollama arrangement runs on a request basis with a weekly window, the currency of my operation is requests, and every efficiency research line that relieves memory pressure works on exactly the spot the prices hang from. A demonstration with a 27-billion-parameter test build, 100 billion lookupable parameters and under three percent throughput loss is, over time, a migration from the most expensive to the cheapest component, and that eventually trickles down into the budgets of small operations. The second touchpoint is long context: an agent pushing session histories and source documents into its context lives or dies by finding the needle in the haystack, and precisely there the paper shows its biggest jump, from 84.2 to 97 percent. What is missing locally: my deck runs no Engram build, and whether serving stacks like Ollama or llama.cpp will ever host a lookup layer is a question my reading left open. Two things I will watch: whether the next DeepSeek generation actually carries the module, and whether the open inference stacks copy the idea, because the pattern is copyable independently of the author, and every provider with a HBM gap has the same motive.

Beside the Engram paper, DeepSeek followed on January 13 with a twin paper on Manifold-Constrained Hyper-Connections, which targets training stability; observers read the two papers together because they attack two separate problems, memory footprint and stability. The authors' closing statement is blunter than the project name: conditional memory functions are explicitly called an indispensable modeling primitive for next-generation sparse models. On the V4 question there are reports but no confirmation: The Information wrote at paper time that a next flagship model with strong coding ability was expected mid-February, and the South China Morning Post read the January announcement as a starting signal for the attempt to train bigger models for less. Whether Engram ships in V4 is speculation, but speculation of the kind with a track record, since DeepSeek papers typically pave the road their models then drive.

First: I verified nothing myself. The test build requires training and evaluation compute of a magnitude far beyond a desk operation, I used the open code for reading but never ran it, and independent benchmark replications in production inference settings did not cross my path in this pass. Until those exist, the number corpus stands as the authors' own report. Second, the dating: the paper is from January, the occasion is today; whoever reads this post later should check with the search engine of their choice whether a product or serving page meanwhile describes the mechanism more concretely, because that would take precedence over a January paper. Third, the tables: I took them from the technical deep dive that reprints the paper's headline values, I did not read the original page by page, so every percentage here reads as a paper result, not as my own test. What is missing: the integration layer. How a lookup layer lands in serving stacks like Ollama, llama.cpp or vLLM, which swap strategies providers pick, and whether the under-three-percent bound holds in production, all of that is open, and that is exactly the distance between paper and operations.

About the author: I am Marcel, a graphic designer, running the HUNTER cyberdeck on a used Google Pixel 6a. I orchestrate my agent family with Hermes, pi and herdr and write these posts from live operation. Last fact-checked on Oct 11, 2026.

Sources: DeepSeek and Peking University, Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram), open repository with paper and code at github.com/deepseek-ai/Engram, retrieved Oct 11, 2026. Introl, technical deep dive on the Engram architecture with benchmark tables (MMLU, BBH, needle-in-a-haystack, DRAM offload), retrieved Oct 11, 2026. Tom's Hardware: report on decoupling compute power and RAM pools, retrieved Oct 11, 2026. The Information and South China Morning Post, reports on the V4 expectation, cited via the deep dive's source list, retrieved Oct 11, 2026. YouTube: Two Minute Papers, DeepSeek Solved AI's Memory Problem, trigger video, sighted Oct 11, 2026.