ctrlb-decompose separates what repeats in your logs from what actually changes. This is the math, the single-pass algorithm, and the type-aware encoding behind it.
01 · The problem
More logs in, worse answers out
Past a certain point, pasting more logs into an LLM makes it worse at helping you. Answers turn vague, earlier context gets dropped, and the model loses track of the error you started with.
The cause is mechanical. A single log line tokenizes to roughly 80 to 150 tokens, and most of them are structure: timestamps, separators, UUID hyphens, key names repeated on every line. The information is closer to 30 or 40 tokens. The rest is scaffolding that the tokenizer dutifully splits into pieces and the model dutifully attends to.
Repetition multiplies the waste. Most of a log file is a handful of templates printed over and over with different values, so the context window fills with near-duplicates while the one line that matters competes for attention. Agent workflows compound it: every debugging loop re-reads the same logs and pays for the same noise again. No amount of prompt engineering fixes this, because the damage is done before the model reasons about anything. The fix has to happen before tokenization.
02 · The solution
Deduplicate before the model sees a token
ctrlb-decompose is a single Rust binary that does the deduplication at the source. It comes out of CtrlB's research on log storage, where the decomposition module was deliberately built as a self-contained pure function, from raw message bytes to typed output, precisely so it could be released independently of the platform around it.
It splits logs into what repeats, the template, and what actually changes, the values, so you stop paying for the same structure twice.
The research paper defines this formally, and the formalism is worth reading because everything else follows from it. Let L = {m1, m2, …, mN} be a corpus of N log messages, each a string over alphabet Σ. The decomposition maps every message to a pattern-variable pair:
where pi ∈ P is the pattern, the static text template with variable positions replaced by typed placeholder tokens, and Vi = (v1(i), v2(i), …, vki(i)) is the ordered sequence of extracted variable values. The decomposition is lossless: given (pi, Vi), the original message reconstructs exactly by substituting each value back into its placeholder position. Nothing is summarized away. The repetition is simply stored once.
03 · Why it works
Entropy separation
Log files are not prose. They are the output of a finite set of code paths, and each path prints one fixed sentence with a few blanks in it. This observation underlies a whole family of log parsing techniques, Drain, Spell, and CLP among them, and CtrlB's paper sharpens it into a clean information-theoretic statement. The entropy of a raw message corpus decomposes as:
where H(P) is the entropy of the pattern distribution and H(V | P) is the conditional entropy of the variables given the pattern. The decomposition is effective because these two components are wildly unequal. The pattern set P has very low cardinality: a typical 200K-row file contains |P| < 1,000 unique patterns, with the top 10 covering 80-95% of messages in a Zipf-like distribution. Writing f(p) for the frequency of pattern p:
Plug in a file where the top 10 patterns account for 90% of messages and H(P) comes out to roughly 3-5 bits per message. The raw message it replaces is 50-200+ bytes. That gap, three orders of magnitude between the structural component and its raw representation, is the entire budget this tool spends. Almost everything you pay to store, ship, or tokenize in a log line is carrying almost none of its information.
The consequences split cleanly by component. Patterns are so few that dictionary-encoding them is nearly free: with |P| < 1,000, each pattern ID needs ⌈log2 1000⌉ = 10 bits in place of a string of hundreds of bytes, and in CtrlB's storage engine the fully compressed pattern column typically occupies less than 0.5% of the raw message size. The variable component H(V | P) carries most of the information, but once separated from the pattern, variables stop being opaque substrings. They can be typed and encoded with codecs that exploit their statistical structure, which is where the rest of the compression comes from, and which section 05 covers.
04 · The algorithm
A single pass, no regex
The decomposition runs in one sequential pass over each message. No regex engine is invoked anywhere in the pipeline. The algorithm tokenizes each message using configurable delimiters, then classifies every token with lightweight character-class predicates:
Each recognizer is a constant-time character-class check per character. There is no backtracking, no automaton construction, no NFA simulation. The recognizers evaluate in priority order, so a token that matches UUID is never also tested as HexID, and classifying a token of length |t| costs exactly O(|t|). Tokens classified as Static become part of the pattern pi; every other token is extracted as a typed variable, its position marked in the pattern by a type-specific placeholder byte.
Where it sits relative to CLP and Drain
The paper is explicit about its lineage. CLP performs a similar delimiter-based tokenization with character-class checks, but classifies variables into only two kinds: dictionary variables, repetitive identifiers stored in a dictionary, and non-dictionary variables, numbers encoded as fixed-width 64-bit values. Equation (4) extends that with a richer type system, which is what unlocks the type-specific binary encoding in the next section. Drain goes the other direction: it builds a fixed-depth parse tree that clusters messages into templates by progressively matching prefix tokens. That yields deeper pattern extraction, recognizing for instance that user_login and user_logout share a prefix pattern, but at higher computational cost and with state that must be maintained across messages. The paper's classifier is stateless per message, which makes it trivially parallelizable across ingestion threads.
The open-source CLI combines both ideas. After CLP-style typed normalization, structurally identical lines already share a logtype; a Drain3 clustering stage then merges logtypes whose token similarity clears a threshold, 0.4 by default, and any position where tokens still diverge becomes a <*> wildcard. The clustering is incremental, so each line is still processed exactly once, with no second pass over the data.
What the pass costs
A reasonable worry is that all this analysis makes ingestion expensive. The paper bounds the total work for a message of length n with k tokens:
Tokenization is a single linear scan, O(n). Classification is O(|tj|) per token with no branching beyond character-class checks. Encoding, hex to binary and integer parsing, is linear in each value. The total stays O(n) with a small constant, comparable to the UTF-8 validation any JSON parser already performs. Measured inside CtrlB's ingestion pipeline, the whole decomposition adds less than 5% overhead versus writing raw messages directly, because the dominant costs live elsewhere: JSON parsing upstream, columnar encoding downstream, network I/O. The delimiter scan itself is SIMD-accelerated, using AVX2 mm256cmpeq_epi8 to compare a full vector of characters against the delimiter set in parallel.
05 · Type-aware encoding
Every type gets its natural representation
Once variables are extracted and typed, each type routes to a specialized encoder that exploits its structure instead of treating it as an opaque string:
Type | Encoding | Bits | Rationale |
|---|---|---|---|
UUID | 128-bit binary | 128 | strip dashes, hex to binary |
IPv4 | 4 × 8-bit packed | 32 | dot-decimal to uint32 |
Integer | delta + varint | 8-64 | exploit temporal locality |
Float | int conversion + delta | 64 | lossless float to int64 |
HexID | binary packed | 4 · |v| | hex to binary halves the size |
DictVar | dictionary index | ⌈log₂|D|⌉ | high-frequency repeated tokens |
The arithmetic is satisfying. A UUID stored as its 36-character ASCII string costs 288 bits; strip the four dashes and pack the 32 hex characters into binary and you get 128 bits, a 2.25× reduction before any general-purpose compressor touches it. An IPv4 address like 192.168.1.1 shrinks from 88-120 bits of ASCII to exactly 32. Delta encoding on integer sequences, monotonically increasing request counters and the like, collapses most values to single-byte varints.
Inside CtrlB's storage engine, the decomposed components land as three separate columns: dictionary-encoded pattern IDs, dictionary-variable sequences, and fixed-width encoded values. Analytics queries that never touch the message skip all three, and a substring search like WHERE message LIKE '%task_12%' resolves against the dictionary-variable column without reconstructing a single full message. The open-source tool wraps this same decomposition core behind report formats instead of a storage layer, which is the part that matters for the LLM use case.
06 · What you get
A ranked report, not a wall of lines
Each pattern comes back with its count and share, typed variables with percentiles and top values, anomaly flags for shapes worth a look, and a severity rank that puts errors above noise. The top of the report is the first place to look.
The statistics underneath are all fixed-size sketches, which is the detail that makes the whole thing streamable. Quantiles come from DDSketch at roughly 200 bytes per numeric slot, with no raw values retained. Cardinality estimates come from HyperLogLog++ at about 200 bytes per high-cardinality variable. Example lines are reservoir-sampled into a bounded buffer, and the pattern set itself is capped by LRU eviction at 10,000 clusters, so even a pathological file with unbounded template variety cannot blow up memory.
On top of those accumulators sit the analysis passes: frequency-spike and error-cascade detection, low-cardinality flags for values that suddenly stop varying, bimodal-distribution detection for numerics, severity scoring (ERROR > WARN > INFO > DEBUG), and cross-pattern correlation through temporal co-occurrence and shared variables. None of it requires a second pass, because everything derives from the sketches.
07 · Where it runs
Flat memory, your machine
Because every accumulator is bounded, working memory stays flat whether you feed the tool ten thousand lines or ten million. That budget discipline is what lets one core run as a terminal binary, embed as a Rust library, and compile to WebAssembly for the browser.
Logs are also among the most sensitive artifacts a system produces, so it matters that this is a pure pipe: MIT licensed, stdin to stdout, no service, no upload step, no account. Raw lines never need to leave the machine, and the compact report is small enough to read before you send it anywhere.
08 · Try it
Three commands
# macOS
brew tap ctrlb-hq/tap && brew install ctrlb-decompose
# ranked patterns in your terminal
cat app.log | ctrlb-decompose
# compact markdown for a model
ctrlb-decompose --llm app.log--top N controls how many patterns you see (20 by default), --context N attaches example lines to each, and --json emits the full structured report for tooling. Debian packages and a source build are in the repository.
Pipe something big at it and skim what comes back. You will read 43 lines instead of 1.2 million and so will your model.