Language Data Format, v0.1
Tables a language model can read.
LDF turns JSON records into typed tables: one header, one line per record, child tables for nested arrays. Nothing is lost on the way in or out, and a frontier model reads it as well as JSON for 35 to 45 percent fewer tokens.
- Lossless. Byte-exact round trip on ten of twelve benchmark datasets; the two sparse ones come back with keys reordered, nothing lost.
- Typed. Integers, floats, strings, booleans, JSON cells and child tables, declared once in the header.
- Measured. Four models, three sizes, ten formats. The misses are in the report, not hidden.
@records order_id:s,customer.id:i,customer.name:s,items:t,total:f,status:s,note:s?
ORD-000001,4654,Val Thompson,2,715.94,shipped,context compare
ORD-000002,50,Sol Ritchie,1,2757.03,cancelled,null
@records.items order_id:s,sku:s,qty:i,price:f
ORD-000001,E5,3,28.78
ORD-000001,C3,4,157.4
ORD-000002,A1,1,2757.03[{"order_id":"ORD-000001","customer":{"id":4654,"name":"Val Thompson"},
"items":[{"sku":"E5","qty":3,"price":28.78},{"sku":"C3","qty":4,"price":157.4}],
"total":715.94,"status":"shipped","note":"context compare"},
{"order_id":"ORD-000002","customer":{"id":50,"name":"Sol Ritchie"},
"items":[{"sku":"A1","qty":1,"price":2757.03}],
"total":2757.03,"status":"cancelled","note":null}]The format
Seven rules. That is the whole spec.
A document is a sequence of tables. Each table is a header line and rows. Everything else follows from the data.
@records a:i,b:sA header names the table and types each column: i integer, f float, s string, b boolean, j any JSON, t child table. ? marks a column that may be null or absent.
1,helloA row is one record, cells separated by commas. Strings are bare unless they need quoting, then they are JSON strings.
, and nullAn empty cell means the key is absent. null means it is present and null. JSON's distinction survives.
customer.name:sObjects with a stable shape flatten into dotted columns. Irregular ones stay as a JSON cell.
@records.itemsArrays of objects become a child table joined on the parent's first column. The parent keeps a count.
~:j?Keys present in fewer than half the rows travel in a small per-row object instead of a mostly empty column. Measured to matter for smaller models.
1.0 stays 1.0Numbers keep their kind and shortest round-trip text. Integers above 2^53 do not get mangled.
Numbers
Tokens, as a share of compact JSON.
o200k_base tokens on the first 1 000 records of the benchmark datasets, from the report's own tables. CSV ties LDF where CSV works at all; CSV cannot hold nested data and drops the difference between missing and null.
| dataset | LDF | TOON | CSV |
|---|---|---|---|
flat_users | 0.61 | 0.65 | 0.60 |
nested_orders | 0.64 | 1.08 | n/a |
sparse_events | 0.80 | 1.22 | 0.61, lossy |
wide_table | 0.43 | 0.44 | 0.43 |
empty_values | 0.36 | 1.27 | n/a |
deep_nesting | 0.80 | 1.82 | n/a |
Below the frontier tier no format is reliably better than another, and the report says so. Read the methodology before quoting any number.
The middle man
Nobody has to adopt a format. Put it between the data and the model.
ldf-mcp is an MCP server. In proxy mode it wraps any stdio MCP server you already use, forwards every message, and re-encodes JSON tool results as LDF when that is shorter. The model keeps calling the same tools and receives cheaper data. In ClickHouse mode it is the server: read-only SQL over the HTTP interface, rows returned as LDF.
claude mcp add clickhouse -- ldf-mcp clickhouse --url http://127.0.0.1:8123
claude mcp add my-data -- ldf-mcp proxy -- <any stdio MCP server> [its args]Status
A v0.1 prototype with its evidence attached.
One Rust workspace: the format, ten baseline formats implemented faithfully (TOON passes its official conformance suite), a benchmark harness, an evaluation harness and the MCP server. Every dependency pinned, no network calls outside the MCP server's own query.