NDJSON to JSON

Turn newline-delimited JSON into a single array. Every line is parsed independently, and any line that fails is reported with its number rather than taking the whole file down.

NDJSON is what you get from log pipelines, BigQuery exports, Elasticsearch bulk files and streaming APIs, and it is what causes the "Extra data" and "unexpected non-whitespace character" errors when it is parsed as one document.

What NDJSON is

One complete JSON value per line, separated by a newline. No commas between records, no wrapping array. Blank lines are ignored. The conventional extensions are .ndjson and .jsonl.

JSON Lines and NDJSON are effectively the same format described by two small specifications that agree on everything that matters. Some tools name one and some the other; a file written for either is read correctly by both.

Why it exists

Three real advantages, all of which come from records being independent.

It streams

A consumer processes one record at a time and never holds the whole file. A 50 GB export is fine; a 50 GB JSON array is not.

It appends

Adding a record is a single write to the end of the file. Appending to a JSON array means rewriting the closing bracket, which is not an append at all.

It survives corruption

One malformed line costs you one record. One malformed byte in a JSON array costs you the file.

Which way to convert

To an array when the data is going somewhere that expects one document: a browser, an API body, a config file. To NDJSON when it is going into a pipeline, a log, an append-only file or anything that streams. Both directions are available above.

How to do this in code

Reading and writing NDJSON in code.

import json

# Read
with open('events.ndjson') as f:
    records = [json.loads(line) for line in f if line.strip()]

# Write
with open('events.ndjson', 'w') as f:
    for r in records:
        f.write(json.dumps(r) + '\n')

# pandas knows the format
import pandas as pd
df = pd.read_json('events.ndjson', lines=True)

# NDJSON to an array
jq -s . events.ndjson > events.json

# An array to NDJSON
jq -c '.[]' events.json > events.ndjson

# Filter a huge NDJSON file without loading it all
jq -c 'select(.level == "error")' events.ndjson

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const rl = createInterface({
  input: createReadStream('events.ndjson'),
  crlfDelay: Infinity,
});

for await (const line of rl) {
  if (!line.trim()) continue;
  const record = JSON.parse(line);
  // one record at a time, constant memory
}

Questions

Is NDJSON the same as JSON Lines?

For every practical purpose, yes. They are two small specifications that agree on the important parts: one JSON value per line, UTF-8, newline separated. The .jsonl and .ndjson extensions are used interchangeably.

Can a record span several lines?

No. That is the whole point of the format: a line break is the record separator, so every record must be on exactly one line. Minify each record before writing it.

Why does my NDJSON file fail to parse as JSON?

Because it is not one JSON document, it is many. JavaScript reports "Unexpected non-whitespace character after JSON" and Python reports "Extra data". Both mean the parser finished one value and found another.