Convert NDJSON to Parquet

NDJSON (Newline Delimited JSON) is a handy format for storing or streaming structured data that you can process one record at a time. Each line in an NDJSON file is a valid JSON object, with a newline character separating them.

This format works great for handling large datasets. You can easily parse and process data line by line without loading the entire file into memory.

Key Features

• Each line is a complete JSON object
• Easy to read and write sequentially
• Great for streaming and logging
• Allows partial processing of large datasets

NDJSON is popular in data pipelines, log processing, and situations where you need to add new data to existing files without changing the whole structure.

Apache Parquet is a columnar storage format that works great with big data processing frameworks. It compresses and encodes data efficiently, which saves a lot of storage space and makes reading data faster.

Parquet can handle complex nested data structures. It lets you query and work with specific columns without having to read the entire dataset.

Compression

Parquet supports several compression algorithms like Snappy, Gzip, and LZO. These compression techniques help reduce storage space and speed up data processing tasks.