Menu

Parquet vs Avro vs ORC for Data Engineering
📰

Parquet vs Avro vs ORC for Data Engineering

DEV Community: database·Keshav·5 days ago
#SsZAdl7k
#devtodatabase#devto#dev#parquet#avro
Reading 0:00
15s threshold

The first question is row or column? Parquet and ORC store data by column; Avro stores it by row. That determines what each is good at: columnar formats for scanning a few columns of many rows, row formats for writing and reading whole records one at a time, as in a message stream.

Parquet ORC Avro

Layout Columnar: row groups, then column chunks Columnar: stripes, then column streams Row-based: blocks of records

Schema In the file footer In the file footer JSON schema in the file header (or in a schema registry for messages)

Statistics for skipping Min/max/null count per row group and page; optional bloom filters Min/max per file, stripe and every 10,000 rows; optional bloom filters None

Splittable Yes, at row groups Yes, at stripes Yes, at sync markers between blocks

Schema evolution Add columns; readers match by name (or field ID) Add columns; readers match by name or position Explicit reader/writer schema resolution rules

Typical ecosystem Spark, Delta Lake, Iceberg, Hudi, warehouses, DuckDB,…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.