I THOUGHT A FILE FORMAT WAS JUST A FILE FORMAT. I WAS WRONG.

Today I learned something that completely changed the way I think about data engineering.

Before today, if someone asked me to save data, I would probably choose whatever format came first to my mind. CSV? Sure. JSON? That works too. They all store data, right?

That's what I believed.

Then I started building a big data pipeline.

I quickly realized that choosing a file format isn't a small implementation detail. It's an architectural decision. It affects how fast your queries run, how much storage you pay for, whether your data stays consistent, and even whether your system can scale when the data grows.

The same data can behave completely differently depending on how you store it.

That surprised me.

CSV: Great for Humans, Not for Big Data

CSV is probably the first format every programmer learns. It looks simple.

textuser_id,event_type,price,country
u001,click,29.99,US
u002,purchase,149.00,UK

You can open it in Excel. You can read it with your eyes. Almost every programming language and tool supports it. For sharing data with someone or quickly exporting a report, CSV is perfect.

But production systems are a different story.

One thing I didn't realize before is that CSV has almost no protection against bad data. Imagine one row stores the price as 29.99 and another row stores $29.99. Both look reasonable to a person. But for a machine, they're completely different values. CSV won't complain. It simply stores whatever you give it.

Another problem is performance. Suppose you only want the average price from a file with millions of rows. Even though you only need one column, the system still has to read every row and every column. That means reading user IDs, countries, timestamps, product IDs — everything. It's like opening an entire book just to find one sentence.

JSON: Better Structure, Same Problem

JSON solves one problem that CSV cannot. It supports nested data. Instead of flattening everything into columns, you can group related information together. For example:

json{
  "user_id": "u001",
  "metadata": {
    "product_id": "p99",
    "price": 29.99
  }
}

This makes APIs much easier to design. That's why almost every web API returns JSON. Kafka messages are also commonly sent as JSON.

But JSON still isn't designed for large-scale analytics. It's verbose. It repeats field names over and over again. It's still row-based. And it still doesn't enforce a strict schema.

The biggest lesson I learned was this:

JSON is excellent while data is moving between systems. It is not the best place to store years of analytical data.

Think of JSON as raw material arriving at a factory. You don't leave it there forever.

Then I Met Parquet

This was the moment everything clicked. Parquet stores data completely differently. Instead of storing rows together, it stores columns together.

Imagine a table like this:

textUser   Event      Price
A      Click      20
B      Purchase   100
C      View       30

CSV stores it row by row.

textA Click 20
B Purchase 100
C View 30

Parquet stores it more like this:

textUsers:
A
B
C

Events:
Click
Purchase
View

Prices:
20
100
30

At first this felt strange. Then I realized why it's brilliant.

If I only need the average price, Spark only reads the Price column. It completely ignores everything else. That means less disk reading, less memory usage, less network traffic, and much faster queries.

The Hidden Superpower

The feature I found most interesting wasn't even the column layout. It was the metadata.

Parquet keeps statistics about each block of data. For example:

textPrices in this block:
Minimum = 10
Maximum = 50

Now imagine you ask:

Give me every purchase above $100.

Spark checks the metadata first. If an entire block only contains prices between 10 and 50, it skips reading that block entirely. No disk access. No unnecessary computation. Just skip it.

I thought that was incredibly clever.

What About Avro?

At first I wondered: if Parquet is so good, why does Avro even exist? The answer is simple. They solve different problems.

Parquet is designed for data at rest. Avro is designed for data in motion.

When Kafka sends millions of messages every second, it needs something compact and fast. That's exactly what Avro provides. Unlike JSON, Avro stores data in a binary format, making messages much smaller and faster to transmit.

It also supports schema evolution. That means you can add new fields later without breaking existing consumers. That's a huge advantage in streaming systems where producers and consumers evolve independently.

The Mental Model I'll Remember

After today's learning, I simplified everything into one picture in my head.

• CSV → Share data with people.
• JSON → Move data between applications and APIs.
• Avro → Stream data through Kafka efficiently.
• Parquet → Store data for analytics and machine learning.

And if you're using Delta Lake... you're already using Parquet underneath.

My Biggest Takeaway

Before today, I thought file formats were just different ways to save data. Now I understand they're actually different tools built for different jobs.

Choosing the wrong one can make your pipeline slower, more expensive, and harder to maintain. Choosing the right one can save storage, speed up queries dramatically, and make your entire architecture scale.

It turns out one of the biggest decisions in a data pipeline isn't the algorithm. Sometimes... it's just the file format.