10 min read

How OpenAI Scaled PostgreSQL to 800M Users, and What to Do in Your System

OpenAI kept one PostgreSQL primary alive for a massive read-heavy workload by moving reads to replicas, pooling connections, protecting caches, rate limiting spikes, and pushing write-heavy workloads elsewhere. The useful lesson is not "Postgres scales forever." It is knowing which pressure you are actually under.

Data Engineering

A practical guide to the OpenAI PostgreSQL scaling story: what they did, why it worked, and what normal engineering teams can do with batching, Kafka, idempotent writes, read replicas, caches, and workload isolation.

0 comments
9 min read

How Flink Works Internally

Flink is not just a library that runs a loop over events. It turns a streaming program into a distributed dataflow, splits operators into parallel subtasks, places those subtasks into task slots, and keeps state recoverable with checkpoints.

Data Engineering

A practical walkthrough of Flink internals: what problem Flink solves, how operators become tasks and subtasks, how parallelism and keyBy scale a job, and how TaskManagers, slots, state, and checkpoints fit together.

0 comments
7 min read

Why Netflix 4K Can Look Worse Than a Theater at 1080p

Resolution is only the pixel count. The picture also needs enough bits to describe those pixels. A 4K stream can have more pixels than a theater image and still give each pixel far less data.

Data Engineering

4K streaming is not automatically better than a lower-resolution theater image. Bitrate, compression, mastering, color, motion, and viewing conditions all decide how much detail survives.

0 comments
8 min read

How to Deduplicate Streaming Events at Scale

Redis, Flink, databases, and Bloom filters can all remove duplicates, but they are not interchangeable. The right choice depends on where the duplicate appears, how long you must remember it, and what happens if you are wrong.

Data Engineering

Streaming deduplication is not one trick. It is a state-management problem: pick the event identity, choose the memory horizon, and put the dedup state where the failure mode is acceptable.

0 comments
6 min read

Why Kafka Needed ZooKeeper

Before KRaft, Kafka used ZooKeeper as its coordination layer: broker membership, controller election, topic metadata, partition leaders, and ISR state all needed a consistent place to live.

Data Engineering

ZooKeeper was not just extra infrastructure. It was the consensus-backed metadata brain that kept Kafka brokers from disagreeing about cluster state.

0 comments
5 min read

Why Columnar Databases Feel Like Cheating

Analytical databases like BigQuery and Redshift are fast because they stop pretending every query needs every field. They read columns, compress patterns, and skip the bytes your question never asked for.

Data Engineering

If your query asks for average transaction value, a row database may drag every field through memory. A columnar database can go straight to the one column that matters.

0 comments
Menu