The duplicate is not the hard part

A streaming system rarely duplicates events because someone wrote send() twice in the obvious place.

It duplicates events because distributed systems retry:

  • a producer times out after the broker already accepted the write
  • a connector restarts from an old offset
  • a consumer processes an event and crashes before committing progress
  • a webhook sender retries because your 200 OK was lost
  • a backfill overlaps with live traffic

So the first instinct should not be "use Redis" or "use Flink."

The first instinct should be:

What makes two events the same event?
How long can a duplicate arrive late?
Where is the side effect that must not happen twice?

Deduplication is memory with a boundary. You remember enough event identities for long enough to reject repeats.

Everything else is implementation detail.

Start with the identity

You cannot deduplicate until you define the dedup key.

For a payment event, the key might be:

payment_id

For a tracking event, it might be:

source + event_id

For CDC, it might be:

table + primary_key + log_sequence_number

For an IoT sensor, it might be:

device_id + sequence_number

This choice is more important than the technology.

If the key is too narrow, different events collapse into one. If the key is too wide, duplicates slip through because retries are not byte-for-byte identical. If the key is generated downstream, it cannot protect upstream side effects.

dedup identitywhat makes two events the same

Then choose the memory horizon

The second question is how long the system must remember a key.

There are three common answers.

Short horizon: seconds or minutes
Medium horizon: hours or days
Permanent horizon: forever, or as long as the business record exists

Short horizons fit cache-based dedup well. Medium horizons fit stream processors with state and time. Permanent horizons usually belong in the destination database, ledger, or warehouse table.

memory horizonhow long a duplicate can arrive late
shortseconds – minutes
Redis
mediumhours – days
Flink
permanentforever
Database

The cost grows with:

unique keys per second x retention window x bytes per key

If you receive 100,000 unique event ids per second and remember them for one hour, you are storing up to 360 million ids before overhead. That is the real scaling problem.

At scale, dedup is mostly about controlling state size without lying to yourself about correctness.

Approach 1: Redis as an idempotency gate

Redis is the simple and useful version for many systems.

The common shape is:

SET dedup:{event_id} 1 NX EX 86400

If Redis returns success, this worker is the first one to claim the event inside the retention window. Process it. If Redis returns no write, treat the event as a duplicate and skip it.

That makes Redis a fast idempotency gate.

event -> Redis SET NX EX -> first time? -> process
                          -> duplicate?  -> drop

redis idempotency gateSET dedup:{id} NX EX 86400

This works well when:

  • the duplicate window is bounded
  • the event id is stable
  • losing dedup memory after the TTL is acceptable
  • the protected side effect happens after the Redis claim
  • operational simplicity matters more than stream-native replay semantics

The danger is treating Redis as magic exactly-once processing.

If you write the Redis key, then crash before the real side effect, a retry may be suppressed even though the work never finished. If you do the side effect first, then crash before writing Redis, a retry may do the side effect twice.

You can reduce that gap with status values, Lua scripts, leases, transactional outbox patterns, or by making the final side effect idempotent too. But the gap is the important lesson.

Redis dedup protects a boundary. It does not automatically make the whole pipeline exactly-once.

Scale this fits: Redis works well for short-lived idempotency at service edges: API retries, webhook deliveries, worker retries, and retry windows measured in seconds, minutes, or sometimes hours. The practical limit is memory. If you ingest 100,000 unique keys per second and keep each key for one hour, Redis must hold up to 360 million keys before object overhead, replication, fragmentation, and clustering costs.

Drawbacks: Redis keeps the dedup set in memory. That is why it is fast, and also why long horizons get expensive quickly. It also sits outside the stream processor's checkpoint model. If Redis is unavailable, evicts keys, loses data during failover, or accepts a claim before the worker crashes, the rest of the pipeline has to decide what correctness means.

Flink is a better fit when deduplication is part of the stream computation itself.

For example:

key the stream by event_id
remember whether that key has been seen
emit the first event
drop later events for the same key until the state expires

In the DataStream API, the core shape is keyBy plus keyed state:

events
  .keyBy(Event::eventId)
  .process(new KeyedProcessFunction<String, Event, Event>() {
    private ValueState<Boolean> seen;

    @Override
    public void open(Configuration config) {
      StateTtlConfig ttl = StateTtlConfig
        .newBuilder(Time.hours(24))
        .build();

      ValueStateDescriptor<Boolean> descriptor =
        new ValueStateDescriptor<>("seen", Boolean.class);

      descriptor.enableTimeToLive(ttl);
      seen = getRuntimeContext().getState(descriptor);
    }

    @Override
    public void processElement(
      Event event,
      Context context,
      Collector<Event> out
    ) throws Exception {
      if (seen.value() == null) {
        seen.update(true);
        out.collect(event);
      }
    }
  });

That is the whole idea: after keyBy, every event with the same dedup key is routed to the same logical state partition. The operator checks local keyed state, emits only the first event, and lets TTL or event-time timers clear old keys.

The implementation is stateful. Flink must remember keys it has seen, and it must know when that memory can be cleared. That is where state TTL, event-time timers, watermarks, checkpointing, and backend choice become central.

This works well when:

  • events are already flowing through Flink
  • dedup is event-time aware
  • late events need a principled policy
  • the stream is partitionable by dedup key
  • replay and checkpoint recovery matter
  • downstream consumers should see a cleaned stream

Flink's advantage is not just speed. It is that dedup state participates in the stream processor's fault-tolerance model.

If the job restarts, state can come back from checkpoints. If events arrive late, event-time logic can decide whether they still belong to the valid horizon. If the stream is keyed correctly, the state scales across workers.

The cost is operational complexity.

You now manage state growth, checkpoint size, backpressure, RocksDB or another state backend, watermark behavior, schema evolution, and job upgrades.

Scale this fits: Flink is the better shape when the dedup set is too large or too stream-native for a single service cache. Compared with Redis, Flink can usually support much larger dedup horizons because its state is partitioned and can be disk-backed, while Redis dedup is primarily constrained by memory cost. With a disk-backed state backend such as RocksDB, remembered keys can be spread across task slots and stored on local disk, with checkpoints copied to durable storage. That makes it a reasonable fit for high-throughput streams, multi-hour or multi-day windows, and replayable pipelines.

Drawbacks: Flink does not remove the state problem. It makes the state managed, partitioned, checkpointed, and recoverable. Large key cardinality still means large state, slower checkpoints, more disk IO, longer recovery, and more painful rescaling. Dedup only scales if the key distribution is healthy; a hot key still becomes a hot partition.

Approach 3: Database uniqueness at the sink

Sometimes the best dedup layer is the destination.

If the destination has the truth, make it reject duplicates:

create unique index payments_event_id_key
on processed_payments (event_id);

Then the consumer can attempt an insert and let the database decide:

insert into processed_payments (event_id, amount, user_id)
values ($1, $2, $3)
on conflict (event_id) do nothing;

This is boring and strong.

It works well when:

  • the final record naturally has a unique key
  • the dedup horizon is long or permanent
  • the write volume fits the database
  • the side effect is the database write itself
  • correctness matters more than shaving a few milliseconds

The weakness appears when the database write is not the only side effect.

If the consumer sends an email and then inserts the row, a crash can still send the email twice. If it inserts the row and then sends the email, a crash can suppress the email forever unless another process observes the row and sends from an outbox.

That is why durable side effects often use this pattern:

consume event
write business row + outbox row in one DB transaction
separate relay sends side effect from outbox
mark outbox row sent

The unique key protects the durable state. The outbox makes the side effect recoverable.

Scale this fits: Database uniqueness is strongest when the dedup horizon is permanent and the database is already the system of record. It is appropriate for orders, payments, ledger entries, entitlement changes, and CDC materialization where one key should exist once forever.

Drawbacks: The database is durable, not free. A unique index has write amplification, storage cost, lock contention, and hotspot risk. It is also usually the wrong place to absorb every noisy duplicate from a huge stream if most of those duplicates could have been filtered earlier. Use the database as the correctness layer, then add upstream dedup to protect it from unnecessary load.

Approach 4: Bloom filters and approximate dedup

Approximate dedup is useful when the stream is enormous and exact state is too expensive.

A Bloom filter can answer:

Have I probably seen this key before?

It has no false negatives if used correctly: a key that was added should be recognized later. But it can have false positives: a new key may be incorrectly classified as already seen.

That tradeoff is unacceptable for payments and orders.

It can be acceptable for analytics, abuse signals, metrics, clickstream compaction, crawler URLs, or pre-filtering before a more expensive exact check.

Approximate dedup works well when:

  • false drops are tolerable
  • the exact set would be too large
  • the output is statistical or best-effort
  • the filter is scoped by time bucket
  • a second exact layer can catch important cases

Approximate dedup buys memory efficiency by spending certainty.

Scale this fits: Bloom filters fit very large streams where the exact set would be too expensive and occasional false drops are acceptable. They are often useful as a front filter for analytics, crawling, abuse scoring, metrics, and other best-effort pipelines.

Drawbacks: A Bloom filter can say "probably seen" for a key that is actually new. That means it can drop real events. Standard Bloom filters also do not delete individual keys cleanly; production designs usually rotate filters by time bucket or use counting variants with more memory. This is a scale tool, not a correctness tool.

Comparing the options

The simplest comparison is by where the memory lives.

where the memory livessame job, different home for the key

Redis
Memory lives in an external low-latency cache.
Best for bounded idempotency gates near services.

Flink
Memory lives in keyed stream processor state, often backed by local disk and checkpoints.
Best for event-time-aware stream cleanup and keyed transformations.

Database uniqueness
Memory lives in the durable destination.
Best for permanent business records and sink-level correctness.

Bloom filter
Memory lives in a compact probabilistic structure.
Best when false positives are acceptable.

The wrong question is:

Which tool deduplicates best?

The better question is:

Where can I afford to remember this key, and what failure happens if that memory is wrong or unavailable?

A practical decision path

If the duplicate can cause a permanent business error, start at the sink.

Use a unique constraint, idempotency key, ledger key, or provider-level idempotency key. Then add upstream dedup to reduce load, not to be the only correctness layer.

If the duplicate pollutes stream aggregations, handle it in the stream processor.

Use Flink or another stateful processor so dedup, watermarks, windows, checkpointing, and replay behavior are part of the same model.

If the duplicate is a short retry around an API or worker, Redis is often enough.

Use SET NX with an expiry, understand the crash gap, and keep the TTL tied to the maximum retry window.

If the stream is massive and the cost of exact state is worse than occasional false drops, consider approximate filters.

Do not use them for money.

The practical takeaway

Deduplicating streaming events at scale is not a single-system problem.

It is three decisions:

identity: what makes two events the same?
horizon: how long can a duplicate arrive late?
boundary: where must the duplicate be stopped?

Redis, Flink, databases, and Bloom filters are different answers to the boundary question.

Redis is fast in-memory state near the service. Flink is managed keyed stream state that can spill to disk and recover from checkpoints. A database constraint protects durable truth. A Bloom filter trades exactness for compactness.

The strongest designs usually layer these: cheap upstream suppression plus durable idempotency at the place that actually matters.

Sources