MongoDB – Change Streams and Time-Series

Two features that both sit on top of the oplog, and both get oversold. Change streams let your application subscribe to writes as they happen, which removes a whole category of polling code. Time-series collections are a storage engine for append-only measurements — and their headline benefit turns out to depend entirely on your data, in a way the marketing does not mention.

Change streams

A change stream is a tailable cursor over the oplog. MongoDB hands you every write as it is committed, with no polling and no message broker:

const stream = db.reels.watch([
  { $match: { operationType: { $in: ["update", "replace"] } } }
], { fullDocument: "updateLookup" })

while (stream.hasNext()) {
  const change = stream.next()
  print(change.operationType, change.documentKey._id, change.fullDocument.stats.views)
}

Three details in that snippet are the difference between it working and it silently doing nothing.

It needs a replica set. A standalone has no oplog, and the error says so without saying what to do about it: The $changeStream stage is only supported on replica sets.

An update event carries no document by default. Change events describe the change, not the result — so an update gives you the changed fields and nothing else. fullDocument: "updateLookup" makes MongoDB re-read the document and attach it. Without it, change.fullDocument is null on every update, and the cause is not obvious.

Filter on the server. The pipeline passed to watch() runs on the server, so an unfiltered stream ships every write in the collection to your process to be discarded there.

What an event looks like

{
  "operationType": "update",
  "documentKey": { "_id": "..." },
  "updateDescription": { "updatedFields": { "stats.views": 4963 }, "removedFields": [] },
  "fullDocument": { "...": "only present with updateLookup" },
  "_id": { "_data": "8267..." }
}

That last _id is the resume token, and it is the part that makes change streams usable in production. Store it, and after a restart you can resume from exactly where you stopped rather than losing everything that happened while you were down:

db.reels.watch([], { resumeAfter: savedToken })

The window is bounded by the oplog, so a consumer offline longer than the oplog covers cannot resume and must fall back to a full read.

In Java

The cursor blocks forever, so it needs its own thread. Spring Data provides a container that handles that:

var request = ChangeStreamRequest.builder(listener)
        .collection("reels")
        .filter(new Document(
                "$match", new Document("operationType", new Document("$in", List.of("update", "replace")))))
        .fullDocumentLookup(FullDocument.UPDATE_LOOKUP)
        .build();

One trap specific to Spring Data, and it cost real time in ReelCMS: there are two filter overloads. The one taking an Aggregation maps field names against your domain type — and operationType is a field of the change event, not of Reel, so the mapper rewrites it into something that matches nothing. The stream opens cleanly and delivers silence. Passing a raw Document, as above, skips the mapping and works.

What they are good for

Pushing live updates to a browser without polling, invalidating a cache the moment its source changes, feeding a search index, or emitting an audit record. In ReelCMS the dashboard’s live view counter is a change stream forwarded over server-sent events — the counter ticks as a side effect of the $inc, with no publish step anywhere in the application.

What they are not is a message queue. There is no acknowledgement, no dead-letter handling, no consumer group. If a message must be processed exactly once, use a queue.

What you can watch, and for what

watch() exists at three levels, and the wider two are easy to miss:

db.reels.watch()      // one collection
db.watch()            // every collection in the database
db.getMongo().watch() // the whole deployment

The deployment-wide form is what you want for an audit log or a cache invalidator, because it does not need updating every time somebody adds a collection.

The operationType values worth handling:

• insert, update, replace, delete — the ordinary four. Note that update and replace are different events, so a filter that only watches update misses a save() from an object mapper.
• drop, rename, dropDatabase — structural.
• invalidate — the stream is finished. This arrives after a drop or rename, and the cursor cannot be resumed from a token past it.

A delete event carries only documentKey, because the document is gone — updateLookup cannot help. If your consumer needs to know what was deleted, enable fullDocumentBeforeChange, which requires turning on change-stream pre-images for the collection and costs storage.

Time-series collections

A time-series collection is a different storage layout for append-only measurements. You declare a timeField, usually a metaField, and MongoDB buckets documents that share metadata within a time window, storing the measurements column-wise:

db.createCollection("view_events", {
  timeseries: { timeField: "ts", metaField: "metadata", granularity: "seconds" },
  expireAfterSeconds: 7776000
})

Two constraints come with it. The collection must be created with these options before anything inserts — insert first and MongoDB auto-creates an ordinary collection that works fine and quietly costs more. And documents cannot be updated or deleted individually; expiry is by TTL on whole buckets. For measurements that is fine, because a measurement is a fact.

The measured result — and it is not what the pitch says

The claim is large storage savings. Measured on ReelCMS’s actual data — 30,187 view events, after forcing a checkpoint:

time-series, random insert order : 0.84 MB
time-series, sorted by timestamp : 0.85 MB
ordinary collection              : 0.91 MB

A saving of about 1.1×. Effectively a wash. Insert order made no measurable difference either, which was the first hypothesis and it was wrong.

The reason is bucket density:

measurements                    : 30,187
buckets                         : 17,967
average measurements per bucket :      1.7
distinct metadata combinations  :     49

Columnar compression pays off in proportion to how many measurements share a bucket. At 1.7 per bucket there is nothing to compress: 49 metadata combinations scattered across 30 days of hourly windows produce roughly 35,000 possible buckets for 30,000 events.

So the honest rule is: time-series collections are not a free win, and the saving is proportional to bucket density. They pay off with few metadata combinations and a high event rate — one sensor emitting every second fills buckets beautifully. They do nothing for sparse events spread across many metadata values. Check for yourself before assuming a benefit:

db.getCollection("system.buckets.view_events").countDocuments({})   // 17967
db.view_events.countDocuments({})                                   // 30187

The TTL and the time-range query optimisations are still worth having on their own. The compression is a maybe, and it is measurable in ten minutes.

Running one in production

Four things separate a demo consumer from one you can leave running:

1. Persist the resume token, and only after the event has been handled. Storing it first turns a crash into silent data loss.
2. Expect duplicates. Resuming replays from the token, so an event can arrive twice. Make the handler idempotent — which usually means keying on documentKey._id rather than counting.
3. Run one consumer, not one per instance. Every instance of your application opening the same stream means every event handled N times. Either elect a leader or accept the duplication deliberately.
4. Handle invalidate. A dropped or renamed collection ends the stream, and the stored token is no longer usable. The consumer has to start fresh rather than retry forever.

None of that is difficult, and all of it is the sort of thing that gets discovered in production rather than designed in.

The short version

• Change streams need a replica set — they are the oplog, exposed.
• fullDocument: "updateLookup" or update events carry no document.
• Filter server-side; store the resume token so a restart does not lose events.
• Not a message queue: no acks, no redelivery, no consumer groups.
• Create a time-series collection before the first insert; it cannot be converted.
• Its storage saving depends on measurements-per-bucket. Measure yours — ReelCMS got 1.1×, which is nothing.