MongoDB – The Aggregation Pipeline

find answers “which documents match this?”. The aggregation pipeline answers everything else — totals, averages, groupings, reshaping, joins. It is where a MongoDB application stops being a key-value store and starts being able to report on itself.

The mental model is simple and worth getting right immediately: documents flow through an ordered list of stages, and each stage transforms the stream it receives. It is a Unix pipe, not a SQL query.

The shape

db.reels.aggregate([
  { $match: { status: "PUBLISHED" } },            // filter
  { $group: { _id: "$creator.username",           // group
              reels: { $sum: 1 },
              views: { $sum: "$stats.views" } } },
  { $sort:  { views: -1 } },                      // order
  { $limit: 5 }                                   // trim
])

Each stage receives what the previous one produced. After $group the documents no longer look like reels at all — they are the grouped results, and only fields that survived the grouping exist. That is the single most common source of confusion: a $match after a $group can only see the grouped fields.

Note the $ prefix on "$stats.views". Inside an aggregation, a string starting with $ means “the value of this field”; without it, it is a literal string. { $sum: "stats.views" } sums the literal text and returns zero.

$match — and why it goes first

$match uses exactly the same syntax as find. Put it at the top of the pipeline whenever you can, for two reasons that both matter:

It can use an index, but only while it is still the first stage. Once anything has transformed the stream, there is no index to use.
Every later stage does less work, because fewer documents reach it.

The difference is not marginal. A $match that filters ninety percent of a collection makes every subsequent stage ten times cheaper — and moving it below a $group turns an index seek into a full scan.

$group — the one that does the work

_id is the grouping key and it is required. Set it to null to aggregate everything into a single row:

{ $group: { _id: null, total: { $sum: "$stats.views" } } }          // one row
{ $group: { _id: "$status", n: { $sum: 1 } } }                      // one row per status
{ $group: { _id: { creator: "$creator.username", status: "$status" }, n: { $sum: 1 } } }  // compound key

The accumulators you will use:

{ $sum: 1 }                  // count
{ $sum: "$stats.views" }     // total
{ $avg: "$watchSeconds" }    // mean
{ $min: "$publishedAt" }     // earliest
{ $max: "$stats.likes" }     // largest
{ $push: "$title" }          // every value, as an array
{ $addToSet: "$status" }     // distinct values
{ $first: "$title" }         // first in the current order

$addToSet deserves a note: it is how you count distinct values, because there is no $countDistinct accumulator. Collect the set, then take its size with $size in a later stage.

One behaviour to remember from lesson 3: $sum silently skips values that are not numeric. A single string in a numeric field makes the total quietly wrong rather than raising an error.

$unwind — grouping over an array

You cannot group by the elements of an array directly. $unwind makes it possible by emitting one document per element — a reel with three tags becomes three documents that are identical except for the tag:

db.reels.aggregate([
  { $match: { status: "PUBLISHED" } },
  { $unwind: "$tags" },
  { $group: { _id: "$tags", reels: { $sum: 1 }, views: { $sum: "$stats.views" } } },
  { $sort: { views: -1 } },
  { $limit: 10 }
])

That is ReelCMS’s engagement-by-tag report, and it is the same pipeline in Java:

var agg = newAggregation(
        match(Criteria.where("status").is(ReelStatus.PUBLISHED)),
        unwind("tags"),
        group("tags")
                .count()
                .as("reels")
                .sum("stats.views")
                .as("views")
                .sum("stats.likes")
                .as("likes"),
        sort(Sort.Direction.DESC, "views"),
        limit(limit));

Be careful with sums after an unwind. Total views here is the sum per tag, so a reel with three tags contributes its views three times across the whole result. That is correct for “views by tag” and wrong if you read the column as a grand total — the numbers will not add up to the site total, and they should not.

$project and $addFields

$project chooses fields and computes new ones. $addFields (and its identical twin $set) adds without removing anything, which is usually what you want:

{ $addFields: { engagement: { $divide: ["$stats.likes", { $max: ["$stats.views", 1] }] } } }
{ $project:  { title: 1, engagement: 1, _id: 0 } }

$max around the divisor is not decoration. Dividing by zero in an aggregation does not throw — it produces null, which then poisons any $avg downstream and gives you an empty result with no explanation. Guard every divisor that could be zero.

Getting the grouping key back

A detail that catches everyone: $group always puts the grouping key in a field called _id, whatever it was called before. Group by tags and the result has _id, not tags:

{ $group: { _id: "$tags", views: { $sum: "$stats.views" } } }
// → { _id: "motorsport", views: Long("5733") }

{ $project: { tag: "$_id", views: 1, _id: 0 } }     // rename it back if you want

This matters more than it sounds, because the next stage — a $lookup, or a sort — has to refer to _id too.

Two shortcuts worth knowing

Counting and counting-by-something are common enough to have dedicated stages, and both are clearer than the $group they replace:

{ $count: "total" }             // instead of { $group: { _id: null, total: { $sum: 1 } } }
{ $sortByCount: "$tags" }       // instead of $group by tag + $sort desc

$sortByCount is exactly a $group with { $sum: 1 } followed by a descending $sort, which is such a common pair that it is worth recognising when you read one.

It is also worth knowing when not to reach for the pipeline at all. countDocuments, distinct and an ordinary find with a sort are simpler, and for what they do they are no slower. The pipeline earns its complexity when you need to group, join, reshape, or compute across documents — not when you need a filtered list.

Debugging a pipeline

When a pipeline returns nothing, the fastest way to find out where is to truncate it. Run the first stage alone, then the first two, until the results disappear:

db.reels.aggregate([{ $match: { status: "PUBLISHED" } }, { $limit: 2 }])          // 2 documents?
db.reels.aggregate([{ $match: { status: "PUBLISHED" } }, { $unwind: "$tags" }, { $limit: 2 }])

The stage where the count drops to zero is the broken one. Two other tools help: explain() works on aggregate as well as find, and { allowDiskUse: true } lifts the 100 MB per-stage memory limit for a genuinely large job — at the cost of spilling to disk, so treat needing it as a signal to add a $match rather than a flag to set by default.

The short version

Documents flow through stages, in order. Each stage sees only what the last one produced.
$match first, always — it is the only stage that can use an index.
"$field" means the value; "field" is a literal string.
$group requires _id, and puts the key there whatever it was called.
$unwind lets you group by array elements, and multiplies rows while doing it.
Guard divisors — division by zero yields null and poisons the rest of the pipeline.
Truncate the pipeline to find the stage that empties it.