Choosing a shard key: cardinality, frequency, monotonicity

conceptThe concept

A good shard key has:

• High cardinality: many distinct values, so data can be split into many ranges. country with 15 values can never use more than 15 chunks.
• Low frequency: no single value holds a big share of documents. One huge tenant on a tenantId key creates a jumbo chunk.
• Non-monotonic writes: values that do not always increase. createdAt or ObjectId as a ranged key sends every insert to the last chunk on one shard (a hot shard).
• Query isolation: the most frequent queries include the key, so they are targeted.

Shard key simulator

Choose a key, insert orders and watch where they land, how chunks split, which become jumbo, and which queries are targeted.

Open the interactive lesson to use this diagram.

whyWhy it matters

The shard key decides whether the cluster scales linearly or has one overloaded shard while the others idle. Changing it used to require a dump and reload; since 5.0 reshardCollection does it online, but it still rewrites the whole collection, so getting it right first time matters.

Use caseReal-world scenario

For orders, { customerId: 1, createdAt: 1 } gives high cardinality, spreads customers across shards, keeps each customer's orders together for "my orders" (targeted) and supports per-customer date ranges. { createdAt: 1 } would send every new order to one shard. { status: 1 } has 6 values and would never scale.

Try itExamples

Run and edit these in the interactive lesson.

function analyze(field) {
  const total = db.orders.countDocuments()
  const r = db.orders.aggregate([
    { $group: { _id: "$" + field, n: { $sum: 1 } } },
    { $group: { _id: null, distinct: { $sum: 1 }, maxFreq: { $max: "$n" } } }
  ]).toArray()[0]
  return { field, distinctValues: r.distinct, topValueSharePct: +(100 * r.maxFreq / total).toFixed(2) }
}
["status", "shipping.country", "customerId", "orderNo"].map(analyze)

const ids = db.orders.find({}, { _id: 1, createdAt: 1 }).sort({ $natural: 1 }).limit(2000).toArray()
let inc = 0
for (let i = 1; i < ids.length; i++) if (ids[i].createdAt >= ids[i - 1].createdAt) inc++
print("createdAt increases in", (100 * inc / (ids.length - 1)).toFixed(1), "% of consecutive inserts")

// add a suffix to split jumbo chunks (4.4+)
db.adminCommand({ refineCollectionShardKey: "shop.orders", key: { customerId: 1, createdAt: 1 } })
// rewrite to a completely new key online (5.0+)
db.adminCommand({ reshardCollection: "shop.orders", key: { customerId: "hashed" } })
// measure a candidate before deciding (7.0+)
db.adminCommand({ analyzeShardKey: "shop.orders", key: { customerId: 1 }, keyCharacteristics: true, readWriteDistribution: true })

internalsUnder the hood

Since 4.4 you can add suffix fields with refineCollectionShardKey to split jumbo chunks; since 5.0 reshardCollection copies data to a new key in the background, catches up with the oplog, and blocks writes for a short period (default up to 2 s) at cutover. It needs disk space and oplog headroom on every shard. 7.0 adds analyzeShardKey to measure cardinality, frequency, monotonicity and targeting from sampled queries before you decide.

Avoid / preferDo this, not that

sh.shardCollection("shop.orders", { createdAt: 1 })

sh.shardCollection("shop.orders", { customerId: 1, createdAt: 1 })

sh.shardCollection("shop.orders", { status: 1 })

sh.shardCollection("shop.orders", { customerId: "hashed" })

PitfallsCommon mistakes

• Ranged shard key on a monotonically increasing field (timestamps, ObjectId, auto-increment).
• Low-cardinality keys (status, country, boolean).
• Choosing a key that none of the frequent queries include.
• Hashed key on a field you mostly query by range.
• Ignoring the single giant tenant that will become a jumbo chunk.

LimitsWhen not to use it

Do not choose a key purely for even distribution if your main queries do not include it: every such query becomes scatter-gather across all shards. Do not shard small collections at all; they work fine on the primary shard. Hashed keys spread writes but give up efficient range queries on that field.

SecuritySecurity notes

Shard key values are copied into routing metadata (range bounds on the config servers), logs and diagnostic output. Avoid sensitive values such as email addresses or national ID numbers as shard keys; use a surrogate such as an internal customer ID.

ChallengeTry it yourself

const total = db.orders.countDocuments()
const stats = f => {
  const g = db.orders.aggregate([{ $group: { _id: "$" + f, n: { $sum: 1 } } }, { $sort: { n: -1 } }]).toArray()
  return { field: f, cardinality: g.length, topShare: +(g[0].n / total).toFixed(3) }
}
const result = [stats("customerId"), stats("status")]
result

InterviewQuestions you should be able to answer

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.