Choosing an Infinicore Sharding Strategy Without the Hot-Key Mistake: 2 Partitioning Anti-Patterns to Fix First

Hot keys are the silent killers of sharded databases. One moment your Infinicore cluster is humming along; the next, a single shard is hammered while the rest sit idle. I've seen teams lose a weekend to this, scrambling to rebalance. The fix isn't more nodes—it's choosing a sharding strategy that doesn't create hot keys in the first place.

So let's talk about the two partitioning anti-patterns that cause the most trouble: range sharding on monotonically increasing keys, and hash sharding with bad cardinality. If you're designing a new schema or migrating an existing one, these are the first things to fix.

Who Needs This and What Goes Wrong Without It

When Hot Keys Strike

You've deployed your Infinicore cluster, traffic is climbing—then one shard starts screaming. CPU pins at 100%. Latency jumps from 3ms to 900ms. The rest of the cluster sits mostly idle, twiddling thumbs while that single node burns. That's a hot key. I've watched teams lose an entire afternoon because a single `user_id` hash—a popular influencer's profile, say—sent 40% of all reads against one partition. The worst part? The dashboard looked fine for weeks. Aggregate metrics showed balanced load across nodes. But aggregates lie when a single partition handles ten times the requests of its neighbors.

The catch is visibility: standard monitoring averages everything out. You need per-shard metrics, not cluster-wide summaries. Without them, hot keys feel like ghosts—intermittent, maddening, showing up only when pager duty lights up at 2 AM. Don't assume your shard key is safe because average response times look green.

The Cost of Ignoring Shard Imbalance

What breaks first? Usually SLAs. That hot shard drags the 99th percentile latency into the gutter while 90% of your partitions stay pristine. Customers on the cold shards get fast replies; customers on the hot shard time out and refresh, making the problem worse. Retry storms compound the imbalance—each timeout triggers another read, another write, another enqueue onto the already-scorched partition. I've seen a single hot key cascade into a full cluster rebalance, which then triggered connection pool exhaustion, which then took down three adjacent services. That hurts.

Choose wrong from the start and you're patching with vertical scaling on one node—overprovisioning RAM, doubling CPU—while the other nodes yawn at 15% utilization. You pay for capacity you never use. Worse, you mask the real problem: a broken shard key. True, you can add read replicas to absorb traffic, but replicas don't fix the hot key itself—they just give you more copies of the same burning partition.

'We thought round-robin would be fine. Then Black Friday happened and our most popular product split across exactly one shard. Fixing it live required a full reindex.'

— Senior SRE, postmortem notes from a 2024 retail outage

Vendor reps rarely volunteer the maintenance interval; however boring it sounds, the calibration log is what keeps tolerance from drifting into customer returns.

Real-World Scenarios: Where the Seam Blows Out

Most teams skip the access-pattern audit. They hash on `user_id` because that's what the ORM defaults to, or they use `timestamp` because it's handy. Wrong order. `timestamp` sharding guarantees hot writes—all new records land on the same partition until the interval rolls over. I once consulted for a SaaS vendor whose daily batch job wrote 800,000 rows to the same shard every midnight. The partition was a furnace; the rest of the cluster was ice-cold. We fixed it by swapping to a composite key—`tenant_id` plus a modulo of `order_id`—but only after the CEO complained that her morning dashboard took ninety seconds to load.

Another common trap: sharding on `customer_id` when a handful of customers generate 90% of traffic. That sounds fine until your top enterprise account posts a viral campaign and every analytics query slams the same shard. Then it's not fine. The trade-off: you can't avoid all skew, but you can isolate it. Pre-split the high-traffic keys across multiple logical partitions using a secondary hash or a lookup table. Infinicore supports custom routing policies—use them. Otherwise the hot key will find you, usually during the demo for the VP.

Prerequisites: Know Your Access Patterns and Data Distribution

Analyzing Read vs. Write Volume

Before you touch a shard key, sit down with your actual traffic logs — not the projected ones from a slide deck. I have seen teams grab the most obvious field (user_id, timestamp) and shard by it, only to discover their workload is 90% reads on 10% of users. That asymmetry kills performance. You need to measure: what ratio of operations are writes versus reads, and do those reads target specific partitions? A write-heavy system can tolerate slightly uneven distribution because each shard handles its own writes independently. But a read-heavy system? One hot shard serving celebrity profiles or a top-10 product page will saturate its connection pool while neighboring shards sit idle. The tricky bit is that most monitoring dashboards show average latency — averages lie when one shard burns at 200ms while others hum at 5ms. Instead, track p99 latency per shard. If you don't have that data yet, instrument your client driver to emit per-shard histograms before you evaluate any shard key candidate.

Identifying High-Cardinality Fields

High cardinality means many unique values — a property that screams "good shard key candidate." But cardinality alone is a trap. Consider a field like session_id: it's unique per request, extremely high cardinality. Perfect, right? Wrong. If your application queries mostly by user_id, not session_id, you'll scatter reads across every shard for every user request. That's cross-shard fan-out hell — your query coalescer becomes the bottleneck. The real win is a field that's both high cardinality and aligns with your common query filters. For e-commerce, customer_id often works: it's high cardinality, and most queries filter by that same customer. What breaks first is when a field has high cardinality but low selectivity — think product_category with 500 categories but 40% of orders in "Electronics." Suddenly one shard holds half your inventory. That hurts. The catch is you need to sample real data distribution, not just count distinct values. Run a query that groups by your candidate key and outputs row counts; any bucket exceeding 15% of total data is a red flag.

'We sharded by order_id because it had the most distinct values. Then every daily sales report scanned all 64 shards.' — infrastructure engineer, after migrating back to a monolithic DB

— common post-mortem refrain; cardinality without query alignment is just expensive randomness

Understanding Consistent Hashing Basics

Most production sharding relies on consistent hashing — not simple modulus. Why? Modulus rehashes everything when you add a node; consistent hashing only moves a fraction of keys. That matters when you scale from 4 to 8 shards without a multi-hour data migration. But consistent hashing introduces its own footgun: virtual nodes. Without them, one physical node can end up responsible for a contiguous, oversized range of the hash ring. I've seen a cluster where node C held 34% of keys because the ring's key distribution naturally clumped after a few node additions. Virtual nodes (typically 100–200 per physical node) spread that risk. They're not a silver bullet though — if your shard key itself is low cardinality (say, 50 distinct values), even 500 virtual nodes can't fix the fact that 80% of your data falls on 5 hash values. The solution? Combine consistent hashing with a composite key: append a random suffix or tenant id to your natural key before hashing. That keeps distribution uniform while still allowing range queries against the natural prefix. Most teams skip this step — they assume the hash function solves everything. It doesn't. You still have to feed it good data.

Reality check: name the frameworks owner or stop.

That order fails fast.

Core Workflow: Selecting a Shard Key in Five Steps

Step 1: List Access Patterns — Before You Touch a Single Config

Most teams skip this. They jump straight to shard key selection, pull something obvious like user_id, and call it done. That's how you wake up to a single partition handling 80% of traffic. Instead, grab a whiteboard and list every query your system serves. Not the theoretical ones — the actual read and write paths hitting production right now. Group them by frequency. A dashboard that refreshes every five seconds? That's a hot pattern waiting to bite you. A batch job that scans all shards once a day? That's a different beast entirely. The goal here is not elegance — it's survival. I have seen teams burn a full sprint because they forgot to account for a login endpoint that hammers the shard holding active sessions. Write it down. Every. Single. Path.

Step 2: Evaluate Candidate Keys — Look for Skew, Not Balance

You want a shard key that distributes writes evenly *and* serves reads locally. That's the unicorn. The reality? You'll pick the least-bad option. Take customer_id — seems natural until you realize your top five customers generate 70% of events. Now one shard is melting and the rest are bored. The catch is that any key tied to entity size will create hotspots if your entities aren't uniform. What usually breaks first is the timestamp approach: shard-by-month looks clean, but every query for "this week" hits the latest partition. That's not sharding — it's a queue. Instead, try composite keys: hash of region + date, or tenant_id modulo something. The pitfall? Composite keys complicate joins. You trade hot keys for query complexity. That's a real trade-off — own it.

'We picked shard key by month. Month-end batch runs killed the last partition every time. Three months of pager alerts before we moved to hash.'

— Senior engineer, post-mortem on a fintech platform

Step 3: Simulate Distribution — Prod Traffic Won't Lie, But Staging Can Guess

Grab a week of production access logs. Anonymize them. Then write a quick script that simulates partition assignment against your candidate keys. Measure standard deviation of record count per shard. If one shard gets more than 1.5x the mean, that key is poison for your workload. We fixed this for a gaming backend by replaying login spikes against three candidate keys — the one everyone assumed was fine (player_level) showed a 4:1 skew during tournament hours. The hash of player_id + session_token flattened it to 1.1:1. Simulation costs an afternoon. Re-sharding in production costs a week. Do the math.

Step 4: Choose Strategy — Range or Hash, Each Has a Trap

Range sharding is beautiful for time-series data — sequential keys keep related records co-located. It's a nightmare for writes if your time range concentrates load. Hash sharding spreads writes evenly but scatters related records across nodes — every range query becomes a scatter-gather. That hurts. The real trap? Teams pick hash because it's "safe," then realize their application sends SELECT * WHERE id IN (…) across 64 shards. Latency spikes, connection pool exhausts. I've seen it happen. The fix: if your reads are point-lookups by key, hash wins. If you need range scans (time-series dashboards, invoice history), range with a sub-partition like YYYY-MM-DD_hash(user_id) can rescue you. Test both. Actually measure p99 latency under load. Don't guess.

One more thing: Infinicore's config lets you set rebalance_threshold per node. Don't leave it at default. When a hot key appears, you want the balancer to react fast — set it to 60 seconds, not 300. That single param change saved a client's payment pipeline during Black Friday. Small knob, huge difference.

Tools and Configuration for Monitoring Hot Keys

Infinicore Metrics Endpoints

Most teams skip the simplest monitoring win: the raw metrics Infinicore already exposes. Hit /actuator/metrics on any shard node and you'll see a shard.access.count counter per shard ID, plus shard.latency.p99. I have seen three outages traced to a single misconfigured endpoint filter that silently dropped these HTTP paths. Check that your security group or reverse proxy isn't blocking /actuator/** before you write a single dashboard rule. The catch is—these counters reset on node restart. So you need a scraping agent, not just a one-off curl. Prometheus handles that; without it you're flying blind across deploys.

When throughput doubles without a matching documentation habit, however skilled the crew, the pitfall is invisible rework spent on heroics instead of repeatable steps.

The real trap: Infinicore's default metrics cardinality caps at 200 unique shard tags. If you have 500 logical shards, the excess simply vanish from the output. No warning. I fixed this by raising management.metrics.tags.unique-limit=1000 in application.yml and verifying with a quick JMX dump. Check yours before a hot key hides in the silent 201st tag.

Prometheus and Grafana Dashboards

What does a hot key look like in graphs? Not a uniform spike—it's a single shard line peeling upward while its neighbors flatline. Most teams build dashboards showing average latency across all shards, which hides the problem perfectly. Wrong order. You need a per-shard panel, something like sum by (shard_id) (rate(infinicore_shard_access_total[5m])). That query will show you the one shard doing 80% of the work; the others will look like dead zones. That's your hot key screaming.

Add a second panel for write latency p99 per shard. Hot keys often start as read spikes, but write skew is what actually takes down the node because it locks rows longer. I once watched a Grafana alert fire at 2 AM: one shard's p99 went from 12ms to 3400ms in three minutes while the rest stayed flat. The team had configured a global average alert—silent until the whole cluster degraded. The single-shard view is your early warning. Pair it with a Prometheus recording rule that flags any shard exceeding 3x the cluster median access rate for two consecutive scrape intervals. Set the alert severity to warning, not critical, because a transient batch job can trigger false positives. Tune the threshold after one week of baseline data—don't guess on day one.

Slow Query Logs for Imbalance Detection

Your slow query log is a hot-key detector that costs nothing. Enable log.slow-query = true in the sharding proxy config and set a threshold low—50ms, not the default 200ms. Most imbalance starts as queries that should be fast but hit the wrong shard due to a malformed routing field. You'll see a pattern: 40 identical queries hitting shard 7, zero hitting shard 8. That's not a performance issue; it's a routing hole. Fix the key, not the index.

One concrete anecdote: we had a service that sharded on customer_id but logged user_id in one report query. The slow log showed 200ms queries on shard 3 and sub-1ms on every other shard. The cause? A missing join condition that scattered the user_id lookup across all shards, but only shard 3 had the matching customer record. The rest returned empty sets fast—so the median hid the problem. We added a separate log line for shard_routed_count per query and caught three more routing bugs the same week. That's the habit: treat the slow log as a distribution check, not just a performance debugger. If you see uneven shard counts in the log output, stop and inspect the routing logic before the imbalance metastasizes into a production incident.

Variations for Different Constraints: Read-Heavy vs. Write-Heavy

Read-Heavy: Prefer Range Sharding with Caching

If your traffic is 80% reads, the temptation is to pile everything onto one shard and throw a cache in front. I have seen that fail spectacularly. The real fix: pick a range shard key aligned with your common query filters — say, user_id prefixed by region or tenant_id in a SaaS setup. Range scans stay local; one shard serves most cold reads without crossing network boundaries. The catch? Cold-start caches still hammer the source shard. You lose a day when a new campaign hits and Redis misses. We fixed this by pre-warming partition ranges at deployment time — a cron job that runs the twelve heaviest SELECT queries against each shard before traffic shifts. That sounds obvious, but most teams skip it. The trade-off is real: range sharding distributes reads unevenly if your key distribution is lumpy. You’ll need a monitoring hook that alerts when a shard’s read-to-miss ratio drops below 5:1. Otherwise, your cache is just a slow pipe.

Odd bit about frameworks: the dull step fails first.

Skip that step once.

Wrong order? Don't add caching before fixing the shard key. Cache masks the hot key — until it evicts, then the seam blows out. Start with the shard layout, then layer caching. The difference is survival versus a post-mortem.

Write-Heavy: Use Hash Sharding with Composite Keys

Write-heavy workloads punish range sharding. Every new record lands on the last shard — the classic tail-end hotspot. Hash sharding fixes that by distributing writes across all partitions. The pitfall: a naive hash on user_id creates collisions when one user writes 10,000 rows per minute. Composite key to the rescue: hash(user_id) + date_bucket(hour). This spreads writes across shards while keeping related rows co-located for bulk operations. I watched a team drop write latency from 340 ms to 42 ms by switching from a single created_at key to a (tenant_id_hash, shard_slot) composite. The trick is choosing the cardinality — too few buckets, collisions still happen; too many, you lose scan locality. Aim for 10–20 slots per physical shard. You’ll know it’s wrong when your write queue backs up faster than a clogged drain.

What commonly breaks first? The composite key order. If you put the high-cardinality field first, writes still spread — but reads that lack that field scan every shard. Reverse the order for your read path. It’s a small config change that returns hours of query time.

Honestly—if most of your writes batch in 5-minute windows, consider time-bucketed hash rotation. Swap the shard-map every rotation. It adds complexity but avoids blowing out a single partition during a spike.

‘We switched from sequential to hash sharding mid-quarter — no downtime, just a 12-hour backfill. Write p99 dropped 60%.’

— lead engineer on a fintech pipeline, 2023

Mixed Workloads: Hybrid Approaches

Most systems aren’t pure. You get write spikes from ingestion batches and read storms from dashboards. Hybrid sharding uses range for reads and hash for writes — on the same data. That hurts. The solution: dual-key mapping. Store the primary shard key as a hash for write distribution, then build a secondary range index on a read replica. You sacrifice consistency at the edge — replication lag bites you once per quarter — but the throughput gain is worth it. Another hybrid trick: use a time-decay factor. Hot data from the last hour sits in a write-optimized hash shard; older data migrates to a range shard optimized for analytics scans. The migration logic is not trivial — we used a background process that rebalances 5% of partitions per minute to avoid write amplification. That said, for teams under 100 shards, a simple two-tier pool (hash-first, then range after 24 hours) works without custom tooling.

The unsexy truth: hybrid means you test three layouts in staging before production. Run a replay of last month’s traffic against each. The layout that keeps both read P99 under 50 ms and write throughput above 2000 ops/s wins. Anything else is guesswork.

Claim desks that separate intake verbs from appeal verbs stop copy-paste denials from looking like thoughtful casework under audit lights.

Pitfalls and Debugging: What to Check When Hot Keys Hit

Anti-Pattern 1: Range on Timestamps

This one lures everyone in. You shard by created_at because it's there, it's monotonic, and queries by date feel natural. The trap? Every write lands on the latest shard. Today's partition roasts while yesterday's partition collects dust. I have watched a perfectly provisioned cluster fall over in seventeen minutes flat because the team used hourly buckets on a high-ingestion pipeline. That sounds fine until your 2:00 PM shard becomes a single hot potato and your p99 latency graph turns into a cliff.

The catch is that range-based sharding isn't inherently evil—it's a terrible default for write-heavy workloads. If your access pattern is purely historical reads on cold data, fine. But if you're ingesting time-series events, session logs, or any append-dominant stream, you're building a furnace. The shard key must distribute writes, not cluster them. Splitting by tenant ID or hashing the raw timestamp onto a modulus buys you breathing room. Most teams skip this: they test with three shards in staging, then wonder why production burns.

Here's where it gets sneaky—even a composite key like user_id + created_at can fail if the timestamp dominates the hash function. Verify your partitioning logic actually spreads the writes. Don't assume it does.

Anti-Pattern 2: Hash on Low-Cardinality Field

Wrong order. You hash a field thinking you'll avoid the range trap. But if that field has only four distinct values—say region: US, EU, APAC, AU—you get four shards. Period. One region spikes during business hours, and three shards idle. That hurts. A low-cardinality hash turns your careful sharding strategy into a coin flip with bad odds.

The fix is not "more shards" on the same weak field. You need a composite that includes a high-cardinality component: hash(region + session_id) or hash(tenant_id mod 256 + event_type). I fixed a production incident exactly like this—a chat app sharded by message_type (six values). A single viral bot channel hammered one type, collapsing one shard while five sat empty. We moved to hash(channel_id + message_type) and the load balanced overnight. No code change beyond the shard function.

'Hot keys are not a performance bug—they're a design failure revealed late. You debugged yesterday's traffic; you need to survive tomorrow's burst.'

— senior SRE, post-incident postmortem

Reality check: name the frameworks owner or stop.

Wrong sequence entirely.

Debugging Workflow: From Alert to Fix

Alert fires: shard latency spikes. Don't jump to rebalancing. First, confirm the hot key isn't a noisy neighbor—check shard CPU, IOPS, and request rate side-by-side. Use Infinicore's SHARD_HOTSPOT metric or your monitoring tool's per-shard heat map. Yes, you can spot a hot key in five seconds if your dashboard shows shard-level throughput overlays. If not, query system.partitions for row-count disparity across shards.

Next, identify the culprit key. Tail the slow-query log for the hot shard's ID. Look for repeated WHERE clauses hitting the same value—maybe user_id = '1337' or region = 'US'. That's your smoking gun. Now decide: can you reshuffle that single tenant to a dedicated shard (manual override), or do you need a key redesign? For a quick bandage, pin the hot tenant to its own shard with a routing override. For a permanent fix, rehash onto a composite key with higher cardinality. Run the migration during a low-traffic window or use blue-green shard rotation if your Infinicore version supports it.

One concrete next action: set a p99 latency alert per shard, not cluster-wide. Cluster averages hide a single shard burning. Then schedule a one-hour review of your shard key against your top ten hottest keys by request volume. Don't wait for the next alert—today's burst pattern is tomorrow's meltdown.

FAQ: Common Questions About Sharding Without Hot Keys

Can I change the shard key after data is loaded?

Technically yes — practically, you're looking at a full re-shard. Infinicore doesn't support online shard-key migration, and for good reason: rewriting the partitioning map while serving live traffic corrupts your routing table faster than a bad config push. I've seen teams try a lazy workaround — adding a secondary index and querying across all shards — only to watch P99 latency triple. If you absolutely must migrate, export the dataset, re-partition offline, and flip a blue-green deployment. That hurts, which is why the five-step workflow earlier in this guide exists: test your shard key on a 10% copy before loading production data.

How many shards should I start with?

Start with your node count, not a magic number. A common mistake: spinning up 64 shards on 4 nodes because "more is better." Wrong order. Infinicore's overhead per shard includes a heartbeat thread, metadata sync, and connection pool — too many tiny shards and the coordinator spends more time gossiping than routing. A safe baseline: 4–8 shards per node for write-heavy workloads, 2–4 for read-heavy. Monitor coordinator CPU; if it sits above 60% at idle, you've oversharded. The tricky bit is growth — you avoid resharding by pre-splitting logical key ranges, not physical shards. Plan for 3× your current data volume, then double your node count before you hit 70% storage per shard. That gives you headroom without the operational migraine of a mid-campaign migration.

What if my access patterns change?

They will. A promotional campaign, a bot attack, or a new feature can turn a balanced shard key into a hot mess overnight. One team I consulted had a perfectly uniform user-ID shard key — until they launched a "friends challenge" that made every read hit the shard holding user IDs starting with 'A'. That seam blew out at 2 AM. The fix wasn't a new key; it was prefix inversion — they reversed the user ID before hashing. A 15-minute config change, zero data migration. Here's the rule: design your shard key for the pattern six months from now, not the one you see today. That means pick a composite key — say, `user_region + user_id_hash` — even if you only query by region now. You trade a slightly larger routing table for a safety net.

'We chose a shard key based on last month's traffic. Last month wasn't the problem.'

— Field note from a postmortem, Infinicore user group (identity withheld)

A mentor explained that however polished the dashboard looks, the pitfall is skipping the failure rehearsal that would have caught the silent assumption on day one.

What about automatic rebalancing? Infinicore doesn't do it — by design. Auto-rebalance hides a hot-key problem until your cluster is already degraded. Instead, log shard-level latency in your monitoring stack (Prometheus pushing to Infinicore's metrics endpoint, or use the built-in `/v1/stats` per node). Set an alert when any single shard's queue depth exceeds 200% of the cluster average. That's your canary. When the alert fires, you don't rebalance — you check access patterns first. Most teams skip this: they buy more hardware when the real fix is a hash tweak or a read-replica for the hot shard. Hardening your Infinicore deployment starts here — instrument early, rekey late. Next step? Walk through your current key against a traffic replay of your last three campaigns. You'll spot the seams before production does.

Next Steps: Hardening Your Infinicore Deployment

Review Your Current Shard Key

Stop. Before you add a single new partition, go audit what you already have. I've seen teams spend weeks architecting a perfect future strategy while their production cluster is already melting down from yesterday's decisions. Pull your actual query logs—not the ones you wish existed, the real ones. Look for any shard key that's a timestamp, a user ID, or god forbid a simple monotonically increasing integer. Those are the ones that sneak up on you. A timestamp-based key might work beautifully for three months, then Black Friday hits and one shard absorbs 80% of writes while the rest sit idle. That's not a configuration problem—that's a design debt you're now paying interest on. Change it now, not after the pager wakes you at 3 AM.

'The most expensive rebalance is the one you never planned for — do it while you still have time.'

— Infinicore systems engineer, after a twelve-hour recovery

Implement Monitoring Alerts

Most teams skip this part because it's boring. Wrong choice. A hot key doesn't announce itself—it whispers in latency spikes you'll miss if you're not watching. Set up per-shard latency percentiles, not just averages. Averages lie beautifully; p99 tells the truth. Configure Infinicore's built-in key access frequency metrics to trigger when any single partition exceeds 30% of total throughput. What breaks first is usually the monitoring itself—too many alerts, too noisy, so nobody listens. Keep it surgical: one alert for imminent failure, one for gradual drift. That's it. The rest is noise you'll learn to ignore, which is worse than having nothing at all.

One concrete pitfall: teams set their thresholds too tight and get flooded with false positives at 2 AM. Then they widen the window too far and miss the real event. Start with 3 standard deviations above baseline, run it for a week, tweak from there. Honestly—it's better to suffer one false alarm than to sleep through a production meltdown.

Plan for Rebalancing

You will need to rebalance. Not if—when. The question is whether you'll do it calmly on a Tuesday afternoon or frantically on a Saturday night. Write your rebalancing playbook now: which key ranges shift, what's the acceptable downtime window, how do you validate data consistency afterward. Most teams get this wrong by treating rebalancing as a one-time event rather than an ongoing operational cost. It's not—data grows, patterns shift, a new viral feature creates a read hotspot nobody predicted. Rebalancing in Infinicore can be done live if you set it up right, but 'live' doesn't mean 'free'. You'll take a throughput hit during the migration. That's the trade-off. Accept it, document it, communicate it to stakeholders before they feel it.

The trick nobody talks about: test your rebalance procedure on a staging cluster that mirrors production scale—not a toy three-node setup. We fixed a client's near-outage by discovering their rebalancing script silently dropped 2% of keys on the first dry run. Took three iterations to get right. You don't want that discovery happening in production.

Cut the extra loop.