AWS OpenSearch Support
OpenSearch shard sizing strategy.
Too many small shards waste heap and make clusters unstable. Too few huge shards slow recovery and searches. Shard count is also fixed per index when created, so mistakes are costly to undo. We review your shards and set a sizing approach that matches how you ingest and query.
When You Need This
The situations where this work pays for itself.
Signs it is time
- The cluster has thousands of small shards
- JVM memory pressure is high though data volume is modest
- Daily indices are tiny but each has several primaries
- Searches or recoveries are slow because shards are huge
- You are designing a new log or search index layout
What you get
- Shard and index inventory with problems highlighted
- Index templates with primary and replica settings
- Rollover and retention policies
- Reindex or shrink plan for existing indices
How we do it
A clear, safe process you can follow along with.
1. Inventory shards and sizes
We list indices and shards with _cat/indices and _cat/shards, noting size per shard, shards per node and how many shards hold little data.
2. Set a target shard size
AWS guidance for common workloads is shard sizes in the range of roughly 10-30 GB for search workloads and 30-50 GB for log analytics. We confirm against current guidance and your query latency needs.
3. Compute the right primary count
We divide expected index size by the target shard size, then check the total shard count per node against heap, since shards carry fixed memory overhead.
4. Use rollover and templates
For time series, we use index templates plus rollover or Index State Management to cut over by size or age rather than creating fixed daily indices that stay small.
5. Fix existing indices
Small indices can be merged by reindexing into fewer shards or using the shrink API. Oversized shards require a reindex into an index with more primaries.
6. Set replicas deliberately
We choose replica counts for availability and read load, and remember each replica multiplies storage and shard count.
7. Verify
After changes we compare shard count, heap use, search latency and indexing rate, and set alarms so shard growth is watched.
Mistakes we help you avoid
Common ways this goes wrong, and what we do instead.
Using the default primary count for every index
A small index with several primaries is pure overhead.
Adding nodes to fix oversharding
More nodes cost more and the shards remain small.
Reindexing without capacity planning
Reindex adds load and needs temporary space.
Frequently Asked Questions
How many shards should my OpenSearch index have?
Divide expected index size by a target shard size, commonly tens of gigabytes, and keep total shards per node within what the heap supports. Test with your own queries.
Can I change the number of primary shards later?
Not directly. You can use the shrink or split APIs under conditions, or reindex into a new index with the desired count.
What is oversharding?
It means too many small shards, which waste heap and cluster state memory and can cause instability, slow master operations and high JVM pressure.
Want Help With OpenSearch shard sizing strategy?
Schedule an appointment. Tell us about your environment and we will outline the safest, highest-value approach.
Schedule Appointment › No commitment required