Agent Skills: Marketplace-Research OpenSearch + Personalize Caching Best Practices

Marketplace-Research OpenSearch + Personalize Caching Best Practices

A reference distillation of caching strategies for two-sided marketplaces running AWS OpenSearch (search) and AWS Personalize (recommendations behind a microservice). Contains 52 rules across 9 categories, ordered by cascade effect — from the upstream decision of whether to cache, through key design, personalisation boundary, strategy selection, TTL design, stampede protection, observability, and the lower-cascade categories of negative caching and tier composition. Each rule explains the WHY (the cost, latency, or correctness mechanism), shows incorrect-vs-correct code (TypeScript/Node for the microservice layer, Python for batch and analytics, OpenSearch JSON for OS-specific queries, YAML for CDN/Kubernetes), and cites the canonical source — AWS Personalize/OpenSearch/ElastiCache documentation, the XFetch paper (Vattani et al. VLDB 2015), RFC 5861 (stale-while-revalidate), and the engineering blogs of cache infrastructure teams (Netflix EVCache, Pinterest Cachelib, Twitter Twemcache, Cloudflare).

This is the complement to opensearch-function-scoring-algorithms — that skill answers "what should the ranking compute?", this skill answers "how do you scale it to production traffic without burning down OpenSearch or Personalize?"

When to Apply

Reach for this skill when:

• Adding caching to a search or recommendation surface for the first time — start with decide-cache-roi-calculation and decide-hot-key-distribution
• A homepage or category page renders 5+ recommenders and Personalize bills are growing faster than traffic — decide-amplification-multiplier, pers-recommender-fan-out-coalescing, pers-cohort-precomputation
• Cache hit rate is suspiciously low (under 30-40%) and you don't know why — key-canonicalize-query, key-strip-volatile-params, key-bucket-numerical-ranges, obs-key-cardinality-tracking
• Personalize is throttling (HTTP 429) during traffic spikes — decide-personalize-quota-budget, neg-cache-throttled-personalize, stamp-circuit-breaker-on-origin-error
• p99 spikes at TTL boundaries — stamp-coalesce-concurrent-misses, stamp-probabilistic-early-expiration, stamp-serve-stale-on-rebuild, ttl-soft-and-hard, ttl-jitter-to-prevent-thundering
• Recommendations stay stale after a model retrain — key-version-the-model, ttl-personalize-solution-version, strat-async-warm-up
• A read-after-write surface shows stale data (user favourites, saved searches) — strat-write-through-mutations, ttl-event-driven-invalidation
• Cache decisions need to be defensible to finance — decide-cache-roi-calculation, obs-cost-attribution, obs-cache-simulation-from-logs
• Anonymous and logged-in traffic mix on the same routes — pers-anonymous-vs-logged-split, tier-cdn-for-anonymous
• Cache is at memory pressure and you need to know whether to upsize, shorten TTL, or change strategy — decide-hot-key-distribution, tier-l2-elasticache-redis, obs-cache-simulation-from-logs
• OpenSearch CPU is high on common queries — tier-opensearch-request-cache, tier-opensearch-filter-context, neg-cache-empty-results
• A traffic spike from a viral link or crawler is hammering the origin — neg-bloom-filter-against-misses, neg-cache-empty-results, stamp-circuit-breaker-on-origin-error

The rules apply to any AWS-based marketplace with OpenSearch and Personalize fronted by an application microservice, regardless of vertical — accommodation, food delivery, fashion, services, jobs, secondhand goods, real estate. Triggers include "cache hit rate", "cache miss storm", "Personalize throttling", "Personalize cost", "multi-recommender page", "cohort caching", "single-flight", "stale-while-revalidate", "XFetch", "OpenSearch slow queries", "ElastiCache sizing", "CloudFront search caching", "Bloom filter cache penetration", and "thundering herd".

The Caching Pipeline

Categories are derived from the request-time caching pipeline. Earlier stages cascade: a wrong "should we cache?" decision wastes everything below; un-canonicalised keys cap hit rate at a fraction of the achievable ceiling; without observability you can't tell whether any of the rules helped.

Request → [1] Decide → [2] Key construction → [3] Personalisation boundary
        → [4] Strategy (read/write path) → [5] TTL/freshness → [6] Stampede protection
        → [8] Negative/defensive → [9] Tier composition (L1/L2/CDN/OS-internal) → Response
                                                ↑
                                [7] Observability (meta-layer applied to all stages:
                                    hit rate by key class, latency-with-and-without,
                                    cost-per-1k, cardinality, staleness, log-replay)

Rule Categories by Priority

| Priority | Category | Impact | Prefix | Rules | |----------|----------|--------|--------|-------| | 1 | Decision & Cost Calculus | CRITICAL | decide- | 7 | | 2 | Cache Key Design | CRITICAL | key- | 7 | | 3 | Personalisation Boundary | HIGH | pers- | 6 | | 4 | Strategies & Write Paths | HIGH | strat- | 6 | | 5 | TTL & Freshness | HIGH | ttl- | 6 | | 6 | Stampede Protection | HIGH | stamp- | 5 | | 7 | Observability & Empirical Measurement | HIGH | obs- | 6 | | 8 | Negative & Defensive Caching | MEDIUM-HIGH | neg- | 4 | | 9 | Tiered & Edge Caching | MEDIUM-HIGH | tier- | 5 |

Quick Reference

1. Decision & Cost Calculus (CRITICAL)

• decide-cache-roi-calculation — Compute Cache ROI Before Adding the Cache
• decide-cardinality-floor — Skip Caching When Traffic Distribution is Flat
• decide-personalize-quota-budget — Model Personalize TPS Budget Before Choosing a Cache Strategy
• decide-latency-budget — Cache Only When Origin p99 Exceeds the Latency Budget
• decide-amplification-multiplier — Account for Multi-Recommender Page Amplification
• decide-hot-key-distribution — Profile Traffic Distribution Before Sizing the Cache
• decide-search-vs-personalize-asymmetry — Cache Candidate Sets for Search, Full Payloads for Personalize

2. Cache Key Design (CRITICAL)

• key-canonicalize-query — Canonicalise Queries Before Hashing
• key-segment-not-user — Key Recommenders by Cohort When Users Outnumber Cohorts
• key-version-the-model — Include the Personalize Solution Version in the Cache Key
• key-locale-currency-explicit — Make Locale, Currency, and Timezone Explicit in the Key
• key-strip-volatile-params — Strip Volatile and Tracking Params Before Hashing
• key-bucket-numerical-ranges — Bucket Continuous Filters Before Hashing
• key-stable-hash-algorithm — Use SHA-256 over MD5 for High-Cardinality Keys

3. Personalisation Boundary (HIGH)

• pers-cohort-precomputation — Precompute Recommendations Per Cohort Offline
• pers-anonymous-vs-logged-split — Route Anonymous Traffic to Global Cache, Logged-in to Cohort Cache
• pers-cold-start-cache-priority — Serve Cold-Start Users From Popularity Cache, Skip Personalize
• pers-recommender-fan-out-coalescing — Coalesce Multi-Recommender Fan-Out Into Batched Calls
• pers-shared-candidates-private-ranking — Cache Retrieval Candidates Globally, Re-Rank Per-User From Cache
• pers-session-vector-write-through — Maintain Session Vectors in Cache with Write-Through on Every Event

4. Strategies & Write Paths (HIGH)

• strat-cache-aside-default — Use Cache-Aside as the Default Strategy for Read-Heavy Paths
• strat-refresh-ahead-hot-keys — Use Refresh-Ahead Only for the Top 1% of Hot Keys
• strat-write-through-mutations — Use Write-Through When User Mutations Are Immediately Re-Read
• strat-precompute-batch — Precompute the Popular Fraction with Batch Jobs
• strat-tiered-promotion — Promote to L1 In-Process Cache on L2 Hit
• strat-async-warm-up — Async Warm-Up After Deploy, Restart, or Model Retrain

5. TTL & Freshness (HIGH)

• ttl-by-content-volatility — Set TTL From Content Volatility, Not Engineering Convenience
• ttl-soft-and-hard — Separate Soft TTL (Async Refresh) from Hard TTL (Sync Miss)
• ttl-jitter-to-prevent-thundering — Add Random Jitter to TTL to Prevent Synchronized Expiry
• ttl-personalize-solution-version — Pin TTL to Personalize Solution Version, Not Wall Clock
• ttl-event-driven-invalidation — Pair TTL with Event-Driven Invalidation for Critical Freshness
• ttl-bound-by-staleness-tolerance — Bound TTL by Product Staleness Tolerance, Not the Default

6. Stampede Protection (HIGH)

• stamp-coalesce-concurrent-misses — Coalesce Concurrent Misses Into a Single Origin Call
• stamp-probabilistic-early-expiration — Use XFetch Probabilistic Early Expiration for Hot Keys
• stamp-serve-stale-on-rebuild — Serve Stale While Refresh Is In Flight
• stamp-circuit-breaker-on-origin-error — Trip the Circuit Breaker on Origin Errors; Fall Back to Stale
• stamp-distributed-lock-rebuild — Use a Distributed Lock to Coordinate Cross-Instance Cache Rebuilds

7. Observability & Empirical Measurement (HIGH)

• obs-hit-rate-by-key-class — Track Hit Rate by Key Class, Never Aggregate Only
• obs-latency-histograms-with-without — Measure Latency Histograms With-Hit and With-Miss Separately
• obs-cost-attribution — Attribute Cost Per Thousand Requests With and Without Cache
• obs-key-cardinality-tracking — Sample Key Cardinality Daily; Alert on Explosion
• obs-stale-served-ratio — Measure Stale-Served Ratio to Validate TTL Choice
• obs-cache-simulation-from-logs — Replay Production Logs Through a Cache Simulator Before Changing TTL or Strategy

8. Negative & Defensive Caching (MEDIUM-HIGH)

• neg-cache-empty-results — Cache Empty Search Results With a Short TTL
• neg-cache-throttled-personalize — Serve Last-Known-Good When Personalize Throttles
• neg-bloom-filter-against-misses — Use a Bloom Filter to Block High-Cardinality Miss Storms
• neg-poison-pill-detection — Checksum Cache Entries to Detect and Reject Poisoned Writes

9. Tiered & Edge Caching (MEDIUM-HIGH)

• tier-l1-in-process — Use In-Process LRU as L1 for Sub-Millisecond Reads
• tier-l2-elasticache-redis — Size L2 ElastiCache to the Cross-Instance Working Set
• tier-cdn-for-anonymous — Use CDN for Anonymous Traffic; Bypass for Cookies
• tier-opensearch-request-cache — Enable OpenSearch Request Cache for Aggregation-Heavy Queries
• tier-opensearch-filter-context — Put Reusable Predicates in Filter Context for Segment-Level Caching

How to Use

For a focused question ("should I cache this?", "why is my hit rate low?", "how do I survive Personalize throttling?"), jump directly to the relevant rule — each is self-contained with the WHY, code, and citation.

For a full caching-design review of a new or struggling surface, work the categories top-to-bottom. The cascade is real: a wrong decide-cache-roi-calculation wastes engineering effort on a cache that doesn't pay; a leaky key-canonicalize-query caps the achievable hit rate; a missing pers-cohort-precomputation keeps Personalize bills proportional to MAU. Stampede and observability are mandatory once hit rate exceeds 90% — the 10% miss in a thundering herd kills the origin, and without per-class hit-rate dashboards you can't tell.

For tuning an existing cache empirically, start with obs-cache-simulation-from-logs (replay your logs through what-if configs) and pair with obs-hit-rate-by-key-class, obs-cost-attribution, and obs-stale-served-ratio for the dashboards. The trio answers: is the cache doing its job, what does it cost, and are users seeing stale data?

For the multi-recommender homepage problem specifically (the most common Personalize cost-explosion pattern), the priority order is: decide-amplification-multiplier → pers-cohort-precomputation → pers-recommender-fan-out-coalescing → pers-anonymous-vs-logged-split. These four typically cut Personalize spend by 70-90% on consumer marketplaces.

For the "Personalize is throttling under load" incident, the priority is: stamp-circuit-breaker-on-origin-error → neg-cache-throttled-personalize → decide-personalize-quota-budget. The first two stabilise the user-facing impact; the third right-sizes minProvisionedTPS so it doesn't happen again.

For sibling-skill cross-reference, see opensearch-function-scoring-algorithms — that skill covers what to compute in OpenSearch (function_score, kNN, RRF, rank_feature, decay, LTR, MMR, evaluation). This skill covers how to cache it so the cluster survives production traffic.

Read section definitions for the cascade-impact rationale, or the rule template when adding a new rule.

Related Skills

• opensearch-function-scoring-algorithms — Research-backed ranking, retrieval, and evaluation rules for OpenSearch. The "what to compute"; this skill is the "how to scale it."

Reference Files

| File | Description | |------|-------------| | references/_sections.md | Category definitions and ordering by cascade impact | | AGENTS.md | Compact TOC navigation (auto-built; do not edit by hand) | | assets/templates/_template.md | Template for authoring new rules | | metadata.json | Version and authoritative reference URLs |