Personalized search for 5.77M queries a year

AWS documentation search gets about 5.77 million queries a year. 49% of searches were unsuccessful, and 57% of people searched more than once to find what they needed.

Everyone saw the same results whether they were new to AWS or had been building on it for years. The site has no sign-in, so there was no profile to personalize from.

• Re-rank in stages on Amazon Personalize: retrieve the usual results, extract features, then let a personalization model reorder them.
• Use page views, search behavior, and session context instead of an account.
• Model four roles (developer, architect, operations, data scientist) at three experience levels.
• Keep re-ranking under 250 ms. If search got slower, people would notice that long before they noticed better results.
• I wrote 23 functional requirements with acceptance criteria, so engineering and science were building and testing to the same spec.

• Baselines before launch: 41.49% click-through and 0.7469 mean reciprocal rank.
• A controlled A/B test, then two rounds of post-launch analysis and an operational readiness review.

• Click-through rose from 42.95% to 43.75%. The effect was large and statistically significant (Cohen's d = 2.16).
• Clicks spread further down the page. Position 1 went from 61.75% to 54.40% of clicks while positions 2 to 5 rose from 32.48% to 37.26%, which suggests people were finding useful results below the top slot.