Etsy~3 min read
Shoppers wanted to know a person had picked this.
Multi-phase work with Merchandising and Analytics, and a design I argued for and lost.
- Role: Product design and research
- Timeline: Multi-phase
- Team: Merchandising, analytics, me
- Impact: 20% conversion, A/B tested
- Platform: Web
- Platform: iOS, Android
Conversion
Engagement
An A/B test across about a million users, read on Etsy’s internal analytics by the analytics team rather than by me, over the two days immediately after launch. The concurrent control rules out seasonality and site-wide events. It cannot rule out novelty, and two days is not long enough to know whether the lift survives that wearing off. The effect is established. Its durability is not. Still missing: what the 20% and 80% are measured against, across how many sessions.
Overview
Etsy believed its algorithm delivered quality, and ten interviews showed it did not for 60% of users.
Shoppers raised product quality and what the site chose to surface as pain points. Etsy’s answer was that the algorithm had it covered. Personalization says something is relevant to you. Curation says someone with taste chose it. The marketplace had no way to put a point of view on a page.
Worked with: Merchandising, Analytics, and a PM; 10 shoppers in paper-prototype interviews.
What I owned
I owned
- Design end to end, across the editorial page and the main-page banner
- Ten paper-prototype interviews, which moved the internal assumption about the algorithm
- The argument for rule-based authoring over SQL
- The staged build of the merchandising tool: preview, filters, tagging and themes, then ordering
Shared with Merchandising and Analytics
- What a collection needed to contain
- How performance was read after launch: Analytics owned the measurement, including the two-day window
Research
The obstacle was a belief inside the company, not a technical limit.
Ten paper-prototype interviews found the algorithm missed the quality bar for 60% of users. Shoppers wanted both: personalization for relevance, and curation as proof an expert had chosen the item.
A merchandiser also could not publish a point of view without an engineering ticket.
Decision: who publishes
I argued for rule-based authoring over SQL and won, so merchandisers publish without an engineer.
The PM wanted SQL, which gave total expressive power but meant merchandisers had to become technical to use their own tool. Engineer-built collections needed a ticket. I argued for less expressive rules a merchandiser could use unaided.
The cost: removing the engineer removed the last person who read a page before launch, so a wrong rule could reach the banner as quickly as a good one. Tag and theme rules can produce a wrong collection but not an incoherent one.
Decision: where curation lives
I argued for blending curated picks into the feed and lost, so curation got its own page.
The main page drove revenue. Displacing personalized results with editorial ones put that at risk, with no upside we could quantify in advance. A banner on the main page became the doorway to a separate editorial page.
That leaves shoppers to choose to visit. A blended feed would have put curation where they already look. This project did not establish whether the blended version would have converted better.
Redesign
The merchandising tool was built in four stages.
An algorithmic preview, then filters that assembled a product mix from shopper behavior, then tagging and themes, then control over ordering. The tool also supports audience segmentation and prevents someone in two campaigns from receiving duplicate emails.
Scope
An editorial page, a main-page banner, and the tool behind both shipped. A blended feed did not.
Results
Conversion rose 20% and engagement 80% in an A/B test.
The 20% may mostly be a launch spike, and I would not use those two days to call the blended version the wrong bet
My case for blending was not strong enough against a page already doing its job. The effect is established. Its durability is not.
