Improving Pro Response Quality at Yelp
Building a System through Incentives & Guidance
CHALLENGE
I led design for a nine-month, multi-team initiative to fix a longstanding marketplace issue at Yelp: poor pro response quality. This problem was undermining major investments in lead generation and user growth.
My role spanned problem definition, roadmap creation, and design execution for a new AI-powered incentives and guidance system aimed at improving first-contact quality between consumers and businesses.
IMPACT
Launched globally, the work is already showing measurable improvements: average pro response quality scores increased by 2.2% and low-quality response rates dropped by 11%. The behavioral framework from this work also informed longer-term AI-assisted solutions on the 2025 roadmap.
Approach
1. Mapping the gridlock:
I partnered with a Product Manager to synthesize past research (quant survey, pro CSAT) and run cross-functional workshops to understand pro and consumer user needs, and how the system introduced competing tension between them. We identified key friction points:
Generic “Call me” replies (~33% of responses) had the lowest consumer engagement.
Leads often lacked sufficient detail or were outside a pro’s scope.
Status quo incentives rewarded responsiveness over professionalism, overwhelming pros and disappointing consumers.
2. Defining success and metrics:
We also ran an audit over 420 conversations to define “improving poor quality responses” in terms of product outcomes so it could feed into org-level KPIs.
3. Bringing structure to ambiguity:
I facilitated ideation across PM, Eng, and Marketing while fielding executive input.
I introduced an Opportunity Solution Tree to ensure all solutions mapped back to user problems, and organize them in testable pathways.
4. Securing executive buy-in
To get leadership alignment and rally the org behind a single solution track, the OST helped my Product Manager and I build a simple framework for classifying solutions:
1. Behavior-shaping incentives
Concept: A response quality indicator aligns incentives with genuine consumer needs and provides pros with guidance.
2. Automation aids
Partial automation concept: LLM-generated Pro response suggestions & guidance
Full automation concept: AI Receptionist for Pros
3. System-level changes
AI Assistant that aggregates responses from multiple pros and brokers discussion - reducing back and forth and qualifying leads.
We presented these concepts and trade-offs to C-suite, and secured approval to ship the Response Quality Indicator first while using the framework to inform future AI investments.
Key Design Decisions
Call out response quality separately from speed
While I explored combining response quality with other attributes for a singular ‘superhost’-type badge, a standalone Response Quality score on consumer surfaces would:
Increase relative importance of response quality to businesses
Make it easy for consumers to evaluate businesses on both speed and quality
choosing a descriptive score over numerical score or descriptive labels
Based on a rubric of motivation, actionability, ease of understanding and trust - I interviewed pros and consumers to directionally validate the following hypotheses:
The descriptive score was intrinsically motivating for pros to change their behavior.
Pros are motivated by indicators that consumers find the most helpful.
Consumers find the descriptive score easiest to parse.
Optimize for continuous improvement and simplicity over precision
Unlike response speed, quality is subjective and requires iteration - so we started with fewer instead of many quality tiers.
Prioritize carrot over stick in the experience
Reward great responders; create a sense of FOMO instead of punishing bad responders. Only show positive badges to consumers. Make it easier to gain vs lose status in terms of score calculation on an ongoing basis.
Final Designs
I led end-to-end design of the Response Quality Indicator and subsequent real-time guidance feature across business and consumer flows, collaborating with multiple engineering teams.
Phasing of key initiatives. A gradual rollout minimized risk, giving businesses the opportunity to improve their score before Yelp revealed it to customers.
Results & Learnings
Quantitative Impact (Post-Launch):
Quality score improvements: +2.2% vs. baseline.
Price sharing: +12%.
Excellent-tier pros: +5% more originating projects; +12% more engagement.
Low-quality responses: –11% rate; –9% volume (~21.6K/month fewer).
Behavior change: 42% of warned businesses edit their response at least once.
Qualitative Impact:
Pros reported that scores and guidance were motivating and helped clarify expectations.
”It made me more aware on what I could have been improving on my profile rather than just waiting for things to happen. So I really like this gauge... I spent a couple more minutes replying to clients, rather than just throwing in some information and calling it good.” ~ Pet Photographer, Yelp Advertiser
“…If there's a way that customers see that I have an excellent response… that could impact them. So therefore, it helps the business… that's part of why I was motivated. Knowing that I had an excellent response, I wanted to keep that up in case people were like, oh, well, this is someone that obviously cares and will get back to me.“ ~ Event Services Company Owner, Yelp Advertiser
“[It brings quality] to the top of my mind, like more awareness to it… Before, maybe I would have been more focused on… timeliness and I think this would help, you know, expand that awareness into: … it's not just about how quickly I respond, but it's about these other things too.” ~ HVAC Business Owner, past Yelp Advertiser