OVERVIEW
Varsity Tutors
An online tutoring platform that connects learners with qualified tutors through an automated matching system.
The matching process considers the unique attributes, needs, and goals of each learner to optimize tutor-student pairings.
Problem
Despite the platform's advanced matching algorithms and numerous experiments to optimize the matching process, the majority of learners were being assigned tutors with median rankings rather than top-performing matches.
Specifically, only 15% of new requests resulted in assignments to top-decile tutors.
Outcome
Through a series of initial experiments to improve the tutor experience, we increased the percentage of learners matched with top-decile tutors from 15% to 43%, a nearly 3x improvement that directly enhanced learning outcomes and customer retention.
The project also established a foundation for ongoing optimization, with additional experiments on deck to further improve match quality and the tutor experience.
Why is this a problem worth solving?
Better outcomes for learners
Tutors are scored and ranked based on several metrics directly related to customer value, including high session ratings, sessions per student, and low replacement rates.
Help tutors cut through the noise
Top tutors get bombarded with new opportunity notifications. They don’t have time to review and respond to all of them, especially when there's no indication of which ones they are best-suited for.
It’s a big win for the business, too
Clients who are matched with a top-decile tutor have a much higher retention rate, leading to ~50% greater customer LTV vs. being placed with a median-ranked tutor.
PROCESS
Initial Discovery & Problem Analysis
Working with a junior designer and the product manager, I led a workshop to review underlying data and start to understand the root causes of the mediocre matching outcomes.
Our analysis revealed a negative feedback loop:
A small percentage of high-performing tutors consistently ranked highest in their subject areas, leading to a heavily skewed distribution of notifications for this tutor population, which in turn hurts response rates due to a low signal-to-noise ratio of
Key findings included:
Top tutors were flooded with opportunities but had no way to identify which ones they were best matched for
As opportunity and notification volume increased, individual opportunity view and response rates declined dramatically
Top tutors often had full schedules from existing students, making them less available despite being highly ranked
The system was assigning median-ranked tutors because higher-ranked tutors weren't responding in time, if at all
Hypothesis and Solution Strategy
CORE HYPOTHESIS:
User Research and Initial Concept Validation
Tutors responded most positively to having top opportunities separated out in their own section at the top of the existing page. They also liked clear badge indicators clarifying why it's a top match.
Messaging related to the benefits of being a top-ranked tutor for an opportunity, such as early or exclusive access was an enticing concept for tutors. Early access to review and respond felt like a reward for their stellar track record on the platform.
Experimental Design & Implementation
Results:
After two weeks, we observed significant increases in both view and response rates for "top match" opportunities among treatment group tutors, contributing to measurably more top-decile tutor assignments.
Follow-up Experiment: Next, we wanted to test a range of badge and messaging options to see which treatments led to the greatest behavioral changes.
It's one thing to get async user feedback about a mock-up in Slack, but there's a reason for this maxim in product design: pay attention to what users do, not what they say.
Key Insights & Final Solution
The winning treatment combined behavioral psychology with practical functionality:
Priority Placement
Top opportunities displayed in a dedicated section above the general feed
Clear value proposition
"Top Match" badges that were immediately understandable
UrGENCY & EXCLUSIVITY
1-hour early access that transitioned to general "Top Match" after expiration
Minimal disruption
Changes focused tutors' attention without overwhelming the existing interface
business impact & next steps
The final solution was gradually rolled out platform-wide, ultimately achieving the 43% top-decile assignment rate.
Beyond the quantitative improvement, the project demonstrated that small, targeted UX changes to how tutors find new opportunities could drive significant business outcomes.
Areas to explore
How might we:
give tutors more control over the types of opportunities they see?
help tutors understand how they got matched (and how to get more matches)?
distinguish between urgent, bonus, VT4S, and Consumer opportunities in the UI?
help tutors track opportunities relevant to substitute their soon-to-finish students?
support new, unproven tutors (with low initial scores) to get a first match faster?
Going deep to understand the root cause
Digging into the underlying data as a cross-functional team saved us from spending time trying to solve the wrong problems.
recruiting tutors to participate in early concept reviews
This allowed us to address some of our riskiest assumptions and enabled us move forward quickly.
STARTING SMALL, FINISHING BIG
Launching quick-to-build tests instead of jumping into a holistic redesign not only got measurable results sooner, we also earned buy-in from executive leadership for larger improvements in the future.
