Improving Search Relevance Feedback through Human Centered Design

Jan - Aug, 2020

The Master Thesis investigates methods to prompt healthcare professionals to provide Explicit Relevance Feedback on search outcomes to enhance search quality and efficiency. Employing a research-through-design approach, the project intends to formulate a guideline for designing explicit feedback collection in search systems, suitable for platforms such as myTomorrows and beyond.

Role
Graduate Intern (Research & Design)
Platform
Web
Highlights
Obtained a 9/10 for the thesis and influenced myTomorrows Search's product decisions

Your access to Vimeo is restricted :(

Background

A tool that connects patients and new treatments

myTomorrows is a PharmaTech company that aims to help patients discover and access treatments under development (i.e., Clinical trials and Expanded Access Programs), addressing a significant hurdle for patients with unmet medical needs.

To advance the pace of matching, myTomorrows developed a specialized search engine for healthcare professionals to find new treatments all around the world.

an illustration of how patients with unmet medical needs access CT and EAP

screenshot of myTomorrow's original search

The need

The lack of relevance feedback

myTomorrows Search is capable of indexing through all recognized clinical trial registries such as clinicaltrials.gov. However, due to the limited scale of myTomorrows Search users, analyzing user behaviors (implicit relevance feedback) fails to help improve the search system.

Therefore, myTomorrows Search alternates to involve users in providing explicit relevance feedback to improve search quality and efficiency.

an illustration showing how explicit feedback can contribute

User research

Understanding Healthcare Professionals

Literature reviews, expert interviews, and role-playing sessions were conducted to gather healthcare professionals' expectations for myTomorrows Search, their attitude towards providing feedback, and identify key factors influencing their relevance feedback.

Expert Interview

Semi-structured interviews with 9 HCPs, including Medical Ethicists, Physician-Scientists, and Clinicians from the Netherlands, China, and Brazil.

Role-playing session

Conducted with 5 internal and external HCPs by observing their search behaviors and ask for ‘relevance feedback’ as they examines search results.

Not all healthcare professionals are familiar with Clinical Trials (CT) and Expanded Access Programs (EAPs) as treatment options for patients. Those who are aware approach them cautiously yet show openness to information aggregators like myTomorrows Search. To appeal to healthcare professionals, two key expectations they hold are outlined below.

Expectation 01

A Trustful, Unbiased, and Transparent Platform

Expectation 02

Having Complete CT/EAP database

Healthcare professionals typically do not anticipate giving relevance feedback. Their feedback tends to occur in response to extreme situations, like notably unprofessional clinical trial registrations.

Their feedback for the same result may differ depending on patient conditions or as their knowledge expands. A table below outlines the main factors influencing their relevance judgments.

Ideation

Exploring interactions to collect feedback

In the generalized search flow, it was determined that relevance feedback can be provided on SERP (the search results page) by assessing the interventions under study, with assessments unlikely to change after viewing the full study record.

Integrating this with the foundational feedback collection flow, various design approaches for soliciting relevance feedback were explored.

After multiple exploration rounds and speed dating sessions with target users, the explorations were narrowed to three distinct designs as below.

Pre-screener

By tagging search results to either receive recommendations or hide irrelevant ones.

Reminder

By replacing the 'back' button with a feedback collection prompt.

Hearty

By introducing a gamified avatar that requests feedback at the bottom of the page.

The design concept

A Search that evolves with users

User testing revealed that the 'pre-screener' was particularly more effective, transforming personal utility into actionable relevance feedback to improve myTomorrows Search.

The prescreening tool enables healthcare professionals to swiftly indicate their initial choices on SERP, as typically done in medical literature reviews. Additionally, the relevance buckets provide extra functionality to make their search more manageable and efficient.

Get similar recommendations

The 'Interested' bucket displays potentially high-relevance results for different queries. As more items are added to this bucket, increasingly better and more similar studies are recommended, expanding the search scope for users.

Hide irrelevant one with tags

The 'Not Interested' bucket holds results that don't directly meet users' informational needs. Users can add tags to specify reasons and opt to hide these results from their search outcomes.

The generalized guide

A guide to designing explicit feedback collection

After synthesizing research insights and reflecting on the project's progression, a design guide was formulated. This guide details approaches for crafting feedback collection experiences within search systems and extends to broader applications.