User Study: Premium Ad Perception
Understanding how our most popular product is used by entertainment fans.
The Study
In 2018 I worked with a Senior User Researcher at IMDb to conduct a moderated qualitative user research study on our most high impact custom ad product, the Premium Title Page. Premium Title Pages are the best way for an entertainment advertiser to tell their story on IMDb. They are the most expensive product at IMDb and extremely custom. Many movie studios, including Disney and Fox use them as their main movie website. To that end, I decided that as a company we should conduct a user study on them to see exactly how successful they are in our users eyes as opposed to our advertisers.
“Premium Title Pages (PTP’s) are most valuable to the client by providing a superior customer experience and blocking competitive ads. Studios spend higher budgets more frequently in order to unlock them to achieve the aforementioned benefits. I’m of the opinion that this helps the end user (customer) by providing a unique enhanced experience on IMDb that they can’t get anywhere else”
- IMDb Sales Marketing
The Proposal
Before diving into the study, I needed to understand what kinds of questions I actually need to be asking. I had hypotheses on what questions to ask, but the UX Proposal spreadsheet, as outlined by our head of user research, is designed to help pinpoint the direction that the study should go in.
Target user: Prospective PTP viewers, Entertainment Fans looking at the most popular titles at peak page popularity.
Not Target: Power users, users with Ad Block
Questions I needed answers to included:
What do Entertainment Fans notice between PTP vs standard title page vs competitor options?
Are PTPs perceived as ads in the user’s eyes?
What do fans want to do on the PTP experience?
Is the PTP experience distracting to overall page goals?
User Stories
Keeping in mind the goal of the study, I designed 6 in depth user personas with the help of my coworker Billy Thompson. These personae became the basis of our test questions that we would end up giving to our populous. These hypothetical users defined for us how a typical person uses IMDb. These personae were designed with the help of our Fan Needs Frameworks, a framework created from in-depth studies/interviews conducted by our Lead user Researcher. The spectrum of stories captured here is designed to reflect the general demographic of IMDb users.
User Story Breakdown
The next step I took was to break down the in-depth user stories into the actions that one would take in each story. These would manifest into questions like “What kind of browser are they using?” or “How are they arriving on the Title Page?” This exercise helped me think about those seemingly frivolous questions so that there was no ambiguity in the study. Once finished, each user persona had a clearly defined set of actions to accomplish their task. We tied each one of our initial questions in the proposal to each user action to make sure that the behavior of the test participant would answer said question. this also help cross check our questions to make sure they were the right ones to ask.
Discussion Guide
With common questions and user journeys established, I was able to start my discussion guide. Most of the credit for this has to go to Libby Garrett, our Head of Research at IMDb. She is a master of user Research and actually built the initial documentation for me to springboard off of for this study. After she developed the discussion guides, I vetted them to make sure that they were in line with what would fulfill the goals of the study.
In the example here, we requested that the participant find out who the actor was that played Poe Dameron in Star Wars on their mobile device. We made sure to flash the photo of Poe in the participant's face for just a brief moment - the same way they might see a character pop up on screen in a movie. In this case, we wanted to see if they found the PTP useful in their search for actor information.
The Study
Once I had a discussion guide, I was ready to start conscripting users to test with. To avoid overly technical people, (the study was built for empathetical analysis, not technical or design critique) prescreening applicants removed all people with the profession of developer, designer or advertiser.
After the prescreener, fellow UX designer, Billy and I were able to acquire 19 willing participants, a very large number for a qualitative study. With Billy's help, the 2 day-long study was a huge success. The IMDb brand was very strong and the reception of the study itself was fantastic. People didn’t seem nervous or apprehensive so we were confident that the responses we received were honest.
As each participant entered the room, we would first genuinely thank them, shake their hand, and offer them refreshments. We would then remind them of the NDA they signed. We asked for the most brutally honest feedback possible to encourage honest responses. i.e. “We didn't make this, so you won’t hurt our feelings.”
Observations
and Results
After the study, I went through and listened to all of my audio recordings of each participant. I recorded each response to every question in an observations excel doc. From here I was able to identify patterns in the participants’ responses.
Once each recording was transcribed, the doc was complete. I distilled the common responses into a delta chart of themes that using Aaron Walters Hierarchy of Needs. This shows us which parts of the PTP are successful and what elements need improvement. The chart shows a spectrum of meaningful experience all the way down to a frustrating experience. Not every experience needs to be meaningful, in fact, for the purpose of IMDb and the usefulness of the Title Page to entertainment fans, we really only needed the PTP to mostly align with “Usable” and “Comfortable” categories.
Actions Taken
With the success of the study, I presented the findings to senior leadership at IMDb. As a result of the Research conducted, we have built a roadmap to improve our PTP product. With customer anecdotes from the study, my team is now equipped to better serve our customers and push back on requests by advertisers that don’t directly improve the user experience of the title page. You can read more about how the results of this study became a complete redesign of our most valuable product.
Apologies for the blur. Under my NDA, I’m not allowed to show the results of the study.
Special Thanks
Billy Thompson, Senior UX Motion Designer
Libby Garrett, Senior User Researcher