To address one of the most critical problem areas for customers using our software platform, I decided to roll up my sleeves and conduct the fieldwork myself.
For many months, we had been recording feedback from our customers, some direct, some indirect around one particular area of the product. Everyone agreed that we had delivered really powerful features - the common challenge was in the setup and configuration. We had made it too complex to use this valuable piece of the product!
I couldn’t agree more. I knew how hard it was, even for more technical users to configure the tools. I have a deep programming background but even I would struggle trying to ramp up on proprietary schemas with cryptic configuration syntax and poor documentation.
We had been attempting to compensate and even head off frustrated customers by offering to have our own Professional Services team help get customers up and running. But that only served to delay the inevitable as the complaints would then change to “even if you build it for me the first time, I’m not confident I can take it over and maintain it!"
What drove this decision
I had been anticipating a suitable window of time where we could focus on revisiting and rethinking these problems. The teams had been making good progress on some of the pre-requisite components that would be leveraged to rebuild this key feature of our platform. It was time to initiate a round of interviews with customers about how they use the current product, what (if anything) they liked about it and what (again, if anything) they could point to that causes problems for them or their end users.
I wanted to take full advantage of this opportunity to get out of the building, to speak directly to users and reinforce this important research activity with the members of my Product team. Mostly though, I wanted to improve our product and make our customers happy.
The decision: Enlist the help of our User Research expert to coordinate formal interviews with a suitable group of administrators and end users from our active customer base.
I am fortunate to have a talented resource on our team who has raised the bar around user research for our entire company. She continues to guide us all on the finer points of interviewing users early on around product discovery and later on for product usefulness and usability. With her by my side, I felt very confident that we would get to the crux of the problems and ultimately, be able to prioritize the work to come.
While I was eager to get started and optimistic about what we would learn, I heeded my colleague’s advice and adhered to the plan we made to ensure a favorable outcome.
Plan of attack
It was so tempting to just start writing requirements. But I've been wrong before.
I have used our product internally and I have also demoed it hundreds of times to customers so I felt confident that I knew what was wrong and what would need to be fixed. It was so tempting to just start writing requirements.
But I’ve been wrong before. Let me say that again, I’ve been wrong before. I have felt strongly about how our product should work and went straight to the Engineers with requirements only to find out, after they had spent valuable time implementing the stories that there were no customers lined up to use the new enhancements.
Start with hypotheses and a good prototype
Because I was so familiar with this particular part of our platform, I was able to quickly compose a lengthy list of hypotheses of pain points where I would expect to hear complaints from both heavy and casual users:
- We believe Admins are struggling to initially configure a URL to set up a new instance of [tool] in environments like Salesforce.com.
- We believe end users are frustrated with the number of clicks required to complete the ... process, especially if only one document is necessary.
- We believe Admins are resisting adopting and expanding their use of [tool] because it lacks good error handling, is difficult to test, and has too few options for controlling its behavior for the end user.
- We believe end users struggle with scrolling and otherwise navigating through the different areas of the page, e.g. with long lists of templates or with very long data forms and may find it helpful to track incremental progress as they work through sections of a longer form.
- We believe end users are underwhelmed with the stark layout of the [tool] page, the lack of branding, and the lack of any help text.
- We believe end users struggle with using the [tool] with different browser window sizes, specifically when the buttons move around when the page is resized.
I then had the UX team mock up a clickable prototype that addressed some of the primary concerns and that would allow us to better engage with our interviewees.
Recruit and track participants
To build a list of participants, I first reviewed the customers who were already part of our formal “labs” program (always be recruiting!). Then, I put the word out to our Sales and Professional Services teams to identify other viable customers with whom we could talk. The resulting list was manageable enough for us to track using a shared spreadsheet that included contact information, interview dates, and miscellaneous notes.
|Contact Names||Organization||Preferred Phone/Email||Participation||Last Contact Date||Future Participation||Primary Internal Contact||Notes|
|Nellie Bluth||The Bluth Companyemail@example.com||Asked to be included in Fall research initiatives||7/12/15||Also willing to help with...||Maggie Lyes, Acct Mgr||Has been helpful in the past but admits to not being technical|
Conduct and record the interviews
When we had accumulated a sufficient list of suitable users, I crafted and sent out an email invite along these lines:
Not surprisingly, our more fervent users responded immediately. With the others, it sometimes took a little more effort to secure the interviews. We mostly used virtual web meetings to conduct and record the interviews where the customer did most of the driving while we watched.
Our agenda for the interviews followed this path:
- Show us how you use the product today [50% of the interview time was spent here]
- As an [administrator, end user], what would you say if anything, are your favorite things about [tool]? [10%]
- What, if anything causes you or your users problems when using [tool]? [20%]
- Switching gears, we'd like to show you our ideas for a new design... [20%]
I had an intern transcribe notes from the recordings, putting all the material in a central repository so we could index it properly and share it with the rest of Product and Engineering.
The customer interviews went smoothly giving me exactly what I had hoped to obtain. There was plenty of validation around our current shortcomings, some surprises about certain attributes that users found valuable, and an abundance of constructive griping. The early prototype seemed to be a hit with users and gave me additional confidence to pursue that path.
I expect to deliver a first round of product improvements in the very next release and will continue to engage these customers to validate our efforts. Look for more reports from theProductPath around customer research, product validation, and feature prioritization here on PM Decisions.