Conducting Effective User Sessions
There are certain ways to gather user feedback.
Some PM’s may like the focus group / formal approach. I’m much more of a fan of doing “discounted” or informal user research. I like to be right there, asking the questions, and watching the users get frustrated when something doesn’t work or they can’t figure something out. This is the best way to experience the pain with them so you can get a sense of how urgent or severe it is and prioritize it effectively.
The Problem with Focus Groups
The way that i see it (and I’m not the only one), focus groups leave PMs open to false data. Why? Well, it’s pretty easy. Typically, getting a group of strangers together in a foreign setting (where they feel uncomfortable) and asking them questions about some product they have probably never seen nor used creates a very contrived environment.
They will feel like, and try to, answer how they think they should instead of providing the truth. Of course, there are exceptions to this rule - not every participant will do this. But it will happen more often than not, potentially leaving you with incorrect guidance from your set of users.
So what to do?
Discounted User Feedback
The way I will schedule these sessions is very simple - with individual users. Time for me to sit down with them, asking them questions and really get involved with how they are experiencing the product and what it offers. I really am interested in knowing what sucks and what’s going to make them tear their hair out.
Once you get past the “it’s OK to tell me this is awful” threshold, you’ll find the feedback you receive is really honest. And for the most part, very informative. Of course, like with all sessions similar to this you will want to always ask “why?” to ensure you fully understand what the user is saying.
These types of sessions are called “discounted” because they aren’t heavily controlled - they are informal, and that’s what you want. Think about it. What’s better? Putting a user in a weird environment where they feel like they are being tested - and having a consultant ask them the questions (who doesn’t really know the product all that well to begin with) sitting with a group of their peers, answering really specific questions. The answers to which may be perceived as making them look really smart or really stupid.
You want your users to relax so they can speak freely. That’s going to give you the best feedback.
Collecting the Data
The best way, at least in my experience, is to use both a combination of note-taking and activity models. Don’t worry - it sounds a lot harder than it is.
Really, activity models are very basic sketches of what a user is doing and where they get blocked while using the product. You can see a rough / example activity flow below.
Again, this doesn’t have to be fancy; just capture the user activity (each major step) and then note where they got blocked / frozen so you can look for patterns throughout all user research that’s executed - and hopefully find & solve some problems.
Asking the Questions
The key to all this is knowing the right questions to ask. Ideally, at least to me, you would start out with asking the user (who presumably has never seen the product before) to accomplish the products main goal. For example, imagine working for Microsoft and you are doing this style of user feedback for the Word product. I’d probably start out with a question like (from a blank Desktop): “OK, create, save, and print a Word document with the text ‘1, 2, 3, 4′ in it.”
Some other example questions might be:
- Create a bulleted list
- Adjust the properties of the document to ensure the ‘Author’ field reads ‘Bill Smith’
- Insert a table with 4 columns and 4 rows and make sure the text is center aligned vertically, and left-aligned horizontally
These are some leading questions that should drive the user forward without them just sitting there staring at a blank screen. You want to keep the sessions moving, but you don’t want to push the user to answer for which you are seeking. If a user has difficulties, the user has difficulties - note it and move on. If they can’t overcome them and have to to continue the next pieces of the session, teach them.
They should be able to offer up some ideas to make things much easier - and even if they don’t, you would have just experienced a pain point with a real-life user. Much better than doing anything in a vacuum.
High Satisfaction is Good, but…
Remember, above all else, you want negative feedback / criticisms to come out of user feedback sessions. You don’t want to have to report back to your boss / management that everything is 100% - that’s like saying you don’t have any competitors in the marketplace.
You need to sometimes really hunt for the gems. Users don’t usually know what’s going to be the solution to the problem they are having - that’s not their job. But they (in concert with additional market research, diving in to product analytics, competitive analysis and other such inputs) should offer up a very clear picture of ways in which to proceed.
Feature Plateau Trap
There is a big trap when managing an early-stage product. I like to call this the “feature plateau trap.”
When you have a product that is relatively new in the market there is a strong temptation to not stop building. It’s exciting to add new features - I totally understand. Everything appears sexy, and when you first start off, it’s easy to get caught up in, “well this next release is looking sparse compared to the last two or three - we better throw more stuff in there.”
As soon as you catch yourself, or your organization doing / saying this, you need a full stop.
There are some key things I think we can all take away from, and acknowledge about product management - many of which I have talked to death here. That is, you want to build products people want. You can’t build a product that is all things to all people - they don’t exist and attempts to create such a monster typically will fail.
So how do you know you have reached this plateau. For me, I like to constant evaluate against my roadmap. Where is the product today in comparison to where it was 6 months ago? What did we define as being important 6 months ago, and are those things in the product today? How is your current roadmap different from your previous 2 iterations on that roadmap?
If you start to see a lot of variance, you may be in starting to slide down a slippery slope.
Remember, folks - being market driven is so much more than responding to every feature request that comes in and making it a reality. Even when they are coming from really important people.
For example, would I act faster if Bill Gates asked me to implement a feature than I would if someone in Texas asked for a different feature? No.
But, would I take notice (not necessarily implement - but take notice) if both someone in Texas and Bill Gates requested the same feature? Yes. Much more so than a single opinion on what will make the product I’m managing the “best ever.”
But, this type of thinking has to start from the top. The PM can only do so much before they will be run down and just start writing requirements for whatever the CEO wants. You can push back and push back — but eventually, if the CEO wants the features they want and you aren’t willing to give them to him or her, they will find someone who is. That’s what makes this tricky.
All things aside, the feature plateau trap is very real and a very common problem in start-ups. There is a very fine line between building out a robust / complete product that solves a problem and just adding features. Maintain your perspective and don’t be afraid to put something new in, even when you aren’t getting the user feedback early on (because it’s hard to come by when you are building out your user base) and don’t be afraid to curtail feature development later in the product lifecycle.
It has to happen eventually, even if you look like an idiot for being the first to suggest it. You can’t always develop features at the pace you do the first year or so of the product’s existence. The thing has to breath, even if Bill G makes that “need to have” feature request. All of them should be evaluated equally and evenly and against the core product definition.
Otherwise, you are just building a group of features that happen to show up together and nothing cohesive.
Don’t Get Discouraged
When you are in the thick of things, it’s easy to lose sight of the bigger picture. That picture may be 1 month, 3 months, or 6 months (plus!) away, but people (myself included) can get wrapped up in not being perfect, seemingly not doing enough, or thinking you are doing things wrong.
You can’t get worried. Well, you can - if you don’t have certain key things in place.
It’s easy to get all down and razzled if you see constant new “competition” popping up all over the place every week, or other products getting press yours isn’t. Quite frankly, it’s just not worth it. Take it as inspiration, learn what you can from it, and press on.
Remember - it comes down to a couple of things. Are you solving a problem people have, and does your product add some good value while solving it? If everything from the top-down (I’m not talking organizationally, really) is aligned, that means you have placed your bets and can just worry about execution.
It’s all about placing a bet and executing. “Wow” features and “silver bullets” are not what you rely on to build successful products. It’s all in the fundamentals. Those things may come along, or maybe you are developing one without even realizing it - but you can’t hold things up waiting for those ideas to come along.
You need to just do a few things really well. In fact, things I’ve already mentioned:
- Ensure you are solving a problem
- Align things form top to bottom
- Execute — quickly and with confidence
This “top-down” stuff I speak of is from the vision / problem identification, to the roadmap, to the requirements, to each release being pushed out the door. There are a few strategies that go into shipping a product which I may detail in other posts (for example, is this product brand new? does it already exist? at what stage of the lifecycle is it?), but for now take solace in having a clear vision to which all decisions are baselined.
If you are focused on something (almost obsessively so) you are doing everything right. Of course the organization could fail - but all you can do is pick a problem and go after it as hard as you can. You can’t solve all of the World’s problems by building 15-20 products at once.
Remember — it’s all about being good enough. Well, in most cases. Tony Stark may have a problem doing that, but start-ups building Web applications should not.
