Conducting Effective User Sessions
There are certain ways to gather user feedback.
Some PM’s may like the focus group / formal approach. I’m much more of a fan of doing “discounted” or informal user research. I like to be right there, asking the questions, and watching the users get frustrated when something doesn’t work or they can’t figure something out. This is the best way to experience the pain with them so you can get a sense of how urgent or severe it is and prioritize it effectively.
The Problem with Focus Groups
The way that i see it (and I’m not the only one), focus groups leave PMs open to false data. Why? Well, it’s pretty easy. Typically, getting a group of strangers together in a foreign setting (where they feel uncomfortable) and asking them questions about some product they have probably never seen nor used creates a very contrived environment.
They will feel like, and try to, answer how they think they should instead of providing the truth. Of course, there are exceptions to this rule - not every participant will do this. But it will happen more often than not, potentially leaving you with incorrect guidance from your set of users.
So what to do?
Discounted User Feedback
The way I will schedule these sessions is very simple - with individual users. Time for me to sit down with them, asking them questions and really get involved with how they are experiencing the product and what it offers. I really am interested in knowing what sucks and what’s going to make them tear their hair out.
Once you get past the “it’s OK to tell me this is awful” threshold, you’ll find the feedback you receive is really honest. And for the most part, very informative. Of course, like with all sessions similar to this you will want to always ask “why?” to ensure you fully understand what the user is saying.
These types of sessions are called “discounted” because they aren’t heavily controlled - they are informal, and that’s what you want. Think about it. What’s better? Putting a user in a weird environment where they feel like they are being tested - and having a consultant ask them the questions (who doesn’t really know the product all that well to begin with) sitting with a group of their peers, answering really specific questions. The answers to which may be perceived as making them look really smart or really stupid.
You want your users to relax so they can speak freely. That’s going to give you the best feedback.
Collecting the Data
The best way, at least in my experience, is to use both a combination of note-taking and activity models. Don’t worry - it sounds a lot harder than it is.
Really, activity models are very basic sketches of what a user is doing and where they get blocked while using the product. You can see a rough / example activity flow below.
Again, this doesn’t have to be fancy; just capture the user activity (each major step) and then note where they got blocked / frozen so you can look for patterns throughout all user research that’s executed - and hopefully find & solve some problems.
Asking the Questions
The key to all this is knowing the right questions to ask. Ideally, at least to me, you would start out with asking the user (who presumably has never seen the product before) to accomplish the products main goal. For example, imagine working for Microsoft and you are doing this style of user feedback for the Word product. I’d probably start out with a question like (from a blank Desktop): “OK, create, save, and print a Word document with the text ‘1, 2, 3, 4′ in it.”
Some other example questions might be:
- Create a bulleted list
- Adjust the properties of the document to ensure the ‘Author’ field reads ‘Bill Smith’
- Insert a table with 4 columns and 4 rows and make sure the text is center aligned vertically, and left-aligned horizontally
These are some leading questions that should drive the user forward without them just sitting there staring at a blank screen. You want to keep the sessions moving, but you don’t want to push the user to answer for which you are seeking. If a user has difficulties, the user has difficulties - note it and move on. If they can’t overcome them and have to to continue the next pieces of the session, teach them.
They should be able to offer up some ideas to make things much easier - and even if they don’t, you would have just experienced a pain point with a real-life user. Much better than doing anything in a vacuum.
High Satisfaction is Good, but…
Remember, above all else, you want negative feedback / criticisms to come out of user feedback sessions. You don’t want to have to report back to your boss / management that everything is 100% - that’s like saying you don’t have any competitors in the marketplace.
You need to sometimes really hunt for the gems. Users don’t usually know what’s going to be the solution to the problem they are having - that’s not their job. But they (in concert with additional market research, diving in to product analytics, competitive analysis and other such inputs) should offer up a very clear picture of ways in which to proceed.
Giving Effective Feedback
I’ve seen several posts written, and written several myself, about gathering effective feedback when you are in product management. But what about everyone else that’s interfacing with product? How can you give PdMs good, solid feedback on a consistent basis?
First, lets take a quick step back - why do we encourage anyone and everyone to give feedback? Isn’t it the PdM’s job to identify the ways to make the product better?
it’s true that one of the core tasks of product managers is to gather and filter input, make decisions based on it and/or make recommendations to senior management about directions to take or priorities to pay attention to. But, there’s one important thing that everyone in an organization should always remember - it’s everyone’s job, all the time, to help make products the best they can be.
Many PMs will tell you that they’d rather receive what someone thinks to be a crappy idea than no ideas at all. Ensure there is in fact a reason for the suggestion and it does provide value - or, at least, you think it does.
I will always try to ask, “what do you recommend?” (or more simply, “why?”) when receiving ideas / notes / feedback to ensure I’m capturing the full thought, and that one is actually there. I want to be sure I get an entire notion or idea - not just cut it off and only take a potentially negative criticism. If a PM in your organization is blindly accepting things like, “this doesn’t work” or “I don’t like this,” they aren’t doing their job.
They need to dig deeper to get the entire request. This is why it’s important people interacting on the front lines (with real users) are trained to inquire and gather that feedback properly. Many times, users will just write a 1/2 finished idea without even realizing it. They may written at a time of frustration or being super happy while using the product; but in many cases they will leave off the “why?”
The “why?” is the most crucial part, because it’s what tells product manager’s what’s actually important to the user giving you the request. If that’s not there, it’s all very binary - a thumb’s up or down would suffice.
This doesn’t just go for product feedback, but delivering feedback in general; especially when managing others. One of the best lessons to help someone that may be more junior (or, depending on how you look at things - less jaded) learn is how to deliver proper feedback. Whether that’s a product idea, making a PowerPoint deck look better, etc… The earlier people get comfortable with the practice of actually explaining why they are saying something, the better. But, I digress.
Product manager’s should always be willing to listen to feedback. And those that want to provide it to them should make sure they have these critical elements in their delivery. Heck, anyone should feel comfortable even asking their PdM if they are providing their ideas to them in the most effective way possible. That should help clear any confusion / misinterpretations up pretty quickly.
Why Mahalo
One of the most frequent questions I get is: “why do you use / like Mahalo?” Well, I figured it was time to address this in some more depth.
First, those that discover (for the first time - and maybe beyond) that I use the product over Google are quite surprised. It’s almost like a “how could you abandon Google?” type response. Well, the truth is, I’m not. But we’ll get to that in a little bit.
Really, it comes down to believing in what Mahalo is all about as a product more than anything. Sure I use Wikipedia. Could I rely on it for day-to-day information? No. Sure I read a ton of RSS feeds, but can I rely on those to provide with answers about a bunch of stuff (really, anything) throughout the day? No. Does Google mesh both of those things (current + news) — yeah, kinda.
But not in a way that completely satisfies me.
Let me fill you in on what I deem to be the secret of Mahalo. It’s not trying to be all information to all people (Google), and their guide notes.
This little blurb of text provides me with Wikipedia type information, but in a fast and easily referenced way. This is especially true when you get to the homepage. This is human-monitored and managed information, folks.
So this should cover why it’s better than Wikipedia, even though Mahalo makes no promise to be a Wikipedia replacement. It it did, I’d surely be less interested. But I see these two things I’ve mentioned already as key time savers throughout my day.
Now, if you look at the screenshot above carefully, you’ll notice a section called “Help Build.” I’m not going to get in to the intricacies of the Mahalo Greenhouse. But certainly, one of the best ways I got hooked as a user was when they rolled out Mahalo Social. Yes, I contribute regularly.
I actually started, and contributed, several links to help build the product management page, as I’ve mentioned on this blog before. And you know what’s funny? I actually want to submit content. When I see a spelling error on a guide note, for example, I want to draft a quick message on the message board for that page to let the guide that created it know. Why? Because I’ve actually received timely responses that are well-spoken and seen results.
You don’t get this from Google. Mahalo knows it’s capital intensive to run a people-driven content business like this, but it’s imperative. Regular old, unimportant users like me actually get to see there is stuff going on behind the scenes and they really do care about the quality of the content they are creating.
One of the other great things is their browser plug-ins that make it drop-dead simple for shmoes like me to easily supply them with content.
Now, this is only the direct and immediate type stuff that you get when interacting with the product. And I can’t lie - I know there are certainly critics out there. Any product will have them. But anything that really saves me time and helps me to find the content I’m looking for quickly is a big winner in my book.
