Conducting Effective User Sessions
There are certain ways to gather user feedback.
Some PM’s may like the focus group / formal approach. I’m much more of a fan of doing “discounted” or informal user research. I like to be right there, asking the questions, and watching the users get frustrated when something doesn’t work or they can’t figure something out. This is the best way to experience the pain with them so you can get a sense of how urgent or severe it is and prioritize it effectively.
The Problem with Focus Groups
The way that i see it (and I’m not the only one), focus groups leave PMs open to false data. Why? Well, it’s pretty easy. Typically, getting a group of strangers together in a foreign setting (where they feel uncomfortable) and asking them questions about some product they have probably never seen nor used creates a very contrived environment.
They will feel like, and try to, answer how they think they should instead of providing the truth. Of course, there are exceptions to this rule - not every participant will do this. But it will happen more often than not, potentially leaving you with incorrect guidance from your set of users.
So what to do?
Discounted User Feedback
The way I will schedule these sessions is very simple - with individual users. Time for me to sit down with them, asking them questions and really get involved with how they are experiencing the product and what it offers. I really am interested in knowing what sucks and what’s going to make them tear their hair out.
Once you get past the “it’s OK to tell me this is awful” threshold, you’ll find the feedback you receive is really honest. And for the most part, very informative. Of course, like with all sessions similar to this you will want to always ask “why?” to ensure you fully understand what the user is saying.
These types of sessions are called “discounted” because they aren’t heavily controlled - they are informal, and that’s what you want. Think about it. What’s better? Putting a user in a weird environment where they feel like they are being tested - and having a consultant ask them the questions (who doesn’t really know the product all that well to begin with) sitting with a group of their peers, answering really specific questions. The answers to which may be perceived as making them look really smart or really stupid.
You want your users to relax so they can speak freely. That’s going to give you the best feedback.
Collecting the Data
The best way, at least in my experience, is to use both a combination of note-taking and activity models. Don’t worry - it sounds a lot harder than it is.
Really, activity models are very basic sketches of what a user is doing and where they get blocked while using the product. You can see a rough / example activity flow below.
Again, this doesn’t have to be fancy; just capture the user activity (each major step) and then note where they got blocked / frozen so you can look for patterns throughout all user research that’s executed - and hopefully find & solve some problems.
Asking the Questions
The key to all this is knowing the right questions to ask. Ideally, at least to me, you would start out with asking the user (who presumably has never seen the product before) to accomplish the products main goal. For example, imagine working for Microsoft and you are doing this style of user feedback for the Word product. I’d probably start out with a question like (from a blank Desktop): “OK, create, save, and print a Word document with the text ‘1, 2, 3, 4′ in it.”
Some other example questions might be:
- Create a bulleted list
- Adjust the properties of the document to ensure the ‘Author’ field reads ‘Bill Smith’
- Insert a table with 4 columns and 4 rows and make sure the text is center aligned vertically, and left-aligned horizontally
These are some leading questions that should drive the user forward without them just sitting there staring at a blank screen. You want to keep the sessions moving, but you don’t want to push the user to answer for which you are seeking. If a user has difficulties, the user has difficulties - note it and move on. If they can’t overcome them and have to to continue the next pieces of the session, teach them.
They should be able to offer up some ideas to make things much easier - and even if they don’t, you would have just experienced a pain point with a real-life user. Much better than doing anything in a vacuum.
High Satisfaction is Good, but…
Remember, above all else, you want negative feedback / criticisms to come out of user feedback sessions. You don’t want to have to report back to your boss / management that everything is 100% - that’s like saying you don’t have any competitors in the marketplace.
You need to sometimes really hunt for the gems. Users don’t usually know what’s going to be the solution to the problem they are having - that’s not their job. But they (in concert with additional market research, diving in to product analytics, competitive analysis and other such inputs) should offer up a very clear picture of ways in which to proceed.
Assessing Opportunity
So what does it take to bring a new product to market? Well, a lot of things. But it all starts somewhere; and that somewhere is assessing the potential market opportunity.
Really, this is about constructing a business - however lightweight or heavy you and your organization need it to be - and making the appropriate decisions early on.
I’ve been reading through the Product Manager’s Desk Reference these last couple of weeks, and it has really started to round out my skill set as a product manager. It’s one thing to be able to take instruction top-down (from your CEO or Board of Directors) and bring a product to market that they say needs to do, “x, y, and z.” But it’s another to fully understand the product and how it aligns with the opportunity / customer need and overall business strategy.
When you are in a start-up, you always walk a very fine line between overkill and getting 3-4 months in to something and realize you’ve missed key things. It’s important not to rush, but it’s really important not to skip over key steps.
So I look at this step in product development / definition as identifying a few key things, that aren’t too different from the Party of Four model I’ve identified in the past.
It all starts with identifying the problem itself. This is essentially stating the obvious - what the opportunity is. For example, maybe it’s the fact that people always complain about chewing gum never lasting long enough and always losing its taste far too quickly.
The second aspect is identifying the market segments / target customers for whom the problem will be solved. I have this clearly identified within the Party of Four model as “market segments.” In keeping with the chewing gum example (in case you haven’t figured it out yet, I’m referring directly to Stride gum). Now, this isn’t your parents gum that’s been around for 50 years, so I would pinpoint their target customer as a younger crowd - maybe 18-40 or something like that.
You could go on about additional characteristics of their target market - more than likely it’s predominantly Male if you look at the front page of their website. More of a professional type than blue-collar worker, so they are probably looking for folks that are college-educated, etc… Anyway, you get the picture.
Then you move in to how the problem is going to be solved. Remember, the problem is chewing gum not lasting long enough - so the problem is solved by Stride, probably using a proprietary formula / technology or unique composition in order to allow each piece of gum to actually last longer than other competitors in the market.
You should also discuss (provided the company is larger than one product) how the product aligns to their parent company (Cadbury Adams) as it is right in the candy / gum / etc… grouping of products. They already have a line of gums in Trident, etc… so they have to be careful not to cannibalize their existing business by marketing very carefully and effectively.
Now that you have gotten through all that, you need to clearly assess why the opportunity is attractive at all - obviously, if the marketing is successful and this product catches on, everyone wants to have their gum last longer. Nothing pisses me off more when I pay money for a pack of Juicy Fruit and each stick of gum lasts about 30 seconds.
Identifying key competitors takes the Party of Four model a little bit further than what it is today. Of course, there are a lot of players in the chewing gum space - Trident, Juicy Fruit, Bubbalicious, etc… So - Stride would probably remove any direct competitors (like bubble gum manufacturers) and stick to a more everyday / lifestyle type of gum. It’s not a breath freshener play - Excel and Clorets may not be viewed as direct competitors. This needs to be evaluated carefully in order to help ascertain what’s going to be unique about the product in the market place for the target customers - which leads to a strong value proposition.
When creating high-level / rough financials, it’s critical to be conservative. Don’t go out guns blazing and state you are going to clear 20mm in first-year revenue, because there is just now way. Get your cost measurements under control and start to figure out how you are going to price the market and how many units you think you are going to move in the first fiscal year. Someone from Finance should be a part of your cross-functional product team, so they can really help you out here to get the numbers right - especially if you are like me and you’re not the strongest in accounting and finance.
But most folks with any business experience should be good enough to sketch our a strawman model, which is all that’s needed at this point.
Remember to keep in mind who the audience is for this type of document - and that it’s going to keep evolving and changing as you go throughout the evolution of fleshing out the business case with other cross-functional team members, senior management, and possibly the Board of Directors.
But don’t let folks tell you this isn’t important - it is. Even if it seems like a lot of work and overhead, this business case approach (even though it’s really rough sketches at the beginning) will help you out a great deal down the road.
Statistics
Product managers need to always remember this:
There are lies, damned lies, and statistics.
Too often, numbers will be used as the only basis for decision-making - or they will be held up as the guiding light of “what to do.” The wikipedia article actually sums this up rather nicely:
The statement refers to the persuasive power of numbers, the use of statistics to bolster weak arguments, and the tendency of people to disparage statistics that do not support their positions.
I’m not advocating a complete disregard of numbers / stats - but if you have them, make sure you also exercise common sense. And if you can at all, try to validate any larger choice or decision by talking to real users.
Sometimes, when users click on things it’s just users clicking on things.
Feature Plateau Trap
There is a big trap when managing an early-stage product. I like to call this the “feature plateau trap.”
When you have a product that is relatively new in the market there is a strong temptation to not stop building. It’s exciting to add new features - I totally understand. Everything appears sexy, and when you first start off, it’s easy to get caught up in, “well this next release is looking sparse compared to the last two or three - we better throw more stuff in there.”
As soon as you catch yourself, or your organization doing / saying this, you need a full stop.
There are some key things I think we can all take away from, and acknowledge about product management - many of which I have talked to death here. That is, you want to build products people want. You can’t build a product that is all things to all people - they don’t exist and attempts to create such a monster typically will fail.
So how do you know you have reached this plateau. For me, I like to constant evaluate against my roadmap. Where is the product today in comparison to where it was 6 months ago? What did we define as being important 6 months ago, and are those things in the product today? How is your current roadmap different from your previous 2 iterations on that roadmap?
If you start to see a lot of variance, you may be in starting to slide down a slippery slope.
Remember, folks - being market driven is so much more than responding to every feature request that comes in and making it a reality. Even when they are coming from really important people.
For example, would I act faster if Bill Gates asked me to implement a feature than I would if someone in Texas asked for a different feature? No.
But, would I take notice (not necessarily implement - but take notice) if both someone in Texas and Bill Gates requested the same feature? Yes. Much more so than a single opinion on what will make the product I’m managing the “best ever.”
But, this type of thinking has to start from the top. The PM can only do so much before they will be run down and just start writing requirements for whatever the CEO wants. You can push back and push back — but eventually, if the CEO wants the features they want and you aren’t willing to give them to him or her, they will find someone who is. That’s what makes this tricky.
All things aside, the feature plateau trap is very real and a very common problem in start-ups. There is a very fine line between building out a robust / complete product that solves a problem and just adding features. Maintain your perspective and don’t be afraid to put something new in, even when you aren’t getting the user feedback early on (because it’s hard to come by when you are building out your user base) and don’t be afraid to curtail feature development later in the product lifecycle.
It has to happen eventually, even if you look like an idiot for being the first to suggest it. You can’t always develop features at the pace you do the first year or so of the product’s existence. The thing has to breath, even if Bill G makes that “need to have” feature request. All of them should be evaluated equally and evenly and against the core product definition.
Otherwise, you are just building a group of features that happen to show up together and nothing cohesive.
What Are Good Requirements?
What makes a good requirement?
That’s something that is completely open to interpretation. In my humble opinion, a good requirement is one that is clear and direct. It communicates what it is supposed to and then moves on.
No muss, no fuss.
There are many out there that will disagree with me on this. That’s OK - it’s all open to interpretation and what you’ve seen work before, or have tried, have succeeded with, have failed at, etc…
See to me, in the scenarios where development is not run by product (which should be every scenario if you want to do things right) you need to have a good partnership there. If that doesn’t exist, things will go down the tubes pretty quickly. That being said, you don’t have to love each other and hold hands - but something has to be shipped.
Now, to boil this down pretty quickly, I really see there being only two types of requirements:
1. Agile requirements; and
2. Waterfall requirements
These are really polar opposites of one another. You can’t get much different, aside from the fact that you are communicating the same problem you want solved within each - they are both just different ways to communicate them.
The thing I have come to realize is that for agile to work - for it to really work - you need everyone behind it and you need pretty smart developers who are on the ball. The main reason for this is that you want great code to be written, but the time you are saving by not writing long (and I mean looong) requirements documents produces ambiguity, and that must be filled in by the developer as they go.
They must ask questions. They must talk to you (the product wonk). And above all, you must trust each other.
Some developers have serious issues working this way - I get it. I understand it’s not easy. It puts a lot of onus on those that are writing the code, as opposed to the waterfall way….
Now in waterfall, at least in my experience, you are pushing away communication in favor of documentation. Developers want every single piece of a feature mocked-up, written down, and worked out before they even start thinking about the code. I always picture this style of process as taking things and lobbing them over walls. Requirements over a wall to dev, code over a wall to QA, QA over a wall to production, etc, etc…
Especially if your team is small, working this way is not leveraging one of the team’s crucial strengths - it’s size.
A lot of companies work in waterfall, and it does have its place. Specifically in organizations that demand a heavy process workflow through a release - or something that has a ton of formal check points. Two examples here (respectively) could be a bank and a consultancy
If you are going gangbusters trying to get products out the door, the last thing you need are formal check points. The last thing you need are 50-100+ page requirements documents. But that is what happens. This gums up the works for several key reasons.
The main one is, people stop talking. They stop asking questions. They just assume they can blindly follow the documentation - and if it doesn’t turn out right, oh well - wasn’t documented that way, so it’s becomes a bug. People way smarter than me came up with the agile manifesto for a reason. It can and does work.
People are smarter than this and everyone should expect them to be.
Of course developers need things to go on. You can’t expect them to read your mind and they can’t design anything (in most cases). But they are smart. After all, they are computer programmers.
The bottom line is this - no one can tell the future. As features get implemented, they change. Sure you know a lot of things up-front (button goes here, text goes there, save this, etc…) but you are 100% bound to miss things. If your product release process doesn’t account for things being missed down stream, it’s wrong.
Remember one of the key tenets of agile - put things off as long as possible until the last possible moment you can decide to ensure you have the best set of information at hand to make that decision.
Those things don’t happen in closed-door offices with a product manager huddled over his keyboard typing out requirements and acceptance criteria fast and furious. They happen in talking to those coding out those ideas and breathing life in to them.
But, not all teams are cut out for that sort of challenge. And that’s OK too - just make sure your management team knows you are going to need a solid chunk of time (1 1/2-2 months) per release cycle. And make sure you have some spare toner and paper ready for your printer. You’re going to be doing a lot of editing on those requirements docs.
