The Importance of QA
I am in the process of hunting for a great senior QA analyst. That being said, I wanted to take some time to talk about how important QA is in the product management process; or at least, how important I view it to be.
In my experience, if you are able to find and bring in the right QA person, that’s really into QA (and not just using it as a stepping stone), they will have a strong / noticeable impact on your product as early as its next release. I’m not kidding. I was lucky enough to work with a very talented QA engineer / analyst while at MusicIP, and his work ethic and results were absolutely perfect for helping us with several key initiatives.
So, what are some of the things that I’ll rely on QA for? I’m glad you asked…
Primarily, you don’t want defects in your product. I view the role of QA to help identify and manage bugs, but also prioritize them and ensure they are getting fixed. This way, I can stay focused on planning the product and trust that when a certain set of users can’t do something they are supposed to, it will get taken care of accordingly.
So, this means setting up your QA team to succeed. And usually, that means metrics and objectives. The key metric I’ll look to that’s directly tied to QA is product quality. Essentially, this is an amalgamation of high severity defects throughout the product (1s, 2s, and 3s usually). A great QA person will derive these priority values by assigning visibility and class rankings to them. For example, how likely is it a user will encounter a specific defect? And, if they do encounter it, what is the likelihood of it completely hindering their product experience?
You need to have these checks in place in order to ensure what you are shipping (especially as it grows and becomes more and more complex) is of the highest possible quality.
Now, the other key aspect I look for in QA is sanity checking my requirements and making sure their associated test plan runs the gamut. I may state that a user “must be able to create an account” and that “the username and password fields are required,” but the QA analyst will take this one step further and actually determine all of the permutations (that either may or may not be documented in the requirements) development must cover.
In this example specifically, maybe the e-mail address the user enters is malformed. Maybe their password doesn’t match to the 6-character standard. When you are working on releasing code quickly, especially in an agile system, this type of coverage becomes crucial. I need that constant stream of discussion and feedback.
Essentially, I’m looking for trust - not only from the product team, but the QA team (regardless of wether they sit in product or engineering), the development team, the marketing team, etc… I want to develop the trust that the QA team will catch critical user flaws (hopefully) before requirements even get to development for implementation so we can work out the kinks and ensure they are as clear as possible.
Beyond this, QA can really help development too — with structuring automated test suites, tracking their results per build and really grinding out the processes to ensure maximum build effectiveness (especially when using continuous integration tools) and insist the quality is really high and stays there. They may also be able to help when running core A/B tests on a site to help determine if user’s are encountering critical flaws in one implementation and/or another and what the outcomes are of possible change.
Remember, the “Q” in the abbreviation stands for quality. That’s their job. It’s not to administer servers or to be an authority for developers. However, it’s one of the most critical jobs in an organization - especially one that is just building / starting out. If the product you are releasing is poor quality, chances are it’s going to get very far.
Bring in an experienced QA person early and really maximize their value as much as possible. You won’t regret it, and your users will thank you for it.
Staying Focused and Making Bets
I had a really great conversation today with a super smart individual. Whenever I can step away from a conversation / meeting knowing that I’ve learned something, it’s awesome. Today was one of those meetings.
One of the topics that came up was, with relation to a start-up, how imperative it is to do four things to give your organization the best chance at succeeding:
- Develop a vision and focus on it
- Place your bets
- Be heavily critical about the business
- Obsess about results
I know the term “vision” doesn’t always have the best connotations. I could easily see someone hearing the word and picturing those cheesy motivational posters hanging in SVP/EVP and C-level offices all over corporate North America. Take a look at the picture to see what I mean.
What I’m talking about is a very centered description of your business and what you are trying to accomplish. This isn’t a PHd or Masters paper here, folks. It’s a very clear and concise paragraph at most. You want something that everything else is aligned to - if you don’t have it, you are going to start spinning your wheels and going in circles; trust me on that one.
Once you have the vision (and yes, it can change - this is a start-up after all) you need focus. At the exclusion of all else, everyone in the organization must be 100% committed to executing that pre-defined vision. Your product roadmap must be committed to it. Your finance people must be committed to it. Your support reps, sales reps, etc…
If “really awesome / cool” product ideas start to seep into the culture, you’re sunk. Kill everything that exists that doesn’t support the vision and focusing on it.
Of course, let’s be realistic. You’re certainly going to want to assess whether your vision is attacking an appropriate market opportunity. If it’s not, that’s also a major cause for concern unless you know how to create new markets. That’s not entirely out of the question, but it is quite difficult.
Once these pieces are put together, you need to place your bets. You could be wrong. Trust me, it wouldn’t be the first time a start-up executed 100% correctly but failed due to external consequences (i.e., wrong market, wrong product, wrong timing, etc…). But if you can get something out the door that solves a problem for a set of people, are you light years ahead.
Again, and I can’t stress this enough, you very well may be wrong. Your board / investors have to be comfortable with the fact that not everyone hits a home run every time. Sure, take input - but don’t let things veer off course. Remember, you have to maintain focus in order to place a bet on the vision you’ve constructed.
OK, so say you get all of these things in place - what’s next? Well, you must be critical of the business. Let me repeat - “the business” does not mean “other people.” For example, you have an amazing quarter and are really rocking along (blowing away estimates, shipping stuff ahead of schedule, improving productivity, landing some key talent, etc…) how are you going to do better next quarter?
On the flip side, what if you have a horrible quarter? Don’t blame Jimmy in the mail room. Take ownership. Are you maintaining morale? Are you still maintaining your focus on the bets you’ve already placed? If not, you have some re-evaluating to do. Namely, identifying how things slipped off the rails during this time period. But always be heavily critical of the business. This dovetails into the next point…
Obsess about results. If you aren’t concerned with the results you are delivering, something is out of whack. If you can’t deliver the results you need to, change something. If you are delivering, see the previous point - heavily criticize the business.
Of course, this all may seem like common sense. I guess that’s kind of the point. It’s all the little things that really make the difference - and ultimately, being able to execute.
Ask a Product Manager Question
I finally found some time to put my thoughts down on paper from a question Jeff Lash sent my way regarding how to gather data in an organization (either B2B or B2C) that may not want or support PMs talking directly to prospects or customers.
Crazy idea, I know. Believe me, it happens quite a bit.
So, here it is: Can a product manager get feedback without talking to customers?
Listening Labs
Not sure how many of you out there have been a part of usability studies or testing. My personal opinion is that “studies” put people in contrived situations and expect them to yield real-world data about how they are interacting with products.
This research is inherently faulty. Why? Namely because when asked, the natural response is for people to tell a researcher (or “expert”) what they think they want to hear.
Funny enough, Harry Beckwith covers this very, very well in his book The Invisible Touch.
I just watched one of Robert Scoble’s early interviews at his new FastCompany.tv gig with Mahalo founder Jason Calacanis. It’s not secret I’m a big Jason fan, and I think the Mahalo product is outstanding; in fact, it has replaced Google as my homepage.
The whole interview (parts 1 and 2) are really worth watching. However, about 13 minutes in to the first segment, they interview Mahalo’s Director of User Experience. He talks specifically about what i started this post discussing. His insights about what Mahalo does are fascinating - check it out:
To follow this up, Eric has created a Mahalo page with his insights and information about user testing. This is fascinating stuff, and I hope it can bring some additional depth to your product development process in the future.
Agile Risks
It’s easy to get taken with the simplicity of agile. Less requirements to write, less documentation, less time planning - but it all pans out in the end. I find that once you are in the thick of it, there are intricacies that can really bite you in the ass if they don’t have the proper attention paid to them.
Let me provide an example that has to do with requirements - writing them, providing them to development, and just their overall management.
If you write a one-line requirement, develop your workflows, and then your wireframes and send it on over to development, is there a risk they won’t get it? Um, yes. For sure.
Now, on the flip side, could you spend a lot more time writing those requirements, break out each permutation and case, and then send it on to development and still have them make incorrect assumptions? Yep - probably much less of chance, seeing as the requirement would box the developer into exactly what needs to be executed, but there is still a chance.
But, will it take you much longer to write that requirement? Yes. Will there be more documentation produced as a result? Yes. Will people talk more, or read more? Read more. Is that bad? In my book, yes it is folks.
See, you want people talking as much as possible. Communication. Remember that thing people did before e-mail and telephone? They actually talked to one another - face-to-face. Why people don’t do this as much now is left up to a myriad of reasons. However, the “agile” style of simple requirements (i.e., capture the damn idea and what it is supposed to do, design it so you don’t have developers opening up Photoshop and churning out graphics, and write all the necessary associated copy, etc.) will intentionally drive more and more communication.
The product manager should be ahead of the release curve. We do one release per month and I’m already planning May. This is highly ideal (while not always possible in every scenario) because I have a ton more bandwidth to provide the necessary attention to, and answer the questions about, how certain requirements are to work.
Now, developers of course need to know when to ask the questions. Because this style of requirements leave a lot more flexibility (by design), they need to know it’s OK to say, “alright, product flunky, I don’t get it. Why is it supposed to work this way?” To which I would reply, “for reasons X, Y, Z” or, “um, I dunno. Good call - you got me. Let’s make a change.”
Now, how does this relate to risk? Well, of course, this method leaves you open to having development mis-intepret the requirements, not ask about them because they believe to understand them and they are seemingly clear, and they get implemented wrong.
This is bad, but hey - agile is all about iterations. Sure, expect to get possibly lambasted for not getting something out the door because the requirements weren’t understood (PMs, I’m talking to you - a release is your responsibility), but at the end of the day, it’s one of the key agile risks.
You could toil away in your office, spend three weeks writing requirements for a single feature (been there, done that) and then always be “catching up” or “not getting stuff done on time due to having a lack of time to do it.” Or, just document what it’s supposed to do and trust your developers (because they are really super smart people) to interpret and understand, and know when to ask questions.
Then, expect you’ll have some good conversations, and possibly some healthy conflicts, over what something is supposed to do. And that, my dear friends, means the product is going to be a lot better at the end of the day.
