The Importance of QA
I am in the process of hunting for a great senior QA analyst. That being said, I wanted to take some time to talk about how important QA is in the product management process; or at least, how important I view it to be.
In my experience, if you are able to find and bring in the right QA person, that’s really into QA (and not just using it as a stepping stone), they will have a strong / noticeable impact on your product as early as its next release. I’m not kidding. I was lucky enough to work with a very talented QA engineer / analyst while at MusicIP, and his work ethic and results were absolutely perfect for helping us with several key initiatives.
So, what are some of the things that I’ll rely on QA for? I’m glad you asked…
Primarily, you don’t want defects in your product. I view the role of QA to help identify and manage bugs, but also prioritize them and ensure they are getting fixed. This way, I can stay focused on planning the product and trust that when a certain set of users can’t do something they are supposed to, it will get taken care of accordingly.
So, this means setting up your QA team to succeed. And usually, that means metrics and objectives. The key metric I’ll look to that’s directly tied to QA is product quality. Essentially, this is an amalgamation of high severity defects throughout the product (1s, 2s, and 3s usually). A great QA person will derive these priority values by assigning visibility and class rankings to them. For example, how likely is it a user will encounter a specific defect? And, if they do encounter it, what is the likelihood of it completely hindering their product experience?
You need to have these checks in place in order to ensure what you are shipping (especially as it grows and becomes more and more complex) is of the highest possible quality.
Now, the other key aspect I look for in QA is sanity checking my requirements and making sure their associated test plan runs the gamut. I may state that a user “must be able to create an account” and that “the username and password fields are required,” but the QA analyst will take this one step further and actually determine all of the permutations (that either may or may not be documented in the requirements) development must cover.
In this example specifically, maybe the e-mail address the user enters is malformed. Maybe their password doesn’t match to the 6-character standard. When you are working on releasing code quickly, especially in an agile system, this type of coverage becomes crucial. I need that constant stream of discussion and feedback.
Essentially, I’m looking for trust - not only from the product team, but the QA team (regardless of wether they sit in product or engineering), the development team, the marketing team, etc… I want to develop the trust that the QA team will catch critical user flaws (hopefully) before requirements even get to development for implementation so we can work out the kinks and ensure they are as clear as possible.
Beyond this, QA can really help development too — with structuring automated test suites, tracking their results per build and really grinding out the processes to ensure maximum build effectiveness (especially when using continuous integration tools) and insist the quality is really high and stays there. They may also be able to help when running core A/B tests on a site to help determine if user’s are encountering critical flaws in one implementation and/or another and what the outcomes are of possible change.
Remember, the “Q” in the abbreviation stands for quality. That’s their job. It’s not to administer servers or to be an authority for developers. However, it’s one of the most critical jobs in an organization - especially one that is just building / starting out. If the product you are releasing is poor quality, chances are it’s going to get very far.
Bring in an experienced QA person early and really maximize their value as much as possible. You won’t regret it, and your users will thank you for it.
Breaking into a Sprint
I just stepped through a session planning out the next couple of iterations on a product halo with a colleague of mine. The process we have taken to using is quite cool. I don’t proclaim it to be unique in any sense of the word, but definitely something I’m liking the feel of, and one I could see really developing and growing with the organization.
I started by writing out a set of bullet points, broken in to feature groups on the white board. This was our baseline. We talked through each point, made some revisions, removed some, added some others, and proceeded forward. This really allowed us to reset, and get in to what it was we were trying to accomplish.
We then proceeded to prioritize each bullet point. Since we have a lack of customer data, and some things are just obvious (especially when dealing with halo items), it was pretty easy to go through and assign low / med / high or 1 / 2 / 3 to each bullet we had written up there.
After the bullets were all prioritized, we then assigned gross hourly estimates to them - nothing under 2 hours allowed. Since the bullets weren’t full requirements, this kept things padded to a reasonable amount, allowing us to continue marching towards to goal (yes, details on what exactly what is forthcoming).
We then decided, right or wrong, that since these are halo items and not product “features” in the true definition of the word, we didn’t need to rev the version number at all, really. So it was just a matter of adding up all the gross estimates and then identifying sprint length and doing some simple division.
So, we landed on each sprint being 10 days in length for the project. With all the tasks grossly estimated, this ended up being something like 9 sprints total. We backlogged a few things, but couldn’t really get that gross-level estimate to budget too far, so we stuck with it.
My colleague now has the lucky job of fleshing out those bullets in VersionOne and assigning them to the correct sprints. And then of course, gathering up all of the materials that do already exist and attaching them to the right items, and then proceeding to execute on some key market research tasks that were identified.
Yeah, I guess I’ll help out at some point.
So, what did we end-up with? Well, a pretty common thing - a set of requirements, associated to a release project and sprints in our requirements / product management tool (VersionOne). “Big deal” you say? Well, it’s important to recognize that when you have a crap load of things to do (especially halo items, which can be convoluted), using a framework / brief / flexible process that helps get all that stuff that’s currently up in your head down on paper and in front of others, can be a huge weight off your shoulders.
Keep tweaking, keeping writing things down - but when you find you have it all up in your head, take a step back, get some people in the same room as you and start to write out bullet points and take it from there. Worst case you end-up with an empty head and an eased mind. Best case, you have developed a quick and dirty roadmap for the next several weeks that can be iterated on.
Agile Requirements
I’ve written long requirements documents. Really long. Coupled with vision documents as well.
They suck up a lot of time, and usually, the argument is that they are needed in order to keep people on track.
Well, recently, I’ve been experimenting with agile requirements writing and let me say this: it’s one heck of a lot easier.
I’ve combined a basic requirements table / roadmap along with product definition to create my own PRD-style document. The TOC looks like this:
- Document Overview
- Problem Statement
- Vision
- Market Segments / Target Market
- User Goals
- Requirements
The overview section is pretty boilerplate. I’ll include information regarding the document as well as any overarching requirements themes that I have in it. For example, within a user-based Web application, one requirements theme coul be “user management.”
Aside from that, the other sections that are included are pretty self-explanatory.
The requirements portion though - that’s the difference.
I have all requirements laid out in tables. One table per version currently planned, and the one table at the end of the document for the backlog of functionality that’s important, but not yet assigned to a release.
The table I’m referring to consists of 6 columns:
- ID
- Name
- Description
- Estimate
- Risk
- Notes
Each of these columns play an important role when they are applicable. I say that “when they are applicable,” because not all of them always are. For example, small internal projects need documented requirements, but doing a risk assessment is probably over the top there.
Each row in this table is a “story” by proper definition. And, as most of us should recognize, stories are (in effect) just a more “modern” version of saying requirement. But to be clear, they are not use cases.
Oh, I should stay away from this - I’m verging on religious discussions. Eek!
Now, there is another person in our company working on defining requirements for the first time. When discussing methods to do this, she had a hard time, and felt very overwhelmed with the death-march style of waterfall project management.
And no, the pros / cons of waterfall did not lend well to this project. A more agile approach was highly applicable.
But, the benefit here was, by following this simple structure, she was able to get a great handle on the project / product she’s working to ship quickly and had a much easier time explaining it to others in this format.
I have the same feelings.
Some so straightforward lends itself quite well when needing to get these types of things in front of developers, project managers, QA folks, and writers - all working on the same product.
I’ll post more about this structure / format in the future. Just wanted to get some thoughts down on it this evening.
Requirements vs Solutions
There is a big different between creating requirements and creating solutions (or specifications, in some circles). It may sound odd, but this is a tough thing to learn. Yes, Scott always gets me thinking about writing requirements 
A lot of folks, especially if they have experience with implementation / coding / whatever, will have an inclination to state a solution and not a requirement. The trouble with that is, they can be irrelevant and confusing, and not make sense to the dev team.
My advice: leave the solutions and specs to the programmers and developers and architects and even program managers. This is a very hard split to make, but I was very lucky it’s lesson I learned early on in my career and carry with me to this day.
So, what exactly do I mean? Let’s look at an example.
Say you wanted to call attention to a group, or type, of items in a report. What’s the difference between the requirement and solution? Well, a solution (or the “spec”) would look like this:
“When running the SQL query to extract data out of the database for the report, loop through it, and if an item equals [some value meeting a condition] set it’s associated <span> tag to have a background color of #f5f5f5.”
Or, even to a greater degree:
“Connect to the database using the ADOdb connection library, execute the SQL query (SELECT…), and loop through it like this: (foreach…), and make sure the HTML looks like this: (<span>). This isn’t rocket science and should only take you an hour or so.”
The problem with this, even if you can do it right, is that the PM is a) not a developer, b) you will inevitably develop disdain with developers and (to be candid), c) you are wasting time.
See, the requirement is much shorter. Something like this:
“Highlight items that meet condition X in report Y.”
You could go further and describe the business value / context. Maybe there are some permutations in there. I will typically do this in a quick conversation with the developer, but that’s because I am afforded the flexibility. I have written things like that out in the past.
Now, yes, you could state a non-functional requirement as well:
“Make items of type X in report Y yellow.”
So yes, understand the difference between functional (do stuff) requirements and non-functional (look & feel stuff) requirements.
Now, to me, this makes perfect sense. You can bullet out a million requirements this way. OK a million is probably excessive. But you get the picture. If you were stating SQL, code, this and that (i.e., the spec), it would take you a lot longer.
And your time is better spent writing what? Well, more requirements of course (or release notes). Or doing BAT testing, or being on a Sales call, or a myriad other things.
Pulled Every Which Way
Product managers get pulled in a ton of different directions. Leading by indirect influence can also put you in the middle of a lot of things you don’t have the authority to prevent or solve yourself. Or, think you are guiding folks down the right path, only to have it turn out needing to be completely different at the end of the day? it’s gross.
I will say this, for all of the process and procedure Scott goes through for requirements, I’ve been working with the alternative (i.e., not doing much at all), and while beneficial from a speed perspective, it’s very easy to get people off the page you are on and/or not be on the same as them.
To all those that are trying to avoid 1,000+ page requirements documents, I will tell you this (from experience):
Walk over to people’s desks and talk. Or, if that’s not possible, get on the phone.
Get on the phone and talk through key product enhancements and projects early and often.
Get prospects on the phone to interview them about what you are doing. Get clients on the phone and treat them almost like design partners. (hint: you can do that even when writing large requirements docs).
Get Sales & Marketing on the phone. Get in front of support. Client Services.
And of course, get to your Senior Management team. Being “too busy” to talk about key product-level initiatives is not an excuse for them — or a product manager. I don’t care if you are Bill Gates or Steve Jobs. If management isn’t in the loop, when they get something that isn’t what they expected, you will get a phone call, and it won’t be to say “happy birthday.”
You don’t need forms or signatures. You need to talk. The worst case thing that can happen is development working on something that is a) not attuned to the market and/or b) not communicated well to key folks in the organization, no matter how big that company may be
To be an effective product manager, you must not fall into a trap of being “too burned out” or overloaded with other tasks to talk to customers. Conversely, if you absolutely cannot talk to customers / prospects regularly, you better make sure you have folks covering that aspect, and more importantly actually writing things down, that you trust.
This all ties together. It may seem crazy, but it’s true.
Don’t stop talking. And don’t avoid getting in front of prospects and customers. Your shareholders and VC’s will thank you later, even if you do take a bit of heat for it up front.
