The Importance of QA
I am in the process of hunting for a great senior QA analyst. That being said, I wanted to take some time to talk about how important QA is in the product management process; or at least, how important I view it to be.
In my experience, if you are able to find and bring in the right QA person, that’s really into QA (and not just using it as a stepping stone), they will have a strong / noticeable impact on your product as early as its next release. I’m not kidding. I was lucky enough to work with a very talented QA engineer / analyst while at MusicIP, and his work ethic and results were absolutely perfect for helping us with several key initiatives.
So, what are some of the things that I’ll rely on QA for? I’m glad you asked…
Primarily, you don’t want defects in your product. I view the role of QA to help identify and manage bugs, but also prioritize them and ensure they are getting fixed. This way, I can stay focused on planning the product and trust that when a certain set of users can’t do something they are supposed to, it will get taken care of accordingly.
So, this means setting up your QA team to succeed. And usually, that means metrics and objectives. The key metric I’ll look to that’s directly tied to QA is product quality. Essentially, this is an amalgamation of high severity defects throughout the product (1s, 2s, and 3s usually). A great QA person will derive these priority values by assigning visibility and class rankings to them. For example, how likely is it a user will encounter a specific defect? And, if they do encounter it, what is the likelihood of it completely hindering their product experience?
You need to have these checks in place in order to ensure what you are shipping (especially as it grows and becomes more and more complex) is of the highest possible quality.
Now, the other key aspect I look for in QA is sanity checking my requirements and making sure their associated test plan runs the gamut. I may state that a user “must be able to create an account” and that “the username and password fields are required,” but the QA analyst will take this one step further and actually determine all of the permutations (that either may or may not be documented in the requirements) development must cover.
In this example specifically, maybe the e-mail address the user enters is malformed. Maybe their password doesn’t match to the 6-character standard. When you are working on releasing code quickly, especially in an agile system, this type of coverage becomes crucial. I need that constant stream of discussion and feedback.
Essentially, I’m looking for trust - not only from the product team, but the QA team (regardless of wether they sit in product or engineering), the development team, the marketing team, etc… I want to develop the trust that the QA team will catch critical user flaws (hopefully) before requirements even get to development for implementation so we can work out the kinks and ensure they are as clear as possible.
Beyond this, QA can really help development too — with structuring automated test suites, tracking their results per build and really grinding out the processes to ensure maximum build effectiveness (especially when using continuous integration tools) and insist the quality is really high and stays there. They may also be able to help when running core A/B tests on a site to help determine if user’s are encountering critical flaws in one implementation and/or another and what the outcomes are of possible change.
Remember, the “Q” in the abbreviation stands for quality. That’s their job. It’s not to administer servers or to be an authority for developers. However, it’s one of the most critical jobs in an organization - especially one that is just building / starting out. If the product you are releasing is poor quality, chances are it’s going to get very far.
Bring in an experienced QA person early and really maximize their value as much as possible. You won’t regret it, and your users will thank you for it.
Some New Tools
Well, there have definitely been some cool new Web-based apps and tools released. But, would a day really be a true day right now if a new Web-based app wasn’t released?
It’s true - they are everywhere.
And while some of them don’t in fact serve any purpose whatsoever, a large part of them do. I don’t know what that is attributed to, but there are some very interesting concepts out there and some very interesting solutions to different types of problems we are encountering day-to-day.
I wanted to post and draw some attention to Twitter and Pownce.
Now, I use both. I find I use Twitter more for boring, throughout the day style posts. Where I’m at, what I’m doing, that type of thing. Think old-school blogs. Like those very first blogs that started showing up on the ‘net back in the day when blogs first started gaining ground. Here’s an example:
“I burnt my toast this morning, so I ate Fruit Loops for breakfast.”
And that was a single entry of many throughout the day. Twitter is more a medium / application that lends itself to this style of posting, and everyone on there (even the cool kids) is doing it.
Since I’m the furthest thing from “a-list.,” I’m fine following that trend.
Pownce on the other hand I’m using entirely different. We are in the the thick of trying to get v1.0 of a new product out the door. It’s a hectic week, no matter how much work you put in up-front. Launch weeks are very stressful and exciting. Add the fact that you have never done a v1.0 launch together as a group, and it increases the complexity.
Now, I’m one of the lucky PM’s that have a solid Product Analyst to work with (you’re rockin’ Jenn!). She’s learning the ropes of product management, and doing a fantastic job.
She and I have been using Pownce to get through the last few weeks in terms of defect tracking. Yes, we have formal bug tracking software. But, on a v1.0 in a start-up, the name of the game is speed. I don’t want to be sitting around filling out CR entries, assigning them, forcing developers to read them, re-assign, etc, etc… It would kill our momentum.
So, each morning, we compile our list for the day for development to work through (in true “agile” like style) and that includes notes we have logged / passed back and forth to one another using Pownce. It’s been insanely helpful for speed and completeness.
This week, we have the added fun of leaving the office, going to a coffee shop at night to step through the days work and re-assess where we are at for tomorrow morning so we don’t hold-up dev.
Now, I realize this can’t go on for every release, and I hope to god I’m not so lazy it does. We are learning where our balance is between long, streched out waterfall-style docs and short, brief agile style docs. I want to keep things trim, but we are learning with our dev team what’s going to help for future releases; not only v1.0’s but other revs and upgrades as well.
So, I’d recommend trying it out. If you need invites, let me know. But, it’s helped me a great deal. Not only for bugs, but also if you can get your entire team on there, for updates throughout the day. Even though we’re a really small company, you can’t spend your entire day giving and collection status updates - then you’re wasting time.
But, spending 2 seconds to jump into Pownce, select your group of work friends and post a quick update like, “Steve is done with feature A - test it out and give feedback.” That rocks. It’s instant, immediate, folks know there’s progress and they can take part right away if they like. If not, the message is there for them later on to review at their leisure and give feedback on.
XBox Breakage
So, last week while I was working towards getting some hardcore achievements, my xbox 360 crapped out on me. It’s been a pretty frustrating week, considering it happened while my wife has been out on business.
All my pizza eating, xbox playing plans went right out the window.
But, it was an interesting experience to go through all of the points of support before deciding, yes it needs to go in for repairs. I pretty much tried everything.
My first step was to head over to xbox.com. It was really simple to find their knowledbase, and the articles I worked with were well laid out and written. Very simple to diagnose / find what was wrong with my console and get their tips.
Once I had played out as much as I could, I actually managed to get the DVD-ROM drive working to the point of getting back in game and getting some stuff done. But, it didn’t last too long before my entire console was frozen again.
So, on to email support. They wrote back quite a friendly message and told me to call support. They had all the helpful info: phone number, reference number based on my email, and the name of the support rep that replied, which I really like. I actually have our support reps do the same thing.
Now, the one beef I had with support was, the rep was CLEARLY reading right off of a script. Now, I realize that she probably didn’t even work for Microsoft or the XBox team, but it would have been nice if she did. I always got this feeling that the XBox team was this cool little start-up like thing inside of the bigger borg that is MSFT.
I lost that when I talked to support. Now, don’t get me wrong, she did a good job and gave me everything I needed, but it seemed like returning the console was the very first thing she jumped to. She didn’t even sound like she owned an XBox, let alone understand what a DVD-ROM drive was and what I was saying to her.
I think had she actually been more passionate about the console, and had something like, “yeah, my friends went through that, what date was your console issued? Yeah, we’ve had reports of those failing” — or something more than just going down her paper script, it would have continued me buying into the small, start-up like XBox team brand.
Maybe I’m crazy, though. I could very well be the only person on Earth that gets that feeling about the XBox team. But, that’s one of the reasons why it’s so important that Support is a key part of Product Management. You get it built into their thinking that there is something more they could do to enhance the customer’s experience. A rep that was saying, “hey, we noticed that 3,000 of the 10,000 XBox’s manufactured during date X to date Y had DVD-ROM problems” would allow MSFT to put the right literature on their site.
I’m not saying that they don’t do that, but based on who I spoke with, I have a hard time buying into that type of ambition going on from their front lines.
Come Visit at CES!
Just a quick post to let anyone know that’s going, we’ll be chilling at CES in the Sands — we’re over next to the Slingbox booth. Stop by and say hi if you’re around, and we’ll show you some really cool stuff…
Watching TV on Your Lap
There’s just something so cool about watching TV on your lap. In addition to getting my new Macbook Pro this week, I also found out that SlingMedia (the makers of SlingBox) release the beta client for OS X. Sweet!
There’s nothing better than watching David Letterman guess what kind of pies his mother made for Thanksgiving. I wish that everyone could have as great of a first American Thanksgiving experience as I did.
To all those wondering what the different is between Canadian and American thanksgiving, I don’t think there’s much. I’d say sports, but The Leafs weren’t even playing tonight, and there’s a whole slew of NFL football thats available.
And for all those thinking about how cold it is — trust me, it could be much colder. You could be in Antarctica.
