Exploratory Testing: A Lost Art?

Discuss the feature articles on the Tech Home Page.
User avatar
Posts: 490
Joined: Wed Jul 18, 2007 3:38 pm
Location: Salt Lake City, Utah

Exploratory Testing: A Lost Art?

Postby McDanielCA » Fri Feb 19, 2010 11:49 am

Exploratory Testing: A Lost Art? was originally posted on the Projects page of LDSTech. It was written by David Kosorok.


When I was a kid, I would use a hammer to set my tent stakes or to pound nails into two-by-fours to build tree forts. I thought I could use the hammer for anything. If I hit a board hard enough, it would break, and I wouldn’t need to use the saw. If I hit the plywood just right, I could punch a hole through the board without using the drill.

Obviously, using one tool to do it all may get quick results, but it won’t look pretty. Many in the testing industry use a similar single-tool approach, attempting to solve every problem using automated testing. No time? Slap some automation on your software and ship it out the door. No money? Cut down on testers and have an automation engineer do it all.

While I’m a proponent of automation in its proper context, you still need the interactive tester to dig deep using exploratory testing and find bugs. Let’s look into what exploratory testing is and a method to find great exploratory testers.

Exploratory testing is an art—know the requirements, understand the features, seek out their weaknesses, all while building the test plan. Cem Kaner, defines exploratory testing as “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

In other words, exploratory testing combines ad hoc testing with intelligent randomized testing. This testing approach requires a skilled tester with knowledge of the product and well-honed testing instincts. James Bach, another expert on exploratory testing, defines exploratory testing as “test design and test execution at the same time.”

I still don’t shun the automation engineer—we must have automation to reduce repetitive tasks and give us more time to dig deeper with exploratory testing. But sometimes people overemphasize automation. One team I used to manage had a few expert automation engineers that even refused to log bugs. They said their job wasn’t to find bugs but to write automation. Huh?

How, then, do you find an exploratory tester? You need to know how to recognize the attributes of a successful exploratory tester. Over the years, I have seen the following attributes continue to bubble up in great candidates:

  • Natural test aptitude
  • Current technical knowledge
  • Desire to continually learn
  • Deep curiosity of how things work
  • Strong problem solver
  • Good communicator
  • An ability to handle ambiguity
I needed to find a way to help filter out the best potential candidates. A number of years ago, in conjunction with another senior tester, I developed a sample application with built-in bugs. It is a simple application, with GUI defects, memory leaks, functional quirkiness, documentation errors, etc.

I give an interviewee a 15 minute limit to find as many defects as they can. I look for somebody who enthusiastically dives into this activity and tries to uncover many bugs. This model has consistently guided my search for testers with great aptitude and has helped me identify strong candidates. The best candidates apply a structured methodology to find bugs rather than spending too much time running redundant tests.

I don’t want to give away all my secrets, since I may be interviewing some of you in the future, but I do want to point out that after I’ve verified that the candidate has tested the basic features, he or she should also hit boundaries, accessibility, error cases, usability, and edge cases.

The success rate I’ve had using this tool, while not scientifically documented, has been very high—many of the candidates testing high have proven themselves to be great exploratory testers.

Automation will always be part of testing, but an exploratory tester brings insight and intelligence to the testing process. Finding the right tester will improve the testing process and ultimately create a better product for the end user.

David Kosorok is a QA Team Manager for the Church.

New Member
Posts: 2
Joined: Mon Aug 27, 2007 7:01 pm

Agreed, and willingly borrowed

Postby MarkEWaite-p40 » Mon Feb 22, 2010 7:39 am

I agree with your technique (hire testers who love testing, and who show initial aptitude for testing). I've had great results with that sort of "testing challenge" as one of the aspects of hiring exploratory testers.

Other techniques we've used to assure we were hiring good exploratory testers included (1) interview team includes product owners (scrum phrase for business analyst), programmers, managers, and other testers, (2) grant veto power to any member of the interview team that objects to the candidate, and (3) background check the entries on their resume (trust but verify).

I've also liked the results we obtained from having testers attend the "Black Box Software Testing" course from the Association for Software Testing, and the "Bug Advocacy" course from the same group.

New Member
Posts: 2
Joined: Thu Oct 22, 2009 2:33 pm
Location: Lehi, UT

there are a few other things we can do to assess testers...

Postby dkosorok » Fri Feb 26, 2010 12:41 pm

Good comments, Mark. I've also been looking at using online assessments to validate general testing knowledge. This can be both good and bad - good for consistency, bad if the tester is brilliant but learned the processes under a different set of vocabulary.

New Member
Posts: 2
Joined: Thu Dec 26, 2013 11:00 am
Location: Salt Lake, Utah

Re: Exploratory Testing: A Lost Art?

Postby vovnik » Thu Dec 26, 2013 11:09 am

I agree with you, dkosorok. Standardization is always such a two edged sword. It's good to get things down to a science in terms of efficiency, but it's also a struggle because sometimes you can miss out on some really talented individuals who have a self-taught or sort of "quirky" background that is led them to where they are today.

Return to “Featured Article Discussions”

Who is online

Users browsing this forum: No registered users and 0 guests