The doctoral dissertations of the former Helsinki University of Technology (TKK) and Aalto University Schools of Technology (CHEM, ELEC, ENG, SCI) published in electronic format are available in the electronic publications archive of Aalto University - Aaltodoc.
Aalto

Empirical Studies on Exploratory Software Testing

Juha Itkonen

Doctoral dissertation for the degree of Doctor of Science in Technology to be presented with due permission of the School of Science for public examination and debate in Auditorium T1 at the Aalto University School of Science (Espoo, Finland) on the 18th of November 2011 at 12 noon.

Overview in PDF format (ISBN 978-952-60-4339-5)   [2236 KB]
Dissertation is also available in print (ISBN 978-952-60-4338-8)

Abstract

Exploratory software testing (ET) is a practically relevant approach to software testing that lacks scientific knowledge. In ET, the tester's work is not based on predesigned and documented test cases. Instead, testing is guided by a higher-level plan or mission, and the testing work involves parallel test design, test execution, and learning. One of the distinct characteristics of ET is that the tester designs the tests during ET and uses information gained to design new and better tests continuously. The ET approach relies on testers' skills and experience. The main claimed benefits of ET are the tester's ability to apply personal knowledge and creativity during testing as well as effectiveness, efficiency, and agility in terms of adapting to changes and working with imperfect documentation.

In this thesis, the ET approach has been studied using empirical research methods. Two case studies, one controlled experiment, and two field studies were performed to address three research goals: defining ET and understanding its applicability based on the literature; empirically investigating the benefits and shortcomings of ET; and providing empirically based results on how the ET approach is applied in practice.

This research identifies different approaches to ET in industry and describes concrete testing practices. The role of the tester's personal knowledge is identified in the literature, and this research provides a detailed analysis of the application of personal knowledge in failure detection using ET.

The main conclusions of this work are that ET can be as effective as test case-based approaches and even more efficient in certain contexts. The testers are capable of utilizing their personal knowledge in failure detection, and the role of personal knowledge is important in the ET approach. In addition, software testing in product organizations seems to involve multiple diverse organizational groups, and ET was found to be an applicable approach to engage domain experts in testing.

The main implications of this thesis are introducing the exploratory testing approach to the research community and motivating its relevance by providing empirical studies in industry. In addition, the results of the effectiveness and efficiency of ET as well as the qualitative data on exploratory testing practices and the detailed analysis of knowledge in exploratory testing work are valuable for the research community. The main practical implications include presenting the benefits and applicability of the ET approach along with the potential shortcomings and providing empirical evidence regarding the benefits of ET.

This thesis consists of an overview and of the following 6 publications:

  1. Juha Itkonen, Kristian Rautiainen, and Casper Lassenius. 2005. Toward an understanding of quality assurance in agile software development. International Journal of Agile Manufacturing, volume 8, number 2, pages 39-49. © 2005 International Society of Agile Manufacturing (ISAM). By permission.
  2. Juha Itkonen and Kristian Rautiainen. 2005. Exploratory testing: A multiple case study. In: Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE 2005). Noosa Heads, Queensland, Australia. 17-18 November 2005. IEEE. Pages 84-93. ISBN 0-7803-9507-7. © 2005 Institute of Electrical and Electronics Engineers (IEEE). By permission.
  3. Juha Itkonen, Mika V. Mäntylä, and Casper Lassenius. 2007. Defect detection efficiency: Test case based vs. exploratory testing. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007). Madrid, Spain. 20-21 September 2007. Los Alamitos, California, USA. IEEE Computer Society. Pages 61-70. ISBN 978-0-7695-2886-1. © 2007 Institute of Electrical and Electronics Engineers (IEEE). By permission.
  4. Juha Itkonen, Mika V. Mäntylä, and Casper Lassenius. 2009. How do testers do it? An exploratory study on manual testing practices. In: Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement (ESEM 2009). Lake Buena Vista, Florida, USA. 15-16 October 2009. IEEE. Pages 494-497. ISBN 978-1-4244-4842-5. © 2009 Institute of Electrical and Electronics Engineers (IEEE). By permission.
  5. Juha Itkonen, Mika V. Mäntylä, and Casper Lassenius. The role of knowledge in failure detection during exploratory software testing. IEEE Transactions on Software Engineering, May 2011, 17 pages, submitted for publication. © 2011 by authors and © 2011 Institute of Electrical and Electronics Engineers (IEEE). By permission.
  6. Mika V. Mäntylä, Juha Itkonen, and Joonas Iivonen. 2012. Who tested my software? Testing as an organizationally cross-cutting activity. Software Quality Journal, volume 20, number 1, pages 145-172. © 2011 by authors.

Keywords: software testing, exploratory testing, defect detection, effectiveness, experience, domain knowledge, case study, controlled experiment, field observation

This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

© 2011 Aalto University

Last update 2012-02-16