Skip to main content
Springer Nature Link
Log in

Studying Software Engineers: Data Collection Techniques for Software Field Studies

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software engineering is an intensively people-oriented activity, yet too little is known about how designers, maintainers, requirements analysts and all other types of software engineers perform their work. In order to improve software engineering tools and practice, it is therefore essential to conduct field studies, i.e. to study real practitioners as they solve real problems. To do so effectively, however, requires an understanding of the techniques most suited to each type of field study task. In this paper, we provide a taxonomy of techniques, focusing on those for data collection. The taxonomy is organized according to the degree of human intervention each requires. For each technique, we provide examples from the literature, an analysis of some of its advantages and disadvantages, and a discussion of how to use it effectively. We also briefly talk about field study design in general, and data analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anquetil, N., and Lethbridge, T.C. 1999. Recovering software architecture from the names of source files. Journal of Software Maintenance: Research and Practice 11: 201–221.

    Article  Google Scholar 

  • Baddoo, N., and Hall, T. 2002. Motivators of software process improvement: An analysis of practitioners’ views. Journal of Systems and Software 62: 85–96.

    Article  Google Scholar 

  • Baddoo, N., and Hall, T. 2002. De-motivators of software process improvement: An analysis of practitioners’ views. Journal of Systems and Software 66: 23–33.

    Google Scholar 

  • Basili, V. R. Software modeling and measurement: The Goal/Question/Metric paradigm, Tech. Rep. CS-TR-2956, Department of Computer Science, University of Maryland, College Park, MD 20742, Sept. 1992.

  • Beecham, S., Hall, T., and Rainer, A. 2003. Software process improvement problems in twelve software companies: An empirical analysis. Empirical Software Engineering 8: 7–42.

    Article  Google Scholar 

  • Bellotti, V., and Bly, S. 1996. Walking Away from the Desktop Computer: Distributed Collaboration and Mobility in a Product Design Team. Cambridge, MA: Conference on Computer Supported Cooperative Work, pp. 209–219.

    Google Scholar 

  • Berlin, L. M. 1993. Beyond Program Understanding: A Look at Programming Expertise in Industry. Empirical Studies of Programmers. Palo Alto: Fifth Workshop, 6–25.

    Google Scholar 

  • Bratthall, L., and Jørgensen, M. 2002. Can you trust a single data source exploratory software engineering case study? Empirical Software Engineering: An International Journal 7(1): 9–26.

    Google Scholar 

  • Briand, L., El Emam, K., and Morasca, S. 1996. On the application of measurement theory in software engineering. Empirical Software Engineering 1: 61–88.

    Article  Google Scholar 

  • Buckley, J., and Cahill, T. 1997. Measuring Comprehension Behaviour Through System Monitoring, Int. Workshop on Empirical Studies of Software Maintenance, Bari, Italy, 109–113.

    Google Scholar 

  • Budgen, D., and Thomson, M. 2003. CASE tool Evaluation: Experiences from an empirical study. Journal of Systems and Software 67: 55–75.

    Article  Google Scholar 

  • Chi, M. 1997. Quantifying qualitative analyzes of verbal data: A practical guide. The Journal of the Learning Sciences 6(3): 271–315.

    Article  Google Scholar 

  • Curtis, B., Krasner, H., and Iscoe, N. 1988. A field study of the software design process for large systems. Communications of the ACM 31(11): November, 1268–1287.

    Article  Google Scholar 

  • Damian, D., Zowghi, D., Vaidyanathasamy, L., and Pal, Y. 2004. An industrial case study of immediate benefits of requirements engineering process improvement at the australian center for unisys software. Empirical Software Engineering: An International Journal 9(1–2): 45–75.

    Google Scholar 

  • Delbecq, A. L., Van de Ven, A. H., Gustafson, D. H. 1975. Group Techniques for Program Planning. Scott. Glenview, IL: Foresman & Co.

    Google Scholar 

  • DeVaus, D. A. 1996. Surveys in Social Research. 4th edition. London: UCL Press.

    Google Scholar 

  • Draper, S. 2004. The Hawthorne Effect. http://www.psy.gla.ac.uk/~steve/hawth.html.

  • Ericcson, K., and Simon, H. 1984. Protocol Analysis: Verbal Reports as Data. Cambridge, MA: The MIT Press.

    Google Scholar 

  • Foddy, W. 1994. Constructing Questions for Interviews and Questionnaires: Theory and Practice in Social Research. Cambridge, MA: Cambridge University Press.

    Google Scholar 

  • Hassan, A., Holt, R., and Mockus, A. 2004. MSR 20004: The international workshop on mining software repositories. Proc. ICSE 2004: International Conference on Software Engineering, Scotland, UK, May pp. 23–28.

  • Herbsleb, J., and Mockus, A. 2003. An empirical study of speed and communication in globally distributed software development. IEEE Transactions Software Engineering 29(6): 481–494.

    Article  Google Scholar 

  • Hungerford, B., Hevner, A., and Collins, R. 2004. Reviewing software diagrams: A cognitive study. IEEE, Transactions Software Engineering 30(2): 82–96.

    Article  Google Scholar 

  • Iivari, J. 1996. Why are CASE tools not used? Communications of the ACM 39(10): October, 94–103.

    Article  Google Scholar 

  • Jick, T. 1979. Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly 24(4): December, 602–611.

    Google Scholar 

  • Jordan, B., and Henderson, A. 1995. Interaction analysis: Foundations and practice. The Journal of the Learning Sciences 4(1): 39–103.

    Article  Google Scholar 

  • Jørgensen, M. 1995. An empirical study of software maintenance tasks. Software Maintenance: Research and Practice 7: 27–48.

    Google Scholar 

  • Keller, R., Schauer, R., Robitaille, S., and Page, P. 1999. Pattern-based reverse engineering of design components. In: Proc, Int. Conf. Software Engineering, Los Angeles, CA, pp. 226–235.

  • Kemerer, C. F., and Slaughter, S. A. 1997. Determinants of software maintenance profiles: An empirical investigation. Software Maintenance: Research and Practice 9: 235–251.

    Article  Google Scholar 

  • Kensing, F. 1998. Prompted Reflections: A Technique for Understanding Complex Work. interactions, January/February, 7–15.

  • Lethbridge, T. C. 2000. Priorities for the education and training of software engineers. Journal of Systems and Software 53(1): 53–71.

    Article  Google Scholar 

  • Lethbridge, T. C., and Anquetil, N. 2000. Evaluation of approaches to clustering for program comprehension and remodularization. In H. Erdogmus and O. Tanir, (eds), Advances in Software Engineering: Topics in Evolution, Comprehension and Evaluation, New York: Springer-Verlag.

    Google Scholar 

  • Lethbridge, T. C., and Laganière, R. 2001. Object-Oriented Software Engineering: Practical Software Development Using UML and Java. London: McGraw-Hill.

    Google Scholar 

  • Miles, M. B. 1979. Qualitative data as an attractive nuisance: The problem of analysis. Administrative Science Quarterly 24(4): 590–601.

    Google Scholar 

  • Miles, M. B., and Huberman, A. M. 1994. Qualitative Data Analysis: An Expanded Sourcebook. 2nd edition. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Mockus, A., Fielding, R. T., and Herbsleb, J. D. 2002. Two case studies of open source software development: Apache and mozilla. ACM Trans. on Software Engineering and Methodology 11(3) 209–246.

    Article  Google Scholar 

  • NASA, SEL COTS Study Phase 1 Initial Characterization Study Report, SEL-98-001, August 1998, http://sel.gsfc.nasa.gov/website/documents/online-doc.htm.

  • Nielsen, J. 1997. The Use and Misuse of Focus Groups. http://www.useit.com/papers/focusgroups.html.

  • Perry, D. E., Staudenmayer, N., and Votta, L. 1994. People, organizations, and process improvement. IEEE Software July, 37–45.

  • Pfleeger, S. L., and Hatton, L. 1997. Investigating the influence of formal methods. Computer February, 33–43.

  • Pfleeger, S., and Kitchenham, B. 2001. Principles of survey research Part 1: Turning lemons into lemonade. Software Engineering Notes 26(6) 16–18.

    Article  Google Scholar 

  • Porter, A. A., Siy, H. P., Toman, C. A., and Votta, L. G. 1997. An experiment to assess the cost-benefits of code inspections in large scale software development. IEEE Transactions Software Engineering 23(6): 329–346.

    Article  Google Scholar 

  • Punter, T., Ciolkowski, M., Freimut, B., John, I. 2003. Conducting on-line surveys in software engineering. Proceedings Int. Symp. on Empirical Software Eng. ‘03, pp. 80–88.

  • Rainer, A., and Hall, T. 2003. A quantitative and qualitative analysis of factors affecting software processes. Journal of Systems and Software 66: 7–21.

    Google Scholar 

  • Robbins, S. P. 1994. Essentials of Organizational Behavior. 4th edition. Englewood Cliffs, NJ: Prentice Hall.

    Google Scholar 

  • Robillard, P. N., d’Astous, P., Détienne, D., and Visser, W. 1998. Measuring cognitive activities in software engineering. Proc. 20th Int. Conf. Software Engineering, Japan, pp. 292–300.

  • Sayyad-Shirabad, J., Lethbridge, T. C., and Lyon, S. 1997. A little knowledge can go a long way towards program understanding. Proc. 5th Int. Workshop on Program Comprehension. Dearborn, MI: IEEE, pp. 111–117.

    Google Scholar 

  • Sayyad Shirabad, J., Lethbridge, T. C., and Matwin, S. 2003. Applying data mining to software maintenance records. Proc CASCON 2003, Toronto, October, IBM, in ACM Digital Library, pp. 136–148.

  • Scacchi, W. 2003. Issues and experiences in modeling open source software processes. Proc. 3rd. Workshop on Open Source Software Engineering, Portland, OR: 25th. Int. Conf. Software Engineering, May.

    Google Scholar 

  • Seaman, C. B., and Basili, V. R. 1998. Communication and organization: An empirical study of discussion in inspection meetings. IEEE Trans. on Software Engineering 24(7): July, 559–572.

    Article  Google Scholar 

  • Seaman, C., Mendonca, M., Basili, V., and Kim, Y. 2003. User interface evaluation and empirically-based evolution of a prototype experience management tool. IEEE Transactions on Software Engineering 29: 838–850.

    Article  Google Scholar 

  • Seigel, S., and Castellan, N. J. 1988. Nonparametric Statistics for the Behavioral Sciences. 2nd edition. Boston, MA: McGraw-Hill.

    Google Scholar 

  • Shull, F., Lanubile, F., and Basili, V. 2000. Investigating reading techniques for object-oriented framework learning. IEEE Transactions on Software Engineering 26: 1101–1118.

    Article  Google Scholar 

  • Sim S. E., and Holt, R. C. 1998. The ramp-up problem in software projects: A case study of how software immigrants naturalize. Proc. 20th Int. Conf. on Software Engineering, Kyoto, Japan, pp. 361–370, April.

  • Sim, S. E., Clarke, C. L. A., and Holt, R. C. 1998. Archetypal source code searches: A survey of software developers and maintainers. Proc. Int. Workshop on Program Comprehension, Ischia, Italy. pp. 180–187.

  • Singer, J., Lethbridge, T., Vinson, N., and Anquetil, N. 1997. An examination of software engineering work practices. Proc. CASCON. IBM Toronto, 209–223, October.

  • Singer, J. 1998. Practices of software maintenance. Proc. Int. Conf. on Software Maintenance. Washington, DC, November, pp. 139–145.

  • Singer, J., Lethbridge, T. C., and Vinson, N. 1998. Work practices as an alternative method to assist tool design in software engineering. Proc. International Workshop on Program Comprehension. Ischia, Italy, pp. 173–179.

  • Singer, J., and Vinson, N. Ethical issues in empirical studies of software engineering. IEEE Transactions on Software Engineering, 28: 1171–1180.

  • Snelling, L., and Bruce-Smith, D. 1997. The work mapping technique. Interactions 25–31, July/August.

  • Somé, S. S., and Lethbridge T. C. 1998. Parsing minimizing when extracting information from code in the presence of conditional compilation. Proc. 6th IEEE International Workshop on Program Comprehension. Italy, June pp. 118–125.

  • Teasley, S., Covi, L, Krishnan, M., and Olson, J. 2002. Rapid software development through team collocation. IEEE Transactions on Software Engineering 28: 671–683.

    Article  Google Scholar 

  • von Mayrhauser, A., and Vans, A. M. 1993. From program comprehension to tool requirements for an industrial environment. Proc. of the 2nd Workshop on Program Comprehension, Capri, Italy, July, pp. 78–86.

  • von Mayrhauser, A., and Vans, A. M. 1995. Program understanding: Models and experiments. In: M. C. Yovita and M. V. Zelkowitz, (eds), Advances in Computers, Vol. 40, Academic Press, pp. 1–38.

  • Walenstein, A. 2003. Observing and measuring cognitive support: Steps toward systematic tool evaluation and engineering. Proc. the 11th IEEE Workshop on Program Comprehension.

  • Walz, D. B., Elam, J. J., and Curtis, B. 1993. Inside a software design team: Knowledge acquisition, sharing, and integration. Communications of the ACM 36(10): October, 62–77.

    Article  Google Scholar 

  • Williams, L., Kessler, R. R., Cunningham, W., and Jeffries, R. 2000. Strengthening the case for pair-programming, IEEE Software July/Aug, 19–25.

  • Wolf, A., and Rosenblum, D. 1993. A study in software process data capture and analysis. Proc. 2nd International Conference on Software Process February, pp. 115–124.

  • Wu, J., Graham, T., Smith, P. 2003. A study of collaboration in software design. Proc. Int. Symp. Empirical Software Eng. ‘03.

Download references

Author information

Authors and Affiliations

  1. School of Information Technology and Engineering, University of Ottawa, Ottawa, Ontario, K1N 6N5, Canada

    Timothy C. Lethbridge

  2. Department of Informatics, University of California, Irvine, 444 Computer Science, Irvine, CA, 92697-3425, USA

    Susan Elliott Sim

  3. National Research Council Canada, Institute for Information Technology, Montreal Rd, Building M-50, Ottawa, Ontario, K1A 0R6, Canada

    Janice Singer

Authors
  1. Timothy C. Lethbridge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Susan Elliott Sim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Janice Singer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Timothy C. Lethbridge.

Additional information

The authors appear in alphabetical order and contributed equally.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lethbridge, T.C., Sim, S.E. & Singer, J. Studying Software Engineers: Data Collection Techniques for Software Field Studies. Empir Software Eng 10, 311–341 (2005). https://doi.org/10.1007/s10664-005-1290-x

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-005-1290-x

Keywords