Integration of Remote Usability Tests in eXtreme Programming: A Literature Review

Keywords: eXtreme Programming, Remote Usability Test, Integration, RUT, XP


This study is intended to answer the question of how to integrate remote usability tests into eXtreme Programming. The study is based on a literature review involving search, selection and results classification. Keywords, emerging categories, and search strings in relevant databases were used. An analysis of the current state of the components and processes that have been used in this type of integration has led to a combined approach in areas such as: 1. Requirements management, 2. Management of team members including developers, usability evaluators, experts and final users, 3. Management of tests, data collected and feedback, 4. Design, and 5. Activities carried out within an iteration. It is concluded that more research, experimentation and documentation are necessary to be able to recommend integration processes that satisfy demands and take advantage of eXtreme Programming and remote usability tests. Finally, it is necessary to confront the challenges posed by the difference in the holistic visions of both approaches, as well as communication problems between usability developers and evaluators, adaptation of eXtreme components, and the limited priority assigned to usability efforts.


Andreasen, M. S., Nielsen, H. V., Schrøder, S. O., & Stage, J. (2006). Usability in open source software development: opinions and practice. Information technology and control, 35(3), 303-312.
Bailey, J., Budgen, D., Turner, M., Kitchenham, B., Brereton, P., & Linkman, S. (2007). Evidence relating to Object-Oriented software design: A survey. In Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on (pp. 482-484). IEEE. doi
Beck, K. (1999). Embracing change with extreme programming. Computer, 32(10), 70-77. doi
Beck, K. (2000). Extreme programming explained: embrace change. Addison-Wesley professional.
Budwig, M., Jeong, S., & Kelkar, K. (2009). When user experience met agile: a case study. In CHI'09 Extended Abstracts on Human Factors in Computing Systems (pp. 3075-3084). ACM. doi
Bustamante, J. (2010). La herramienta de tests de usabilidad a distancia Loop11. El profesional de la información, 19(4). doi
Carnaghan, I. (2014). An expeditionary learning approach to effective curriculum mapping formalizing the process by exploring a user-centered framework (Doctoral Dissertation). Retrieved from University of Baltimore, ProQuest, UMI Dissertations Publishing. (3626119)
Chalil Madathil, K. & Greenstein, J. S. (2011). Synchronous remote usability testing: a new approach facilitated by virtual worlds. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2225-2234). ACM. doi
De Lima Salgado, A., Amaral, L. A., Freire, A. P., & Fortes, R. P. M. (2016). Usability and UX practices in small enterprises: lessons from a survey of the Brazilian context. In Proceedings of the 34th ACM International Conference on the Design of Communication (p. 18). ACM. doi
Draxler, S., & Stevens, G. (2006). Getting Out of a Tailorability Dilemma. In GI Jahrestagung (1) (pp. 576-579).
Highsmith, J. A. (2002). Agile software development ecosystems. Addison-Wesley Professional, Vol. 13.
Hosseini-Khayat, A., Hellmann, T. D., & Maurer, F. (2010). Distributed and Automated Usability Testing of Low-Fidelity Prototypes. In AGILE Conference, 2010 (pp. 59-66). IEEE. doi
Kawakita, J. (1991). The original KJ method. Tokyo: Kawakita Research Institute
Khayat, A. H. (2009). Distributed Wizard of Oz Usability Testing for Agile Teams. (Doctoral dissertation). UNIVERSITY OF CALGARY.
Kurosu, M. (Ed.). (2013). Human-Computer Interaction: Human-Centred Design Approaches, Methods, Tools and Environments. 15th International Conference, HCI International 2013, Las Vegas, NV, USA, July 21-26, 2013, Proceedings (Vol. 8004). Springer. doi
Lee, J. C. (2006). Embracing agile development of usable software systems. In CHI'06 Extended Abstracts on Human Factors in Computing Systems (pp. 1767-1770). ACM. doi
Lizano, F. (2014). Integrating Usability Evaluations into the Software Development Process: concepts for, and Experiences from, Remote Usability Testing (Doctoral dissertation). Institut for Datalogi, Aalborg Universitet.
Lizano, F., Sandoval, M. M., & Stage, J. (2014). Integrating usability evaluations into scrum: a case study based on remote synchronous user testing. In International Conference on Human-Computer Interaction (pp. 500-509). Springer, Cham. doi
Salah, D., Paige, R. F., & Cairns, P. (2014). A systematic literature review for agile development processes and user centred design integration. In Proceedings of the 18th international conference on evaluation and assessment in software engineering (p. 5). ACM. doi
Sohaib, O., & Khan, K. (2010). Integrating usability engineering and agile software development: a literature review. In Computer design and applications (ICCDA), 2010 international conference on (Vol. 2, pp. V2-32). IEEE. doi
Tullis, T., Fleischman, S., McNulty, M., Cianchette, C., & Bergel, M. (2002). An empirical comparison of lab and remote usability testing of web sites. In Usability Professionals Association Conference.
Wolkerstorfer, P., Tscheligi, M., Sefelin, R., Milchrahm, H., Hussain, Z., Lechner, M., & Shahzad, S. (2008). Probing an agile usability process. In CHI'08 Extended Abstracts on Human Factors in Computing Systems (pp. 2151-2158). ACM. doi
Yetim, F., Draxler, S., Stevens, G., & Wulf, V. (2012). Fostering continuous user participation by embedding a communication support tool in user interfaces. AIS Transactions on Human-Computer Interaction, 4(2), 153-168. doi
How to Cite
Sánchez-Hernández, D., Lizano-Madriz, F., & Sandoval-Carvajal, M. (2020). Integration of Remote Usability Tests in eXtreme Programming: A Literature Review. Uniciencia, 34(1), 20-31.
Original scientific papers (evaluated by academic peers)