Empirical Analysis of Widely Used Website Automated Testing Tools

Balqees Sani; Sadaqat Jan

doi:10.4108/airo.7285

Authors

Balqees Sani University of Engineering and Technology, Mardan
Sadaqat Jan University of Engineering and Technology, Mardan

DOI:

https://doi.org/10.4108/airo.7285

Keywords:

Automated testing tools, Selenium, Performance testing, Cloud-based testing

Abstract

In today's software development, achieving product quality while minimising cost and time is critical. Automated testing is crucial to attaining these goals by lowering inspection efforts and discovering faults more effectively. This paper compares widely used automated testing tools, such as Selenium, Appium, Java Unit (JUnit), Test Next Generation (TestNG), Jenkins, Cucumber, LoadRunner, Katalon Studio, Simple Object Access Protocol User Interface (SoapUI), and TestComplete, based on functionality, ease of use, platform compatibility, and integration capabilities. Our findings show that no single tool is inherently superior, with each excelling in certain areas such as online, mobile, Application Programming Interface (API), or performance testing. While Selenium and Appium are the dominant online and mobile testing frameworks, TestComplete and Katalon Studio offer complete, user-friendly cross-platform testing solutions. Despite the benefits of automation, obstacles such as tool maintenance, scalability, and cost issues remain. The report finishes with advice for picking the best tool for the project and offers potential approaches for enhancing testing frameworks, such as AI-driven optimisation, cloud-based testing, and greater Continuous Integration/ Continuous Deployment (CI/CD) integration. This study offers useful information for developers and testers looking to optimise their testing methods and increase software quality.

Downloads

Download data is not yet available.

References

[1] Homès, B. (2024). Fundamentals of software testing. John Wiley & Sons.

[2] Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024). Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering.

[3] Alshazly, A. A., Elfatatry, A. M., & Abougabal, M. S. (2014). Detecting defects in software requirements specification. Alexandria Engineering Journal, 53(3), 513-527.

4] Xu, G., Khan, A. S., Moshayedi, A. J., Zhang, X., & Shuxin, Y. (2022). The object detection, perspective and obstacles in robotic: a review. EAI Endorsed Transactions on AI and Robotics, 1(1).

[5] Latifi Rostami, S. A., Ghoddosian, A., Kolahdooz, A., & Zhang, J. (2022). Topology optimization of continuum structures under geometric uncertainty using a new extended finite element method. Engineering Optimization, 54(10), 1692-1708.

[6] Yu, X., Liu, L., Hu, X., Keung, J., Xia, X., & Lo, D. (2024, September). Practitioners’ Expectations on Automated Test Generation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 1618-1630).

[7] Kolahdooz, A., Nourouzi, S., Bakhshi Jooybari, M., & Hosseinipour, S. J. (2014). Experimental investigation of thixoforging parameters effects on the microstructure and mechanical properties of the helical gearbox cap. Journal of Mechanical Science and Technology, 28, 4257-4265.

[8] Moshayedi, A. J., Khan, A. S., Yang, S., & Zanjani, S. M. (2022, April). Personal image classifier based handy pipe defect recognizer (hpd): Design and test. In 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP) (pp. 1721-1728). IEEE.

[9] Hanna, M., Aboutabl, A. E., & Mostafa, M. S. M. (2018). Automated software testing framework for web applications. International Journal of Applied Engineering Re-search, 13(11), 9758-9767.

[10] Zarei, M., Moshayedi, A. J., Zhong, Y., Khan, A. S., Kolahdooz, A., & Andani, M. E. (2023, January). Indoor UAV object detection algorithms on three processors: implementation test and comparison. In 2023 3rd International Conference on Consumer Electronics and Computer Engineering (ICCECE) (pp. 812-819). IEEE.

[11] Camacho, N. G. (2024). Unlocking the Potential of AI/ML in DevSecOps: Effective Strategies and Optimal Practices. Journal of Artificial Intelligence General Science (JAIGS) ISSN: 3006-4023, 3(1), 106-115.

[12] Moshayedi, A. J., Uddin, N. M. I., Khan, A. S., Zhu, J., & Emadi Andani, M. (2023). Designing and Developing a Vision-Based System to Investigate the Emotional Effects of News on Short Sleep at Noon: An Experimental Case Study. Sensors, 23(20), 8422.

[13] Amit, Y., Felzenszwalb, P., & Girshick, R. (2021). Object detection. In Computer Vision: A Reference Guide (pp.875-883). Cham: Springer International Publishing.

[14] R. N. Khan and S. Gupta, “Comparative Study of Automated Testing Tools: Rational Functional Tester, Quick Test Professional, Silk Test and Loadrunner,” Int. J. Adv. Technol. Eng. Sci., vol. 3, no. 01, pp. 167–172, 2015

[15] R. K. Lenka, S. Mamgain, S. Kumar, and R. K. Barik, “Performance analysis of automated testing tools: JMeter and TestComplete,” in 2018 International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), IEEE, 2018, pp. 399–407.

[16] M. Karthik and V. K. Jananisivapriya, “Comparison of Software Test Automation Tools-Selenium and UFT,” Am. Int. J. Res. Formal, Appl. Nat. Sci., p. 5, 2019.

[17] H. Wu et al., “Peculiar: Smart contract vulnerability detection based on crucial data flow graph and pretraining techniques,” in 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), IEEE, 2021, pp. 378–389.

[18] 18. Samlı, R., & Orman, Z. (2023). A comprehensive overview of web-based automated testing tools. İleri Mühendislik Çalışmaları ve Teknolojileri Dergisi, 4(1), 13-28.

[19] Chaves, L. C., Oliveira, F. C. M., Tiago, L. A., & Castro, R. G. V. (2024, May). Robert: An Automated Tool to Perform Mobile Application Test. In Proceedings of the 2024 10th International Conference on Computer Technology Applications (pp. 33-36).

[20] Moseh, M. A., Al-Khulaidi, N. A., Gumaei, A. H., Alsabry, A., & Musleh, A. A. (2024, August). Classification and Evaluation Framework of Automated testing tools for agile software: Technical Review. In 2024 4th International Conference on Emerging Smart Technologies and Applications (eSmarTA) (pp. 1-12). IEEE.

[21] Haas, R., Nömmer, R., Juergens, E., & Apel, S. (2024). Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis. IEEE Transactions on Software Engineering.

[22] Aburas, A. (2024, May). Choosing the Right Automated Software Testing Tools. In 2024 IEEE 4th International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA) (pp. 31-35). IEEE.

[23] Garousi, V., Joy, N., & Keleş, A. B. (2024). AI-powered test automation tools: A systematic review and empirical evaluation. arXiv preprint arXiv:2409.00411.

[24] Jha, N., Popli, R., Chakraborty, S., & Kumar, P. (2022). Software Test Automation Using Selenium and Machine Learning. In Proceedings of First International Conference on Computational Electronics for Wireless Communications: ICCWC 2021 (pp. 419-429). Springer Singapore.

[25] Abdullin, A., & Akhin, M. (2024, April). Kex at the SBFT 2024 Tool Competition. In Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing (pp. 65-66).

[26] Prasad, L., Yadav, R., & Vore, N. (2021). A systematic literature review of automated software testing tool. In Proceedings of 3rd International Conference on Computing Informatics and Networks: ICCIN 2020 (pp. 101-123). Springer Singapore.

[27] Moshayedi, A. J., Roy, A. S., Ghorbani, H., Lotfi, H., Zhang, X., & Liao, L. (2024, May 9). A novel IoT-enabled portable, secure automatic self-lecture attendance system: Design, development and comparison. International Journal of Electronic Securi-ty and Digital Forensics.

[28] Ma, X. (2024). Development and Automation of a Web Applications Using FastAPI, Jenkins, and Robot Framework.

[29] Mughal, A. H. (2024). Advancing BDD Software Testing: Dynamic Scenario Re-Usability And Step AutoComplete For Cucumber Framework. arXiv preprint arXiv:2402.15928.

[30] Moshayedi, A. J., Roy, A. S., Liao, L., Lan, H., Gheisari, M., Abbasi, A., & Bamakan, S. M. (2021). Automation attendance systems approaches: a practical review. BOHR Int. J. Internet Things Artif. Intell. Mach. Learn, 1, 23-31.

[31] Safaat, G., & Tjhin, V. U. (2024). Analysis of Quality Assurance Performance in the Application of Manual Testing and Automation Testing for Software Product Testing. Indonesian Interdisciplinary Journal of Sharia Economics (IIJSE), 7(2), 1987-1996.

[32] 32. Raj, V., & Varri, U. S. (2024). 13 An Automated Approach. Cloud of Things: Foun-dations, Applications, and Challenges, 223.

[33] Prasad, L., Yadav, R., & Vore, N. (2021). A systematic literature review of automated software testing tool. In Proceedings of 3rd International Conference on Computing Informatics and Networks: ICCIN 2020 (pp. 101-123). Springer Singapore.

[34] Dias, T., Batista, A., Maia, E., & Praça, I. (2023, July). TestLab: An Intelligent Automated Software Testing Framework. In International Symposium on Distributed Computing and Artificial Intelligence (pp. 355-364). Cham: Springer Nature Switzerland.

[35] Moshayedi, A. J., Soleimani, M., Marani, M., Yang, S., Razi, A., & Andani, M. E. (2023, June). Fingerprint Identification Banking (FIB); Affordable and Secure Biometric IOT Design. In 2023 4th International Seminar on Artificial Intelligence, Net-working and Information Technology (AINIT) (pp. 384-390). IEEE.

[36] Melyawati, N. L. P., Asana, I. M. D. P., Putri, N. W. S., Atmaja, K. J., & Sudipa, I. G. I. (2024). Comparison of Automation Testing On Card Printer Project Using Play-wright And Selenium Tools. Journal of Computer Networks, Architecture and High Performance Computing, 6(3), 1309-1320.