Detecting Programming Flaws in Student Submissions with Static Source Code Analysis
Abstract
Static code analyzer tools can detect several programming mistakes, that would lead to run-time errors. Such tools can also detect violations of the conventions and guidelines of the given programming language. Thus, the feedback provided by these tools can be valuable for both students and instructors in computer science education. In our paper, we evaluated over 5000 student submissions from the last two years written in C++ and C# programming languages at E¨otv¨os Lor´and University Faculty of Informatics (Budapest, Hungary), by executing various static code analyzers on them. From the findings of the analyzers, we highlight some of the most typical and serious issues. Based on these results, we argue to include static analysis of programming submissions in automated and assisted semi-automatic evaluating and grading systems at universities, as these could increase the quality of programming assignments and raise the attention of students on various otherwise missed bugs and other programming errors.
References
[2] Bardas, A. G., et al. Static code analysis. Journal of Information Systems & Operations Management 4, 2 (2010), 99–107.
[3] Birillo, A., Vlasov, I., Burylov, A., Selishchev, V., Goncharov, A., Tikhomirova, E., Vyahhi, N., and Bryksin, T. Hyperstyle: A tool for assessing the code quality of solutions to programming assignments. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1 (New York, NY, USA, 2022), SIGCSE 2022, Association for Computing Machinery, pp. 307––313.
[4] Blau, H., and Moss, J. E. B. Frenchpress gives students automated feedback on java program flaws. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (New York, NY, USA, 2015), ITiCSE ’15, Association for Computing Machinery, pp. 15-20.
[5] Clang Team. LLVM - Clang-tidy - cppcoreguidelines-slicing. https://releases.llvm.org/13.0.0/tools/clang/tools/extra/docs/clang-tidy/checks/cppcoreguidelines-slicing.html, Accessed: 2023-02-25.
[6] Edwards, S. H., Kandru, N., and Rajagopal, M. B. Investigating static analysis errors in student java programs. In Proceedings of the 2017 ACM Conference on International Computing Education Research (New York, NY, USA, 2017), ICER ’17, Association for Computing Machinery, pp. 65—-73.
[7] ELTE. TMS – Task Management System. https://tms-elte.gitlab.io/, Accessed: 2023-02-27.
[8] Ericsson Ltd. CodeChecker. https://codechecker.readthedocs.io/, Accessed: 2023-02-25.
[9] Eurostat. ICT education - a statistical overview. https://ec.europa.eu/eurostat/statistics-explained/index.php?title=ICT education - a statistical overview, Accessed: 2023-04-16.
[10] Gomes, I., Morgado, P., Gomes, T., and Moreira, R. An overview on the static code analysis approach in software development. Faculdade de Engenharia da Universidade do Porto, Portugal (2009).
[11] Ihantola, P., Ahoniemi, T., Karavirta, V., and Sepp¨al¨a, O. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (New York, NY, USA, 2010), Koli Calling ’10, Association for Computing Machinery.
[12] Keuning, H., Heeren, B., and Jeuring, J. Code quality issues in student programs. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (New York, NY, USA, 2017), ITiCSE ’17, Association for Computing Machinery, pp. 110—-115.
[13] Kremenek, T. Finding software bugs with the clang static analyzer. Apple Inc (2008).
[14] Loyalka, P., Liu, O. L., Li, G., Chirikov, I., Kardanova, E., Gu, L., Ling, G., Yu, N., Guo, F., Ma, L., Hu, S., Johnson, A. S., Bhuradia, A., Khanna, S., Froumin, I., Shi, J., Choudhury, P. K., Beteille, T., Marmolejo, F., and Tognatta, N. Computer science skills across china, india, russia, and the united states. Proceedings of the National Academy of Sciences 116, 14 (2019), 6732–6736.
[15] Marjam¨aki, D. Cppcheck. https://cppcheck.sourceforge.io/, Accessed: 2023-02-23.
[16] Martignano, M., and Spazio, I. A new static analyzer: The compiler. ADA USER 40, 2 (2019), 99–103.
[17] Microsoft. Async return types (C#). https://learn.microsoft.com/en-us/dotnet/csharp/asynchronous-programming/async-return-types, Accessed: 2023-02-23.
[18] Molnar, A.-J., Motogna, S., and Vlad, C. Using static analysis tools to assist student project evaluation. In Proceedings of the 2nd ACM SIGSOFT International Workshop on Education through Advanced Software Engineering and Artificial Intelligence (New York, NY, USA, 2020), EASEAI 2020, Association for Computing Machinery, pp. 7––12.
[19] Orr, J. W. Automatic assessment of the design quality of student python and java programs. arXiv e-prints (2022).
[20] Porkolab, Z., Brunner, T., Krupp, D., and Csordas, M. Codecompass: an open software comprehension framework for industrial usage. In Proceedings of the 26th Conference on Program Comprehension (2018), pp. 361–369.
[21] Quinlan, D., Vuduc, R., Panas, T., Haerdtlein, J., and Saebjoernsen, A. Support for whole-program analysis and the verification of the one-definition rule in C++. In Static Analysis Summit 2006 (6 2006).
[22] Striewe, M., and Goedicke, M. A review of static analysis approaches for programming exercises. In Computer Assisted Assessment. Research into E-Assessment (07 2014), vol. 439, Springer, pp. 100–113.
[23] Sundstrom, J. Assessment of Roslyn analyzers for Visual Studio, 2019. [24] Vasani, M. Roslyn Cookbook. Packt Publishing Ltd., 2017.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
When the article is accepted for publication, I, as the author and representative of the coauthors, hereby agree to transfer to Studia Universitatis Babes-Bolyai, Series Informatica, all rights, including those pertaining to electronic forms and transmissions, under existing copyright laws, except for the following, which the author specifically retain: the right to make further copies of all or part of the published article for my use in classroom teaching; the right to reuse all or part of this material in a review or in a textbook of which I am the author; the right to make copies of the published work for internal distribution within the institution that employs me.
