Transforming New Zealand's ICT workforce using digital personalized interactive training

The growth of NZ’s software industry is essential for NZ’s vision of a productive economy. With 12,000 firms, 35,000 employees, a $5.3Bn contribution to GDP and a strong export focus, the software sector is a high-value sector.

Underpinning this economic value are highly skilled professionals. To build high-quality software, 'transferable' skills, including team and intercultural skills, communication, negotiation and empathy are significant. However, many ICT graduates and professionals lack these skills, and teaching them is expensive and time-consuming.

This research aims to develop an online, interactive and personalised learning approach for transferable skills based on 'active' video watching. Novel scientific contributions are:

1. AI-based model to provide personalized support and learning during video-watching
2. An interaction model for engagement with video-based learning material that integrates interactive activities to ensure consistent, active engagement based on a learner’s profile
3. A computer-based training platform geared towards transferable skills relevant to different types of ICT roles based on scientific rigor and practical relevance.

We will identify psychological and cognitive factors that facilitate video-based learning of transferable skills and define skills suitable for video-based learning. The research will enhance conventional passive video watching with novel mechanisms to increase learning experience, and push beyond a conventional classroom setting into professional development in industry.

The research will help small and medium-sized ICT companies in NZ train employees in an effective and time-efficient manner, resulting in quantified gains in productivity and economic performance. In particular, it will extend reach and accessibility to more effective training and lower barriers to adoption.

Our research will reinforce NZ’s position as an international leader in AI in education, and software engineering research, science and technology.

Overview

Soft skills, such as communication, teamwork and problem-solving skills, are crucial for software engineers to develop and maintain quality software products and services. Software development is a human-centred activity, and having good soft skills enables professionals to interact with others more effectively, influencing developer and team productivity and job satisfaction. Soft skills are those skills which enable positive social interactions and include non-technical and domain-independent skills that influence our behaviour in the workplace. 

The goals of our project were: To determine which soft skills are critical in New Zealand's software industry; To develop a soft skills assessment framework for software professionals; To develop a novel, interactive, and personalised training solution for soft skills, based on active video watching.

Which soft skills are needed in New Zealand’s software industry? Through analysis of recruitment advertisements for software professionals, reviewing existing literature, and interviewing software professionals, we show that soft skills are keenly sought after by employers, regardless of company size or business domain, and that professionals consider them to be critical in their roles. Soft skills were even highlighted as more important than technical skills and that their development requires attention from both employees and employers. The most important soft skills identified for the software industry were communication skills (both verbal and written), teamwork, leadership and interpersonal skills.

Soft skills assessment framework for software professionals: Soft skills can be challenging to assess. This is the perspective of software professionals but also reinforced by the limited availability of tools to assess them in a software engineering context. We therefore developed a Soft Skills Assessment Framework (2SAF) to guide professionals in the assessment of these skills. The 2SAF is a step-by-step guide for software development professionals and companies when considering soft skills assessment and development. 

We also developed instruments to measure Team Empathy and Face-to-Face Meeting Communication skills for the context of software engineering. These scales can be used by software developers to assess their own soft skills and to support them in their professional development. We conducted various studies using these scales, supporting their reliability and validity. We also showed that these skills can be improved with training, using the developed training solution, AVW-Space.  

A novel, interactive, and personalised training solution for soft skills based on active video watching: We developed AVW-Space, an online portal for video-based learning which supports engagement during video watching and provides personalised hints to focus learners' attention on important aspects in videos. Video-based learning is appropriate for teaching soft skills, as it is necessary for the learner to see the skill from various viewpoints as well as to reflect on their own experience.

In collaboration with Professor Vania Dimitrova from the University of Leeds (UK), we have developed the AVW-Space platform, a controlled video-watching platform that supports active learning.

Our goal was to develop a platform for teaching soft skills, although AVW-Space can be used for teaching other types of skills, including supporting flipped classrooms and other forms of video-based learning.

AVW-Space allows the teacher/facilitator to select videos from YouTube, and select options provided by the platform for supporting users. Different types of options are available, such as specifying aspects (i.e. the theme for a comment) which focus users' attention on important parts of videos.

Interaction with AVW-Space happens in two phases:

In Phase 1, users watch and comment on videos individually. The screenshot below shows the progress bar and the list of videos used in studies on presentation skills.

The initial studies we performed with AVW-Space demonstrate that learners who watch videos actively, that is those who write comments and also rate comments written by their peers, improve their knowledge of soft skills. In order to increase learning, we developed AI-based support for learners.

Based on the learner's behaviour, AVW-Space provides personalized nudges. For example, if a user is passively watching videos and not writing comments, they will get a nudge motivating them to write about the video:

AVW-Space tracks all activities that a learner performs and maintains the learner model, which is a representation of the learner's knowledge inferred and maintained by the platform. The learner model is used to provide personalised hints to the learner, in cases when the user behaviour is suboptimal. These hints encourage the user to write about particular aspects of the video, to elaborate on the video content and also to reflect on their past experience with the skill.

As soon as the user writes a comment on a video, AVW-Space analyses it and shows the comment quality to the user.

The user can then ask for the explanation of how comment quality is determined. AVW-Space contains AI-based components which allow comments to be analysed and provide explanations on comment quality.

The facilitator can later open comments to the whole group of users, who can see each others' (anonymised) comments, rate and reply to them. In this way, AVW-Space supports social learning. 

The learner model is also used to provide comments for rating tailored towards each individual learner. AVW-Space provides comments for ratings to users adaptively, based on their learner model. This is done by identifying knowledge concepts the user has not covered yet, or has not written about enough, and select good quality comments to present to the user. In this way, the learner has an opportunity to learn more about the relevant soft skill.

We have also developed a gamified version of AVW-Space. This version provides badges to learners based on the activities they perform during learning. Badges are given for watching videos, making high-quality comments on videos, and rating comments written by peers. The evaluation studies we performed with this gamified version show a huge effect on motivation for completing activities, which also increase learning.

During this project, we developed three spaces in AVW-Space, for selected soft skills: presentation skills, face-to-face communication in software development meetings, and team empathy. For each of these skills, we developed a taxonomy of the relevant knowledge components. The taxonomy is used as the basis for the learner model. As the development of AI-based components require data, we conducted several studies to collect data about how users interact with the platform. After generating AI-components, we conducted studies to evaluate their effect on learner engagement and learning.

The evaluation studies were performed with students enrolled in courses at the University of Canterbury and also at the Ateneo de Davao University in the Philippines. All studies demonstrate significant increases in the levels of student engagement and learning. We have also conducted three studies with software professionals, recruited from New Zealand and worldwide. The studies with professionals showed significant increase in user engagement and learning, due to the support provided by AVW-Space.

Please email tanja.mitrovic@canterbury.ac.nz if you are interested in trying AVW-Space and collaborating with us.

Industry report: Soft skills assessment framework (2SAF)

This report introduces a Soft Skills Assessment Framework (2SAF) for practitioners to guide software engineering and human resource management professionals in the assessment of soft skills. We discuss the importance of soft skills for software engineering, challenges and opportunities for soft skills assessment, and a framework that can be used to guide soft skills assessment. We also provide some examples of recently developed assessment instruments.

Further Documents

• Soft Skills infographic
• Poster - Development of Empathy Skills in Software Professionals Using Video-Based Training
• Infographic - Soft Skills Assessment Framework (2SAF) Industry Report
  

For More Information

Please see the publication list below.

Please contact Professor Tanja Mitrovic (tanja.mitrovic@canterbury.ac.nz) if you would like to try AVW-Space.

Team

Acknowledgements

Prior work on AVW-Space was supported by two regional grants from the Southern hub of Ako Aotearoa, an ImREAL grant (EU-FP7-ICT-257184), as well as funding from the University of Canterbury.

List of Publications

1. Peiris, P., Galster, M., Mitrovic, A., Malinen, S., Bojnordi, E., Holland, J. Impact of Gamification on Engagement and Learning in Video-Based Platforms. Accepted for ICCE 2025 (full paper). 
2. Baldwin, Chantelle, Mitrovic, A. Assessment of comment quality in active video watching using deep learning. Accepted for ICCE 2025 (short paper)
3. Lumapas, R.V., Mitrovic., A., Galster, M., Malinen, Sa., Holland, J. Investigating the Effectiveness of Explanations in Active Video Watching. Accepted for ICCE 2025 (full paper).
4. Malinen, S.K., Sankara, S., Galster, M. & Mitrovic, T. (2025). Development of empathy skills in software professionals using video-based training. European Association of Work and Organizational Psychology (EAWOP) congress, May 21-24, 2025, Prague, Czech Republic.
5. Bojnordi, E., Mitrovic, A., Galster, M., Malinen, S., Holland, J. (2025). Fostering Interactive Engagement in Active Video Watching via Adaptive Comment Recommendations. In: Cristea, A.I., Walker, E., Lu, Y., Santos, O.C., Isotani, S. (eds) Proceedings of the 26^th International Conference on Artificial Intelligence in Education AIED 2025. Lecture Notes in Computer Science, vol 15882, pp. 83-90. Springer, Cham. https://doi.org/10.1007/978-3-031-98465-5_11 
6. Malinen, S., Galster, M., Mitrovic, A., Iyer, S.S., Peiris, P., Clarke, A. (2025) Soft Skills in Software Engineering: Insights from the Trenches. IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice ICSE-SEIP 2025, 296-306. https://doi.org/10.1109/ICSE-SEIP66354.2025.00032
   
7. Mitrovic, A., Galster, M., Malinen, S., Iyer, S.S., Lumapas, R.L., Mohammadhassan, N.,  Holland, J. (2025). Video-based Empathy Training for Software Engineers. IEEE/ACM 37th International Conference on Software Engineering Education and Training CSEE&T 2025, 264–269. https://doi.org/10.1109/CSEET66350.2025.00033 
8. Bojnordi, E., Mitrovic, A., Galster, M., Malinen, S., Holland, J. (2024) Personalized comment reviewing in active video watching: investigation of learners’ cognitive load. Akihiro KASHIHARA, Bo JIANG, Maria Mercedes RODRIGO, Jessica O. SUGAY (Eds) Proceedings 32^nd International Conference on Computers in Education ICCE 2024, Vol. 2, pp. 780-782, Manila, Philippines. https://doi.org/10.58459/icce.2024.5067
9. Lumapas, R.V., Mitrovic, A., Galster, M., Malinen, S., Holland, J., Peiris, P. (2024). Integrating Explanations in Active Video Watching. Akihiro KASHIHARA, Bo JIANG, Maria Mercedes RODRIGO, Jessica O. SUGAY (Eds) Proceedings 32^nd International Conference on Computers in Education ICCE 2024, Vol. 2, pp. 755-758, Manila, Philippines. https://doi.org/10.58459/icce.2024.5055
10. Bojnordi, E., Mitrovic, A., Galster, M., Malinen, S., Holland, J., Mohammadhassan, N. (2024) Enhancing Social Learning in Active Video Watching.  Akihiro KASHIHARA, Bo JIANG, Maria Mercedes RODRIGO, Jessica O. SUGAY (Eds) Proceedings International Conference on Computers in Education ICCE 2024, Vol. 1, pp. 171-176, Manila, Philippines. https://doi.org/10.58459/icce.2024.4834
11. Lumapas, R.V.,  Mitrovic, A., Galster, M., Malinen, S. (2024). Exploring Explainable Artificial Intelligence in Active Video Watching. Akihiro KASHIHARA, Bo JIANG, Maria Mercedes RODRIGO, Jessica O. SUGAY (Eds) Proc. 32^nd International Conference on Computers in Education ICCE 2024, Vol. 1, pp. 116-118, Manila, Philippines. https://doi.org/10.58459/icce.2024.4824
12. Galster, M., Mitrovic, A., Malinen, S., Iyer, S. S., Musa, J. A., Holland, J. (2024) Video-based Training for meeting communication skills. In Proceedings of the 46^th International Conference on Software Engineering: Software Engineering Education and Training (pp. 170-179). April 14-20, Lisbon, Portugal. ACM Distinguished paper award. https://doi.org/10.1145/3639474.3640080
13. Lumapas, R.V., Mitrovic, A., Galster, M., Malinen, S., Holland, J., Mohammadhassan, N. (2023). Evaluating the Assessment of Comment Quality in Learning Communication Skills using Active Video Watching, Proc. 31^st International Conference on Computers in Education ICCE 2023, Vol II, pp. 1045-1047. https://doi.org/10.58459/icce.2023.4785
14. Bojnordi, E., Mitrovic, A., Galster, M., Malinen, S., Holland, J. (2023). Adding Interactive Mode to Active Video Watching. Proc. 31^st International Conference on Computers in Education ICCE 2023, Vol II, pp. 1042-1044. https://doi.org/10.58459/icce.2023.1499
15. Peiris, P., Galster, M., Mitrovic, A., Malinen, S., Lumapas, R. (2023). Learner Perceptions on Gamifying Active Video Watching Platforms, Proc. 31^st International Conference on Computers in Education ICCE 2023, Vol II, pp. 1038-1041. https://doi.org/10.58459/icce.2023.1498
16. Lumapas, R.V., Mitrovic, A., Galster, M., Malinen, S., Peiris, P., Holland, J. (2023). Question-Driven Design Process for XAI in Active Video Watching, Proc. 31^st International Conference on Computers in Education ICCE 2023, Vol II, pp. 878-880. Matsue, Japan. https://doi.org/10.58459/icce.2023.4771
17. Mitrovic, A., Galster, M., Malinen, S., Holland, J., Lumapas, R. V., Mohammadhassan, N., Musa, J. (2023) Effectiveness of video-based training for communication skills: evidence from a three-year study. ACM Transactions on Computing Education, 23(4), 1-25. https://doi.org/10.1145/3631532.
18. Galster, M., Mitrovic, A., Malinen, S., Holland, J. (2022). What soft skills does the software industry *really* want? An exploratory study of software positions in New Zealand. In Proceedings of the 16^th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (pp. 272-282). September 19-23, 2022, Helsinki, Finland. https://doi.org/10.1145/3544902.3546247
19. Mohammadhassan, N., Mitrovic, A., (2021) Providing personalised nudges for improving comments quality in active video watching. 11th International Conference on Learning Analytics & Knowledge LAK 2021, Companion proceedings, demo.
20. Mohammed, A., Dimitrova, V. (2020) Characterising video segments to support learning. In: So, H.J. et al. (Eds.) Proceedings of the 28th International Conference on Computers in Education, pp. 11-20. Asia-Pacific Society for Computers in Education. https://library.apsce.net/index.php/ICCE/article/view/3892
    
21. Mohammadhassan, N., Mitrovic, A., Neshatian, K., Dunn, J. (2020) Automatic quality assessment of comments in active video watching using machine learning techniques. In: So, H.J. et al. (Eds.) Proceedings of the 28th International Conference on Computers in Education, pp. 1-10. Asia-Pacific Society for Computers in Education. ISBN978-986-97214-5-5. Nominated for the Best overall paper and Best student paper awards. https://hdl.handle.net/10092/101306
    
22. Mitrovic, A., Gordon, M., Piotrkowicz, A., Dimitrova, V. (2019) Investigating the effect of adding nudges to increase engagement in active video watching. In: S. Isotani et al. (Eds.) Proc. 20th Conf. AIED 2019, LNAI 11625, pp. 320-332, Springer Nature Switzerland.
    https://doi.org/10.1007/978-3-030-23204-7_27 
    https://ir.canterbury.ac.nz/handle/10092/101306
23. Taskin, Y., Hecking, T., Hoppe, H.U., Dimitrova, V., Mitrovic, A. (2019) Characterizing comment types and levels of engagement in video-based learning as a basis for adaptive nudging. ECTEL 2019, Delft, 16-19 September 2019, LCNS 11722, pp. 362-376. https://doi.org/10.1007/978-3-030-29736-7_27
    
24. Mitrovic, A., Gordon, M., Piotrkowicz, A., Dimitrova, V. (2019) Investigating the effect of adding nudges to increase engagement in active video watching. Proc. 20th Int. Conf. Artificial Intelligence in Education AIED 2019, LNAI 11625, pp. 320-332, Springer Nature Switzerland. https://link.springer.com/chapter/10.1007%2F978-3-030-23204-7_27
25. Abolkasim, E., Lau, L., Mitrovic, A., Dimitrova, V. (2018) Ontology-based domain diversity profiling of user comments. In: C. Penstein Rose et al. (Eds.) Proceedings of the 19th Conf. Artificial Intelligence in Education, Part II, Springer, LNAI 10948, pp 3-8, London, 27-30.6.2018. http://dx.doi.org/10.1007/978-3-319-93846-2_1
    
26. Piotrkowicz, A., Dimitrova, V., Mitrovic, A., Lau, L. (2018) Self-Regulation, Knowledge, Experience: Which Characteristics are Useful to Predict User Engagement? HAAPIE workshop, Adjunct Proceedings of the 26th ACM UMAP conference, Singapore 8-11 July 2018, pp. 63-68. https://doi.org/10.1145/3213586.3226196
    
27. Piotrkowicz, A., Dimitrova, V., Mitrovic, A., Lau, L. (2018) Using the Explicit User Profile to Predict User Engagement in Active Video Watching. Proc. 26th Conf. UMAP 2018, pp. 365-366, Singapore, 8-11.7.2018. https://doi.org/10.1145/3209219.3209262
    
28. Sjödén B., Dimitrova V., Mitrovic A. (2018) Using Thematic Analysis to Understand Students’ Learning of Soft Skills from Videos. In: Pammer-Schindler V., Pérez-Sanagustín M., Drachsler H., Elferink R., Scheffel M. (eds) Lifelong Technology-Enhanced Learning. EC-TEL 2018. Lecture Notes in Computer Science, vol 11082. Springer, Cham (pp 656-659) 
    https://api.ltb.io/show/BCFPK
29. Abolkasim E., Lau L., Dimitrova V., Mitrovic A. (2018) Diversity Profiling of Learners to Understand Their Domain Coverage While Watching Videos. In: Pammer-Schindler V., Pérez-Sanagustín M., Drachsler H., Elferink R., Scheffel M. (eds) Lifelong Technology-Enhanced Learning. EC-TEL 2018. Lecture Notes in Computer Science, vol 11082. Springer, Cham (pp 561-565).
    https://api.ltb.io/show/BVFND
30. Hecking, T., Dimitrova, V., Mitrovic, A., Hoppe, U. (2017) Using Network-Text analysis to characterise learner engagement in active video watching. In: Chen W. et al. (Eds), Proceedings of the 25th International Conference on Computers in Education ICCE 2017, Christchurch, 4-9 December 2017, pp. 326-335. Asia-Pacific Society for Computers in Education. Best Technical Design Paper Award
    https://ir.canterbury.ac.nz/handle/10092/15125
31. Mitrovic, A., Gostomski, P., Herritsch, A., Dimitrova, V. (2017) Improving presentation skills of first-year engineering students using Active Video Watching. In: N. Huda, D. Inglis, N. Tse, G. Town (Eds.) Proceedings of the 28th Annual Conference of the Australasian Association for Engineering Education (AAEE 2017), Sydney, 10-13 December 2017, pp. 809-816. https://search.informit.org/doi/10.3316/informit.392216803020572
    
32. Galster, M., Mitrovic, A., and Gordon. M. (2018). Toward Enhancing the Training of Software Engineering Students and Professionals Using Active Video Watching. In Proceedings of 40th International Conference on Software Engineering: Software Engineering Education and Training Track, Gothenburg, Sweden, May27-June 3 2018 (ICSE-SEET’18), pp. 5-8, ACM. https://doi.org/10.1145/3183377.3183384
    
33. Dimitrova, V., Mitrovic, A., Piotrkowicz, A., Lau, L., Weerasinghe, A. (2017) Using Learning Analytics to Devise Interactive Personalised Nudges for Active Video Watching. In: Bielikova, M., Herder, E., Cena, F., Desmarais, M. (Eds.) Proc. 25th ACM UMAP conference, Bratislava, Slovakia, 9-12 July 2017, pp. 22-31.
    http://dx.doi.org/10.1145/3079628.3079683, ACM.
    https://ir.canterbury.ac.nz/handle/10092/14529
34. Mitrovic, A., Dimitrova, V., Lau, L., Weerasinghe, A., Mathews, M. (2017) Supporting Constructive Video-based Learning: Requirements Elicitation from Exploratory Studies. In: E. Andre, R. Baker, X. Hu, M. Rodrigo, B. du Boulay (Eds.), Proc. 18th Conf. Artificial Intelligence in Education, LNAI 10331, pp. 224-237. https://doi.org/10.1007/978-3-319-61425-0_19
    
35. Mitrovic, A., Dimitrova, V., Weerasinghe, A., Lau, L. (2016) Reflective experiential learning: using active video watching for soft skills training. In: Chen, W. et al. (Eds.) Proc. 24th Conf. Computers in Education, pp. 192-201. Mumbai, India, Nov 28 – Dec 2 2016. Asia-Pacific Society for Computers in Education (ASPCE). https://doi.org/10.58459/icce.2016.1166
    
36. Lau, L., Mitrovic, A., Weerasinghe, A. and Dimitrova, V. (2016) Usability of an Active Video Watching System for Soft Skills Training. Zagreb, Croatia: Proc. 1st Int. workshop on Intelligent Mentoring Systems IMS 2016 held in conjunction with ITS 2016, 7-10 June 2016. https://imsworkshop.wordpress.com/wp-content/uploads/2016/03/ims2016_paper_7_lau.pdf

Presentation: Developing Personalised Nudges for Improving Comments Quality

Invited Talk

Tanja gave an invited talk at ICCE 2018, titled 'Towards personalised support for learning transferable skills via active video watching'.