Empowering Teaching for Digital Equity and Agency: IFIP TC 3 Open Conference on Computers in Education, OCCE 2020, Mumbai, India, January 6–8, 2020, Proceedings

Due to recent developments in the field of artificial intelligence (AI) and its impact on many areas of life, this paper provides an overview of that field, focussing on current approaches, especially in schools. After a clarification of the particular terminology in a wider context, and after a short journey into the history of AI in schools, current initiatives and AI-related approaches on a school level are described. The disciplinary aspect of AI is highlighted. This paper concludes with some implications for the practice of AI education.


IFIP -The International Federation for Information Processing
IFIP was founded in 1960 under the auspices of UNESCO, following the first World Computer Congress held in Paris the previous year. A federation for societies working in information processing, IFIP's aim is two-fold: to support information processing in the countries of its members and to encourage technology transfer to developing nations. As its mission statement clearly states: IFIP is the global non-profit federation of societies of ICT professionals that aims at achieving a worldwide professional and socially responsible development and application of information and communication technologies.
IFIP is a non-profit-making organization, run almost solely by 2500 volunteers. It operates through a number of technical committees and working groups, which organize events and publications. IFIP's events range from large international open conferences to working conferences and local seminars.
The flagship event is the IFIP World Computer Congress, at which both invited and contributed papers are presented. Contributed papers are rigorously refereed and the rejection rate is high.
As with the Congress, participation in the open conferences is open to all and papers may be invited or submitted. Again, submitted papers are stringently refereed.
The working conferences are structured differently. They are usually run by a working group and attendance is generally smaller and occasionally by invitation only. Their purpose is to create an atmosphere conducive to innovation and development. Refereeing is also rigorous and papers are subjected to extensive group discussion.
Publications arising from IFIP events vary. The papers presented at the IFIP World Computer Congress and at open conferences are published as conference proceedings, while the results of the working conferences are often published as collections of selected and edited papers.
IFIP distinguishes three types of institutional membership: Country Representative Members, Members at Large, and Associate Members. The type of organization that can apply for membership is a wide variety and includes national or international societies of individual computer scientists/ICT professionals, associations or federations of such societies, government institutions/government related organizations, national or international research institutes or consortia, universities, academies of sciences, companies, national or international associations or federations of companies.
More information about this series at http://www.springer.com/series/6102 Preface This volume contains selected papers from the Open Conference on Computers in Education (OCCE 2020), organized by Technical Committee 3: Education (TC3) and its working groups. The conference was held in Mumbai, India, during January 6-8, 2020, and was hosted and supported by the Tata Institute of Social Sciences and the India Didactics Association. OCCE 2020 was open to researchers, policy makers, educators, and practitioners worldwide. The conference title, which has also been selected as the book title, Empowering Teaching for Digital Equity and Agency, reflects the ongoing commitment to and current interests in research and practice in learning and technology that members of TC3 and its working groups have fostered over many years, and continue to nurture today. Submissions to the conference were invited to address the following eight key themes: Altogether, 57 submissions of full and short papers, symposia, industry foresights, learner and teacher presentations, and system presentations were received and reviewed by an International Program Committee and additional reviewers in a double-blind peer-review process. Among these submissions were 47 full and short papers, from which 15 were accepted for publication in the volume at hand. The overall acceptance rate was 31.9%. Each one of these papers was reviewed by at least three reviewers. The papers in this book arise from contributions from (in alphabetical order) Australia, Austria, Denmark, Finland, Germany, India, Ireland, Japan, and the UK, which reflects the conference's success in bringing together and networking experts from many countries worldwide.
The book is structured into five topical sections. The first section focuses on aspects of "Computing Education" at school levelan important facet in the process of developing digital agency among school learners. In this section, you will find practical approaches and examples as well as a theoretical framework. Micheuz reports on the important and current topic of artificial intelligence and provides an overview of relevant initiatives and approaches for school education. Weigend combines aspects of learning with digital technologies and about digitization in the classroom by presenting examples of an introduction to programming using a variety of digital media. Nayak, Keane, and Seeman describe a conceptual framework based on technacy theory with the potential to form a working model for teachers teaching computer science/digital technologies in K-12 classrooms.
The second section is about "Learners' and Teachers' Perspectives," which address important concerns when designing learning and teaching processes for digital equity and agency. In this section, you will find papers about learners' and teachers' conceptions developed in classrooms, through everyday experiences and in teacher education programs. Hillier, Kumar, and Wijenayake investigate the impact of technology problems on students' perceptions of computerized examination technologies and procedures. Keil, Batur, Kramer, and Brinda report on upper-secondary school students' images of computer scientists. Butler and Leahy describe pre-service primary school teachers' understandings of computational thinking after having completed a course in digital learning. Hayes reports on an early exploration of gender imbalance in computing, by analyzing trainee teachers' images of computing classrooms.
The third section covers "Teacher Professional Development." In this section, you will find papers about the development of professional and digital agency of teachers. Paltiwale, Sarkar, and Charania provide a descriptive analysis of a community of practice of teachers, which was enabled by the use of digital technologies in rural India. Misquitta and Joshi report on how Universal Design of Learning can be implemented in an Indian context based on outcome data of a six-month professional development program. Andresen presents results of a case study exploring design and certification of e-learning courses with a focus on the professional agency of teachers.
The fourth section contains papers that take "The Industry Perspective" and focus on the information technology (IT) industry's need for engaging a workforce with professional digital agency. The papers in this section cover two aspects: alignment of IT curricula and industry needs; and conclusions to be drawn from female career paths into the IT industry with regard to education in this field. Garscha and Wöhrer report on their findings concerning the question, whether cloud computing is adequately dealt with in the IT curricula of Austrian universities according to the requirements of the IT labor market. Hyrynsalmi, Islam, and Ruohonen analyze the stories of 23 women who were working in the ICT industry to learn more about the motivation, challenges, and best practices for different career paths that can lead to working in the IT industry.
The fifth and final part of the book on "Further Aspects" contains three more papers. Fluck analyzed the terms and conditions of 48 online services and discusses the estimated professional cost of their perusal with regard to the use of such services by teachers in the classroom. Shiozawa, Hoenigman, and Matsuzawa developed a tool to visualize the course selection in interdisciplinary study programs such as social informatics and report first findings of its application. Aoki, Sakka, Emi, Kobayashi, and Okamoto report on the results of a text data analysis on answers written in Japanese to free text questions obtained at astronomical lectures using co-occurrence network diagrams.
The editors offer leading-edge work through this choice of papers that they hope will be of interest to further inspire your own work.

Introduction
Writing a contribution about artificial intelligence (AI) in schools is a challenging task. One reason is the abundance of relevant existing on-and off-line resources, and the other is the difficulty to offer an overview of the already many studies and initiatives in that field. However, in this paper, an attempt is made to give a comprehensive overview of this multifaceted and broad field.
The pace of recent developments in AI has surprised not only insiders, but also the public and schools. If anyone thought that the subject of computing in schools is already completed and curricula do not need any major revisions, then they have recently been shown a different picture.
Although there is a plethora of insightful and valuable books, papers and a vivid blogosphere about AI terminology (the particular AI glossary of Wikipedia consists already of more than 300 terms), a compact overview is given in section two.
About thirty years ago, when the subject of computing was in its infancy, AI already played a certain role in school education. Section three gives some insight into this historical period. The following section provides an exemplary overview of recent initiatives and AI-related projects, and is concluded by an illustrative rapid run-through of some approaches to impart AI at various levels of education. In section five, the interdisciplinary nature of AI is explored, followed by a short conclusion in which some practical aspects are addressed.
The challenge for providing good practices in AI education and to convey a complete picture of this field is open. It can be expected that AI is not a fad and is here to enter school education, and to stay. From the title, this paper is concerned with "learning about AI", and does not elaborate on "learning through AI" in the context of educational technologies, except for the following notes on this issue. AI has the potential to play an important role in educational technology, with many potential applications, from inspiring ones like personalised learning, (automatic) assessment facilitation, assisted language learning and translation to less favourable ones like advanced cheating. However, we still do not know how the digitalisation of education and the adoption of AI will shape learning in this decade [1].

What Is It All About?
Let us start with a term sometimes used in an educational context: "Deep learning". It stands for meaningful learning, in contrast to human surface and rote learning [2]. "Deep learning" with regard to AI is a method that mimics the workings of the human brain in processing big data for use in predictions and decision making [3]. Its results affect our lives in a way which could not have been foreseen some years ago. It is very likely that most of us have unknowingly been using deep learning models already on a daily basis. A deep learning model is almost certainly used every time we use an internet search engine, a face recognition system on a social media website, a translation system or a speech interface to a smart device. Accordingly, deep learning can be regarded as one of the most powerful and fastest growing applications of artificial intelligence within the sub-field of machine learning.

From Deep Learning to Artificial Intelligence
All three areas, deep learning (DL), machine learning (ML) and artificial intelligence stand in a hierarchical relationship to each other ( Fig. 1), although the concept of what defines AI has changed over time. But, at the core, there has always been the idea of building machines (computers) which are capable of "thinking" like humans.
The field of research -already with impacting and fruitful applications in recent years -has become known as "machine learning". Even moreso, it has become so integral to contemporary AI that the terms "artificial intelligence" and "machine learning" are sometimes used interchangeably.
Machine learning is one of the primary approaches to artificial intelligence, but by far not the only one, as will be seen later. There are many similar definitions around, varying just in the wording, but with the same semantics. Wikipedia's definition is: "Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. It is seen as a subset of artificial intelligence. Machine learning algorithms build a mathematical model based on sample data, known as "training data", in order to make predictions or decisions without being explicitly programmed to perform the task. Machine learning algorithms are used in a wide variety Fig. 1. Relationship between DL, ML and AI [4] of applications, such as email filtering and computer vision, where it is difficult or infeasible to develop a conventional algorithm for effectively performing the task" [5].
Used to solve problems which were previously considered too complex, and using the model of neural networks involving large amounts of data and rapidly growing computational power, AI has revolutionised the quality of speech recognition, language processing and computer vision, vehicle identification, driver assistance and other domains.
Generally, AI comprises advanced algorithms based on advanced mathematics, which can handle higher processes similar to humans. Examples include visual perception, speech recognition, decision-making, and translations between languages. Among other trends in information technology, such as the internet of things (IoT), robotics, 3-dimensional (3D) printing, big data, blockchain technology, virtual and augmented reality, AI is one of the leading topics of our digitally penetrated world. AI is often accompanied by misleading stories and thus leads to diverging feelings in the general public, ranging from utopian enthusiasm to dystopian fear. Accordingly, there is a huge challenge for education and schools to provide all pupils with a solid understanding in that field.
As with every new technological achievement, deep learning as the main application of machine learning has its dystopian implications. It is potentially worrying that the trail of data and metadata we are leaving and delivering voluntarily, and largely unnoticed when moving through the online world, is also being processed and analysed using deep learning models. This is why it is so important to at least understand what artificial intelligence is, how smart technologies work, and what they are capable of, and what their current limitations are [6].
It is important to recognise also that AI is a constantly moving target. Things that were once considered within the domain of artificial intelligence -optical character recognition and computer chess, for example -are now perceived as routine computing. Today, robotics, image recognition, natural language processing, real-time analytics tools and various connected systems within the IoT are all increasingly using AI in order to be augmented with more advanced features and capabilities.
Deep learning and, or with, neural networks, meanwhile, gain the most attention because they are particularly well suited for tasks involving image, video, and audio data [7]. For text and numerical information, though, the older methods can still be more suitable.
AI's transformative effects on technology will increase over the coming decades, with the development and adoption of deep learning continuing to be driven by rapidly growing datasets, the development of new algorithms, and improved hardware. These trends are not stopping.

Artificial Intelligence in a Wider Context
We cannot discuss AI without considering the highly related and overlapping wide area of data science (Fig. 2). Simply put, data science is the study of data, involving developing methods of recording, storing, and analysing data to effectively extract useful information. The goal of data science is to gain insights and knowledge from structured and unstructured data. It is the science which brings the saying "data is the new oil" to life.

Fig. 2.
Relationship between data science and AI [9] Data are worth very little if there are no highly-skilled professionals who can derive actionable insights from it. Undoubtedly, the competence to understand, use, process and interpret data has become indispensable and a requirement for an expanding range of jobs and careers. (Big) data are ubiquitous. It is estimated that about ninety percent of the world's data has been created in the last two years [8].
Mastering data science and harnessing data requires a solid grounding in: mathematics for processing and structuring (big) data; statistics; programming not just in one specific programming language; and last but not least analytical and computational thinking, including problem solving and logical reasoning.
The new data-driven world requires individuals to be constantly separating fact from fiction. In short, the need to analyse and interpret data is no longer confined to engineering or computer programming; it has become an essential life skill. Yet, the K-12 education landscape is lagging behind. Schools have not recognised today the changes the data explosion has made to society. Curricula we teach currently should be revised and provide more practical utility for the 21 st century. There is a widening gap between competencies students need in life and what is taught in schools. It can be argued that data science including AI should be building blocks of a modern school education.
Machine learning is one of the primary approaches to artificial intelligence, but it has to be seen in the wider context of data science, which encompasses important areas such as the often underestimated and arduous work of data preparation on the one hand, and the fascinating field of data visualisation on the other.
Before learning machines and machine learning provided us with suggestions and predictions in (still) particular situations, (contrary to "general artificial intelligence" with super intelligent robots exceeding the abilities of human beings), they had to be trained with (big) data sets which could be accomplished through "supervised", "unsupervised" or through "reinforced learning". In short, supervised learning requires the supply of training data and correct answers; unsupervised learning occurs when machines learn from a dataset on their own, and reinforcement learning is based on permanent feedback from the environment. Machine learning uses algorithms to learn from data and data patterns, and the knowledge acquired can be used to make predictions and decisions.
Whereas AI is just at the beginning of being included in curricula and lessons, data have already been for a long time a building block of computing education. In some countries, the term "Electronic Data Processing" was the predecessor for the later subject "Informatics". Accordingly, AI is naturally embedded in all aspects and fields around data; that is, data literacy and data management. Recently, a comprehensive model of data key concepts and a competence model of data literacy have been published [10,11] (Fig. 3). This holistic view on data shows convincingly that data driven computing education is very broad, with AI playing an increasingly important role.  3. Data management and data literacy competence model [10,11]

Historical Context
Although AI may be regarded as the 'hot topic' of the moment in (digital) technologies, and the driving force behind many technological breakthroughs of recent years, the term is not all that new. During the last decades, AI has moved out of the domain of science fiction and into the real world, while the theory and the fundamental computer science which makes it possible has been around for decades.
There are many very useful resources on the web which describe this timeline in general and AI milestones in particular [12]. Such content on the web can be harnessed and discussed by pupils in a historical, interdisciplinary context.
Long before robots were dominating utopian and dystopian arenas in science fiction, in the 17 th century the scientist and philosopher Rene Descartes thought of thinking and decision-making machines. While he was wrong in stating that they would never be able to talk like humans, he already distinguished between machines which might one day learn about performing one specific task, and those which might be able to adapt to any job. Today, these two fields are known as specialised and general AI.
The origin of the term "artificial intelligence" goes back to a conference at Dartmouth College (in the United States of America) in 1956, years before the subject computing (computer science, informatics, information technology) came into play into the curricula of timetables in some countries, and accordingly into textbooks about computing/informatics. In their seminal and modern introduction to computer science, Goldschlager and Lister [13] locate artificial intelligence in the chapter "Algorithms in action: some computer applications", subdivided into "Can machines think?", computer games, understanding language, visual perception, knowledge representation and expert systems. The book ends with the (philosophical) question "Superfluous human?" Their comforting assertion that human beings with their creative capabilities, innovation power and originality will never be superfluous can be judged to be correct, but were they right with their statement that "Computers carry out (only) repetitive tasks"? About ten years later, the first German comprehensive textbook on "Didaktik der Informatik" contained a full chapter about "artificial intelligence" [14], referring to AI as an academic discipline with its subsections of natural language systems, expert systems, robotics, computer vision, followed by a short historical summary and general explanations about cognitive science, including a sceptical view on neural networks. Herein, the author Baumann dismissed statements (from his students) such as "neuronal networks can learn" as an improper use of language.
A similar misjudgement can be read in Rechenberg's "Was ist Informatik" [15], where neuronal networks are denoted as an offspring, and not really belonging to artificial intelligence. "It looks like this idea is doomed to failure. […] Neuronal networks have proven some applicability, but its performance should be estimated very carefully. There are many examples for which much more efficient solutions have been found with other mathematically trustworthy processes." What followed was an "AI winter", also in schools, until recently. Currently, we are witnesses of an "AI springtime", not only in research and business, but slowly in schools too. However, there is much evidence that AI in schools is still in its infancy.

Current AI Initiatives and Approaches in K-12 Education
Following the hype about AI in the 1980 s that used a classical rule-based approach, expert systems and the programming language Prolog [15], we currently observe a regaining of momentum in this field, fuelled by promising approaches as shown below. From this work, it appears that AI can be introduced in curricula and computing lessons in an appropriate and sustainable manner.
Perhaps the most advanced development in implementing AI in K-12 education can be found in China, where a textbook on AI based on computational thinking has been rewritten and published [16]. The old textbook in that field mainly included knowledge representation, reasoning, expert systems, search, etc., whereas the core concepts of the new textbook encompass intelligent systems, artificial neural networks, and machine learning. The shift has been from focussing on expert systems to the analysis and design of intelligent systems incorporating state of the art AI concepts.
In England, the initiative "Computer Science for Fun", an (online) magazine where "the digital world meets the real world" offers many ideas and teaching materials for machine learning (http://www.cs4fn.org/machinelearning), with the so called "Sweet learning computer (a simplified chess game)" as one example [17].
In Germany, there are various initiatives to pilot AI-related projects and studies in schools. These range from unplugged activities in that field [18], an activity-based explanation of how neurons work and learn in robot controlling [19], to machine learning in the context of data science [20].
Computer vision, which perhaps for pupils is the most striking and impressive aspect of AI, can already be treated and discussed in a phenomenological way for a young agegroup, using Google's Autodraw as an example. The question if and how a computer recognises animals (such as the fat giraffe in Fig. 4) can be a starting point for stimulating interest in lessons. Another fertile question can address the future of so-called "Captchas" (see Fig. 4, an applied Turing test), including a discussion on how much longer it will be before DL will also solve these puzzles. The right-hand face in Fig. 4. is fake; it does not exist in reality and is AI-generated.  [21] Due to deep learning, great progress has been made with character recognition. This aspect of AI is used almost as a daily practice by all internet users and is therefore suited for computing lessons, addressing (un)supervised and reinforcement learning through training with data, underpinned by theoretical foundations.

AI is Interdisciplinary
Like hardly any other science, AI is interdisciplinary. It uses results from such diverse fields as mathematics, logic, operations research, statistics, control engineering, image processing, linguistics, philosophy, psychology and neurobiology. In addition, in many AI projects, the field of the respective application has to be taken into account. To successfully work on AI projects is therefore not always easy, but almost always exciting and challenging [22]. The challenge is at least as great when thinking of introducing AI into school curricula, as interdisciplinary aspects in this context are rather the exception than the rule.
In Germany, a nationwide initiative about imparting a holistic view on AI within the "Science Year" has been launched [cmp. 20]. Its ambitious goals, aiming at the target group of 12-18-year-old pupils, comprise: • a sound explanation of how AI works.
• stimulation of a social discourse on AI.
• a reduction of existing misconceptions.
The course consists of (up to) six modules, containing teaching material, arranged around: "Introduction -students' everyday experiences with AI"; "How does machine learning work?"; "What's the difference between man and machine?"; "Historical overview of the development of artificial intelligence"; "The distribution of roles of man and machine -ethical and societal aspects"; and finally, "In which AI world do we want to live?" Contemporary computer lessons (should, it can be argued) make use of contextualised teaching concepts such as IniK which means "Informatik in Kontext" [23]. IniK is based on the assumption that solely technical computing competences do not suffice to understand the digital world of information technology (IT) systems and digital media. Pupils should be able to use them in a self-determined way. To this end, questions are placed at the centre of computing that go beyond technical issues, and include the social context, aspects of economics, culture, politics or law [24]. Answering these questions makes it possible to develop IT content in a cross-disciplinary way and can lead to sustainable computing competences. Aspects of AI are almost predestined to supplement this approach of IniK with already recommended and elaborated topics, such as "E-mail only(?) for you", "My computer is talking to me!", "Smart and rich through apps", "Social networks", and "Don't trust a picture!" Lessons according to "Computing in Context" are per se interdisciplinary and subject-linking. Assuming a real-life context, there are manifold references to different subjects. But, most likely, there are only a few teachers who are able to deal with multidisciplinary challenges with professional competence. Maybe appropriate interdisciplinary teacher training in which colleagues from relevant subjects join together to form a team may be the key to remedy this situation. However, the challenge of a sound teacher education in AI is still ahead of us.

Implications and Final Remarks
From a theoretical point of view, some studies indicate that many seemingly "intelligent" systems and AI can be demystified in computing lessons, and sooner or later, this muchdiscussed area of digital technologies will reach school informatics on a broader basis. However, we have to be honest and realise that a constructive and meaningful approach to teach the (currently) 'hot topic' of AI in depth requires a deep understanding of the field, represents a big pedagogical challenge for teachers and teacher trainers, and of course, a cognitive one for many pupils.
Obviously, it makes a big difference to approach this topic in schools (comprehensively): • from a social and philosophical viewpoint (talking and reasoning about AI).
• by conscious awareness of AI applications on a phenomenological level (knowing about AI and using AI applications in a reflective way). • by applying a grey box model of AI, requiring a basic knowledge of its key concepts and programming languages and environments (applying AI). • through putting the mathematical and computational perspective into the foreground (understanding the foundations of AI and constructing AI).
From a practical point of view, and with the focus on learning outcomes, it is useful to consider the seminal (revised) Bloom's taxonomy [25] in mind, which starts from mere recalling and basic understanding to the creation of AI applications. Referring to the "Tale of Three Learning Outcomes" [26] with three categories "No learning", "Rote Learning" and "Meaningful Learning", the question arises as to how much time can be allocated to AI in (an always) overcrowded curriculum. "Meaningful Learning" is recognised as an important general educational goal and occurs when pupils build their knowledge and cognitive processes that are needed for successful problem solving. This begins with an appropriate mental representation of the problem and ends with the problem solution, in which the pupils devise and carry out a plan for solving the problem.
With regard to didactical aspects of AI education, it is a key question how to deal with the plethora of its possible approaches. The spectrum begins with a discipline of AI in its own right, providing a holistic picture of the field with sequenced und structured lessons, ranging across fragmented approaches within subjects such as computing, mathematics or philosophy. These include singular bottom-up initiatives such as the simulation of a neuron with the programming environment Scratch, and unplugged activities in the form of a role-playing simplified chess-game to demonstrate reinforcement learning. All these approaches in schools are still in their infancy worldwide and at an experimental stage lacking empirical results.
However, recent developments and the obvious progress of machine learning and its impact on all of us suggests the implementation of AI education in school education on a broader and deeper basis. This would have the potential to extend and enrich not only the subject of computing in schools, but education in general.
However, there is still a long way to go to find appropriate approaches for particular age groups with reasonable levels and requirements. Above all, it needs curious and engaged educators, teachers and teacher trainers able to incorporate this important and prospective field into general, specific and vocational education.

ICT and Computer Science Education
There is a permanent discussion about what to teach in Computer Science (CS) classes in schools. According to Webb et al. [1], there is a consensus in the community of educators and curriculum developers that digital literacy -the skills to use information and communication technology (ICT) -and CS are complementary and that both are needed in the school curriculum. CS curricula usually focus on principles and practices related to creating digital artefacts. In the CSTA K-12 CS standards [2], basic competences related to using ICT are mentioned just for level 1a (ages 5-7 years), like "Select and operate appropriate software to perform a variety of tasks" (1A-CS-01) or "Store, copy, search, retrieve, modify, and delete information using a computing device" (1A-DA-05). At higher levels, the focus is on creating, understanding and evaluating digital technology. Among the 102 competences listed for levels 1b to 3b there are only 3 on using digital tools. Bridging ICT and CS can be done in different ways. Brinda et al. [3] suggest connecting basic CS concepts like finite state automata and object-oriented modelling (OOM) with ICT usage. State transition diagrams may help understanding of how to use audio players, and OOM concepts help understanding of the philosophy of text editors. For example, when Tina sees characters and paragraphs as objects of different types, it might be easier for her to understand how to change attributes (like size and colour of characters and line spacing and alignment of paragraphs). Dagienė [4] suggests that activities like "Computer Science Unplugged" [5] and the tasks of the Bebras Challenge on Informatics and Computational Thinking lead to a deep understanding of ICT. Although the Bebras tasks cover ICT as a topic, they require little or no ICT usage. All information necessary to solve the task can be found in the task description and the children can principally solve the tasks in their minds or with pencil and paper.
In contrast to this approach, this paper focuses on ways to encourage ICT usage for creative programming projects. Obviously, students use an integrated development environment (IDE) in each programming project. But beyond that, in CS lessons, often only a little technology use is required. There is a huge supply of free resources on the Internet that can be used for programming projects: images, movies, data, texts and digital tools (apps). In an ICT-rich programming project, these resources are used extensively for creating and collecting data that are needed for programming projects.

Example Projects
In this section, I present four projects with extensive usage of ICT, sorted according to required programming skills.

Visiting Cool Places
An algorithm is the precise description of instructions to solve a problem. An example is the directions guiding a person from one place in a city to another. If the directions are correct and correctly executed, the person following the direction arrives at a certain place. The first ICT-rich project is to create directions using Google Maps. This task does not require any specific pre-knowledge and has been used as part of an introduction into algorithms with 14-year-old students. It is probably also suitable for younger age groups.
Task 1: Pick a place on Google Maps somewhere in the world. Write a formatted text document, with images, that explains how to walk from this place to another place by using Google Street View. Street names are only allowed for describing the starting point.
At the end of the text, ask the reader a question that the reader can only answer, if she or he has arrived at the correct place.
First, try this mini example. Open Google Maps and go the Brandenburger Tor in Berlin. Put the orange Street View Pegman on the Pariser Platz. Turn around until you see the Brandenburger Tor right in front of you. Walk through the Brandenburger Tor on the right-hand Street View path. Immediately behind the Brandenburger Tor, there is something on the right-hand side. Which of these items can you see on your right-hand side immediately behind the Brandenburger Tor? A: Wastepaper basket; B: Manhole; C: Spotlight; D: Traffic sign.
To create their own directions, the students are encouraged to use Google Maps, Google Street View, text editing and image editing tools. Some students tend to use directions that are then automatically generated by Google Maps. To avoid this, the task contains the rule that position names of streets may be used only for the specification of the start. The general idea is to write down directions you would give to a person that has asked you for the way to a certain place.
Regarding the algorithmic nature of this task, the students face three challenges: • The starting point and the direction in which the reader has to look first must be explained very precisely using information that is available on Google Maps. • On the way, they have to exploit Street View images to find appropriate landmarks that are easy to describe and easy to find. A challenge is to find good wording. Frequently used phrases are: "Turn around until you see….", "Walk along the road until …" indicating repetitions. • They have to create questions for checking whether or not the reader executes the algorithm correctly. At the same time, the reader's correct answers indicate that the algorithm (directions) is correct. Checking correctness is a computational competence.
Computer programs may contain assertions that guarantee correctness to some extent. In this case, the students need to use ICT to find good questions. Ninth-graders from a German high school, who did this project, had these ideas: Tina (name changed) created a section from a Google Street View screenshot at the goal, depicting a statue, removed the text using Windows Paint and asked: "What is written on the statue?" Tom asked the reader to use the Google Maps tool for measuring distances and to measure the length of a house at the goal.

Mining Mark Twain
The second project is an example of creative coding exploiting textual data that are available on the Internet. In Project Gutenberg (http://www.gutenberg.org), one can find more than 60,000 free e-books, including the complete work of Mark Twain (5,598 pages). Creative coding (CC) is developing computer programs for personal expression or artistic purposes. CC has been used in CS education for many years [6][7][8]. Pioneers in CC are artists like Hiroshi Kawano (China, Japan) and Francois Morellet (France), who have used computers to generate paintings since the early 1960s. In 1960-61, Morellet used textual data to create a painting. He distributed 40,000 squares according to even and odd numbers in a telephone directory (Zentrum für Digitale Kunst in Karlsruhe, Germany). In literature, there are sonnets, haiku, or other forms written by computer programs [9]. The website "Bot or Not" (http://botpoet.com/) offers a Turing Test for poems. Chris Peck and Eleanor Bauer developed the text of the theatre play "New Joy" (premiere 2019 at the Schauspielhaus Bochum, Germany) using simple algorithms. The following task has been used in introductory programming classes (secondary school and university) in the context of string processing. To be able to understand and to improve the starter program, the students should have basic programming skills (control structures, files, strings and lists). Task 2: Write a Python program that creates a short text (say for a birthday card) picking parts from the complete work of Mark Twain.
The following starter program illustrates basic Python techniques, that can be adopted. The text file containing Mark Twain's work must be downloaded and stored in the project folder. The program reads this file and creates a string object named text. The statement in line #2 creates a list of sentences (strings ending with ".") In the while-loop, items from this list of sentences are randomly picked again and again until a sentence with a length between 20 and 50 characters is found. Three of these randomly chosen sentences are concatenated to a string.

Output (example):
He was competent and satisfactory. The matter ended in a compromise, I submitted. Still he could not understand.
The basic idea of a starter program is that students copy it, run it and change it until they understand it completely. Then they improve and extend it, for example: • Add documentation/trancing features: how many random picks were necessary? How many sentences of a certain length contains Twain's work? • Develop/refine the functionality: search for sentences that contain certain words (e.g. the first name of a friend); write computer poems, using words that rhyme.
More examples for simple Python projects exploiting textual and visual data from the Internet (twitter tweets, webcam life images, etc.) can be found in Weigend [10].

Evaluating Interviews
What are hot topics and controversial issues in the school community? In a classroom project at a German high school, students tried to find out, and interviewed school friends and teachers using their smartphones. The goal of the interviews was to get a collection of words indicating controversial issues of school life.
The students developed a small set of questions that were supposed to evoke statements on conflicts (e.g., do you remember a conflict in the classroom today? Did something on the schoolyard bother you today?) They interviewed each other first and then interviewed teachers and students on the school campus using their smartphones and a voice recorder app with voice-to-text functionality. The text files (created from audio) were collected in a folder. This can be done by sending the files to a common Google Drive Folder, which has been created and made accessible to everybody by one member of the group. Each student copies these text files with the interviews into a subfolder named "data" in his or her project folder.
The project is suitable for Python programming classes in secondary education, especially if the school supports interdisciplinary teaching. The students should be familiar with sequences and sets. As part of a computer science curriculum, the project can serve as an example for modelling with dictionaries.
The starter program illustrates basic Python programming techniques with files, dictionaries, and sets. The program reads all files in the data folder and creates a string containing all interviews (the loop starting in line #1). It splits the string into a list of words and creates a set of words by eliminating duplicates (#2). Using this set and the original list of words the program creates a dictionary, which maps each word from the set to its frequency (the loop starting in line 3). import os interviews = "" for fn in os.listdir("data"): There are several approaches for extension and improvement, for example: • Improve the usability: the program should filter the set of words and ignore irrelevant words like numbers or pronominals. A simple way to filter is considering only words with more than five letters. Students who are familiar with the Python interface Tkinter may create an application with a graphical user interface that displays the most relevant issues in a comprehensive way (see Fig. 1). • Improve the technical quality: define functions or classes and split the program into smaller parts.

Evaluating a Google Forms Survey
With the free version of Google Forms, students can conduct surveys. They create a questionnaire, make it accessible in the Cloud and use it for interviews on the schoolyard. With the free version of Google Forms it is possible to get an overview of the answers. Although the questionnaire might contain some personal questions like age and gender, unfortunately, filtering is not supported. For example, it is not possible to compare the responses of girls and boys. However, it is possible to create a comma-separated values (csv) text file. The csv text file is structured like this: • The first (very long) line contains the questions of the questionnaire as strings separated by commas. There is no comma at the end of the line. • The other lines contain the answers given by the people who completed the questionnaire. If a respondent has not answered a question, this non-answer is represented by an empty string.
Consider this mini survey: Question 1: Which of these statements do you agree with?
• Listening to music should be allowed (wearing headphones), while quietly working in the classroom. • The first lesson should not start before 9:00. • There should be a vending machine for snacks in the lobby. Question 2: What is your gender?
The csv file starts with two lines similar to these: "Time Stamp", "Which of these statements do you agree with?", "What is your gender?" "2019/09/17 8:54:20 AM OEZ", "The first lesson should not start before 9:00.; There should be a vending machine for snacks in the lobby.", "Male" Each line ends with the invisible new-line character (escape sequence: \n). Since lines in a csv file can be very long, on paper it might require several lines to print a single line from the file. In the example, the second data line starts with an (automatically generated) time stamp, indicating when this response was submitted. The second string contains the two checked options of the first multiple choice question separated by a semicolon. The third string of this line is the selected gender.
Unique Options Method. The example questionnaire was designed in way that makes it very easy to program an analysing tool. Each option is unique. It occurs only once in the questionnaire. That implies that the data analysis can be done simply by counting occurrences of strings.
This programming project should be a part of an interdisciplinary project in upper secondary education. To be motivated for the program development, students must be involved in the design of the underlying research questions.
The following listing is a starter program that students can test, modify and extend (a few lines are shortened for printing reasons). It illustrates a few basic techniques: • Reading the csv file and creating a list of lines containing the answers of all participants (#2). The starter program can be changed and extended by the students in a number of ways: • Adapt the starter project to a different -self-made -survey.
• Make it interactive. The user could choose the filter criterion.
Parsing the CSV Text. The example program from the previous section only works with very special questionnaires. Programs that are supposed to handle every kind of survey need to parse the csv text. This requires more programming experience. One approach would be to create a list of dictionaries, each dictionary representing the data of one participant in this format: {question1:answer1, question2:answer2, …} The key of each item (question1, …) is a string containing a question from the first line of the csv text. The value of each item (answer1, …) is a tuple of strings, representing the answers, given to the question.

Conclusion
This contribution presented four examples of ICT-rich programming projects. In these projects, the students experience that programming competence implies empowerment. With programming knowledge, it is possible to use digital tools (e.g., Google Forms) and other resources like free literature in new ways.
The second benefit from ICT-rich projects is that students discover new digital resources and new features of tools that they already use every day. For example, when students write precise directions based on Google Maps and Street View, they use different functionality from that when looking up a bus connection. This advanced usage of digital tools also implies reflecting their potential and limitations. In the case of the Google Maps project, students reflect on the difference between directions using landmarks that are easy to find and to describe, and automatically generated directions.
Finally, students get a deeper understanding of the internal structure of digital resources. For example, when programming a tool for analysing Google Forms survey data, they become aware of the structure of csv representations.
ICT-rich programming projects can easily be integrated into a conventional CS curriculum that covers coding-oriented competences like those in the CSTA standards [2]. The usage of ICT is not systematically introduced, but teachers assume that students already know most of the tools and can learn things they do not know yet en passant. This fits with the idea that the CS curriculum focusses on principles and concepts that are long-lasting and difficult to learn, whereas the elements of digital literacy change quickly and are easy to learn. Many apps that people download and use do not require a systematic introduction. ICT-rich programming projects encourage the students to improve their digital literacy on their own. The ideal environment for this is a Parkhurstian laboratory, fully equipped with relevant literature and a teacher that can give expert support through a self-controlled learning process.
In students' perceptions, there is a strong connection between ICT and CS. Students who code are also interested in ICT. The results of an Estonian study suggest that programming knowledge from high school education motivates students to enrol in ICT subjects (not just computer science) at university [11].
Fluck et al. [12] have stated economic, social and cultural rationales for computer science as a school subject. These rationales are not only relevant in the general discussion of school development; they are also relevant for teenagers seeking their way into society by developing views, and planning their professional future. ICT-rich programming projects illustrate the relevance of computer science in the economic, social life culture in a modern society, and might influence students' career decisions. Here it is not just claimed by the teacher that CS is relevant, but students actually experience the benefits of CS concepts while working with ICT.
Many students love science not just because of the concepts they learn, but because they have the opportunity to get exciting experiments to work with real chemicals and laboratory equipment. In the same way, the combination of using interesting ICT and programming (including learning fundamental CS concepts) might motivate teenagers to engage in CS.

Introduction
The last decade has seen exponential growth in digital technologies leading to an influx of innovative solutions. Students as young as seven years of age are now exposed to several digital devices and are generally quick to engage with these technologies. There is growing awareness that students should not just be consumers of technology but increasingly creators of innovative solutions. To achieve this, students from a very young age should be encouraged to create products and solutions whilst developing their problem-solving skills, critical thinking skills, digital literacy, creative thinking skills, collaboration and communication [1]. These are key elements of computational thinking, a term coined by Wing [2], defined as "thought processes involved in formulating a problem and expressing its solution(s) in such a way that a computer (or human) can effectively carry out". In more recent times, education policy makers have realised the importance of developing computational thinking skills and have endeavored to incorporate them across various subjects like mathematics, humanities and computer science. Mannila et al. [3] demonstrated that computational thinking skills can be developed within all disciplines, while Selby and Woollard [4], through a systematic literature review, stated that computational thinking is inherently embedded in computer science education. Other studies [5,6] have also shown that computational thinking skills can be acquired by students through the teaching of computer science concepts, programming and robotics. There has been much debate and discussion on how schools can change their curriculum to incorporate computational thinking from a young age. To illustrate this point, in 2015, the Australian Curriculum Assessment and Reporting Authority (ACARA) endorsed Digital Technologies as one of the two core subjects of the Technologies Curriculum [7] for students in Foundation to Year 10. The Digital Technologies curriculum aims to develop knowledge, understanding and skills to create digital products with programming as the vehicle for learning.
Teaching a new curriculum can pose challenges for existing teachers who have taught a very different curriculum and may not have sufficient technical knowledge and/or skills to adapt their pedagogical content knowledge to the new curriculum content [8]. Studies have indicated that teachers need adequate professional development to support teaching and implementation of new curriculum. Hill, Keane and Seeman [9] advocate for a clear set of guidelines, framework and practices that support teachers to improve student learning outcomes. While other disciplines such as English and Mathematics have developed pedagogies for literacy and numeracy respectively, little research exists to examine equivalent ideas for technological areas of learning such as Digital Technologies/Computer Science education. This paper will explore a promising new area known as Technacy Genre Theory as a framework to engage students in programming tasks and improve their computational thinking skills. Technacy provides a holistic framework with social, environmental and human factors, with significant discourse around developmental indicators in technical education. The Technacy Theory is discussed in detail in Sect. 3 of this paper.

Computational Thinking
Wing highlights the importance of computational thinking when she states that it is the "thought process involved in formulating problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an informationprocessing agent" [2]. Progressive thinkers such as Margaret Mead have long argued that: "children must be taught how to think and not what to think". It can be argued that computational thinking skills are important for students. Papert [10], one of the designers of the LOGO programming language aimed at school-aged students, has for many decades advocated that learning to program at an early age allowed children to develop problem solving and logical thinking skills.
At the core of computational thinking is the ability to solve complex problems by breaking it down into small procedures [3,11]. These procedures include the reliance of the following skills: logical thinking, algorithmic thinking, problem-solving skills with understanding of abstraction, generalisation, and decomposition.

Literacy in Digital Age
Bers [12] defined literacy as "the ability to use a symbol system and a tool to comprehend, generate, communicate and express ideas or thoughts by making a sharable product that others can interpret". Education and policy makers acknowledge the importance of technological literacy to help prepare children for the 21 st century. Keane, Keane and Blicblau [13] assert that digital literacy is as important as being literate and numerate. A review of scholarly literature, documents and reports over the last 30 years suggest that digital literacy is also referred to as computer literacy, information literacy and technological literacy. Often, the contextual meaning of the term is in relation to the technology of the given period. For example, computer literacy in the 1980s referred to working knowledge of desktop computers, whilst in the early 2000s information literacy incorporated web development skills. Currently, there are several initiatives to define and mandate assessment of technological literacy [14,15].

Defining Technacy
While being literate and numerate are fundamental skills, with the dynamic and evolving nature of the digital environment, it is important to also be technate (to comprehend technologies) [16]. Technacy and its adjective technate [17] is defined as "the holistic understanding of technology in relation to the creation, design and implementation of technology projects". Drawing parallels between literacy, numeracy and technacy, Seemann and Talbot [18] argue that: Just as there are levels of competence in literacy from writing one's name to writing profound poetry, and in numeracy from adding a few numbers together to compiling a fundamental formula in physics, so too there is a range in technacy from being skilled in joining materials together or repairing equipment to being innovative in the design and development of appropriate technologies and systems. Table 1 offers a comparative structure and parallel proposition of literacy and technacy with attention to the higher order cognitive demands of forming abstractions and inferences [19].

Background to Technacy Theory
Technacy Theory was derived from existing practices in cross-cultural technology education among Indigenous Australian communities. The Australian Science, Technology and Engineering Council [ASTEC] acknowledged that an improvement in science and technology education was necessary, stating that: "technacy -will be as vital to students of 21st century as literacy and numeracy were to Australians who grew up in the 20th century" [20]. ASTEC provided funding to look into innovative ways of teaching and assessing technological knowledge. One of the recommendations was to incorporate technacy education in primary and secondary school curricula across Australia [21]. Additionally, a generic framework, the Technacy and Innovation Chart ( Fig. 1) was developed to identify, assess and measure developmental indicators of technological thinking and doing, in the student learner.

Technacy and Innovation Chart
The Technacy and Innovation Chart (Fig. 1) is a 3x3 grid that provides a model to understand how people respond to increasing degrees of complexity when solving technological challenges. Along the horizontal axis are the three phases Emergent, Competent and Sophisticated which form Technacy Expectations and Complexity whilst the vertical axis are the three domains of innovation: Play, Consolidation and Pioneer, which describe the level of innovative responses to the technological challenges addressed by the student learner. The chart is organised by child Developmental Domains and Phases of Learning Complexity in technologies. The grid has been developed to reflect the learning journey of the student. The learning journey of the technology student typically begins with Emergent Play. As the learner gains confidence, they progress from left to right of the chart, attempting more complex tasks. The column progression from left to right (across the rows) shows the degree of task complexity. The degree of complexity is often set by the teacher. The row progression from bottom to top is attributed to the degree of personal initiative shown by the learner. The vertical progression is demonstrated by the student with the assistance of educational scaffolding and is subject to factors such as developmental ability [23]. The highest category in the framework aims to develop skills that prepare students to become independent creators, innovators and pioneers in their field. The centre of the Technacy Chart (Competent Consolidation) and the perimeter of the edges around it is  [22] where most learners tend to settle before engaging with higher demand technological challenges and pioneering expectations [23].

Mapping Key Dimensions of the Technacy and Innovation Chart
To understand the relationship between programming concepts, computational thinking, child development and technacy as discussed above, these have been mapped by the authors of this paper against the Technacy and Innovative Chart as shown in Table 2.
Play Domain. Students create knowledge in different ways. Play as a strategy has been identified as a powerful way to engage students in their learning, especially to form cognitive schema. Piaget [25], Vygotsky [26], Resnick [27], and Fleer [28] assert that play is essential for intellectual and cognitive development in children. Practical activities provide students with the opportunity to experiment and build ideas from their playful experiences.
In computer programming, play involves students interacting with on-screen objects such as Scratchy the cat in the programming language called Scratch, or the Turtle in the programming language called Python. The focus in the Play Domain is to foster early stages of schema development. The teaching of computer programming using onscreen objects assists in developing problem-solving skills, logical thinking skills and algorithmic thinking skills. The manipulation of on-screen objects requires the ability to comprehend technology knowledge being learned. Hrbacek, Kucera and Strach [29]  Programming Concepts Low-level concepts [24] Mid-level concepts [24] High-level concepts [24] Programming Tool Drag-n-drop Text-based determined that the age of 8 years is the optimal time to introduce early constructs in programming to students. Scratch and Blockly, block-based visual programming languages, are popular in the classroom with younger students due to their drag-and-drop features.
As the student develops confidence, teachers can facilitate their progress from Emergent Play to Competent Play towards Sophisticated Play by setting more higher order tasks to match the complexity of the project. Students will need assistance, guidance and directions from teachers [23] to progress through the three phases of the Play Domain.
Consolidation Domain. Block-based programming creates an interactive and engaging environment making it possible for students to create simple games without writing a single line of code. However, block-based languages have several limitations on what students can do. Text-based programming languages, on the other hand, offer greater flexibility at the expense of having pre-affirmed knowledge. At some stage, students need to be encouraged to move to a text-based programming language to extend themselves. There are several high-level programming languages, each with its own syntax and semantics; however, ultimately, from a pedagogical point of view, they all have similar underlying programming constructs. Butler and Morgan [24] grouped generic programming concepts into levels of difficulty -low, mid and high -based on their complexity as seen in Table 2.
The Consolidation Domain has three phases -Emergent Consolidation, Competent Consolidation and Sophisticated Consolidation. In the Consolidation Domain, the focus is on progressing from early discovery to higher cognitive demand in difficulty thus advancing deep knowledge of the subject in a meaningful way [30]. In the Emergent Consolidation Phase, teachers can assign tasks that focus on early discovery concepts [24]. As students develop confidence, they should be encouraged to be stretched into the Competent Consolidation Phase using programming concepts such as decisions, loops and arrays: the Competent Phase introduces codified established knowledge and techniques. The Technacy Chart is consistent with Papert's idea of low floor, high ceiling [10], a metaphor where a medium with a low floor assures a low barrier to entry, but the high ceiling simultaneously does not constrict creativity.
Pioneer Domain. The Pioneer Domain is where students demonstrate a high capacity to manage novelty and express new ideas. The Pioneer Domain has three levels of capability demonstrated by the learner. The Emergent Pioneer accommodates student learners who demonstrate a high and consistent capacity for imagination but have not yet affirmed competencies to execute their ideas. The Competent Pioneer also has a capacity to imagine new ideas, but unlike the Emergent Pioneer, has knowledge and skills of known techniques to execute their ideas. The Sophisticated Pioneer not only presents novel and creative ideas and has the skills to execute them, but also demonstrates an ability to create new methods and frameworks to solve novel problems.

Conclusion
There is a growing recognition that "coding is the new literacy" [31]. Most technology researchers agree that introducing coding to students at an early age is considered a long-term investment in bridging the skills gap between technology demands of the labour market and the availability of people to fill them [32]. The Technacy Developmental Framework as presented in this paper can be applied to the Australian Digital Technologies subject or any computing course which demands students to have skills in programming and computational thinking. This Framework provides a method in which to map and describe how a student responds to increasing demand associated with learning computer science content such as developing programming skills. This paper asserts that there is promise in combining cognitive development research and computational thinking theory with Technacy Theory. The Technacy Innovation Chart offers considerable opportunities for further research when combined with computational thinking concepts, including offering a common dialogue to describe the learning taking place. 24

Introduction and Background
The study extends a series of computerised examination (e-exam) studies following an Australian Government funded project on the topic [1]. The results of previous studies are available [2][3][4][5]. The Australian-based work is in company with a number of similar projects underway in other countries [6][7][8]. This study aimed to explore the viability of re-purposing biology teaching laboratories for use as an examination (exam) space for students undertaking STEM courses. As a result of this circumstance, the definition of an e-exam in this study was simplified in a pragmatic manner over that espoused in previous studies in the series where the focus was previously on the development of authentic assessment capabilities [9,10] in the exam room [3,11,12]. The previous studies explored ideas of authentic tasks in exams and the use of bring-your-own-device (BYOD) [3,12]. While the use of BYOD is beyond the scope of this study, prior work [13,14] included exploration of issues relevant to the student experience of computerised testing. In this case study, we limited the e-exam to an event supervised by human invigilators undertaken on controlled, institution-owned computers with the exam content served via the quiz functionality of a learning management system (Moodle). This study was situated as the first phase of a broader approach to implementing digital exams at the host university. As such, this study sits within the context of a broader digital exams deployment (see Fig. 1) that includes off-campus via remote invigilation, on-campus in classrooms using BYOD, in exam halls and, as in the case of this study, in computerequipped spaces such as a laboratory where computers were already being used for formative learning tasks. We next explore literature related to user acceptance of technologies, with a focus on the role technical reliability can have in students' attitudes towards the use of computerised examinations.

Literature
As we have previously explored in prior work [3,4], it is the perceptions of users of a technology that play a large part in the acceptance of that technology by individuals and organisations. We review those elements of the literature that are relevant to this particular study. The information systems literature [15,16] has long regarded ease of use and usefulness as being key issues when it comes to people accepting new information technology. It is acknowledged that while a range of other variables can be taken into consideration [17], this study will focus on user perceptions. Prior work undertaken in the area of student acceptance of e-exams has shown that relevance, ease of use, reliability and security have been key concerns [3,4,13,14,18]. Given that students are those with the most at stake with a change to e-exams, it behoves educational institutions that their views are taken into consideration.
Researchers at the University of Bradford in the United Kingdom (UK) undertook a survey of students' opinions following their use of the online quiz-based testing tool "QuestionMark Perception" [18]. The range of issues that were canvased served as a starting point for the survey tools used in this study. Subsequent work in Australia has included a survey [13] that sought to tap into students' preconceptions of e-exams with follow-up work at three universities also examining students' opinions after they had taken an e-exam [3,4,14] using similar technologies.
In the aforementioned studies, it became apparent that the issue of technical reliability had a significant place in the minds of students. The news media in Australia recently publicised a high profile example of this risk playing out [19,20], in which a high stakes medical board e-exam failed half way through resulting in the exam being cancelled. In the researchers' own work, it was previously reported [13] that students yet to undertake an e-exam placed the fear of technology failure high on their list of concerns. Fortunately, this concern was shown to dissipate after students had undertaken one or more e-exams that went well [3][4][5]14].
In seeking to minimise outages, modern information system designers endeavour to build in technology failure mitigation measures. In terms of an e-exam system, this can include adding a frequent auto-save trigger to an online quiz system (as per Moodle quiz), adding redundant components into the network infrastructure such as hot swappable routing components, multiple network connections and placing the server on scalable cloud infrastructure. Yet the cost increases of such measures, or the lack of control over all parts of the connection chain, can mean that there is less than complete coverage for all components in the chain between the exam candidate and the exam server. Such measures may also have their limits in terms of how effective they will be in a crisis, when a highly time-sensitive and stressful event such as an exam is being run. The extent of the risk and fallout is further highlighted by lists [21] and high profile cases [22]. When client-server software systems are used, not all fail-over measures are immediate and a total severing of the connection can still result in an unscheduled end to the exam or a delay to the start of the exam [23]. The latter also occurred in the current study. However, in our series of work, it remained to be seen what the nature of the impact on students' feedback about their e-exam experience would be in the event that technical issue did occur that impacted the running of the exam event. This paper explores just this scenario.

Study Context
The study draws on ideas and techniques previously developed as part of a completed Australian national project on e-exams using bring-your-own (BYO) laptops [1][2][3][4]12]. The same user survey instrument was used in this study to enable comparisons to the previous cited work (ibid). This study differs because it examined the use of a lockdown browser "Safe Exam Browser" (SEB) installed within university-owned computers located on biochemistry laboratory benches, instead of the alternative operating system boot method used on BYO laptops utilised in the previous studies (ibid). These laboratories are the same spaces used for scheduled practical classes for the course where formative learning activities take place. This study was undertaken at the University of New South Wales, Australia within an undergraduate second year biochemistry course (subject). The exam trials were carried out using online Moodle quizzes within the institutional learning management system (LMS). SEB provided a key capability to allow white-listed access to selected resources and software tools. This feature allowed us to go beyond a typical 'locked-down quiz' to allow controlled access to students' prepared notes in a digital format during the exam. In this case, the online LabArchives service [24] was used as the host environment for students' notes. Assessments were undertaken in-class, under supervised conditions at the mid-point of the term and at the end of the term as a final exam. The practice session did not involve any grading, while the mid-term test and final exam were each weighted at 25% of the course. Multiple classes were split over four laboratory rooms with morning and afternoon sessions run to allow all students to use a computer. Questions in both exam sessions were in the format of selected response (multiple-choice questions) and an extended essay style response. The study was run in conjunction with the third author who was the course coordinator and lead teacher in the classes in which the e-exam trials were conducted.

Research Questions
Given the issues raised in the literature review and the circumstances in which the study was carried out, we focus this paper on two main research questions: 1) "Would the approach of using SEB for Moodle quiz-based exams undertaken within a biochemistry laboratory setting be acceptable for students in terms of perceived suitability, ease of use, reliability and security?", and, given the events as they transpired, 2) "What was the impact on these measures of any technology problems that occur during an exam event?" In seeking to explore the impact of technical problems, the severity of the problem is worth considering. A problem that is catastrophic will prevent the exam from starting or will cause a complete break in proceedings. This would negate any data collected, because the students would not have experienced a complete exam. However, problems that occur that are moderate and cause inconvenience rather than a complete failure mean that students experience the full exam procedure. This provides an opportunity to explore the extent to which such technical difficulties influence students' perceptions of various aspects of the e-exam process.

Method
This study utilised a case study approach that sought to capture students' perceptions via a quantitative survey. Procedures followed those used in prior work [4,18]. The procedure used is outlined in Table 1 and was designed to be similar to that described in previous work [3,4] in order to facilitate comparison. This study included a short series of events that comprised one practice session, then two weeks later one mid-term test and four weeks onward one final exam. The study was conducted in term 2 in 2019 involving 215 undergraduate university students.
Previous experience has shown that a zero stakes practice run was important in providing an opportunity for students to become familiar with e-exam procedures and the software environment before undertaking a real e-exam. The ungraded practice session was run in class time with all students participating. All students were asked to use a computer for the exams, and therefore this study contrasts to another study [4] where students were provided a choice to type or hand-write their exam responses. However, university special consideration (alternative assessment) was available to students in accordance with university procedures that would have allowed an opt-out. Students were able to preview the exam process by following the 'digital exam start guide' printed instruction sheet. Students could complete the practice questions that used the same format as those presented in the real exam. Data were collected using observation and a survey of students' first impressions (Pre-survey). Following the session, data analysis of the surveys was carried out to detect any concerns 2. Real exam session(s) For the mid-term and final exams: benches were set up with a paper copy of the 'digital exam start guide' and post-exam survey. Each student was provided with a desktop computer equipped with a wired network connection and the Safe Exam Browser software. All students in the course were asked to use a computer to undertake the exam 1. Students entered the room and were seated at a bench 2. Students could read the printed instruction sheet 3. Once logged into the desktop computer, students then logged into the LMS using a Chrome browser and then clicked on a link leading to an SEB setting file. This then launched Safe Exam Browser. Students were required to again login to the LMS and were then taken to the Moodle quiz start page 4. When all students in the room were ready to begin, the invigilator announced the start of the exam by providing the Moodle quiz password. This enabled all students to start at the same time. The Moodle quiz was set to auto-save responses each 30 s 5. Exam end: students used the Moodle quiz submission button and then exited SEB. The quiz was set to automatically submit if the quiz timer expired 6. Students were requested to log-off from the computer 7. Students completed the post-exam survey before leaving the room 3. Grading In the following week, the teacher did the grading. Students were given grades and feedback comments. Surveys results were analysed A sub-set of the survey items selected for analysis is displayed in Table 2. These questions directly asked for students' perceptions of the issues detailed in the research question. These items were Likert-style items asking for agreement ranging from 1 equating to strongly disagree to 5 strongly agree with 3 being neutral. The analysis procedure followed that of a previous study on e-exams [4]. The responses were analysed using SPSS v24 with an alpha level of .05. Likert items pertaining to students' perceptions were treated as non-parametric [25]. This stance was also used in another study [18] when they analysed students' perceptions of a quiz-based e-exam system. Mann and Whitney's U test [26] was used to test the variance between groups (morning versus afternoon) on Likert items. When comparing paired Likert items (Post 1 and Post 2), a Sign Test was used [27].
It is important to note that specific conditions prevailed during this pilot. As such, the results are only descriptive of this group and do not represent a generalised view of e-exams by university students. Like Dermo [18], we take the position that statistical tests serve as a tool to summarise the body of opinion from this particular group and, as such, we do not present this as a search for an objective truth regarding e-exams.

Findings
This study involved 215 undergraduate students, 60% were female and 40% were male. The total number of students varied at each stage by about 2% because not all students participated in each event or responded to each question. Approximately 65% of the students were undertaking a computerised exam for the first time.
Following each exam, the respondents were asked to reflect on the technology approach used for the exam. The results for agreement items pertaining to perceived suitability, ease of use, reliability and security are displayed in Table 2. Students were also asked if they would recommend the approach to others (also shown in Table 2). The majority of items received positive agreement, most with mean agreement ratings at 3 or greater out of 5 (strongly agree). The sentiment within this group overall was relatively uniform as evidenced by the small standard deviations (Table 2), although some divergence is addressed later. Parametric statistics are shown here to provide clarity to the reader in terms of what the responses were for each item at each stage.  The impact of a technical issue in the form of a substantial drop in the performance (i.e. a slow-down in server response times) of the LMS resulted in disruption to the start of the exams held during the last event (Post 2). Exam candidates were asked: "Did you experience any technical difficulties during this exam?" Those who indicated 'yes' during the mid-term exam were 12% (27), while following the final exam this increased to 43% (88). It should be noted that all those who undertook the exams were able to successfully submit the exam. The disruption mainly impacted the ability for candidates to enter into the exam quiz, but once candidates had begun the quiz, further impacts were not noticeable.
The differences in students' opinions between the mid-term and final assessment events also demonstrate the impact of the technical issue. The results of a Sign Test are shown in Table 3. Note the requirement of a normal distribution of differences was not met for a Wilcoxon Signed Ranks Test [28] to be used because Shapiro-Wilk Tests [29] for each pair were all <.000. The items related to ease of use, reliability and security were significantly marked down by students in the second exam event. It is worth noting that the infrastructure problems impacted the morning and afternoon sessions of the final exam to a different extent. A comparison of the morning and afternoon groups was made using a Mann-Whitney Test (Table 4). Note, means are shown for reader clarity as to the direction of differences. Results show a statistically significant divergence of opinions when looking at the final (Post 2) event. The differences between morning and afternoon sessions at the practice (pre-) event and mid-term event (Post 1) were generally not significant, although the afternoon group tended to provide lower ratings across all events.
The implications for practice in dealing with technical issues during e-exams are considered in the following section.

Discussion and Conclusion
Despite the technology issues encountered in the final exam session, the introduction of e-exams was a success in the context of the use of bench-top computers in a biochemistry course. Generally, positive responses (i.e. above 3 on the 5-point scales) were received from students across the survey questions. What did become apparent was that the technical issue that was encountered during the final exam event did have a statistically significant impact on student ratings of reliability, ease of use and to a lesser extent on their perception of the system security. Their recommendation of the computerised exam approach was not impacted when comparing Post 1 and Post 2 events (Table 3). However, when focusing on the Post 2 event (Table 4), it was found that there were significant differences between the opinion ratings of those in the morning and afternoon groups, with the afternoon session experiencing more severe technical problems than the morning. The extent of the impact is worth reflecting upon, given the relatively minor extent of the technology problem. This was no more than an inconvenience, causing a delay to the start of the exam. The interruption did not prevent the students from undertaking and submitting the exam, yet it still had a statistically measurable impact on students' perceptions. It is likely that the stress of an exam event means that students are extra sensitive to anything that is not perfect on the day.
When considering the possibility of mediating factors in the findings, it is possible students may have regarded the final exam as being higher stakes than the mid-term and that may have also contributed to an elevated level of pre-existing stress in the second event. However, the study design was such that both exam events (Post 1 and Post 2) were run in a near-identical manner, sessions were led by the same team members, with the same groups participating in each. Yet, the greater impact of technology problems in the afternoon session appears to coincide with the statically significant drop in that group's ratings on the survey. Therefore, this suggests that the technology problems did influence the changes seen in the students' responses to the survey following the second event.
The particular technical incident occurred the day following an overnight upgrade that saw additional modules added to the LMS. Overall, the experience suggested that the mitigation measures in place for the infrastructure verified it was capable of hosting an e-exam event even when server performance degradation was being experienced.
The findings from this study show that technical performance of an e-exam system does impact exam candidates' perceptions of the system and therefore their acceptance of an e-exam approach. This highlights that test administrators and teachers need to have open communication with the technical support department and with students about how problems will be handled in order to minimise any additional stress for students.

Motivation and Aim
Technologies shape today's society and the basis of these technologies is computer science. The demand for workforce in the IT industry has risen strongly in Germany in recent years and it is to be expected that this process will continue [1, p. 11]. Therefore, the computing field provides attractive future job perspectives for today's students. Additionally, there is a skill shortage in the computer science industry in Germany [1, p. 12]. The proportion of students with study courses related to computer science is rising only slowly [2]. Furthermore, women are strongly under-represented both in the computer science industry [1, p. 15] and in study courses related to computer science [2]. Individual beliefs and stereotypes about computer science are considered to be one explanation [3], so that targeted action against wrong conceptions of people in this field and the field itself are needed. In this paper, we describe the design and results of an online survey (which took place in North Rhine-Westphalia) that aimed to identify students' stereotypes of people in the computer science field as well as influencing factors in order to provide the basis for developing targeted actions.

Theoretical Background and Related Work
The reasons for the slowly growing number of students as well as the lack of specialists and in particular the under-representation of women in the IT field are very diverse. Ashcraft et al. divide the reasons for the under-representation of women in "Formal and Informal Education", "Families, Communities, and Role Models", "Peer Influences", and "Media and Popular Culture" [4, p. 17].
In their last category, they include aspects such as the media portraying the computing field as masculine and "geeky" and people in computer science as "geeks without social skills doing boring and solitary jobs" [4, p. 27-28]. Osunde et al. group the factors into "Biological differences (Essentialist Theory)", "Structural factors", and "Socio-cultural factors" [5, p. 18]. The latter includes conceptions of computer scientists and the ones found in this study are negative and imply that computer science is for "nerds" or men only [5, p. 19]. Another literature review by Sanders grouped the factors amongst other aspects into "Age, Stage and Pipeline Issues" as well as "Societal Influences" [6, p. 2], in which the media are addressed as a factor. An analysis of computer advertising, the Internet, television and movies revealed gender stereotypes of people in technical roles [6, p. 5]. According to Leyens et al., stereotypes are defined as "shared beliefs about person attributes, usually personality traits, but often also behaviours, of a group of people" [7, p. 11]. Stereotypes are thus generalised judgements and ideas, which are assigned to a group [8, p. 315]. The term focuses on the similarities of group members, while differences move into the background. Often, stereotypes are negative shared beliefs. In rarer cases, they could also be positive or have positive components [8, p. 315]. Since stereotypes often played a role in the above phenomena, we now look into studies focusing on stereotypes.
Studies dealing exclusively with stereotypes towards people in the field of computer science show that students have stereotypes towards people in this field and that these stereotypes can have an influence on their attitudes towards computer science. Negative attitudes can then prevent students from working towards a future career in this field while positive conceptions could encourage them. The results of the first part of a two-part study by  with around 300 participants showed that stereotypes exist for both females and males. The evaluation of the participants' descriptions of people in computer science showed in particular that they are described as having features that are incompatible with the female gender role. Participants often mentioned that these people lack interpersonal skills or are singularly focused on computers [3, p. 63]. In addition, it has been found that women, who have not taken a computer science class, have more often mentioned one of the stereotypical categories in describing people in the computer science field than those, who have had at least one computer science class [3, p. 64]. The results of the second part with around 50 participants showed that stereotypes can affect attitudes towards computer science (especially for women) and that media can have an impact on stereotypes [3, p. 66-67]. Another study with 100 female students indicated that an interaction with a stereotypical person in the field of computer science has a negative effect on the interest in computer science for women both in the short and in the long term [9, p. 74-75].
The above-mentioned studies give indication that stereotypes can influence both the skill shortage in the computer science industry as well as the underrepresentation of women by having a negative impact on their interest. There are also indications that perceptions can also be influenced by the media and by prior computer science education. Since the presentation of computer science in the media is also a subject of change, the results found in earlier studies and in other countries might not be fully applicable to the situation in Germany, so that we decided to investigate German school learners' stereotypes about computer science. We decided to survey 10th and 11th graders, as they may have already thought about future career plans. Summing up, the main research question of the study described in this paper is: Which stereotypes towards people in the field of computer science can be identified among German upper-secondary school students? The question is split into the following sub-questions: 1. Which specific ideas of people in the field of computer science do the students have? 2. Which influence do factors such as gender, previous school experience in the field of computer science, interest in computer science, or the personal environment of the students have on these ideas?

Data Collection
The study was carried out at four upper secondary schools in the Dusseldorf area (Germany, North Rhine-Westphalia) in May 2019 using an online questionnaire 1 .
The questionnaire was structured as follows: The first part contained openended questions about students' understanding of computer science as a discipline or a working field ("Describe your conception of the subject or the industry'computer science'.") and their ideas of people in the field of computer science ("Describe a person who deals with computer science. What do you think are their characteristics?"). The second part of the questionnaire contained closed questions. On the one hand, the students were asked to evaluate predefined statements about people in the field of computer science (e.g.: "People in the field of computer science are intelligent."), and, additionally, they were given a predetermined set of characteristics to choose from which, in their opinion, most likely applied to people in the field of computer science. The third part of the questionnaire was designed to collect students' demographic and sociocultural information in order to build subgroups based on this information. The subgroups were built depending on the gender information, whether the students had received computer science education at school ("How many years have you had computer science lessons or were in another computer science group until now?"); their interest in computer science ("Are you interested in computer science?"), and if they had people in their lives that worked or studied in the computer science field ("Do you have family members who deal with computer science? Do you have acquaintances or friends who deal with computer science?").
Both the given statements and the given characteristics are based on a theoretically derived category system, which consists of 14 categories. Categories C1 to C6 are based on the categories that are described by Cheryan et al. [3, p. 59-60], categories C7 to C12 were created by analysing computer science related job descriptions and extracting frequently required characteristics. C13 is built on the JIM study 2018. It indicates that digital games "play a central role in the media life of adolescents" [10, p. 79]. Finally, category C14 is based on the results of the paper [11], which shows that many students relate computer science with mathematics and science. Table 1 gives an overview of the described category system. The bold descriptions will be used as abbreviation in the rest of the text.

C14 Good skills in mathematics and science
A total of 55 students from four different courses and from four upper secondary schools in the above mentioned region were interviewed. The relevant sample for the evaluation was 52 students after excluding three answers due to incompleteness. The students were 15 to 20 years old and attended the 10th or 11th grade at the time of the survey. 46% (24) of the students were female, 33% (17) male, and 21% (11) made no indication of gender. 38% (20) of the students had previous school experience in the field of computer science, 35% (18) of the students expressed interest in computer science, and 69% (36) of the students had family members or friends, or acquaintances that deal with computer science.

Evaluation
Overall, the following evaluation focused on the open-ended question about the ideas of people in the field of computer science and the predefined statements about people in the field of computer science. The questions are evaluated on the one hand for the entire sample and on the other hand for the above mentioned subgroups in order to be able to identify any differences depending on the gender, the previous school experience, the interest, or the environment of the students.
For the evaluation of the open-ended questions the deductively derived category system described above was used (see Table 1). After viewing the data, additional categories were added inductively [12, p. 85]. Once all statements were assigned, the categories were sorted according to the frequency of their naming. The evaluation of the predefined statements was done as follows: The arithmetic average i.e. mean was calculated (depending on the distribution and weighting of the answer options per statement). The weighting of the answer options takes into account the formulation of the statement, since some statements were made positive and others negative. A high mean implies that students strongly agree with the category to which the statement belongs, while a low mean implies that students agree little. The lowest possible value is 1 and the highest possible is 4. Table 2 gives an overview of the results of the open-ended question about the ideas of people in the field of computer science. It can be seen that in addition to the above-mentioned deductive category system (categories C1-C14), the categories M1-M6 have been added inductively, as some of the students' statements could not get assigned to the existing categories.

Open-Ended Question
The results of the entire sample can be described as follows: The most frequently identified categories are C4 (Special external feature) with 31% (16) and the categories C1 (Intelligent) and C14 (Good skills in mathematics and science) with 29% (15) each. In addition, the categories C5 (Lack of social skills, introverted, or not empathetic) and C9 (Logical thinking, planning-oriented, structured, and organized) are frequently identified with 23% (12) each and the categories C3 (Interested in new technologies) and C6 (Focused on the computer) with 19% (10) each. Not identified are the categories C7 (Work independently), C8 (Team player), and C10 (Good communication skills). The other categories are identified between 2% (1) and 17% (9). It can be stated that in particular, the categories that are known from previous studies (C1-C6) can be identified in the answers very often. With the exception of category C2 (Male), these are identified in at least 19% (10) of the students' answers. When comparing the results of the two subgroups of self-identified male and female students, strong differences in the responses can be seen for the categories C9 (Logical thinking, planning-oriented, structured, and organized), C5 (Lack of social skills, introverted, or not empathetic), and M5 (No special external features). While 33% (8) of the female students describe people in the field of computer science as logical thinking, planning-oriented, structured, and organized only 6% (1) of the male students do so (C9). 35% (6) of the male students state that people in the field of computer science are lacking social skills, are introverted, or not empathetic. Only 13% (3) of the female students mention aspects in this category (C5). 25% (6) of the female students state that people in the field of computer science do not have any special external features while only 6% (1) of the male students do so (M5).
When comparing the results of the two subgroups of students with and without prior educational background in computer science, the strongest differences can be seen for the categories C14 (Good skills in mathematics and science), C13 (Interested in computer games), and C4 (Special external features). While 38% (12) of the students without school experience in the field of computer science describe people in this field as with good skills in mathematics and science, only 15% (3) of the students with school experience do so (C14). A similar difference can be seen for the category C13. 22% (7) of those students without school experience describe people in the field of computer science as interested in computer games, while none of the students with school experience do so. In addition, 38% (12) of those without school experience describe people in the field of computer science with special external features, whereas only 20% (4) of those with school experience mention such aspects (C4).
The results for the subgroups depending on the interest and the environment of the students also show interesting differences and similarities and can be seen in Table 2.

Closed Questions
All calculations have been done with the R environment [13], including the packages psych [14] and pastecs [15]. Table 3 gives an overview of the results of the predefined statements. The results of the entire sample can be described as follows: The category with the highest mean of 3.4 (SD = 0.69) is C7 (Work independently). Similar high means of 3.38 (SD = 0.66) and 3.34 (SD = 0.58) are to be found in the categories C3 (Interested in new technologies) and C6 (Focused on the computer). Students overwhelmingly tend to agree that computer scientists fulfill these stereotypes. The lowest mean is assigned to category C10 (Good communication skills) with a mean of 2.31 (SD = 0.83). All other categories have a mean between 2.5 and 3.31 (0.54 < SD < 1.02). Looking only at the categories C1-C6 resulting from the mentioned study, it can be seen that categories with a positive connotation such as C3 (Interested in new technologies) or C1 (Intelligent) are in the upper half of the descending order, while categories addressing the often portrayed computer scientist such as C5 (Lack of social skills, introverted, or not empathetic), C2 (Male), and C4 (Special external features) are in the lower half. Of particular interest might be the correlations between the categories. The highest positive correlation of r = 0.58 can be found between categories C8 (Team player) and C10 (Good communication skills). Interestingly, the same category C8 is moderately negative correlated to C4 (Special external features) with r = −0.55, C5 (Lack of social skills, introverted, or not empathetic) with r = −0.58 and C6 (Focused on the computer) with r = −0.63. Due to the presence of a large number of underlying categories, the items are not expected to contribute in the same way to the result. Therefore, coefficients for neither τ -equivalent nor τ -congeneric reliability were calculated. Furthermore, Table 3 shows that the five most agreeable categories in the whole group of assessed students (C7, C3, C6, C14, and C1) are the same for the subgroups of self-identified female and male students. They merely differ in their order. Female students mostly agree with categories C14 (Good skills in mathematics and science), C7 (Work independently), C3 (Interested in new  Table 3, the means are all very close with a range between 3.19 and 3.5. Interestingly, the most agreeable aspect of one group scores only fifth place in the other group and vice versa. As tempting as it might be to infer anything from those numbers, a Wilcoxon rank sum test provided neither differences in the distributions for category C14 (W = 174, p = 0.38) nor for C6 (W = 250, p = 0.21).
Comparing the overall differences in the means of the categories for the female and the male students, large differences can also be observed in the categories C4 (Special external features) and C11 (Motivated, hardworking, and eager to learn). In neither case did the Wilcoxon rank sum test provide a significant difference in the distributions. This could be a result of a significant part (about 21%) of participants not indicating any gender. Therefore, the already small sample of 52 individuals gets further reduced to 41 and then split into groups. Table 3 also displays the differences of students who have already received computer science education or have participated in an extracurricular lab class. For the students with school experience in the field of computer science, the category with the highest mean is category C7 (Work independently) with a mean of 3.6 (SD = 0.6). The categories with the second and the third highest mean are the categories C3 (Interested in new technologies) with a mean of 3.35 (SD = 0.67) and C6 (Focused on the computer) with a mean of 3.3 (SD = 0.64). For the students without any school experience in the field of computer science, the category C14 (Good skills in mathematics and science) has the highest mean of 3.44 (SD = 0.62). The categories C3 (Interested in new technologies) and C1 (Intelligent) rank second and third with means of 3.41 (SD = 0.67) and 3.38 (SD = 0.55). Again, categories can be found in which the means of the two subgroups differ strongly. Examples are the categories C14 (Good skills in mathematics and science) (3.44 vs. 3.1) and C7 (Work independently) (3.6 vs. 3.28). Yet, as in the former subgroup analysis regarding gender, the Wilcoxon rank sum test provided no significant differences between the groups, although the result for category C14 (W = 231, p = 0.06) could be indicative and should be investigated in further studies.
The results for the subgroups depending on the interest and the environment of the students show interesting differences and similarities, can be seen in Table 3.

Discussion and Conclusions
Results of the first and second part are very different for some categories, while they are very similar for other categories. For example, in the whole group category C4 (Special external features) is the most frequently mentioned in the first part of the survey, but simultaneously one of the least frequently mentioned in the second part. Notably, some students chose different open-ended answers which are in contrast to their selected statements of the closed questions. To obtain more insights into this opposing result, further investigation is needed.
Overall, the results show that students' perceptions of people in the field of computer science are very diverse and individual. This study provides an overview and basis for further investigations. First tendencies regarding rarer and more frequent ideas can be recognised. Compared to the literature (see Sect. 2) this study shows similar results on students' perceptions. In particular, the results of the open-ended questions make it clear that many features have been named that fit the image of the "computer nerd". Additionally, first tendencies for the fact that factors such as gender or computer science education, but also the interest in computer science or the availability of people in the social environment who work in the field of computer science, have an impact on the students' ideas towards people in this field. To make a more general statement in this regard, the sample of subgroups is insufficient. Nevertheless, the results show that female students seemingly do have negative connoted images of people who deal with computer science.
To get a closer look, further studies could focus on factors separately at a time to study its influence more closely. Especially, studies on female students' conceptions on a larger group with different ages should be considered. Concerning targeted actions to deliver a positive and realistic picture of computer scientists among youth, there is still a long way to go. Mandatory and positively connoted informatics education for all might help to create a more realistic picture of the computer science industry in order to reduce problems such as the skill shortage in the industry or the under-representation of women in this field.

Using Classroom Practice as "an Object to Think with" to Develop Preservice Teachers' Understandings of Computational Thinking
Deirdre Butler (B) and Margaret Leahy Dublin City University, Institute of Education, Dublin, Ireland deirdre.butler@dcu.ie

Abstract.
Irrespective of what approach is taken to the development of computational thinking, or at what age it is introduced, the teacher is central to ensuring that the children they work with develop computational thinking. It is therefore essential that their teachers are adequately prepared to include computational thinking as part of their pedagogical classroom practices. Moreover, it is argued that this preparation should begin at pre-service level. Adopting a constructionist perspective of learning, this paper presents and discusses the findings from research that investigated preservice teachers' understandings of computational thinking, having completed a specialism in digital learning, the final activity of which entailed using computational tools with children in the classroom as part of a primary school science curriculum. Findings indicate that working with the children in the classroom helped the preservice teacher develop their own understandings of what computational thinking (CT) looks like "in action", enabling them to reflect more deeply on the fundamentals of CT and on how to use CT in their own classroom practice as qualified teachers.

Introduction
Computation and the development of computational thinking at primary school level is an area of research that is still in its infancy [1]. In Ireland, as elsewhere, current debates centre on whether there should be a computer science curriculum at primary level with an explicit focus on computational thinking [1,2]. Does computational thinking become a battle cry for coding in primary education [3]? Or should young people be able to develop and use computational thinking concepts in their problem-solving activities as part of subject areas other than computer science, e.g. [4]? Since 2018, the national curriculum body in Ireland, the National Council for Curriculum and Assessment (NCCA), has begun to investigate how computational thinking can be introduced. Irrespective of what approach is taken to the development of computational thinking, or at what age it is introduced, the teacher is central to ensuring that the children they work with develop computational thinking. If young people are to develop computational thinking, it is essential that their teachers are adequately prepared to include computational thinking as part of their pedagogical classroom practices [5,6]. Moreover, it is argued that this preparation should begin at preservice level [7] so that preservice teachers not only develop understandings of computational thinking but are also introduced to ways they can design learning opportunities for their students to develop computational thinking [8].
The study reported in this paper contributes to this debate. Adopting a constructionist perspective of learning, it presents and discusses the findings from ongoing research, specifically focusing on one group of final year Bachelor of Education (B.Ed.) students who have completed a major specialism in digital learning. The aim of the research was to investigate preservice teachers' understandings of computational thinking having completed their specialism in digital learning, the final activity of which entailed using computational tools with children in the classroom as part of the primary school science curriculum [9].

Review of Underpinning Literature
Underpinned by constructionist learning theory and computational thinking perspectives, the primary focus of the major specialism in digital learning is the development of preservice teachers' understanding of the theoretical and practical concepts of computational thinking and coding.

Constructionism
An extension of constructivism, constructionism is both a theory of learning and a strategy for education [10]. Originating in the work of Seymour Papert in the 1980s, constructionism shares constructivism's connotation of learning as building knowledge structures but goes beyond constructivism by emphasising that learning is facilitated by constructing tangible artefacts or objects, which can then be shared and discussed with others [11]. As such, constructionism sees learners as active builders of their own knowledge and asserts that people learn with particular effectiveness when they are engaged in constructing personally meaningful artefacts, "whether it's a sand castle on the beach or a theory of the universe" [11, p. 1]. These artefacts which Papert [12] describes as "objects to think with" (p. 12) support the development of concrete ways of thinking and learning about phenomena. The ability to manipulate these objects, to repeatedly make adjustments and refinements or experiment with them to see how they work lends itself to a concrete style of reasoning. This, Papert argues, changes the process of learning to one which is iterative and cumulative, embracing both planning and bricolage styles. Turkle and Papert [13] refer to this 'validity of multiple ways of knowing and thinking' as "an epistemological pluralism" (p. 129).
Constructionism also draws attention to the social nature of learning, noting that activities such as making, building or programming through which the learner produces artefacts that others can see and critique provide a rich context for learning. The artefacts are a means by which others can become involved in the thinking process while, at the same time, the learner's thinking benefits from multiple views and discussions [14].
In this way, the artefacts or 'objects to think with' provide a link between sensory and abstract knowledge, and between the individual and the social worlds. Shared knowledge is constructed when artefacts and shared understanding are coupled through cycles of representing and interpreting [15]. Through engaging in conversation around their own or another's artefact, the development of a shared understanding is enabled and the foundation for new understandings is provided [16]. Constructionism thus implies a process of building, both in the sense of building artefacts and building new understandings.

Computational Thinking
The concept of computational thinking also originates in the work of Papert [12], when he introduced the "idea of the computer being the children's machine that would allow them to develop procedural thinking through programming" [17, p. 2]. However, it was not until 2006 and the publication of Wing's [18] seminal article that the concept of computational thinking came to prominence. Describing computational thinking as a "fundamental skill" for everyone, Wing defined it as the thought process of formulating and solving problems by "drawing on the concepts fundamental to computer science" (p. 33) when "equipped with computing devices" (p. 35). Within this broader context, she outlined the central components of computational thinking, including algorithms, abstraction, decomposition and automation, all of which can be found in many contexts and disciplines and which assist learners in developing problem-solving skills. Researchers have continued to put forward a number of definitions of computational thinking as they built on the work of Wing but have failed to agree an accepted definition of computational thinking [e.g. 1,4,7,17,[19][20][21]. However, it is broadly accepted that computational thinking is a thought process that utilises the elements of abstraction, generalisation, decomposition, algorithmic thinking and debugging, i.e. detection and correction of errors [1] (see Table 1).  [22], for example, refer to these dispositions as computational practices and computational perspectives. Computational practices are the "problem solving practices that occur in the process of programming" (p. 53) and include: iterative and incremental, testing and debugging, reusing and remixing, and abstracting and modularising. Computational perspectives relate to the "student's understandings of themselves, their relationships to others, and the technological world around them" (p. 53).
A number of implementation frameworks have also been put forward. While most of these frameworks focus on post-primary and third level, a small number have been presented for primary level [e.g. 1,22,23]. Across these frameworks, one of the most frequent methods of providing the opportunity to engage in computational thinking in primary classrooms is through the use of programming languages such as Scratch [e.g. 22]. Although scholars advocate the introduction of computational thinking early at primary school level [24], to date, little attention has been accorded to preservice primary school teachers. While there have been some examples of planned structured teacher preparation programmes [25][26][27], and it is widely accepted that the development of computational thinking for preservice K-8 teachers should be integrated with pedagogical content knowledge [27], framework or models that focus explicitly on preservice teachers are not yet available [28].

Computational Thinking in a Constructionist Learning Environment
From a constructionist perspective, computational thinking can be thought about in much the same way as Papert viewed computer programming; that is, computational thinking is both a skill to learn and a way to learn -"to create, discover, and make sense of the world, with digital technologies as extensions and reflections of our minds" [29].
Computational tools can be a powerful medium for creating contexts for constructing knowledge. However, in keeping with Papert's idea of engaging with "powerful representations", what is essential to consider when designing a learning environment is not so much what programming language and/or computational materials to use, but what personally meaningful ideas the programming language and materials can enable the children to develop and how those ideas will develop computational thinking and form new ideas about the subject area (e.g. mathematics, science). Activities and learning situations should be developmentally appropriate for the children and grounded in meaningful contexts [14].
Drawing together the ideas presented in the literature review, the authors ensured in their design of the major specialism to provide immersive learning experiences for the students which were underpinned by constructionist principles. The students were introduced to a range of expressive computational materials (e.g. Scratch and Lego WeDo) through which the students were scaffolded to progressively develop understandings of computational thinking. As module facilitators, the authors continuously worked alongside the students using a range of pedagogical strategies and practices to help them construct and reflect on their emerging understandings of computational thinking while linking these to classroom practice.

The Context and Participants
The study was conducted with 18 preservice teachers who had taken a major specialism in digital learning as part of their Bachelor of Education (B.Ed.) programme at the Institute of Education, Dublin City University (DCU IoE). The aims of the major specialism were to: expose students to the concepts of CT, raise awareness of the definitions; enable them to build their own interpretation based on experience, literature reading, class discussions, and hands-on learning experiences using a range of computational materials; and finally, to make learning concrete and relevant through the completion of assignments that related to their future teaching and the primary school curriculum. Modules were accordingly designed to include a range of computational materials and contexts to explore strategies that support interest-driven, project-based, collaborative approaches to learning. The final module, 'Designing & Learning with Digital Technologies', was designed to enable students to translate their learning into practice [30]. To this end, students, in groups of three, were required to design a unit of work comprising three workshops of two hours' duration in which they introduced coding and computational thinking concepts using Lego robotic materials within the context of the Primary School Science Curriculum [9] to the 4th or 5th class (children aged 10-11 years) in local schools. The final module was also designed to provide the students with an opportunity to engage in research, thus providing "the foundation of their practitioner-based enquiry stance in the future" [30, p. 23]. As part of classwork, they had been introduced and/or revised participant observation as a data collection method, the deductive approach to data analysis and the ethics of classroom-based research (students had previously completed a research methods course as part of their B.Ed. programme). Students documented the classroom-based work as part of the module assessment

Methodology
The aim of the study was to consider from the perspective of a preservice teacher how their understandings of computational thinking had developed as a result of completing a major specialism in digital learning. The study adopted a qualitative approach. Data collection methods comprised document analysis and group interviews: a) Each of the 18 students wrote reports of their classroom-based experience as the module assignment. Students gave the authors permission to use these reports as part of the data corpus. b) Group interviews with student teachers took place at the end of the semester. Each group interview, approximately 20 min with four to five students, was carried out by the authors. The aim of the interviews was to probe the students' experiences, understandings and reflections in relation to computational thinking.
A typological analysis framework [31] was used as a starting point for analysis. The main themes from the literature review in relation to computational thinking were used to generate typologies, and initial data processing took place within these categories. The categories were re-examined after coding to ensure that they were justified by the data, or if the data excluded contained insights contrary to that proposed. Overall, decisions were driven by the data and, where necessary, new categories of adjustment added. This iterative process helped reduce the data to a small set of themes that then lent themselves to the final narrative.

Findings
Based on the analysis of the data, what was immediately evident was the preservice students' use of CT terminology, demonstrating their development of a "computational thinking language" [32]. They were able to define computational thinking and elaborate on computational processes and practices using classroom examples from their experiences and observations. They stressed how important it was that they had sufficient time and opportunity to engage in problem-based tasks using the computational materials to develop their understandings of CT, before they worked with the primary school children. This first-hand experience helped them understand challenges children could have and they were able to plan for and adapt their classroom practices accordingly. They also observed how computational thinking and, in particular, logical reasoning developed as the children completed the learning tasks. This included some of the practices of computational thinking such as experimentation and iteration, testing and debugging as well as the development of skills or perspectives such as collaboration, communication (expression) and critical thinking (questioning) [22]. As stated in one focus group: …it brought it to life I think, for me anyway… even when we were revising at the beginning just the different components of the computational thinking and being like oh yeah, that's what this is. But then going into the classroom and seeing, like linking the theory of it with children actually putting it into practice really got me to understand the definition of it more. (Focus Group 1) In her essay, Student#4 noted and tracked the development of logical reasoning across the three weeks, stating at the outset that "there was variance in the children's ability to think logically. The majority of children employed basic strategies to solve problems that arose when building and programming their models". However, by the end of the project, she documented the children using strategies of logical reasoning such as predicting, analysing, creating and correcting their algorithm and the build of their model: In the final week, the children were challenged to make their frogs move faster adapting both their builds and code. One child responded with … "Well, if you want to get a car to go faster you need more speed… the speed comes from the acceleration so…. we could try giving him (robot) more acceleration and he might go faster." This child analysed the problem linking it with their own knowledge and experiences "to predict the behavior of…. programs" [33, p. 9]. The child broke down the problem and thought logically about the elements that affect the speed of a car and transferred this knowledge to their own code. By saying, "might", this insinuated that this solution may not work and the child would have to think of an alternative solution, this implies that the child would be tinkering with the problem. (Assignment_Student#4) The preservice teachers observed that the children's self-confidence grew and although they were given challenging tasks, they persevered. They observed that a contributing factor to this perseverance was the design of the learning task (inquiry-based science focus/constructionist principles) combined with the choice of computational materials. Together, these facilitated the creation of a collaborative learning environment as the materials enabled children to externalise their thinking and allowed others to see a physical representation of their understanding. It thus enabled the children to give feedback to each other based on the model they had constructed and the program they had developed. The preservice teachers remarked that they began to realise the importance of using computational materials that enabled children to work with others so that they could benefit from the knowledge and insights that other people brought to the situation. As student teachers said: The findings suggest that clear design-based learning activities need to be planned to ensure that children can experience all the computational thinking skills needed when coding and building these projects. Lego Wedo was a significant factor in this project as it enabled the children to work in groups designing a project and coming up with a solution to a realistic problem. It created a meaningful experience for the children, motivating them to explore and learn, which was key to children's learning (Assignment_Student#4). … makes it a bit more authentic; you can see it actually in front of you. It maybe caters for more learners so that they're tactile and kinaesthetic, as well as maybe the visual… But it's right in front of you and it's actually happening and you can manipulate it. So, it's learning on the go, it's about always progressing. (Focus Group 1) Later in this class, a model failed to move forward in a race that was being conducted. Here another child offered a very strong reasoning for this issue, 'well, if my chain falls off my bike, it doesn't go, and didn't you say the pulley is like a chain? It could be the pulley is after falling off of it'. This observation ratified for us that the children were beginning to not only learn from each other, but make connections with their reasonings from their models to real life. This was also complimentary of a valuable point previously made by Cszmadia based on collaboration with peers to "evaluate each other's code, to isolate bugs, and to suggest fixes." (Assignment_Student#6) Prior to this classroom experience, the preservice teachers had been concerned as to how they could implement the "Digital Strategy for Schools" [34], with reference to the Digital Learning Framework [35] and accommodate the development of computational thinking into the curriculum. However, through their experiences of teaching with the computational materials, they realised the use of computational materials opened new ways of exploring scientific ideas while also providing opportunities for developing computational thinking. Having "an object to think with" as they tried to solve problems and challenges enabled the children they engaged with to develop understandings of science concepts and skills (e.g. inquiry, analysis, observing, predicting, experimenting and designing) but also computational thinking skills (e.g. algorithmic thinking, decomposition, debugging). As student teachers said: This research has shown how both the chosen DLF [Digital Learning Framework] outcome and CT [computational thinking] can be implemented through the science curriculum (Assignment_Student#16) … the teacher has a pivotal role in enabling children to develop this higher order skill-set, thus planning through subject integration is pivotal as suggested by the Irish DLF (Assignment_Student#11)

Discussion and Conclusion
To conclude, working with the children in the classroom helped the preservice teacher develop their own understandings of what computational thinking looked like "in action" and reflect more deeply on the fundamentals of computational thinking and on how to use computational thinking in their own classroom practice as qualified teachers. Their competence and confidence in using the computational materials developed and they were able to connect principles of constructionism and inquiry-based science with an informed understanding of the necessity to plan for the development of computational thinking in primary classrooms. Having the opportunity to engage in this module, rooted in classroom practice, enabled these preservice teachers understand, how computational thinking can be developed by embedding it within existing curricula, leading to the realisation that it is not a case of one or the other (i.e. computational thinking or subject content) but a means of combining both.
The insights gained from this study are particularly relevant for the design of teacher preparation programmes, indicating how computational thinking can be effectively embedded to combine theory and practice. This will ensure that concepts of computational thinking are not developed in a decontextualised manner but are embedded within the prescribed curriculum across a range of subject content in a relevant and meaningful manner.

Computing in English Schools
Reflecting a need in industry to provide for a growing demand of skills for the future, the subject of Computing in English schools has undergone significant change, since Computer Science (CS) replaced Information Communication Technology (ICT) in the curriculum in 2014 [1]. This echoes a pattern found in the subjects of science, technology, engineering and mathematics (STEM), the WISE Campaign (2019) found that just 21% of pupils taking the General Certificate of Secondary Education (GCSE) in CS were girls [2]. Reported reasons for this, range from girls not being interested in the subject, schools not offering it, and perceptions that it was not relevant for future plans, and/or too difficult [3]. Furthermore, subtle aspects of classroom environments, such as the gender ratio of students or posters associated with masculine CS stereotypes, reportedly impacts upon academic engagement of females [4]. As a Senior Lecturer in Computing Education, my role involves observations of lessons in computing classrooms in the North West of England. This paper considers the impact of the curriculum change focussing on classroom design. An aim of the study is to explore images of classroom design through the lens of trainee teachers. The research asks the question of what I can learn in my role as a teacher educator on the gender imbalance in the subject, from an early exploration of computing classroom images.

Background
The research considers how changes introduced in a University programme impact upon trainee teachers when they are in the computing classroom. The course is a one-year course training postgraduate students in secondary school computing education. A key aim of the course is to prepare trainee teachers for the computing classroom in two alternative school placements. As part of the course, I visit the trainees to observe a lesson, and give feedback on their pedagogical practice and progress towards teaching standards [1]. The structures of 'habitus' that exist are unconscious, but form the basis of perception and appreciation of all subsequent experiences. It is important to have a clear understanding of what might be meant by 'habitus'. It is formed as part of our early familial relationships and external factors, for example, morality, tastes, cares [5] thus ultimately influencing our preferences and practice. Using sociologist theory in a study of a computing classroom may not be seen as relevant to some. As with many scientific studies, quantitative data gathering through a positivist lens would be my initial preference. However, this small-scale study considers the use of a visual approach to the data collection as appropriate at this early stage of the research. Through the use of photo-elicitation, a deliberate attempt to move away from my preference of a more positivistic approach to data gathering offered me a methodological approach that allowed an exploration of the study 'beyond the role of traditional interviewees and research subjects' [6]. In an aim to counter the complexities that exist within the field, and calls for innovative approaches [3], it was apparent for the need for credible alternative ways of thinking. Photo-elicitation afforded an opportunity to gather data through this less traditional method in the field, and to understand the social field of the trainee teacher, rather than my own viewpoints. A further reason for this method was to allow trainee teachers to discuss the image they provided without influence from the researcher.

Study Methods
As a University lecturer, I had access to fertile ground in which my data could be gathered, in this early exploration of computing classrooms. All of the trainee teachers were from the 2018-19 Secondary Computing Education course. A representative demographic mix was considered as part of the selection; ages ranged from mid-twenties to midfifties, and an equal number of females and males. The trainee teachers were asked to provide an image of their preferred classroom design in my motivation to understand and explore the unconscious dispositions of the trainee teachers. This was the only request that was made of them. The data gathering was undertaken in two stages. The trainees were sent an email asking them to provide one image of their ideal computer classroom. Out of the 28 trainees, 5 responded to the question providing an image as requested, 2 of the respondents were females and 3 were males. In an aim to limit power relations as a lecturer on their course, the second stage of the data collection was to be a recorded interview. The second stage of the data collection was a group interview. The trainee teachers were asked to discuss the image they had provided. The trainees had not seen each other's images. There was no input from the interviewer; the discussion was recorded and then transcribed.
To ensure the study was following ethical procedures, the participants were reminded they could withdraw from the study at any time, including the images they had supplied. Consent and anonymity were obtained, in line with University ethical considerations. Even with 'signed consents do not imply that researchers have the right to use images in unlimited ways' [6], specifically when taking pictures in schools. Particular care was taken that no pupils were present when pictures were taken. This is a limitation of the study, and in further work ethical clearance would be required to obtain data 'in its entirety' [6]. The images collected through an email allowed the trainee teachers to express their view of a classroom through imagery and not words. The trainee teachers provided images in the form of 4 photographs and 1 graphical representation. All images had computers in the room; 3 of the images had a central space for group work (Fig. 1). A copy of all the images can be found online [7]. Through my analysis of the images and support from the transcripts, the discourse has been drawn out. In this paper, the early themes that I have identified from the images are critically reflected upon, through the lens of Bourdieu's theory of habitus, in an aim to better understand how the trainees' 'habitus', or set of dispositions and values, influenced their choice of image of a computing classroom. A full copy of the transcription can be found online [8].

Discourse: Group Work in Computing
The tensions in the discourse are highlighted by the trainee teacher's identifying that whilst group work takes place in computing lessons how the space is used presents differences.
'…you've got a table area if you want to do group work and a table area to do demonstrations and stuff like that…' (Trainee A) '…that they are all scattered across the room and grouped in four and the first thing I looked at is if I wanted to group work you've already got groups there already so if you plan in advance you can set the tables in such a way that the groups you want them to work in …' (Trainee C) 'I like walking about and I am always roaming around quite easily and with the tables if I choose to do group work I can just stick the tables in the middle and take up the space and all I do is just move the chairs and tables and then I can do group work…' (Trainee E) Trainee C talks of computers in the room in banks of four which are 'all scattered across the room', but important for Trainees A, B and C are spaces without computers to enable group work to take place. Here a preference for computers in the room is evident; the image provided showed computers grouped in four, rather than a space in the centre of the room.
Although Trainees C and D were more accepting of computers in the classroom, Trainee D stated '…I hadn't thought about having tables in the classroom… A big round one in the middle or something like that…' Whilst the accepted practice of Trainee D was in contrast to the other trainees, through reflecting on Bourdieu's earlier statement, what was natural to some of the trainees was not to another [5]. Through the discussion on images, the trainee was able to see, as Nolan states 'a transformed pedagogy would demand more deliberate action on the part of agents to disrupt the balance' [9]. A critical question for educators is how trainee teachers can be encouraged to challenge and disrupt practice in the field. Computer Science Unplugged is recognised as a widely used collection of activities and ideas to engage a variety of audiences with useful ideas from computer science, without having to learn programming or even use a digital device [10]. To what extent the unplugged pedagogical method is used in trainee teacher's computing classroom is an area for further exploration in the field of computer science education. The limitations in the data analysis for this study would allow for further discourse in the field of computing classrooms. What is notable is that the two female teachers have both provided an image with a central space for undertaking group activities away from the computer. Only one of the male trainee teachers provided an image with a space for group work, rather than desks in the middle. This is a point for further exploration in a future study.

Conclusion
The early findings in this paper are very much like the early stages of the research into the practices in the field of CS. The trainee teachers who took part in the study are now in their first year of teaching the subject. As part of the further data collection, trainee teachers will be taking part in a series of workshops at the University, which will be prior to their placements in the classroom. This will then enable them to consider the classroom design as part of the university sessions, and lessons will be planned using alternative approaches to teaching the subject. In using Bourdieu's field theory, the conflicting and developing discourse on computing education has brought insights into the challenges that trainee teachers face in their classrooms. The design of the classroom can be further considered for engagement of gender in the classroom. It is clear that the accepted practices, which typify the field of computing, are not fixed, as evidenced through the trainee teachers' responses. What the study has shown is that the trainee teachers are considering alternative uses within the fixed space of the computing classroom; the 'CS unplugged' pedagogy is a preference in their pedagogical practice. These images, and interviews, could be used to initiate discourse before they enter their practice to encourage a more reflexive stance in the field. The discussion around the images proved valuable; the images provided a stimulus for the trainee teachers to explore their use of the classroom. For future studies, the analysis of data proved to be richer with the use of both methods. The discourse may not have been given such a central theme with the use of images alone. A further point of reflection, and one for further study, is to consider the viewpoints of pupils in the lesson to their preferred teaching spaces.

Introduction
The students of the 21 st century are already experiencing new forms of economic and sociological changes which information and communications technology (ICT) is contributing in society. The education system needs to equip students with knowledge to help them develop skills and attitudes to fit the needs of the 21 st century. Changes have been made at school policy level and OECD countries are working to install networks in schools, connect them to the Internet, and ensure a workable configuration of multimedia computers, educational software, technical support, and ICT-competent teachers [1]. The teacher should not only be equipped with technology skills but also with the knowledge of effectively integrating ICT in the curriculum.
Currently, teacher professional development in India is often designed and administered through a top-down approach, and face-to-face training is often unconnected. Formal professional development activities have their challenges in offering a continuous learning process, but informal knowledge sharing methods such as a community of practice (CoP) can offer a much longer experience through an effective platform.
Lave and Wenger [2] created the notion of 'communities of practice', which refers to groups of people who share a concern, set of problems, or a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis. Research continually indicates that providing continuous support and promoting interaction among teachers are keys to successful teacher professional development. Many researchers have reported the benefits of CoPs in teacher professional growth. Online environments enable people to communicate at any time; thus, Herring, Barab, Kling, and Gray [3] have suggested the creation of online communities of teachers as a new professional development model.

Context of the Paper
The initiative to integrate digital technologies into the curriculum and instructional plans of teachers in government schools and learning centres was launched in 2012 under the Integrated Approach to Technology in Education (ITE), an initiative of Tata Trusts. Adopting a largely constructivist pedagogical framework, the initiative seeks to improve teaching and learning processes and foster authentic and project-based learning for older children and adolescents in some of the most underprivileged geographical regions in India [4]. In ITE, the students are the producers of their own knowledge and not merely consumers of technology resources prepared for them [5].
Teacher professional development is central to this approach, which is offered by the Centre for Education Innovation and Action Research (CEI&AR) at the Tata Institute of Social Sciences (TISS). ITE teacher capacity building at CEI&AR offers a 4-credit certificate course in ICT and education to in-service government teachers and teacher educators. The 4-month course follows a blended module, with face-to-face and online distance and implementation modes. Post certification, teachers are classified as master trainers who then help provide training and expand the reach to other schools. An online CoP platform plays an important role in this process and teachers use this platform to share their reflections, images of lesson plan implementation, ask for suggestions on their lesson plan, exchange ideas, etc., during and after the course.

Objective and Method
The objective of this paper is to analyse the frequency, preferred time of participation, and characteristics of the most active participants in two online CoP groups of teachers. The paper also intends to understand the relationship between active participation in the CoPs and relevant ITE practice in the classroom.
For the analysis, two of the certificate course groups on WhatsApp were considered, where teachers had spent about two years on their respective WhatsApp group, after the certificate course was completed. These CoP groups were formed during the face-toface workshops and consisted of teachers, teacher educators, and one or two government officials, ITE supporting organisation and TISS resource team members in Kolkata and Assam. The Kolkata group had 43 participants including 25 teachers and the Assam group had 55 participants including 27 teachers. The conversation from these two groups were exported in text format. Each message in the conversation consisted of fields such as time-stamps, the sender's name/telephone number, body of the message, and an indicator for media files. Using this structure, a browser-based tool was developed which rendered the analytics of the chats [6]. The tool provided descriptive information of numbers of messages sent by a user, numbers of media files sent by a user, most occurring words in the conversation, a time-wise histograms for distribution of frequency of messages in a day, and a date-wise histogram for distribution of frequency of messages throughout the conversation. The number of messages and media files sent by a user was found from the frequency of occurrence of usernames. For the analysis of the most occurring words in the conversations, the minimum length of the word to be considered was three characters, while the most commonly occurring words in English (known as stopwords) were excluded from the analysis [7]. The timestamp was used to analyse date-wise and time-wise distribution of messages in conversations.

Analysis
For analysis of conversations, chats were split into four periods for both groups: during the face-to-face workshop (P1), during the course (P2), in the first six months after the course (P3), and after the course for the six months to October 2019 (P4). Descriptive statistics for these periods show some notable observations, mentioned in later sections. Table 1 shows numbers of messages and media exchanged in different periods.

Time
Analysis of P1 is not reported here, since teachers were working together for four days in face-to-face mode. Across the periods after P1, a rise in the number of messages was observed during lunchtime in the schools in Assam (12 pm) as well as Kolkata (2 pm). During P2 and P3, in the Kolkata group active times were 1am, 2-3 pm, 6 pm and active times for the Assam group were 12 pm and moderately active at 4-6 pm. During P2 and P3, the chat shows that teachers have mostly shared about assignments and their implementations. During P4, the active hours remained almost the same as P2 and P3.

Dates
During P1, active participation was observed amongst teachers and facilitators and sharing of both text and media files was observed. During P2, the main peaks in frequencies of messages in both groups were observed during the cluster trainings where teacher educators were the resource persons. During this period, the Kolkata group shared images and videos of implementation and had discussion about implementation of the project, while in the Assam group, there was an active discussion related to submission of lesson plans and assignments and navigating the Moodle platform.
As the course ended, the number of messages in the WhatsApp group reduced. During P4, the maximum peak for number of messages in the Kolkata group occurred when the TISS-ITE team visited (204 out of 2,125 messages) which consisted of media and messages related to ITE implementation by teachers where ITE team members also participated by giving feedback or posing questions. In the Kolkata group, a drop in the number of messages (by about four times) was observed in messages exchanged between P3 (2,125 messages) and P4 (482 messages). During the same periods, the number of messages in Assam showed an opposite trend. The number of messages in the Assam group increased to 2.7 times to P2 (912 messages) from P4 (332 messages). Frequent visits by the facilitator in Assam for monthly meetings and follow up meetings on continuous professional development activities could be the reason for these sustained discussions in the group.

Most Frequent Words Used in the Chat
During P2, the most commonly occurring words across both the groups were "lesson" (13.7% for Kolkata and 14.1% for Assam) and "plan" (12.1% in Kolkata and 8.8% in Assam). During P3, the words "student" (17.3% in Kolkata and 16.2% in Assam) and "share" (6.6% in Kolkata and 6.2% in Assam) occurred quite frequently in both groups. During P4, "students" still continued to be one of the most used words in both groups (17.3% in Kolkata and 6.2% in Assam). In the Assam group, the word "sharing" was most prominently used (14.5%).

CoP Heroes
Top contributors of groups who were most active in at least three periods were referred to as "CoP Heroes". In Kolkata, four teachers (one female and three male) and in Assam, three teachers (one female and two male) were identified as heroes (see Table 2).

Summary
The results of the analysis indicated that teachers usually prefer lunchtime for What-sApp chats. Further analysis can reveal if lunchtime was preferred on school days or in holidays. Posting videos for discussion, the TISS-ITE team visit from Mumbai, or other visitors and ITE events, boosted sharing and discussions in groups in both states. Further analysis is required on content of the chats; the most frequent words used highlight that the CoP provided a convenient platform for teachers to communicate with other teachers and facilitators to share their ITE related work. CoP heroes were found to be excelling in the course and most had implemented ITE activities after the course.

Introduction
India has taken several positive steps to serve students with disabilities. Most recently, the Draft of the National Education Policy 2019 [1] highlights the need for equitable and inclusive education. India is a signatory to the 2006 UN Convention on the Rights of Persons with Disabilities [2], making an international commitment to better serve struggling learners. The Right of Children to Free and Compulsory Education Act 2009 [3] guarantees education for all children, including those with disabilities. While India has made provisions at the policy level, the implementation of these policies often falls short of expectations. A 2019 UNESCO report on the Status of Children with Disabilities in India [4] identified several gaps between policy and practice. The most recent census data of 2011 paints a dismal picture, with almost one in two persons with disabilities classified as illiterate.
There are several lines of action proposed by both the Draft of the NEP 2019 [1] and the UNESCO 2019 [4] report, and one identified by both reports as having potential to dramatically improve outcomes for children with disabilities is the use of digital technologies (DT). DT are changing how many of us engage in learning today, and has the potential to transform the learning experiences for persons with disabilities. This paper discusses strategies and tools seen to be effective to serve children with disabilities in mainstream and special education settings in India through the Universal Design for Learning (UDL) framework.

Universal Design for Learning (UDL)
UDL [5] is a framework that uses insights on how humans learn to improve teaching and learning experiences for all students. UDL is based on the premise that learners differ across three broad areas -how they engage with content, how they understand content, and how they demonstrate their learning. These premises are drawn from neurobiological research on how the brain learns. In order to best support learning, UDL outlines three principles when designing lessons: planning for multiple means of engagement, multiple means of action and expression, and multiple means of representation. Keeping these principles in mind when designing lessons ensures that most learners will be able to successfully engage in most learning experiences.
Designing a UDL class involves much more than using technology. However, technology has enabled teachers to easily build differentiated learning strategies, ones that may not have been possible, or have been too time-consuming, if a teacher had to do it manually [6]. For example, consider the principle of multiple means of representation. This calls for a teacher presenting information in different ways so that all learners can access and understand the content. Using technology, a teacher could give students the soft copy of a reading passage. Students could change the font to their comfort level. Some students may choose to have the computer read out to them. Others, who find tracking difficult, could use an online feature of a reading bar that highlights only the line being read. Teachers could embed videos within the text to support students' background knowledge. Words could become dynamic, with links to meanings or other related information.

The Professional Development Programme (PDP)
This paper draws on outcome data from a professional development programme (PDP) designed to equip teachers with strategies and skills to better serve struggling learners. While the PDP covered a range of topics and strategies to serve struggling learners, this paper specifically discusses the DT tools introduced in the PDP and shares how participants were able to take tools back to their settings.
The PDP was a six-month in-service professional development programme, comprising six full-day interactive workshops held once a month for six months. During these workshops, participants were given opportunities to use strategies and guided on how to implement these tools in their classes. Participants set goals at the end of each session on what they would action in the following month. Participants were also assigned coaches who were responsible for supporting them as they implemented strategies in their contexts. Coaches visited the participants' organisations to better understand the context and guide participants in their own settings. Participants also had an opportunity to visit the host school to learn how strategies could be implemented with students. Each month, participants shared video-or photo-evidence of how they had been implementing strategies in their context, which culminated in an end of PDP summary presentation on all their take-aways. A three-month follow up of participants indicated that 90% continued to use strategies learnt in the PDP. This paper shares evidence from two PDP cohorts in 2018 and 2019. Eighty-nine participants from 39 educational organisations in and around Mumbai attended the PDP.
Participants came from across low-, middle-and high-income schools. Figure 1 presents a description of participant roles within their organisations.  Table 1 presents a brief description of the tools that PDP participants were able to implement in their settings. Most tools support across the three principles of UDL and provide for multiple means of engagement, multiple means of representation, and multiple means of action and expression.

Strategies and Tools
Among the tools to support reading and reading comprehension, ReadWorks® [7] was most widely adopted by PDP participants. ReadWorks® is a free software that can be used to build reading comprehension. It has a bank of articles and question sets that are categorised by grade, lexile level, and by the type of text -narrative or informational. In particular, features that support struggling learners include StepReads, which is the same article written in decreasing complexity, read aloud options, vocabulary support which provides the meanings of difficult words, a split screen feature that allows students to read a passage and see questions side by side, and the highlighter ribbon, which supports students with tracking by highlighting only the line being read. The account can be shared by teachers and parents and many of the PDP participants have used ReadWorks® to support extension work at home.
Another tool widely used among the PDP participants was Quizlet [8]. Quizlet is a free online software to build vocabulary knowledge. Using Quizlet, teachers can create vocabulary banks for particular chapters, lessons, or browse through existing sets. Features that support struggling learners include multiple ways to practice words including individual games and tests, read aloud features, and a live gaming format that allows for collaborative group work. PDP participants shared Quizlet sets with parents using a WhatsApp link, which helped parents support school work. Participants also used the live gaming format in classes where all students had access to personal devices. Zearn [9] is a freely available mathematics curriculum to support learning in primary years. Zearn is built on the principles of UDL, providing multiple ways for students to engage with mathematical content. PDP participants have used Zearn to personalise learning in mainstream classes, as well as an additional tutoring or remedial session to supplement learning in the classroom. Features that support struggling learners include differentiated pacing, multiple opportunities to practice, and virtual representations and manipulatives.
Several technology tools can be used to assess learning across and build engagement. Kahoot! [10] and Quizziz [11] are 'fun' ways to assess students' knowledge and skills. The gaming format keeps students motivated and engaged. PDP participants used them as formative assessments when all students had access to a personal device, and when personal devices were not available, teachers could use Plickers [12] to achieve the same purpose. Features of Quizizz that are particularly effective for struggling learners are the option to attempt the quiz again, and the read aloud feature available on the application (app). PDP participants assigned quizzes as homework, so second attempts did not necessarily eat into class time. Another software, Mentimeter [13] combines a presentation with in-built assessment, making the entire class interactive. Students can access the presentation on their personal devices and teachers can build in interactives as needed. PDP participants also used Mentimeter to support in-house professional development.
Video apps like Playposit [14] further build in engagement by letting teachers embed questions and other interaction slides within a video. PDP participants used this software with younger students who might struggle to sustain attention throughout a video. Interactive apps like Wizer [15] let teachers create dynamic worksheets. Students could display their learning in different ways -by audio-recording, video-recording or writing responses. PDP participants found these worksheets particularly useful for primary grade students.
Technology can transform how students work together in class. Tools like Padlet [16] let students work on the same board and share responses in real-time. PDP participants used Padlet to support collaboration during homework assignments. Socrative [17] is another tool that supports collaborative work in class by letting students vote for the best answer. Socrative is particularly useful in helping students understand multiple approaches to answer a question, and in building models that students can use as references for their own work.

Conclusions
This paper presents a select set of tools that have been successfully implemented in Indian classrooms. Through the PDP, we have seen that technology that is available today, at no additional cost, can be used powerfully to enhance learning experiences for struggling learners. Most often, the challenge is for teachers who are either not aware of the existing technology, or do not have the skills to use digital technologies (DT) effectively in their classes [18]. The PDP provided the support that teachers needed in order to implement strategies. It first exposed teachers to the tool through workshops, then helped teachers see how tools could be implemented through visits to host schools. It supported teachers as they explored strategies in their own settings through coaching and mentoring, and most importantly, created a community of learners where teachers could share insights, questions and concerns as they experimented with new tools. Teachers were able to overcome the barrier of students not having personal devices, by bringing in parent support and building home programmes. Further studies can continue to explore innovative ways that DT can be adopted effectively in the Indian classroom.

Introduction
Currently, teachers involved in Danish vocational education have to develop their teaching methodologies to meet the changing demands for interplay between school-based and work-based learning activities. Additionally, they have to adjust to the 'bring your own device' principle, which Denmark was the first country to implement fully in all public education [1]. In addition, they have to design e-learning courses tailored to meet lifelong learning objectives of particular target groups through the application of optimum teaching and assessment strategies [2]. Together, these trends have led to disruptive innovation in the work of teachers [3]. This paper reports on a project exploring sustainable ways to meet the needs of teachers for professional development in this state of disruption. Firstly, the aim of the project was to test a new approach in vocational colleges in Denmark to provide teachers systematically with formative feedback on their design of e-learning courses. Secondly, the aim was to empower teachers to influence, make choices about, and achieve e-learning concepts that suited predefined learning needs of particular target groups through well-founded organisation and use of digital technology.
Previous research provides evidence suggesting that practice-based learning fosters the development of teachers' professional identity and agency [4]. In line with these findings, the project explored an alternative route to traditional in-service courses, workshops, seminars, conferences and degree programmes, in which knowledge obtained is seldom transferred into practice in such a way that it makes a real difference for the participants in teachers' courses [5].
Previous research also provides evidence suggesting that teachers' perceived selfefficacy greatly influences their professional capabilities [6]. Overall, the project aimed to increase this sense of self-efficacy. Unlike teaching methodologies based on intuition and earlier experiences, the project aimed to ensure that the teachers' choices of these methodologies in 1:1 learning environments are based on research knowledge about what influences participants' learning outcomes. In particular, the project aimed to provide feedback for teachers with a view to strengthening their sense of personal mastery, which usually has a positive effect on their agency [7].
Around twenty years ago, researchers carried out a meta-analysis of the provision of feedback, drawing the conclusion that formative feedback has a significant influence on student learning outcomes [8]. Around ten years later, these results were confirmed by means of the well-known meta-analyses of Hattie [9]. Currently, teachers are encouraged to implement these research findings in practice. In the project, the teachers carried out design of assessment activities, including the systematic provision of formative feedback tailored to each of the participants in the e-learning courses. In general, this form of feedback influences their performance as well as learning outcomes of the participants [10].
Until now, Danish teachers involved in education for young people have rarely received formative feedback on their practice from their principals, supervisors or mentors [11]. As a result, they may be inclined to reuse plans that have been used in the past. To avoid this situation, the project aimed to provide formative feedback for teachers regarding high-quality e-learning designs.
This approach, which was new in Danish vocational education, aimed at fostering professional agency. It also sought to ensure that the courses were organised in a consistent manner in relation to learning conditions, competences of potential participants, and their expected learning outcomes. For this reason, the National Danish Knowledge Centre of e-learning (eVidenCenter), in cooperation with the association of Danish Vocational Colleges and Upper Secondary Schools (DEG), piloted a project in the second half of 2018 focusing on guidance, provision of non-automated feedback, and certification of teachers' e-learning course designs. Table 1 gives an overview of the design template used, which does not represent a new standard but encompasses main themes drawing on contemporary standards regarding the design of learning events. This template was used to fill the gap between the rhetoric of current standards and actual course designs. In particular, the aim was to fill the gap between educational research and the practice of vocational teachers. All six themes refer to main findings of educational research, focusing on the factors that usually have the greatest impact on participants' learning outcomes.
When vocational teachers design e-learning courses, they are encouraged to consider these six themes carefully, while evolving professionally. Instead of using the template as a manual, they are encouraged to use it as a means to create educationally sustainable e-learning courses of high quality.
Initially, they learned about research findings and quality issues relating to e-learning courses.
Then they: • Considered their e-learning design • Submitted a well-founded e-learning design to the eVidenCenter • Received formative feedback from staff at this external certifier • Revised their e-learning design.
Finally, they obtained certification of their e-learning design before it was used.

Research Objective
The purpose of the case study was to gain new knowledge about how the quality of the teachers' planning efforts could be ensured by improving their opportunities to receive formative feedback. This included the perceived utility for the teachers of being guided and obtaining feedback when drafting e-learning designs, allowing them to create consistency between the prior knowledge and skills of participants, learning objectives and forms of organisation, activities and assessment in e-learning courses.
Accordingly, the case study generates knowledge about teachers making choices, achieving educationally sustainable e-learning designs, and developing professional agency while receiving feedback from the certifier. Since the teachers had to justify their choices, the study produces knowledge about what the teachers chose to do as well as how they justified their choices by referring to, among other things, the themes of the design template that was used.

Methodology
The case study applied a qualitative design encompassing desk study, observation, document analyses, and interviews. An initial desk study was conducted regarding existing research findings in the area of e-learning course design, focusing on factors that influence participants' learning outcomes. In addition, an initial explorative study was based on observation of the communication and interaction at three national project meetings.
Data were then generated through document analyses of the 23 certified course descriptions and the teacher-certifier dialogue related to these courses. Similarly, data were generated through five interviews with representatives of the teachers and vocational colleges involved. In total, seven teachers were interviewed. To frame their judgements, two additional interviews were conducted with one line manager and one educational consultant. The interviews were semi-structured, with a mix of individual and focus group interviews. The interview guide included the roles of the participants in the project, the design processes aimed at increasing quality of e-learning courses, and the perceived utility of the design template and formative feedback from the certifier.

Results
In total, 18 vocational teachers, who provided the 23 course proposals, participated in the case study, and the eVidenCenter provided systematic guidance and feedback to foster professional judgement and agency. Observations at the final project meeting and the data from the five interviews provided evidence indicating that all teachers greatly appreciated this approach. In particular, the guidelines as well the design template were regarded as relevant and valuable, and the teachers who were not familiar with this template before starting the project wanted to continue using it in future. Other teachers would also use the template, adding more themes, including the visualisation of vocational learning content and the use of multimodal materials representing this content.
All teachers also appreciated the certifier's provision of feedback on their draft elearning course designs. This can be seen in light of the rather limited feedback that teachers normally receive in this context. In general, the provision of feedback, which is closely related to the teachers' drafting of e-learning courses, increased their control over the design process and the choices made.
In all circumstances, the 18 teachers benefited from systematic appraisal and feedback, resulting in course designs taking into account contemporary learning objectives and the prior qualifications of potential participants, duration and blended organisation, teaching and learning activities, and evaluation. The analysis of the e-learning designs submitted and certified provided evidence suggesting that the feedback encouraged the teachers to produce well-founded designs fostering practice-based professional development. Even when starting with plans they had made in the past, they submitted new, well-founded and not yet tested course designs. Most often, the use of digital technology was an integrated part of the description of these designs. The shortest course was organised in the form of pure online learning, whereas 22 other proposals provided by the 18 teachers were organised in the form of blended learning, i.e. combined school-based and work-based learning in technology-rich contexts. The duration of the courses varied from 4 h to 185 h (see Fig. 1). The analyses of the 23 final designs provided evidence suggesting that this process led to clear descriptions of the teaching methods of the vocational learning activities. In addition, these analyses provided evidence suggesting that the certified designs described the potential participants' roles and learning objectives in relatively concrete terms, giving these participants a clear picture of what they were expected to do and what they could expect to learn.
Furthermore, teachers stated that the design process created a solid foundation for their choices and justifications regarding the organisation of e-learning courses and the utilisation of digital materials and platforms. Table 2 gives three representative examples of this kind of explicit justification. Further examples can be found in the formal report of the case study [12].
With regard to course organisation, some teachers provided implicit justification due to circumstances, such as predetermined extent of workplace and school-based learning, and the duration of the courses. For example, some participants had to complete part of their course at work, whereas others completed part of their course in particular learning spaces such as industrial kitchens or areas in which forklift trucks operated.
The teachers also provided explicit justifications of their choices regarding, among other things, blended learning activities, individually or team-based guidance, individual or group feedback, and summative evaluation.
When the teachers submitted their draft designs to the certifier, they received critical appraisal and ideas for improvement. Table 3 below gives two representative examples of these issues (from the full range in the formal report [12]). Organisation "The form of organization is flexible so that the participants can either attend college or be at their workplace to complete the course and obtain online guidance" Organisation "The form of organization is blended learning so that the participants can jointly reflect on their opportunities to use what they have learned in practice and to share experiences and ideas with each other"  The perceived usefulness provided for teachers included recognition of the quality of e-learning courses (formal approval) and the teachers' own efforts and competences (informal approval). All teachers appreciated receiving inspiration, guidance, feedback and informal approval of their efforts and competences in relation to vocational e-learning courses. According to the teachers, the certification process fostered their capacity building and created a better understanding of e-learning events. Table 4 gives three representative examples of the values of the design template reported by certifiers (from the full range in the formal report [12]).

Materials
For the project, a course had to be designed in a consistent way, including detailed descriptions of the continuous feedback strategies and summative evaluation strategies to be certified. Many teachers needed to reconsider these strategies. One theme in the design template was learning objectives. An example of a teacher's thoughts in this area indicated that companies and participants needed to know "what is in it for them in advance", while another teacher regarded the template as very useful but also demanding, since "it depends on teachers knowing what their learning objectives are". Some teachers had to learn to describe learning objectives in a clear and understandable way. This included knowledge about the contemporary terminology of learning objectives, i.e. the European Qualifications Framework (EQS), and some older teachers needed a few hours of introduction to this particular terminology.

Discussion
In principle, certification may reduce teachers' autonomy, but the case study did not indicate that this was happening. If teacher autonomy is limited, it is mostly because of predefined conditions regarding, among other things, learning objectives, course duration and the blend of workplace and school-based learning activities. All the teachers involved in the project developed their professional agency by considering and submitting wellfounded course designs. Consistent with these findings, the teachers greatly appreciated developing researchinformed practice. Research on formative feedback has been relatively influential in Danish education for young people and adults. In general, there has been a shift in focus from summative assessment and grading to formative assessment and feedback. The results of the case study showed that this should include the teachers themselves, since they greatly appreciated receiving formative feedback on their draft designs from the certifier.
In the vocational colleges, many courses are still designed with little, if any, feedback for the teachers in charge of the design process. By avoiding this form of professional isolation, teachers have benefited from the guidance and professional dialogue tailored to their particular needs and design efforts. Accordingly, the project tested a highly relevant and useful way to foster innovation in terms of feedback to teachers.
In particular, the project professionalises the design processes of the teachers. The certified course designs all represent rather detailed descriptions of learning objectives as well as assessment strategies, including formative evaluation to foster learning activities during courses and summative evaluation at the end of courses. So far, it has been difficult to promote research-informed practice owing to teachers' workloads. However, the results show that the project has discovered a sustainable approach to achieve this goal, making research-based knowledge about the quality of e-learning courses more accessible for teachers and easier for them to use.
Around the world, there is currently a tendency to organise teachers into teams [13]. For example, some vocational colleges build learning organisations, which implies shared visions and team learning as well as collaborative course design. Other colleges introduce professional learning communities, which similarly involve team learning. However, these initiatives are not completed in the Danish vocational education and training system. Until now, most vocational teachers have worked independently in relative isolation, even though they meet regularly with colleagues who teach the same subject. Accordingly, they often have to try ideas out themselves without appropriate support. When it comes to course design, there is a huge potential in sharing knowledge and ideas about innovative e-learning design at or across vocational colleges. The results of the case study showed that the project realised this potential.
In this context, teamwork "implies talking about practice and sharing ideas and problems" on a regular basis [14]. If the vocational colleges do more than merely introducing teams, professional teamwork can be regarded as one of the reasons why some colleges succeed whereas others fail when implementing innovative teaching methodologies [15]. Accordingly, the vocational colleges involved in the project succeeded due to the teachers' practice-based and collaborative professional development.
None of the teachers worked in professional isolation during the design and certification process. At some vocational colleges, three teachers, each designing an e-learning course, meet regularly to promote knowledge sharing and to support each other. At other vocational colleges, individual teachers designing e-learning courses communicate intensively with either a line manager or an e-consultant, or both. Additionally, three teachers from three different vocational colleges could build a team to initiate cross-college knowledge sharing related to one common e-learning design.
In this context, the concept of knowledge sharing designates a process whereby vocational teachers increase their professional capacity by sharing knowledge about quality parameters related to e-learning events. Previously, knowledge sharing has often been an illusion, because teachers have simply exchanged their experiences at meetings or during breaks [16]. In these circumstances, only a little knowledge becomes common. This situation calls for knowledge sharing formats in which the teachers inform and inspire their colleagues to such an extent that these colleagues learn to improve their practice. When they build on knowledge obtained from colleagues, they do not have to experiment from scratch with new learning objectives, assessment methods, organisational forms, digital materials, and technology applications.
For this reason, the project used an alternative and educationally sustainable method of overcoming the previous lack of knowledge sharing related to designing e-learning courses. In consequence, the project fostered knowledge sharing between teachers and their line managers. This included scenarios in which teachers shared knowledge, collaborated and acted as highly valued critics of each other's work when planning vocational e-learning courses.

Conclusion
This paper presents the results of a case study conducted in the last part of 2018 on the development of professional judgement and agency in e-learning design. The purpose was to investigate the perceived utility for vocational teachers of being guided and obtaining formative feedback from an external certifier while planning vocational elearning courses. The case study provided evidence suggesting that this approach bridges the gap between research-and practice-based knowledge, and that the teachers strengthen their professional judgement and agency. The provision of formative feedback by an external certifier represents a valuable and sustainable approach to ensuring quality elearning courses as well as teachers' capacity building in the area of innovative teaching methodologies in technology-rich environments. In conclusion, the perceived value of this practice-based professional development is high.
This project could be generalised to apply to providers of e-learning courses wanting to ensure the quality of such courses and develop professional judgement and agency regarding the courses. Future directions of research and development in vocational colleges could then include replication on a larger scale. Currently, a number of new elearning courses have been submitted to the certifier for critical appraisal and ideas for improvement. Since some of these courses involve teachers at general adult education centres, the future directions thus include e-learning designs in the field of general education in a broad sense. Future directions of research could also include the certification of certifiers, enabling vocational colleges to have their own educational supervisors. These supervisors could subsequently act as mentors and provide feedback for teachers who are planning e-learning courses in their colleges using the formative feedback approach tested in the project and reported in the case study.

Introduction
A deeper look at the mass of different (IT-)curricula reveals that current curricula are not always up to date -at least in respect to the year of publication. Developing a curriculum for universities is not an easy process. [1] described the development of an IT-curriculum, that lasted ten months from the beginning of the development process till the proposal of the final version. A development process over several months or even longer is not only necessary for the development of a new curriculum, but also for any adjustments. An example is given in Table 1 in which the development process of an Austrian computer science master programme is shown. As it can be seen, the curriculum was originally The degree to which cloud computing is covered by a curriculum is not easy to answer. First the question must be answered, what is meant by cloud computing, whereupon NIST [3] provides a well accepted definition. In addition, cloud computing itself is not a new technology but based on concepts that have been used for many years, but are not explicitly recognised as cloud computing [4].
Based on the factors that, on the one hand, companies are currently heavily aligning their IT strategies with cloud computing, on the other hand, IT curricula in universities may not meet the needs of the labour market and may be slow to implement curriculum changes, the following scientific research questions arise: -Is cloud computing adequately dealt within the IT curricula of Austrian universities according to the requirements of the IT labour market? -How can proposals for improving IT curriculum be generated automatically in order to better meet current requirements of the IT job market?
The methodological approach is divided into two main phases. The first is to develop a method and corresponding software to create relations between the theoretical concepts and concepts of curricula on the one hand and the practiceoriented requirements in job descriptions on the other hand. For this we will use the online encyclopedia Wikipedia as an external source of knowledge. Within the second phase, appropriate suggestions for optimisation for the respective curriculum should be generated by the developed prototype. The mentioned research questions are to be evaluated by a statistical analysis. The rest of this paper is structured as follows: Sect. 2 introduces background and related work while Sect. 3 elaborates on the high-level architecture and implementation details of our prototype. A kernel part of our paper can be found in Sect. 4 presenting the results together with a critical discussion. We finish with our conclusions and outlook for future work in Sect. 5.

Background and Related Work
Curricula in computer science have been already analysed in the past. By [5] two curricula of the Massachusetts Institute of Technology (Cambridge, USA) and the Open University (Milton Keynes, United Kingdom) have been analysed using LDA 1 and Isomap 2 . While [5] developed a method for analysing curricula in detail, our work focuses on establishing relations between curricula and job descriptions.
In analysing ten different computer science related curricula, [6] found that some universities focus more on human factors, while others focus more on theoretical concepts. In our work this is not elaborated, although we determined that some curricula are more business-oriented while others are more technically.
Key technologies that should be included in a cloud computing curriculum have been described by [7]. It has been also differentiated between fundamental technologies and enabling technologies.
The restructuring of their computer science programme has been described by [8]. Further, a self-assessment strategy for evaluating the effectiveness of study programme changes has been introduced, including a job placement tracking and alumni surveys. In contrast, our work aims to automate such a process.
We investigated 84 curricula of Austrian computer science study programmes, and those study programmes that are also associated with computer science. To put into context, curriculum consists of a number of different courses and every course focuses on a special topic of the study programme. At some universities, the courses of a study programme are grouped in modules. Compared to module descriptions we observed that course descriptions are mostly much longer. Additionally we collected 2,014 IT-related job descriptions from an Austrian Internet job platform. In comparison to rather heterogeneous study programmes, job descriptions are significantly more homogeneous, at least in respect of the text length.
An overview on different methods and tools that we used in our work are shown in Fig. 1. We use the Natural Language API 3 and the Custom Search JSON API 4 of the Google Cloud Platform (GCP) as tools for text analysis. We will use both Google APIs for indexing, which means assigning each document a weighted vector of words and which is part of a typical text analysis process [9]. We use the Natural Language API for identifying salient words within curricula and job descriptions. Using Google's Custom Search JSON API allows to search the web programmatically. In our use case we use the Custom Search JSON API to retrieve Wikipedia pages related to the entities that we have found with the Natural Language API. Although Wikipedia is not an acceptable source for citation (as it is not primary nor secondary literature), it became popular in (computer) science as a new research approach [10], e.g. natural language processing. In our work, we will map the descriptions of courses and jobs to a weighted set of different Wikipedia articles. A metric function can be used to express the semantic similarity of two arbitrary texts. Given two vectors x and y in a two-dimensional space, that represent the affinity of two texts to two different topics A and B. That means, a text can be presented as a vector where each vector component represents a different topic and the value of a vector component indicates how much it is related to the topic. According to [11] we use the following cosine-coefficient, where x and y are vectors, n is the total number of topics represented by x and y, and k the index of the respective vector component. 2 We also need to determine how much a Wikipedia article A is related to a given topic. Assumed that we have already identified that topic by another Wikipedia article B, we can calculate the relatedness to any Wikipedia article A by the link distance of this two articles. More precisely, as probably every Wikipedia article has incoming and outgoing links from and to other articles, we define the topic coherence C between two articles a 1 and a 2 by C(a 1 , a 2 ) := min ({ld(a 1 , a 2 where ld(a 1 , a 2 ) represents an oriented linkage from article a 1 to article a 2 , defined by the number of edges. Note, that C(a 1 , a 2 ) is not necessarily equal to C(a 2 , a 1 ).
We will further use genetic algorithms to optimise current IT-curricula to better meet the requirements of the job market and to increase cloud computing aspects in the curricula. Genetic algorithms are part of algorithms for solving multi-objective optimisation problems or at least to find suitable solutions. [12] provide an overview on different methods for multi-objective optimisation as well as a mathematical introduction and also a brief history on genetic algorithms. Although genetic algorithms are quite powerful, it is not guaranteed that the global optimum will be found [13].

Implementation
The basic concept of our work is the mapping from a text, that can be a job description or a course of a curriculum, to a weighted vector of Wikipedia articles that is being used as a meaningful representation. An example in Table 2 shows such a mapping of a course description about virtualisation technologies of a master programme on Cloud Computing Engineering at the University of Applied Sciences Burgenland in Austria. Although the text has been mapped to a total of 183 articles, only the most valuable with the highest weight are shown. As one of our goals is to establish a relationship between curriculum and a job description, we can map both to two different vectors and compare them with a similarity measure. The first step is the mapping of a text to a weighted vector of words using the Natural Language API of the Google Cloud Platform.
As job descriptions and curricula use different wordings for the same concepts, we will further map these words to Wikipedia articles using the Google Custom Search API. Finally, after we derived two weighted vectors of Wikipedia articles for two texts t 1 and t 2 , the similarity S(t 1 , t 2 ) of the texts can be calculated using a metric. The result is a real number greater than or equal to zero.
Curriculum can be split up to a number of courses that are either mandatory or optional. But both, a course and a job are represented in the same manner by a text. That means we are now able to compare different job descriptions with each other, to compare different courses with each other and to compare a course with a job. Therefore, we define: let C be a set of curricula, c ∈ C, J a set of job-descriptions, j ∈ J, M (c) the set of courses of the curricula c, m(c) a course of curricula c and |X| the number of elements in an arbitrary set X. The similarity between a curriculum c and a job-description j can be expressed by the mean similarity of all courses of the curriculum and the job-description: Quite similar definitions can be used for comparing two courses or two curricula. According to [14], who investigated different similarity measures, there is no clear answer for which is the best method of combining similarity values.
To calculate the topic coherence of a job or a curriculum to any given topic, we define the coherence C(a 1 , a 2 ) of a Wikipedia article a 1 to a topic represented by another Wikipedia article a 2 by the shortest path of links from article a 1 to article a 2 . For a job j, represented as a set of weighted Wikipedia articles A(j) := {w 1 · a 1 (j), ..., w n · a n (j)}, the coherence to a topic a can be defined as: Consequently, let M be a set of modules or courses of a curriculum c, and m ∈ M represented as a set of weighted Wikipedia articles A(m) := {w 1 ·a 1 (m), ..., w n · a n (m)}, the coherence of an entire curriculum c to a topic a can be defined as:

Results and Discussion
Using the Google Natural Language API we retrieved 75,017 salient words which correspond to an average of 29.33 words per course. Considering all curricula as a whole, these 75,017 words decreased to 16,228 unique words. On the other side, from 2,048 job descriptions we got a total of 183,047 salient words on an average of 90.89 words per job description. Considering all job descriptions as a whole, they reduce to 29,179 unique words. In total, of all jobs and courses we got 41,615 unique salient words. 25,387 (61.00%) of them have been found only in job descriptions, 12,436 (29.88%) have been found only in curricula and just 3,792 (9.12%) have been found both in jobs and curricula.
In order to prove that only parts of cloud computing can be assigned to topics and concepts in current IT curricula, we have to define these subtopics. Unfortunately, our proposed model to map curricula to sets of Wikipedia articles and to calculate their distances to a specified Wikipedia article representing a cloud subtopic is not suitable, because some cloud-related topics like e.g. Infrastructure as a Service (IaaS) and none of the common known deployment models of cloud services (as described by [15]) have corresponding Wikipedia articles in the German version of Wikipedia at this time. Hence, we selected all Wikipedia articles that link to the Wikipedia article of cloud computing and vice versa all articles that have a link from cloud computing. We found 464 articles and filtered the list, so that it didn't contain names of companies or related products. For the remaining 58 subjects, of which only a few can be clearly assigned to the topic of cloud computing, we calculated the coherence to all curricula. Ordering the results revealed that the articles that rank best are very common topics like Internet, Software or Computer. That is not very surprisingly as every curriculum is related to IT topics.
According to [16] the IT job market can be divided in five categories: programming languages, web development, database, operating systems and networking. We added cloud computing as a sixth category, as it is of special interest for us. We set a corresponding Wikipedia article for each category and calculated the link distances for all articles, that have been found by the Google Search API for the collected jobs and curricula, to each of these six articles. We calculated the link distance to each job category and determined the top 100 jobs that are closest to each category based on their link distance. Finally, we evaluated the similarity of all courses to the jobs of each category.
The results shown in Fig. 2 indicate that the jobs that are related to cloud computing are not the least targeted jobs by current IT-related curricula at universities in Austria. A reason can be that cloud computing comprises a wide variety of common technologies and subjects (Internet, IoT, security, virtualisation, etc.) and that it cannot be defined clearly as other categories. While jobs that are related closely to the category Webapplication are targeted best by the curricula and jobs in the categories Programming Languages and Operating System are poorly targeted. It seems to be meaningful that jobs close to web applications are targeted more than the other categories because there is an ongoing common trend in transforming existing software systems in enterprises to web-based and service-oriented solutions.
We found that the curricula that matched the job market best are mostly from computer science whereas the curricula that match the job market poorly are subject specific like medical informatics or media informatics. It is much harder to evaluate how far a single course matches the job market as the results are dependent on the text length of the considered course description. The longer Fig. 2. Average similarity between IT curricula and the closest jobs of each category a text is, the more salient words can be found that are not related to IT and only cause noise.
As it is our vision to upload curricula under development to a website to get instant recommendations for improvements towards coverage of the job market and the relatedness to certain topics, we tried to optimise curricula with the use of genetic algorithms. A genetic algorithm typically is based on three genetic operators: a selection-, a recombination-and a mutation-operator. Thus, a mutation can simply be defined as exchanging a random course of a curriculum by a random course of any other curriculum. A crossover-operator can be defined as follows: Select all courses that are contained in both curricula and select the remaining number of missing courses from one of both curricula randomly. For selecting the fittest individuals, we calculated how well every curriculum matches the set of jobs and choose the best ones. We found that the genetic algorithm improves the similarity to the jobs and the linkage distance to the topic cloud computing. On average, the similarity of 56 selected curricula (that consist of at least 20 courses) to the set of job descriptions could be increased by 148% as shown in Fig. 3, whilst the distance to the topic cloud computing could be only increased by 1.01%. At the same time, the text lengths of the curricula drop by 5.86%. In order for the changes not to impact the curricula too much, we set a constraint of a maximum of five courses that may be exchanged. The algorithm achieves most of the optimisation after 25 iterations, hence it converges quite fast.

Conclusions and Future Work
Our main contribution to the research area of computer science education is the development of a new approach for comparing curricula with job descriptions. We wanted to show to which extent the IT job market is targeted by current IT curricula at universities in Austria and applied our developed approach. Additionally, the topic cloud computing was of special interest to us, because we supposed that it is not covered sufficiently by the curricula, accordingly to the demand on the IT job market.
Critically summarised, we only focused on IT related curricula and jobs in Austria and only investigated the IT labour market. Nevertheless, our approach can be also applied to other domains and corresponding results would be very interesting. There are possibly many other approaches instead of using similarity metrics for comparing curricula and job descriptions. Although the use of Wikipedia is very convincing for us and is used by many scientists by now, we have to rely on external data and the trustworthiness has to be kept in mind. We decided to use cloud services and Google as an external provider to identify salient words in curricula and job descriptions. Here, too, we have to rely on external data and we actually do not know how Google identifies salient words in texts.
For future research as well as for application we describe two different issues. The first is the missing framework of curricula, the second is the automated improvement of curricula. Some curricula contains only course descriptions whilst others contain only module descriptions, that put related courses together. Few curricula also contain both, descriptions of courses and modules. There are certainly more differences that can be found in comparing curricula. Due to these variations, it is very hard to compare different courses or modules. We recommend the development and implementation of a common framework for curricula descriptions that are being designed at universities. In the best case, future developers of curricula also have an online tool that they can use to analyse their curricula whilst they are still work in progress.

Remarks
This paper is an outcome of a thesis in the master programme on Cloud Computing Engineering at the University of Applied Sciences Burgenland in Austria. The master thesis can be found at http://bit.ly/occe2020sub7.

Introduction
When we are talking about gender bias in the software industry, we are referring to the under-representation of women in the industry -the sheer fact that there are significantly less women working in the industry than men. There are differences between continents and countries but the overall trend has been clear for years -the ICT industry is dominated by males [1,2]. The research about gender in the software industry is usually focused on the importance of the diversity as a driving force for the success of the industry overall. Currently, there is interest in more diverse skills in the industry because previous studies argued that diversity in the software development teams influenced the creation of new innovations as well as on renewing the industry [3,4].
The under-representation of women in the software industry is not a new issue. For decades, a significant amount of work has been done to attract more women into the industry and efforts has been focused especially on younger generations. Still, the change has been slow and there has not been an increase in women participating in the software industry. That has led to the situation, where some researchers have already pointed out that we would need a new viewpoint if we really want to ensure that we will get more women into the industry in practice [5].
Reasons for the lack of the women in the software industry are complicated. Nevertheless, adult aged women still find the industry attractive and are pursuing jobs in the industry, despite the challenges they face when making a career change at adult age [6]. The peer support for women who are interested in the software industry or would like to train themselves further has been shown to be crucial [7,8].
Identifying different ways to enter to the software industry is important when the industry is facing big changes and more high skilled labour is needed -and this is the situation, where both educators and industry has to respond. At the same time, there is discussion going on that some jobs are disappearing via automation. Therefore, the guidance and different career paths for the adult aged population is growing in importance as well as the effect of lifelong learning. This study looked at the motivational factors of women working in the ICT industry and elements of their different career paths into the industry. The research question for this study is:

RQ1: What motivates women in ICT industry?
The paper is structured as follow: Sect. 2 contains the background research on diversity and gender bias in ICT industry. Section 3 presents the research process and Sect. 4 describes the results of research. In Sect. 5 we go through the key findings of the research and limitations. Finally, Sect. 6 concludes the paper.

Background
The need for more women in the software industry is driven by the need for more diversity in the industry. Diversity is more than just a gender issue -it can refer to culture, age and diversity in terms of experience. While innovations, products and services make up the industry, there is a need for people with different backgrounds, values and ideas to participate. Therefore the paucity of women in the innovation processes could impact detrimentally on our future [3,4,9,10]. It has been also researched that women value different aspects in software professional work than men. It was found that women value the possibility to renew their career and have changes in the daily workdays more than men.
Women also thought that new technologies and agile working methods were more important for them than men [11].
The under-representation of women in the software industry has been a problem for long time and it seems to be that change is coming but it is not going to happen fast. The problem is not only that there are not enough women in the industry, but the problem is also that women job duties divide differently in the industry than men. For example, in the USA's software industry only 11% of chief officers or senior leadership positions were occupied by a woman [12]. It has been studied that women usually do not pursue towards more technical jobs inside the industry and they also doubt their skills more than their male colleagues [10].
The background of the lack of women in STEM (science, technology, engineering, mathematics) industries has troubled researchers for decades. The problem has been researched from different viewpoints starting from the childhood to the academic career opportunities. The early childhood attitudes towards mathintensive fields has been shown to be one of the reasons, why women do not find the software industry attractive [13]. Cultural beliefs do also have their effect on what kind of career choices women and men will make. The possibility to use computers in the early ages and in the adulthood also matters. It has been researched that people who have been using computers do not suffer from computer anxiety as much as those who has not been given the access to the computers. Women also tend to have also more negative attitudes towards computers and their own computers skills than men and it can be because of the lack of computer usage of women [14].
Adult aged women find the software industry attractive. The variety of positions in the industry, an equal wages, meaningful work and interesting career opportunities attracts women. Software communities especially set up to help women who are entering to the industry from another industry are useful. These career changers are seen as one solution for the software industry's labour shortage problem [6]. Peer-support and mentoring from other women has been proven to have a positive impact on getting more women to the industry and also stay in the industry [7,[15][16][17].
The labour shortage of the software industry is also partly one reason why software companies are getting more interested in women [18]. For example, the World Economic Forum has stated that there are already some job categories, where finding new talent is difficult and most of these positions are in the software industry [19]. However a crucial change in future work life skills is needed. The 21st century skills, so called 'soft skills' has become more important, especially in the software industry. These skills are such as problem-solving, self-direction, information-processing and communication skills [20]. This all leads to the situation that there could be potential momentum for women to really take their place in the software industry and fix the gender gap.

Research Process
The research method used in this paper is digital ethnography, also called netnography. Digital ethnography takes it form from the ethnographical viewpoints in the digital world [21]. In this research we are using digital stories, blog posts, of women who are already working in the ICT industry. These blog posts were selected, because there was good representation of women who have chosen the ICT industry as their first choice for career and for women who have made a career change to the ICT industry later in their working life. The motivation to approach the topic via digital ethnography was that the blog posts and stories offer an important place to dive deeper into women's experiences and stories about what motivates them to work in the ICT industry.
The blog posts are from The Finnish Software and E-business Association's #mimmitkoodaa (women who code) project and they are posted in 2019. #mimmitkoodaa is a program, which offers free of charge, hands-on workshops in different technologies, with the help of Finnish software companies. These events have been really popular and at the moment they have over 4,000 women on their mailing list. These #mimmitkoodaa blog posts, used in this research, are open for everyone and collected to give peer support, guidance and inspiration for women who are working or interested to work in ICT industry. Blog posts are interviews, made by the same interviewer and every post presented one career story. The blog post series still continues to publish stories and they publish one post every week.
In our research we went through the stories and identified common themes by thematic analysis [22]. Blog posts were reviewed in two rounds get deeper overview for the shared themes. There were 24 stories in 26 blog posts. Two of the stories were both in Finnish and English and one story did not fit the research purpose, because it was written by one of the authors of this paper. All the other blog posts were included for the research. In thematic analysis, we identified 17 common themes (see Table 1).

Themes
The blog posts were published between March and October 2019. All the blog posts are written in Finnish, two of them have been translated into English to serve the career changers who have an international background. From the blog posts, 12 of the stories were from women who pursued a career in the ICT industry as their first option and 11 stories were from women, who made a career change into the ICT industry later in their working life. From those 11 career changers, eight of them revealed their past profession. Two of the career changers had background in Business Administration, two had past experience in customer service and then there was a music educator, translator, travel consultant and one person who held a Master of Philosophy.
In the next subsections, we are going dive into deeper for four most popular themes common in the blog posts.

Lifelong Interest for the Technical Field
The lifelong interest towards technology, starting from childhood was mentioned in 14 out of the 23 stories. Women usually mentioned that they had access to the computer in their childhood home or they ha strong problem solving skills identified in their early childhood. A couple of women also mentioned that they had software professionals as their relatives (usually their father) and that they found out about the job possibilities in the industry. Related to their own interests and positive experiences using technology, teachers were also mentioned in three stories as important actors for career choice. Early interest in technology did not directly influence women to work in the industry, however it was a critical factor when changing careers in their later working life. The following quote from the blog highlights this.
Since school, Hanne has been fond of mathematics and science, and wanted to study computer science in junior high school, but did not get a place from an optional computer science course. In High School she got a chance to attend some web page courses and after high school her father tried to convince her to choose computer engineering as a major because he saw that her strengths were there.

Diverse and Creative Industry
The versatile job opportunities in the industry were seen as one of the most empowering aspects in working in the industry. From the stories, 14 out of 23 mentioned that they valued the diverse job possibilities of the industry and 9 out of 23 mentioned that they enjoyed the possibility to use their problem-solving skills in their daily job. The following quotes highlight this. I see that best in this industry is the possibility to evolve constantly and finding new things. It can also be really frustrating because this industry is never ready and one has to accept the limitations of own understanding and learn to cope with that.
What is the best in this industry is to create new projects, challenging yourself with different problems, the possibility to improve people daily life with the help of technology and that every working day is different from other. What is challenging is the balancing the working life and free time.

The Importance of Diversity for the Industry
Overall, six women out of 23 highlighted in their stories that they see their gender only as a positive factor in a male dominated working environment and five out of 23 see that working in the diverse teams was one of the most positive aspects of their job and they highly valued this. However, there were some negative experiences. The following quotes highlight our results.
Rosa's team has colleagues with different strengths and backgrounds, from babysitters to journalists and social workers. Rosa admires her systematic and precise colleague because she is more generous. Her strengths, in turn, are her excellent problem solving and brainstorming skills.
Indeed, the industry needs all sorts of diversity, and female coders can bring a lot of good to the industry with a variety of thinking, empathy and softness. It is important to get more women excited about the industry and stay in the industry by changing industry cultures, attitudes and roles so that women can work well and feel safe in their work environment.

Networks and Mentoring
Because stories in the blog post were presented by the #mimmitkoodaa project, which also referred to the strong community of women supporting each other, it was not surprising that mentoring, peer support and networks were mentioned in the stories. From the stories, 10 out of 23 women identified that networks were highly important for them and seven of the of 23 mentioned that mentoring had a big impact on their career success. Also, two out of 23 women mentioned that they found the stories of other women really empowering and supporting. The following quote shades further light on it.
Eevis thinks that she was very lucky, because in the first job, she found a mentor who was able to look after, answer questions, and be a true mentor. Unfortunately, along the way, there have also been cases of reverse mentoring. But for this first mentor is to thank that Eevis is still working in the industry. Mentoring and mentoring skills are key to growing less experienced but learning-capable juniors into top performers.

Key Findings
When it comes to the under-representation of women in the ICT industry, there has been a lot of discussion on how we could support and encourage young girls to become inspired about this field. In this research, we found that the majority of women mentioned in their stories had been interested in technology since childhood. Still, this does not directly mean that these women would choose a technical field as their first choice as a career. The importance of positive experiences and teachers guidance was crucial for choosing a career in the field of technology.
The meaningfulness of the job, the possibility to use their own strengths and skills, such as problem-solving skills or past experiences from other industries were seen important. Women highlighted in many ways that they felt that it was important to identify their own skills and strengths and find the right kind of job in organisations where they can grow as professionals. This also affects the education side of things; that is education providers need to be clearer on their offerings to assist career changers. They could recognise prior experiences that career changers bring and offer more flexible pathways into lifelong learning. Given career changers have identified that meaningfulness and identifying own strength is important to them, education providers could use these two areas to their advantage when developing curriculum.
Women also valued diversity in job descriptions, job possibilities and also in their everyday working teams. They felt that the diversity of people with different backgrounds made their job more interesting and meaningful. They also saw that industry would benefit from more diverse organisations and felt pride in their diverse background or capability to think out of the box. Women also saw their gender as a positive factor in the male dominated industry because they could stand out.
The peer support, mentoring and right networks were important for women working in the ICT industry. That is something which educators and industry could take more advantage when planning the strategies to get more diversity to the industry. The career stories from different kinds of people, the importance of teachers and close relatives and the effect of sharing the best experiences could boost the diversity in the industry. There were no differences identified in shared themes and attitudes between women who were career changers and women who chose the software education path straight away. This aspect could be interesting for future researchers to look into.

Limitations and Future Research
This paper used digital ethnography and thematic analysis to approach the issue of the motivational aspects of women working in the ICT industry. There are some limitations in this research. All the stories were collected from people who wanted to share their story about successful career changes or career choice. This can also lead to the situation where some negative issues were wiped from the stories. Also, the interviews were made only by one interviewer. This can affect to the story line and issues raised in stories. For more precise results more diverse ways to approach the topic is be needed. Overall, this research was about the women in the ICT industry. When we are talking about diversity and inclusion in the industry, more voices and underrepresented groups should have a voice. Also, some comparable data from different countries could be beneficial. For example the World IT project data mentioned in this research [11] could provide more answers.

Conclusion
In this research we examined the job motivation of women working in the ICT industry by going through 23 stories. We used digital ethnography and thematic analysis to find out what is meaningful and important for women working in the industry. We found that women saw lifelong interest in technology starting from childhood and continuing into adulthood as the most important driving force. Women also identified diversity as important in their job and for the industry's success. They also saw that mentoring, peer support and networks highly important for their career success.

Introduction
Intellectual property rights date back to the Venetian Patent Statute of March 19, 1474 [1]. In Great Britain, related but distinct copyright laws were enacted through the Statute of Anne, 1709 [2]. Both these regulations gave a limited term monopoly to inventors or authors/publishers to encourage the development and distribution of innovations. The monopoly allowed the innovator to cover the cost of reproduction and physically distributing material, often as books. In the Internet age, the cost of distribution of intellectual property has reduced to nearly zero, since anyone can download a digital file almost anywhere on the planet for negligible cost.
Between those original Statutes and the modern Internet era, the legal complexities of intellectual property have multiplied. International agreements were made to regularise procedures between nations. The Berne Convention for the Protection of Literary and Artistic Works (1886) [3] harmonised copyright protections amongst participating nations. By 2019, 117 states had become parties to the Berne Convention. Under the Convention, once a work was 'fixed', it was automatically protected. Article 10(2) permits Berne members to provide for a "teaching exception" within their copyright statutes, so this directly affects educators. The Universal Copyright Convention, adopted in Geneva, Switzerland in 1952 [4], gave birth to the © symbol. Placing this with the author and year in a work, established a legal framework for its intellectual property.
Despite these international conventions, the provision of online services has seen a proliferation of legal arrangements. Individuals encounter this diversity almost every time they contract a service using digital communications. At some point in the purchase cycle, the person will be required to click in a box next to a phrase such as "I agree to the terms and conditions". There is no opportunity to negotiate these terms -either you agree with a click, or you will not be able to access the service. This is even true for what will be termed a 'test case' -that of a teacher preparing to use the widely popular Scratch programming environment with her class [5]. The terms of use for Scratch are 3,363 words long, and the associated privacy policy comprises 2,418 words. The following points are included in these documents: • The source code for Scratch 1.4 is available for download.
• Scratch has the right to suspend your account for violations of the Terms of Use or Community Guidelines. • You acknowledge that you are using Scratch at your own risk.
• Scratch is based in the United States. Personal Information that we collect may be transferred to, and stored at, any of our affiliates, partners, or service providers which may be inside or outside the European Economic Area, including the United States. By submitting your personal data, you agree to such transfers.
Therefore, the teacher needs to consider if an offline version is preferable for class use. Also, she needs to undertake a risk assessment -is there the potential of harm to anyone in her class? Lastly, are there legal requirements for her students' personal details to be stored within her local jurisdiction? This last requirement might be overcome by the use of aliases.
The growth of online services and cloud computing has been astonishing. Software as a service provides a more equitable model than downloaded and installed licenced software, since the user pays for actual rather than potential use.
The rise of copyleft and FOSS [Free and Open Source Software] has been enabled by the Internet's capacity to deliver intellectual property for virtually zero cost worldwide. Copyleft licences use copyright law to foster the inalienable right to copy, share, modify and improve creative works [6]. The cost of production becomes the dominant factor in driving innovation, rather than the cost of replication and distribution. This area is particularly attractive to teachers, because immaterial digital resources are not often seen as valid educational resources when it comes to distributing school funding.
The analysis of the data presented below is based upon the Australian context. It is expected that a similar analysis would apply in most other jurisdictions in a similar way, making this a global issue.

Literature
Online T&Cs generally relate to licences for intellectual property rights. Coriat and Weinstein [7] describe the evolution of the US patent system from a position where only individuals could be awarded a patent, to the current corporatised version where organisations use patent pools to control markets. "The transformation of knowledge into a commodity (in the form of marketable intellectual property rights promising future revenue) created the permissive conditions for financial capital to enter into the production of knowledge" (p. 284). Further, they comment on the increasingly legalistic approach to intellectual property: "[because of] the complexification of the market for property rights, the major change is that new categories of players have emerged" (p. 286). This complexity contributes to the wide variety of licence agreements.
Femminella [8] pointed to the emergence of click-wrap licence agreements for online services. Under this arrangement, the customer cannot access the product (to evaluate its value) without agreeing to terms and conditions of the licence. There was some debate as to whether web-wrap T&Cs, etc., on the seller's home web page were legal -and whether the positioning or colour/font (grey on grey?) of these was important (not easily seen at the very bottom of the page). Software providers were "allowing operations with different results in different jurisdictions".
Hartzog [9] contrasted the viewers of free-to-air television transmissions with online listeners or readers. The former are not required to consent to any terms and conditions, but the latter invariably must do so. He raises the serious concern that "empirical and scientific research have demonstrated that an individual's cognitive limitations and the design and presentation of standard-form contracts significantly frustrate an individual's ability to properly read and understand standard-form contracts" (p. 408). In the case of browse-wrap terms and conditions, United States of America (USA) law appeared to hold "The offeree had a legal duty to read them" (p. 416). This legal impost on the consumer is explored further in this study. United Kingdom (UK) law generally requires the capacity to negotiate a contract (not usually possible online) and the Norwegian Consumer Council requires licences to be on-balance 'fair' [10]. According to Canino [11], American courts have generally found clickwrap and scrollwrap terms and conditions legally enforceable. Therefore, the teacher in the 'test case' is probably required to read all the terms and conditions on behalf of her class before using Scratch as a learning tool.

Method
The approach used for this study was pragmatic, with a clear focus on autoethnography. According to O'Hara [12], such an approach requires six steps. Step 1 is analytic and defined the database fields. Step 2 is ethical; since all the data were publicly available, no formal ethical approval was required by the chair of the institutional research ethics committee.
Step 3 determines the theoretical underpinnings, which as stated are pragmatic.
Step 4 involves data collection, which is described below.
Step 5 involves reflection, which is presented in the Discussion section. The final sixth step is to present the report. The analysis presented here is largely quantitative, with some qualitative comments associated with territoriality and privacy.
Data for this analysis were gathered by the author over a calendar year 1 January 2018-31 December 2018 in the course of normal academic life. During that time, the individual worked, socialised, took part in hobbies in the country of domicile and overseas. A wide range of online services were encountered in this time, from government human services to wireless network connections and many others. The terms and conditions (and privacy statements) for these were saved into a single folder, forming a corpus of 153 megabytes (MB) of data. The compiled database contained entries with the following information: date of acquisition; organisation type; type of service; number of T&C words; warranty extent; location when accessed; service limitations; jurisdiction for disputes; indemnity clause; bundled privacy policy; number of words in privacy policy; privacy policy excerpts; and total words to be read.
These data from 48 services joined online for the first time during the year (at the rate of 4 new services a month) were entered into the software package PSPP v. 1.2.0-g0fb4db for analysis. Eighty-five percent of the services used were work-related, and therefore relevant to education/academia.

Descriptives
The nature of the data was quite diverse. The most frequently occurring organisation type was hotel (14.6%) followed by transport, software vendor and manufacturer (12.5% each). Government formed 8.3% of the entries, with airports, publishers, charities, etc., forming the remainder of the organisations.
The most frequently occurring service was WiFi (23%), followed by software as a service (14.6%), training (10.4%), publishing, train, software, grant processing and computer equipment (4% each). The remaining services were 2% each, covering park entry, insurance, human services, geographic information, identity management, restaurant reservations, accommodation, car hire, etc.
In addition to the diversity of organisations and services accessed online, the word counts for their associated terms, conditions and privacy policies (where available) were also diverse. Table 1 shows the number of words in each section of the material. It is notable that the number of words in the T&Cs documents varies greatly, from 95 in one case (a hotel WiFi service), to over 15,000 words in another (a global software-asa-service organisation). Indemnities are generally short, but privacy statements tend to be more expansive than T&Cs. This could be because of the growing awareness of data security, with business confidence rocked when personal data are hacked (e.g. [13]), and legislation such as the European General Data Protection Regulation [14] which applies nearly worldwide, since it has implications for all European trading partners through its extraterritorial applicability.
It was noted that verbose T&Cs appeared to be accompanied by long privacy policies. The Pearson correlation between these two variables came to 0.31 (p = .029). So, there was a small, but significant link between overall length and privacy policy length.

Territoriality
Most of the services accessed online (77%) were contracted from the author's home state of Tasmania, Australia. However, others were accessed from other states of Australia or even overseas (see Table 2). This stands in marked contrast to the jurisdictional details specified in almost every set of terms and conditions. The most common jurisdiction was the Commonwealth of Australia (7 mentions, 14.58% of the sample). This was followed by mention of specific states of Australia (each at 4 mentions, 8.33%). 'The laws of the… country where you reside' had 3 mentions, 6.25% of the sample, alongside 'The jurisdiction where you get service' and 'local law' (1 mention each, 2.08%). Some jurisdictions were tightly defined, e.g. the courts of Santa Clara County, California, USA (1 mention, 2.08%). This particular jurisdiction has a population of about 2 million persons, while the organisation offered services on a global scale.
Looking at the possible sources of disputes, the phrases used in the limitations sections in the documents are enlightening. In the case of Wi-Fi services, some clear limitations were placed as follows: "SMTP blocked", "peer-to-peer blocked", "Sessions are limited to 30 min" and "Our Products… are provided 'as is'".
The assumption of risk falls mainly on the customer, with phrases like these included: "You… take full responsibility of your actions", "customers must not use their personal software", "at your own risk", "use at own risk", "No… freedom from computer virus, is [assured]".
One of the rarer stipulations in a limitations clause was the requirement the customer "must be a 'natural person'". The remainder also excluded supplier responsibility for consequential costs or damages, in one case excluding "Treble damages".

Word Counts
The initial analysis evaluated the total number of words in T&Cs and privacy policies for each online service. The mean was 5,102.56 words with a standard deviation of 4,279 words. The shortest was 95 words, while the longest was 18,319 words.
Over the year, a person would be expected to read all the terms and conditions, and also the privacy statements presented. This comes to 244,929 words.
The economic value of this literary burden can be calculated by looking at average adult reading speeds and median adult wages. Thomas, citing Stewart [15], suggests 200-250 words per minute (wpm) but revises this to 50-75 words per minute for technical material.
A more comprehensive meta-analysis by Brysbaert [16] provides strong evidence that the normal silent reading rate in English is 238 wpm for non-fiction and 260 wpm for fiction. For the purpose of calculating economic impact, the 238 wpm figure is adopted, considering that terms and conditions are generally coached in plain English rather than legal language (which might have justified the lower 50-75 wpm for technical material). This does not consider the Flesch scores for readability which have been considered elsewhere [17].
The annual time impost of reading online agreements therefore equates to 1,029.113 min or 17.15 h. In Australia, average weekly earnings were AU$1,653 in May 2018 [18], for an average of 39.5 h per week [19]. The hourly value of their work was therefore AU$41.85. This means the economic cost of reading all the terms, conditions and privacy policies can be costed at AU$717.70 per person over the year.
How far is this economic burden distributed? One choice would be to make a national estimate based upon the total number of employed persons. This was 12,522,300 [20]. The national cost would therefore be AU$8,987,191,306 or nearly AU$9 billion.
However, given the tendency for government services to go online, we cannot fairly exclude unemployed persons. The best estimate of the population from 15 to 65 years and above comes from the Census [21], and comprises 18,414,107 persons. These additional 5.9 m persons may not be employed, but many of them make social contributions. Even if costed at half the rate of employed persons, the value of the time they spent reading terms, conditions and privacy policies for online services would amount to AU$123,280,460. Putting the two groups together (employed and not employed), provides a total impost value on the nation of AU$9,110,471,767. This is a result worth considering. The jurisdictional splintering from a clear-cut copyright law to individual terms and conditions for nearly every online service we use has the potential to seriously affect the national budget. One may argue that no-one reads them anyway: but where then do you stand if there is a dispute that ends in court?
This example has focused on an Australian consumer, accessing online services from around the world. The impact in other countries would be expected to be similar (in terms of the number of words expected to be read by each individual), but the economic impact may be greater or lesser given local pay rates and hours.

Privacy
Privacy policies were on average 2,193 words long (as per Table 2). These were quite complex documents, stating how personal data would be stored, how it would be used, and how these arrangements fit into national or international legal frameworks.
For instance, a common inclusion was a statement that the customer's data, or metadata (e.g. Internet Protocol (IP) and Media Access Control (MAC) address), would be released to authorised legal bodies.
Some policies directly addressed territoriality by stating "[We may] transfer your information to a[nother] country". Others made reference to the European Union (EU)-U. and Swiss-US Privacy Shield frameworks which were approved in 2016 and 2017 respectively [22]. They rely upon self-certification by businesses, with the only oversight provided by the International Centre for Dispute Resolution-American Arbitration Association.
Beyond providing the customer service, some privacy policies were explicit about what may be done with the data supplied. "We use scientific methods, processes and systems to derive meaning from this data" -clearly indicating that machine learning techniques may be used to segment customers. Further, "some personal information collected may be used in research, statistical analysis, state or national reporting, awareness programs, public statements or training" illustrating how even a government organisation may interrogate customer information. Other organisations go further, explicitly invoking rights to data-match between sources: "we combine data we collect from different contexts" and "[We may] create aggregate reports on user demographics and traffic patterns". Some privacy policies use strong rhetoric about the extent to which personal information will be kept secure: "Your… information will not be shared, sold, rented or bartered", and "we consider the privacy of our users to be extremely important".

Discussion and Conclusion
The main limitation of this study is the paucity of data. However, the auto-ethnographic style provides realism and authenticity to that data. A central recommendation is therefore the study be repeated with a range of subjects, ideally drawn randomly from populations which provide diversity of age, gender and socio-economic background. One may also expect different results in other countries. Fifty-one per cent of citizens in Canada, for example, use government websites compared to a global average of 30% [23].
The data presented here relate mostly to professional and work-related activity for a mid-level academic. It is worth noting that the services accessed online were used through a conventional computer (laptop or desktop). Since smart mobile telephones are becoming so much more popular, it would be useful to incorporate the terms and conditions of any new applications (apps) installed during the collection period.
For teachers, the 'test case' mentioned in the introduction comes from the Scratch poly-lingual programming website. It was easy to create a student account on this website without explicitly agreeing to any terms and conditions (no tick box). However, the bottom of the website home page has a link to 'Legal', which provides access to the web-wrap conditions. All together, these amount to 6,650 words, which would take 0.47 h to read. There are both ethical and practical actions available to the teacher of our 'test case'. One strategy might be for her to read the documents and make note of any points about which she is unsure in the local regulatory context. These could be discussed with the school principal, and any outstanding issues put to parents/guardians. This would focus the latter on a few substantive matters rather than the full text. A more practical approach to avoid student data exports might be to download and install the latest offline version of Scratch. Along with these two strategies, she could consider relying on a 'traffic-light' indicator from 'Terms of service -didn't read' (tosdr.org) for similar online learning materials.
Online shopping is becoming more popular, and there are many discussions about advertising for online gambling. Online gaming is also purported to be economically greater than profits from the Hollywood cinema industry [24]. These areas are absent from the dataset but could be included with a larger population study.
Nonetheless, the initial findings about the economic burden of online terms and conditions remains a problem to be solved. Table 3 provides a comparison of these costs over four countries [25][26][27][28]. For Australia, the AU$9 billion cost in an AU$1.69 trillion gross domestic product budget may seem a small proportion, but a great deal could be achieved with it. The second recommendation from the study is therefore that consideration be given to a second Berne Convention, to harmonise terms and conditions for online services. This could be done in a way to protect intellectual property in the way of the first Convention, but also relieve the general population from the variety of agreements to which they are increasingly being expected to concur without any option to negotiate terms. Some regions are already moving in this direction, with the Association of Southeast Asian Nations preparing a common set of rules on intellectual property and e-commerce as part of the Regional Comprehensive Economic Partnership. "Standardised rules will help streamline intellectual property transactions, support transparency, and lower costs of doing business, supporting Australia's creative and innovative industries trade and invest in the region", the Australian government explained [29].
In conclusion, this study used an autoethnographic approach to capture online terms, conditions and privacy policies for a single individual over one calendar year. The reading impost of these documents amounted to 17.15 h in the year. The extrapolated cost across Australia came to more than AU$9b. These costs are a significant justification for new international agreements on standardised terms and conditions to prevent further escalation of this impost on all societies.
We propose the "Course Space" system for visualisation of courses in interdisciplinary departments. The system features include a visualisation scheme from the network sciences [2] and a user interface for observations by manipulating interactive operations.
Research applying methodologies from the network sciences has resulted in important discoveries in various fields. Hidalgo [3] coined the term "economic complexity" which describes the transitions of gross domestic product (GDP) in the long term by economic analysis using networks. He observed that the transitions of GDP can be described by not only economic indicators but also calculation of the complexity of each country's industries from product exchanges in international trade.
Some research in the education field has applied networks to, for example, syllabuses or bodies of knowledge. Mima [4] proposed the MIMA Search system to obtain structured knowledge, demonstrating use of the system for analysis of the curriculum at the School of Engineering, the University of Tokyo. The MIMA Search system visualises relationships by creating a network based on input words, and has been used as part of the syllabus search system for students there.
Masunaga et al. [5] and Yabuki [6] attempted to validate the formulation process of a body of knowledge (BOK). Their proposed methods were used in efforts to build structured BOKs by reverse engineering unstructured individual class materials in non-traditional, interdisciplinary fields. They successfully built a BOK tree unit in the formulation process by making semantic links between syllabuses.
An experiment by Pentland [7] demonstrated application of the network sciences to data collected on two occasions at a Massachusetts Institute of Technology conference. The data described who talked with whom, attendance, and behaviour throughout the day, and was sufficient to reveal participant affiliations (they belonged to the same institutions). Thus, behaviour was demonstrated as useful in predicting group affiliation. Specifically, Pentland found that the sum of times that two people spent together was related to whether they belonged to the same institution.
A similar concept can be applied to membership in the same social network. Social networks are networks of people who are connected through jobs or social activities. The characteristics of a social network rely on the behaviour of its members. Social networks in this research are formed by networks of students connected via course selection, or by course networks connected via students who have taken the course. Such networks are expected to reflect social aspects of the course selection, thereby revealing network clusters showing, for example, majors or laboratory activities in student networks connected by course selection.

Course Space
We propose the Course Space system as a support system for comparing curricula to reveal actual circumstances and for students considering which courses to choose. The system aims to visualise not only the number of students in classes forming a curriculum, but also connections between students and courses.
The system visualises course-taking patterns with metadata such as major, laboratory activities, and career after graduation. Students can find courses they might not have otherwise found by the course name or syllabus. Students can also adjust their course selections after reviewing courses they have already taken.

Design
Main View. Figure 1 shows the main view of the Course Space system. The network on the left is a student network connected by co-occurrence of courses they have taken. The network on the right is a course network connected by co-occurrence of students taking the same course. Each network graph was drawn using a force-directed algorithm.

Functions for Analysis.
We implemented the following functions for analysis:

Link-adjuster function
When graph density becomes high, the system shows all links between co-occurrence nodes. The user must therefore control link density to interpret the network structure. This function hides links with values below a threshold, which users can set using a slider bar or text box.

Highlight connections function
Use of this function prompts the system to highlight the links of selected nodes, thereby supporting exploration of the network structure.

Cross-filter function
This function connects the student co-occurrence network and the course cooccurrence network. When users select a node in one network, the links and related nodes in the other network are highlighted. This allows users to explore the network by focusing on one student or course.

Student attribute visualisation function
This function colours nodes in the student co-occurrence network according to attribute, namely student metadata such as major, laboratory, or career path after graduation. It supports visual analysis by projecting metadata on the graph.

School year filter function
This function filters nodes by the school year of the students in the student cooccurrence network. This decreases differences between the number of courses taken, which increases with each school year. Users need this function for analysis of each school year because of the fewer number of courses taken by the younger school year students connected by the link-adjuster function.

Implementation
Course Space is implemented as a web application written in JavaScript with the D3.js [8] and crossfilter.js libraries. D3.js draws network graphs based on data from a database. Crossfilter.js filters the data before D3.js processing. The database server interface is a PHP: Hypertext Preprocessor (PHP)-based application programming interface that allows JavaScript to issue structured query language (SQL) database commands via POST requests.

Curriculum of the School of Social Informatics
The SSI at Aoyama Gakuin University aims to develop people with broad insights who will contribute to resolving complex, real-world problems by communicating with people of various backgrounds and who can be active in interdisciplinary fields. Based on this aim, the SSI has three study areas: social sciences; informatics; and humanities.  Figure 2 shows the SSI study areas. Circles in the figure indicate study areas. Courses within a circle are called area courses, such as the "Informatics Area Course". Courses in the intersection between two areas are called liaison courses, such as the "Social Science-Informatics Liaison Course". All students select a major programme comprising two areas, namely the Social Science-Informatics Programme, the Social Science-Humanities Programme, or the Humanities-Informatics Programme.

Data Collection Experiment
We conducted an experiment to collect data using the developed system. The input items are as follows: • Elective courses students had taken, from among 168 specialty and English courses in the SSI. • Students' personal information (optional) -System display name -Gender -Student year (first, second, etc.) -Major programme -Laboratory -Desired career after graduation -Decided career after graduation -High school course of studies: humanities or science -Participating in a teacher accreditation programme We recruited fourth-year students belonging to any laboratory. All participating students were volunteers.

Data
We collected valid data from 120 students: 51 in the Social Science-Informatics Programme; 11 in the Social Science-Humanities Programme; 32 in the Humanities-Informatics Programme; and 26 who registered no programme data. We included 114 subjects in the analysis.

The Method of Analysis
Using Course Space, we analysed the following points: 1. Correlations between student attributes and courses, indicated by coloured nodes with attributes in the student co-occurrence network, 2. Relations between courses using two course co-occurrence network layouts, and 3. Each student's course selections, using a crossfilter function.
Note that the number of nodes will not be equal to the number of students or courses in the network graph due to the threshold function excluding some nodes. The next section describes this in detail.

Student Co-occurrence Network
Major in the School of Social Informatics Programme. Figure 3(a) shows links in the student co-occurrence network co-occurring with more than 23 courses, coloured according to major. Pink nodes indicate the Social Science Informatics programme, green nodes indicate the Social Science Humanities programme, blue nodes indicate the Humanities Informatics programme, and grey nodes indicate no programme data. The programme colours clearly show three clusters, indicating that the programmes influence course selection. The network shape reveals that although the programmes comprise two areas, most students select courses in one particular area. Figure 3 shows that the Social Science Informatics programme forms two clusters, one at the top formed by students mainly taking information courses, the other at the bottom formed by students mainly taking social science courses. We can verify this phenomenon by using the crossfilter function to check the selections of each node (student).
(b) Coloured according to laboratory activities.
(c) Coloured according to high school course of study.
(a) Coloured according to major in the School of Social Informatics programme. Furthermore, the network shape shows that nodes are dense at the circumference and sparse at the centre. We call this the "donut phenomenon", and consider it to be due to the course selection design. As mentioned above, the programmes include two areas, and students do not need to take classes from the third area. This course-selection system decreases the number of third-area courses that students take, so the link adjuster function disables co-occurring links with third-area nodes, because their number is below the threshold. Figure 3(b) shows a student co-occurrence network in which links cooccur with more than 23 courses, coloured according to laboratory activity. Colours in the network indicate laboratory affiliation. Although the laboratory clusters are less clear than the SSI major clusters, they are observable, and we confirm the features of each laboratory. Figures for each laboratory can be analysed to indicate the research field of each laboratory. For example, the blue-green nodes representing Laboratory M are scattered throughout the graph, indicating that that laboratory emphasises diversity in course selection.

Laboratory.
High School Course of Study. Figure 3(c) shows a student co-occurrence network in which links co-occur with more than 23 courses and node colours indicate high school course of study. Pink nodes indicate humanities, blue nodes indicate sciences, green nodes indicate others, and grey nodes indicate no data. Coloured nodes are randomly distributed in this network, indicating that course selection is unaffected by students' high school course of study.

Course Co-occurrence Networks
Fixed Layout. Figure 4(a) shows a course co-occurrence network with a fixed layout in which links co-occur with more than 20 students. This network indicates courses that tend to be taken simultaneously. For example, "Database Basics" and "Programming Basics" in the informatics area and "Microeconomics I" and "Macroeconomics I" in the social science area tend to be taken at the same time. In contrast, "Introduction to Mathematics" links with courses in each area, such as "Programming Basics", "Cultural Humanities" or "Macroeconomics I". It can therefore be considered as a basic course.
Force Layout. Figure 4(b) shows a course co-occurrence network with a forced layout and links co-occurring for more than 10 students. Dark blue nodes indicate the informatics area, brown nodes indicate the social sciences area, dark green nodes indicate the humanities area, light blue nodes indicate the humanities-informatics liaison area, purple nodes indicate the social sciences-informatics liaison area, and light green nodes indicate the social sciences-humanities liaison area. This network indicates that students take courses following the aim of their study area's curriculum, because liaison course nodes fall between area course nodes. Figure 4(c) shows a course co-occurrence network showing only area courses with a forced layout in which links co-occur for more than 10 students. Blue nodes indicate the informatics area, brown nodes indicate the social sciences area, and green nodes indicate the humanities area. This network shows clusters for each area because students seem to take courses in the same area.

Student Evaluations
In trials, students used the main view of the proposed system. We conducted a questionnaire survey after the trial, asking if the system offered useful information for course selection, along with any comments or suggestions for improvement. Table 1 shows excerpts of the answers we received. Response S1 mentions bias in their course selections, and response S2 suggests that visualising courses taken provided useful information for future course selections. The proposed system could thus support course selection decisions.

Faculty Evaluations
We showed course selections as visualised by the proposed Course Space system to five SSI curriculum design committee members and discussed the results. Table 2 shows some remarks made during that discussion.
Comment T1 suggests that fewer students than expected select courses from across all three study areas. Comment T2 shows that faculty think highly of students taking a ID Comment S1 Viewing the data, I could clearly see bias in my course selections and which areas I didn't realize I was interested in S2 I thought my course selections might have some bias, but I didn't realize how biased I had been. It is interesting to see which courses I should take to maintain balance or specialize in my major by visualizing the courses I've taken just by selecting them Table 2. Excerpts from comments from curriculum committee faculty.
ID Comment T1 I expected the node density to be gathered closer to the center. I thought my laboratory has only students with scattered course selections, but in fact only students with biased course selections are in the SSI. I don't think this is a good situation T2 Some students are somewhat in the middle of the Informatics-Humanities or Social Sciences-Informatics areas. I don't know whether the faculty aims for this, but I think it is good, curriculum-wise. I don't know if I should think of this number of students who are in the middle as a lot or a little, but it's better than if nobody chooses courses that cross study areas balance of courses in two study areas because the aim of the SSI is for students to not take courses from all three areas. These comments confirmed that from this viewpoint, the goals of the current curriculum are being met. The results of the discussion confirmed that the proposed system is useful for curriculum evaluation, which is one of its aims.

Discussion
We propose Course Space for visualising course selections by applying methodologies from the network sciences. We demonstrated the system using data collected from 114 subjects in an interdisciplinary department. The generated network structures provided clear evidence that students select courses based on their majors, supporting what department faculty members already feel. However, the visualisations in this research confirmed such feelings and revealed the extent to which this phenomenon holdsinformation that can be used for curriculum development.
The system could interactively change network link thresholds for analysing student attributes or course selections in the network visualisations. It allowed exploratory observation by functions allowing users to change node positions by drag and drop or to switch the network layout algorithm. These processes demonstrated the usefulness of the system as an interactive application.
Course co-occurrence networks showed courses taken simultaneously and that course selections reflect the curriculum for interdisciplinary learning. We also found that student majors could be predicted from the network structure.
We thus consider that faculty will be able to formulate curricula based on data visualised in systems like Course Space. We reported the results of this research to the curriculum development committee at the SSI. Such information is necessary for school development, not only at the SSI, but also at other schools focusing on interdisciplinary studies, to observe and understand student course selections.
By having students register their data, they could view Course Space as part of the trial, allowing them to find biases and subconscious interests in their course selections. We believe such discoveries will help students choose courses in the future.
One limitation of this research is the small amount of student attribute data we collected, and therefore few student attributes with biases were shown. However, this does not affect bias effects because new findings are based on the network architecture. Further development of the system and detailed analyses are required in future studies.
consists of multiple-choice questions and free text questions. In this paper, we show the result of the text analysis applied to answers written in Japanese by participants to free text questions in the questionnaire (see Fig. 1(a)).

Features of the Data
We analyzed the data obtained at "AstroTalks" from July 2011 to May 2019. The number of participants was 1,168 and 740 participants responded to the questionnaire, so the response rate was about 74.5%. The number of male and female participants was almost equal (see Fig. 1(b)). More than half of the participants were over the age of 50 years (see Fig. 1(c)). In this paper, we show the result of statistical data analysis with KH Coder [2] on the answers written by participants in Japanese to free text questions 2 "What were the impressive things on our lecture?" (556 participants answered) and 3 "What topics of astronomy are you interested in?" (353 participants answered) [3]. KH coder is widely used in various research for analysing text data [4].

Text Data Analysis of the Answers to Free Text Questions
As a pre-process of the analysis, we unified words or terms that had the same meaning in the answers. In fact, "4D Theater", "4D Space Theater" and "Theater" were unified to "4D Space Theater". Then, we performed statistical text data analysis on the answers to the free text questions using KH Coder. Table 1 shows the top 16 words and terms by the number of appearances, neglecting particles and auxiliary verbs in the answers to question 2 "What were the impressive things on our lecture?" The data showed which words or terms were had impressions on participants. In Table 1, "4D Space Theater" appeared 229 times. Thus, many participants had a strong impression of the "4D Space Theater", which is the special lecture in "AstroTalk". In addition, the "Sun" (21 times) and "Mars" (20 times) appeared many times in the answers. These were often topics in the "AstroTalk". Although the "Moon" was a topic in "AstroTalk" only 3 times, it appeared 17 times in the answers. This is probably due to the fact that the Moon is explained in every "4D Space Theater".  Figure 2 and Fig. 3 are co-occurrence network diagrams of answers to the questions. We can see which words and terms are strongly connected with the words and terms based on how frequently they appeared in the sentences in the answers.

Result of the Analysis of the Answers to a Free Text Question
(b) (a) Fig. 2. (a) The co-occurrence network diagram of the answers to the free text question "What were the impressive things in our lecture?" (b) The enlarged view of the area inside the dotted rectangle in Fig. 2a. The minimum number of appearances is 5 and the maximum is 60. In each diagram, the lines indicate strong connections among words or terms.
As for the answers to question 2 "What were the impressive things on our lecture?", the "Moon" has a strong connection with "watch" (see the upper left dotted ellipse in Fig. 2(b)). In fact, there were answers by participants such as "I was able to watch the back side of the Moon", "I watched the video to show how the Moon was formed".  Fig. 3(a). The minimum number of appearances is 3 and the maximum is 60. In each diagram, the lines indicate strong connections among words or terms.
So, we can say that the simulation video of the "Moon" formation and the back side of the Moon were strongly impressed on participants as the lecturer intended. On the other hand, connections among "Orion", "constellation", "distance" and "position" are also strong (see the lower right dotted ellipse in Fig. 2(b)). It is probably because the three-dimensional structure of the Orion constellation and the position of the stars and distance among the stars that form the Orion constellation were shown by the lecturer at almost every "AstroTalk". In fact, there are answers such as "the positions of stars that form the Orion constellation" and "the distance among the stars that form the Orion constellation". Text data analysis has identified major trends in the text data statistically, but it is not easy to find minor trends such as misconceptions.
As for the answers to question 3 "What topics of astronomy are you interested in?", it can be seen from the diagram that the popular set of words and terms for participants were, for example, "astronomy" in "ancient times", "collision" of "asteroid" to the Earth and "planets" of "solar system" (see dotted ellipses in Figs. 3(b) and 3(c)). In order to improve the design of "AstroTalk" based on the results of the analysis, we adopted several sets of words and terms seen in the diagram as topics for "AstroTalks".

Summary and Future Research
We performed statistical text data analysis using KH Coder on answers written in Japanese to free text questions which were obtained from participants at the astronomy lectures "AstroTalks". Words and terms related to topics of "AstroTalks" appeared many times in the answers. From co-occurrence network diagrams, it was shown quantitatively that participants were impressed with the topics which the lecturers wanted to convey in "AstroTalks". We adopted sets of words and terms seen in the diagram as topics for "AstroTalks" to improve lecture design. There are regional disparities in educational opportunities in astronomy in Japan, because lectures and seminars are usually held in cities. This is because lecturers are usually professors of colleges and universities located in cities. To reduce disparities, applying the system shown, which was very effective in the education of agriculture skills [5], we are developing an active learning system on astronomy with an electronic blackboard connecting a lecturer in a city with participants at a town through the Internet (Fig. 4). Through the system, participants can easily access learning contents through a large size touch panel for group work connecting with lecturers online.