Reflection on How to Write the Learning Outcomes for an Online Programming Course for Teachers

. The EU Commission for Higher Education through the Bologna Process declaration has put into action a series of reforms. One of the reforms is the development of learning outcomes in the European Credit Transfer and Accumulation System (ECTS). One part of these reforms requires European universities to identify and describe learning outcomes students will achieve after attending a course or program. There is no exact way of writing the learning outcomes, and there seems to be an indistinctly use of terms when explaining outcomes (1). This can naturally cause some possible confusions in relation to what the learning outcomes should consist of, and make it difficult to write the learning outcomes for a course or a program. The purpose of this paper is to promote some explanations and clarifications that can ease the writing of the learning outcomes. The research question is How to write meaningful learning Outcomes? This paper presents a reflection on how the learning outcomes might be written. It is taken into account recommendations from relevant literature and framework intended for Europe. In this paper, the online course Applied Programming for Teachers is an example of how meaningful learning outcomes may be written with a focus on digital competence.


Introduction
Right competencies are essential for companies but are equally important for students, to perform a particular profession.Higher education systems strive to produce graduates with the right competencies whatever field of study.On the other hand, students are not always aware of the competencies they developed during their education (2), which can lead to students with low confidence (3).Some researchers suggest that learning outcomes could be used as a learning resource to make students aware of competencies they can gain.Learning outcomes are statements of what a learner knows, understands, and can do when completing a learning process.In this way, the students with knowledge of where they are going and what is expected of them at the end of a course.In addition, learning outcomes can be used as a reflection tool for students to create awareness of achievements throughout a course or program (2).To archive this, learning outcomes must be identified and clearly articulated to the students (4), and to be used productively by students, it needs to be emphasized, explained and exemplified (2).
The identification and description of competence areas is the first step towards the writing of the learning outcomes (5 p. 4), but one problem is that courses and programs in Higher Education often use different terms when describing the learning outcomes.Competence is also a term used for learning outcomes, together with learning objectives, aims, and objectives.The indistinctly use of terms might be challenging when it comes to writing the learning outcomes, select learning outcomes that are appropriate and reflect the particular purpose, and the context in question (1).
There might be several competencies students can achieve through a course or a program.Which competencies a course or program will give depends on the content of the course (2).Further, there seems to be essential to understand the complexity of each competency, since some may be transversal key competencies.Transversal key competencies consist of several underlying competencies like language, mathematics, learning to learn and cultural awareness.In the course Applied Programming for Teachers, there are listed several competencies students will achieve by attending this course.One competence listed is Good digital competence, and this is a typical example of transversal key competencies.Digital competence is an important key competence which also is listed by the European Parliament as one of eight competencies for lifelong learning (5 p. 1).
In this paper, we will give an example of how the learning outcomes could be written reflecting on previous studies with digital competence as the starting point.The paper is organized as follows.Next section presents a literature review on related work and positions the paper in the context of learning outcomes in the area of digital competencies.Chapter 3 presents a description of the method and case.Chapter 4 presents the results combined with a discussion.This chapter outlines some examples of how the existing learning outcomes can be changed to approximate recommendations identified in the literature review.Chapter 5 present a conclusion and further work.

2
Literature review

Clarification of the terms
An aim is usually a definition of overall achievements that teachers, courses, or program are trying to reach.It tells participants what the course or a session is about and how to ensure that students have achieved knowledge about this in a final test or exam.
Learning objectives were earlier stated as the observable and measurable behaviors that learners should show as a result of participating.Today learning objectives provide more broad-based learning outcomes as goals that are intended to arise as a result (6).The learning outcomes, on the other hand, are pointed out as broader intuitive and userfriendly because this describes students' expected knowledge and skills and what is expected that they will be able to do as a result of engaging in the learning process (7; 8).When students demonstrate what they can do at the end of a course or program, it shows that they have specific competencies.It is the focus on competencies that brings in the concept of learning outcomes.The term competence is widely used throughout Europe, and in several countries substitutes the term learning outcomes (9).There are many different definitions and interpretations with the term competence, that can create some confusion when operating internationally.Fortunately, in The European Qualifications Framework (EQF), it is pointed towards a shared approach where competence means: "the proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations" (10 p. 11).What students know, understand and are able to demonstrate after completion the process of learning is stated in the learning outcomes, and learning outcomes are validated by their relationship to competencies (9 p. 31).The shared approach creates a mutual relationship between learning outcomes and competence (11).Compared to competence, learning outcomes are more distinct.Even if learning outcomes and competencies substitutes each other the term learning outcomes seems clearer.Therefor learning outcomes are easier to use than competence when describing what students are expected to know, understand and/or be able to demonstrate at the end of a module or program.The "fuzziness" of competence disappears in the clarity of learning outcomes.The characteristics of successful courses are that they consist of a clear idea of what can be achieved at the end (11).

Competence
As mentioned in the introduction, the identification of competencies is the first step towards writing the learning outcomes.One competence may have several specific outcomes within knowledge and skills, so a course typically contains more outcomes than competencies (6 p. 31).Digital competence is one of eight key competencies for lifelong learning in the 2006 European Recommendation on Key Competencies, and is defined broadly as: "the set of knowledge, skills, attitudes (thus including abilities, strategies, values and awareness) that are required when using ICT and digital media to perform tasks; solve problems; communicate; manage information; collaborate; create and share content; and build knowledge effectively, efficiently, appropriately, critically, creatively, autonomously, flexibly, ethically, reflectively for work, leisure, participation, learning, socializing, consuming, and empowerment."(5 p. 43) Digital competence is seen as a transversal key competence because it also consists of several underlying competencies like language, mathematics, learning to learn and cultural awareness.This competence is also related to the so-called 21st century skills which all citizens should hold to ensure their active participation in society and the economy (5 p. 1).
For teachers, digital competence is more complex and is also seen up against 21stcentury competencies.The most prominent 21st -century competencies found in international frameworks that have been shown to offer benefits in multiple areas of life are associated with critical thinking, communication, collaboration, and creativity and innovation (7).Based on the literature and previous studies in this field, it is identified four competencies (learning competency, educational competency, social competency, and technological competency) that are theorized as core competencies for teachers' innovative teaching.The term innovative teachers are adopted as a result of teachers' responsibility to attract student's interest and attention in new ways.Innovative teaching is a necessity for all teachers in order to meet the educational needs of the new generations (8).

Teachers digital competence
The technological competence is critical because technology creates great opportunities for teachers to inspire curiosity, imagination, and their students' interest.Competence refers to that teachers are aware of how to integrate educational technologies, provoke critical thinking, and deepen student understanding.Teachers also need to be able to find the necessary information among information available on the internet, to integrate this information which often coming from multiple sources, and to effectively use this information to solve teaching problems (13).Taking in concern the definition of digital competence (also referred to as core competencies, key competence, and generic competence), the definition indicates that being digitally competent involves more than having technical skills.Technical skills and the ability to use specific tools are only two of many underlying competencies within digital competence.However, digital competence for teachers include more than technological competence and is more complex than citizens' average use and other professional groups.While teachers need to hold basic competencies, they also need pedagogical-and didactic judgments based on how technology can expand the learning possibilities for students in certain subjects (14).This leads to the term pedagogical digital competence (PDC) and includes a pedagogical aspect of digital competence into consideration (15).The main characteristic of PDC is thus the ability to develop/improve pedagogical work by means of digital technology in a professional context, primarily in web course/online teaching.However, PDC involves all kinds of pedagogical work in professional contexts where digital technology is used and adds a wider sense to PDC.There are three levels to PDC, and their internal relationship.First, there is on a microlevel an interaction level which involves the pedagogical interaction with students.Second, there is a meso-level, which is on a course level and involve design and implementation of courses, and infrastructure of education which can typically contain integration of resources.Third and last is the macro-level which is on an organizational level.This level focuses on educational management and the development of the organization.Strategic pedagogical leadership is a central component of PDC on all three levels, and the complexity of PDC makes this a competence that can be defined in different ways.Which competence a course will lead to will depend on methodological choices of theories of learning, but it is always something that finds expression in concrete action.Regardless of theory applied PDC can always be evaluated, documented and developed.In principle developed PDC always results in better support for students' learning (15).
Based on this, PDC can be defined as: "The concept of pedagogical digital competence refers to the ability to consistently apply the attitudes, knowledge and skills required to plan and conduct, and to evaluate and revise on an ongoing basis, ICT-supported teaching, based on theory, current research and proven experience with a view to supporting students' learning in the best possible way."(15 p. 48)

Writing the learning outcomes
The Examples of ambiguous verbs are "know", "understand", "enjoy", "determine" and "appreciate".Examples of precise words are "identify", "distinguish between", "assemble", "adjust" and "solve" (9 p. 49).Even if a framework might ease the writing of the learning outcomes, outlining knowledge, skills, and competence as the main elements, there are still some challenges.First, one needs to be careful about treating outcomes of learning as information bits that can be selected and combined at will.This can ignore the extent to which knowledge, skills, and competence are related and interdependent and lead to neglect of the conditions (9 p. 45).Second, it might be challenging to select learning outcomes that are appropriate and reflect the particular purpose and context in question (1).Bloom's taxonomy has been reviewed and adjusted several times, but some describe the original Bloom's taxonomy from 1956 as a good starting point for writing the learning outcomes (7).The taxonomy is a hierarchical level categorization of cognitive learning, where one moves from basic knowledge and comprehension to increasingly complex skills.The action verbs associated with each level of this taxonomy is a good starting point for writing the learning outcomes.Table 1 shows in the left column the levels of the taxonomy, while the right column is the verb.A good learning outcome will have a verb that identifies what students should be able to do, and under what conditions they should be able to demonstrate this.According to Bloom's taxonomy, well-written learning outcomes will consist of verbs that identify what action the students should be able to perform, under which conditions students should demonstrate the mastery, and how the mastery may be evaluated (7).
Knowledge is stated as one of three elements in the written learning outcomes, and when describing the knowledge, it should consist of words like define, identify, label, state, list, and match.Comprehension, application, analysis, synthesis are all levels that refer to the element skills (7).Each level includes verbs that should be mentioned in the learning outcomes as shown in the right column in Table 1.The last element in the learning outcomes, competence, is when students use knowledge and skills in work or study situations (10).The main distinction between competencies and true learning outcome is that the learning outcomes are written so that it can be measured or assessed.For example, to state that a student will understand or know some facts or topics are good objectives, but it is not easily measured (7).Having in mind that objective is roadbased learning outcomes as goals that are intended to arise as a result (6).Readers are referred to the Taxonomy of educational objectives: the classification of educational goals (16), for a detailed explanation of Bloom's taxonomy.
In the introduction, it was pointed out that students need specific outcomes to reflect on throughout and after a course.The written learning outcomes are important for the students throughout a course to be able to identify which way they are going.The learning outcomes are also important for students after a course to be made aware of the competencies that they have developed during a course.Without, the learning outcomes it would be like planning a journey and don't know where to go before the journey start, you may end up somewhere you do not want to be (6).However, it is worth mention that students are unlikely to seek out, read or reflect on the learning outcomes unless they are specifically encouraged to do so by teachers, or communicated through assessment practices (4).

Description of method and case
The purpose of this research is to promote some explanations and clarifications that can ease the writing of the learning outcomes.This case study is organized according to the pattern identified in relevant literature, frameworks, and standards common for European countries.Findings in the literature are compared to the learning outcomes as it is written in the course Applied Programming for Teachers today.Each element used in this research is examined separately before considering all of the elements together.
As explained in the introduction, this is not a close examination of the learning outcomes for the course in this research but outlines some example on how to write the learning outcomes, and where to start.The Norwegian University of Science and Technology (NTNU) have created a Development Guide for study plans and course descriptions that is a standard requirement specification for all studies.The Development guide points out that the description of the learning outcomes on a course level should be more concrete and measurable than the descriptions on a program level.This because it is the sum of all courses in a program that ensure that the total learning outcomes of a program are covered.Further, the Development guide provides some example on how to write the learning outcomes focusing on knowledge and skills.One example is that formulation such as "The candidate understands key theories within..." should rather be described as "The candidate shows good understanding of theories within…".Another example is the use of active verb with explicit subjects and rather write "candidate understand instead of "it is understood" (17).The Development guide offers no explanation of what each category knowledge, skills, and competence are or why each of them is important but is a table divided into the three categories knowledge, skills, and competence.Several institutes at NTNU have developed forms based on this guide.Here one can fill in information related to the course or program, and the learning outcomes are one part of the form.The form does not explain what each category knowledge, skills, and competence are or why each of them is important.The form is also used in the description of the learning outcomes in the course Applied Programming for Teachers.
Applied Programming for Teachers is one of two online courses for teachers within ICT Programming.These two courses build on each other where the first one is Basic programming for teachers.The second course, Applied Programming for Teachers, will give students a deeper understanding of basic programming and how this can be applied to solve issues within different subject areas.The learning objective of the second course is to provide in-service teachers with insight on how to use programming to create digital solutions.This course focuses on the knowledge students need to become a teacher, and how they can ease students learning process and understanding of programming.Further, it gives guidance to programming in school and other subjects and activities where programming is used to support learning.The course currently has 83 participants, but this will more than double for the next year.The course has already received applications from 320 qualified applicants, of which 200 of these will be invited to participate in the course.Table 1 shows the description of the learning outcomes as it is written today.Here the teacher started on the top of the form describing the knowledge, the skills and finally the competencies.• Knowledge of programming languages, tools and methodology, both pedagogically oriented solutions and solutions that are used professionally, • Basic understanding of the software's function in electronics and robots • Basic insight into the technique and methods for testing and misfire in major program projects

Skills
• Develop and test programs with some complexity, • Utilize modern programming tools and assess their suitability in teaching and learning; • Identify and evaluate programming that should be included in simple technological solutions, understand the difficulty, scope and suitability in a teaching situation, • Understand how creativity and collaboration can be utilized to promote programming learning.

Generic competence
• Convey knowledge of programming and the didactics of the subject to others, both written and oral, • Discuss, describe and evaluate solutions with some complexity, • Plan varied work tasks and programming projects, focusing on creativity and social learning; • Evaluate ethical issues related to programming, • Demonstrate good digital competence The First author in this paper is currently a Pd.D. candidate in Computer Science, with the aim to highlight what competencies students hold after completing higher IT education.The second author is the primary teacher and responsible for the subject Applied Programming for Teachers and is also responsible for writing the learning outcomes in this course.In this research, the first author has done the writing and review of the literature.Together we have discussed challenges that may arise when writing the learning outcomes and gone through the learning outcomes as they are today for this course.The discussion outlined that without any clear guideline on what knowledge, skills, and competence could consist of and where to start, it is challenging to write the learning outcomes.One competence often mentioned in 21st-century education, and essential for teaching today is digital competence.This is the competence that will be discussed further in this paper related to the learning outcomes.

Result and discussion
A starting point for writing the learning outcomes pointed out in the literature is to identify competencies students can achieve by attending a course or a program.One competence students will hold after attending the course Programming for teachers is stated to be Demonstrate good digital competence.This statement raises questions like how can good digital competence be demonstrated, and what is good digital competence?First, digital competence consists of several underlying competencies because it is a transversal key competence (5).To stat that participants get digital competence by attending one course could be a little ambitious, but may be suited better for a program.Second, to demonstrate something will be connected to skills, and under what condition students should demonstrate the mastery (7).The digital competence for teachers is in additional transversal because it includes pedagogical-and didactic judgments.When pedagogical-and didactic judgments are included the existence of other underlying competencies occur.An underlying competence may be more suited for Applied Programming for Teachers.Example of one underlying competence that might suit the course is technological competence, which includes how to integrate educational technologies, provoke critical thinking, and deepen student understanding.Further, this provides skills such as synthesis and analyze.For example, to integrate information from multiple sources requires the ability to plan and organize, in another word synthesis.To be able to integrate information from multiple sources also requires the skill ana-lyze because the information from several sources needs to be compared.Also, competencies on a micro level can be included like pedagogical interaction with students.The micro level will also lead to strategic pedagogical leadership competence (15).Given that the course focuses on the knowledge students need to become a teacher, and how they can ease students learning process and understanding of programming, it also leaves students with the tree competencies Learning competence, Educational competence, and Social competence.
When it comes to knowledge and skills, there seem to be significant differences between the recommendations given by Cedefop (9) and Blooms Taxonomy described by Hartel and Foegeding (7), and the Development guide provided by NTNU (17).While Cedefop (9) and Hartel and Foegeding (7) refer to words like know and understand as ambiguous verbs, the Development guide recommends these verbs.Thus, it may seem that the Development guide is somewhat indistinct.First, the Development guide recommends ambiguous verbs, and second, it points out that the description of learning outcomes on a course level should be more concrete and measurable than the specifications on a program level.This may lead to some confusions because ambiguous verbs may appear to be a contradiction to something that should be precise.Further, the Development guide does not offer a description of where to start when writing the learning outcomes, and the focus on competence is very limited.Since the various competencies require students to hold specific knowledge and specific skills, competencies are the first step towards writing the learning outcomes (5 p. 49).
When it comes to the knowledge listed in Table 2, this also seems to be ambiguous, if we are to follow the verb described in Bloom's taxonomy.Statements like knowledge of, detailed knowledge of, basic understanding, and basic insight, are all words that not easily are measurable.At the same time, words like this can ignore the relationship with subsequent ability and competence (9 p. 45).Knowledge is information students should remember, and consists of action verbs like define, identify, label, state, list, and match.When students prove that they are capable of doing some of this action, they show that they possess the knowledge the course is meant to provide.Since what is listed within knowledge in Table 2 do not consist of action verb the knowledge may not be sufficiently described as it is today.Two examples here are Detailed knowledge of constructions and structures in modern programming and Basic insight into the technique and methods for testing and misfire in major program prospects.The first example could be replaced with Identify constructions and structures in modern programming.The second example could be replaced with State misfire in major program projects, technique and methods for testing.To state that a student will understand or know some fact or topic is a good objective but are not fitted as learning outcomes (8).Words like know and understand is too ambiguous and might create some confusion about what's actually learned.
Also, the element skills might be confusing for students in this course, and especially Understand how creativity and collaboration can be utilized to promote programming learning.Again, understand together with knowledge of some facts or topic could be a good objective for the course, but may not fit for describing what student actually can do.Creativity is a valid term to use and is also one of the mentioned in teachers' digital competence (12).In Table 2, however, creativity is described as something students should understand instead of with an action verb that typical shows that students can demonstrate creativity.This might be more appropriate if this skill were described with words like putting parts together to form a new and integrated whole, or in other words, design.Using verbs from Bloom's taxonomy, this skill could be written, for example, as Design programming learning through creativity and collaboration.
The writing of the learning outcomes for this course might have been challenging because the competencies that are described sounds more like knowledge and skills except for Digital competence.Example of skills are the words plan, evaluate and convey listed under the element competencies, in Table 2.These are all action verbs connected to skills.and might be more suitable for this purpose.There are listed several competencies in the learning outcomes for the course Applied Programming for Teachers and one competence will typically have several specific outcomes.Looking at the amount of knowledge and skills in Table 2, compared to competencies listed, might be the first sign that this might not be an adequate learning outcomes.In the learning outcomes five competencies are listed, four types of knowledge and four types of skills.This is less than two outcomes for each competence.Also, as mentioned earlier, since this is a course for teacher's competencies attached to pedagogical-and didactic judgment might be more appropriate.Descriptions of pedagogical work in professional contexts or strategic pedagogical leadership might be better choices than Good digital competence for the course Applied Programming for Teachers.This involves all kinds of pedagogical work in professional contexts where digital technology is used.It may also be advisable to look more closely at the remaining competencies listed in Table 2.Here it can be recommended to take into account what competencies are and are not, and Bloom's Taxonomy described briefly in chapter 2.4 Writing the learning outcomes.
Plan varied work tasks and programming projects, focusing on creativity and social learning, seems more like skills.Also, the competence Discuss, describe, and evaluate solutions with some complexity, is more connected to skills than it is a core competence.To be able to state that students have a specific competence, they who write the learning outcomes need to know what is expected within that competence.For example, reviewing relevant literature showed that digital competence is a complex competence because it consists of several underlying competencies.It is also recommended to look into the content of the course and identify if the content is suitable for the competencies that will be listed in the learning outcomes (2).

Conclusions and further work
This research shows that writing meaningful learning outcomes can be challenging, even with a development guide.A development guide needs a description of the meaning of each element knowledge, skills and competence, and where to start.Without this description, there is a risk of neglecting the relationship and interdependent between knowledge, skills, and competence, as seen in this case.For the course Applied Programming for Teachers, several competencies listed in Table 2 are identified as skills., and this might be the reason for the use of ambiguous descriptions listed in knowledge and skills.
Today only the Good digital competence appears as a pure competence, but it may be too broad for a single course.Further, digital competence might not be appropriate because it does not reflect the particular purpose and context in question.Since this is a course for teachers, it should include pedagogical-and didactic competencies.Also, competencies on a micro level like pedagogical interaction with students, strategic pedagogical leadership and learning competence, educational competence, and social competency, may be appropriate for the course Applied Programming for Teachers.
The identification of competencies student can gain by attending a course is the first step towards the development of the learning outcomes (5 p. 4).It is indicated through the literature that there is a need for knowing what is expected within each competence listed, and then look into the course content and identify if this suits the competence.Further, words like know and understand are challenging to measure and asses.This indicates that several of the descriptions listed in Table 2 are too ambiguous.The description of the learning outcomes as it is written today may give unclear dependencies between knowledge, skills, and competencies.The description may also create confusions among students when it comes to where they are going and what is expected that they can do at the end of the course.The examples given chapter 4 Result and discussion might provide some guidelines on how to write the learning outcomes.This paper has not completed a full evaluation of the learning outcomes for the course Applied Programming for Teachers but given some example, based on a review of relevant literature.Further work should evaluate the learning outcomes as a whole for this course, and identify competencies through the course content.As mentioned in the introduction, students' reflection on learning outcomes will promote students with knowledge of where they are going and what they can do at the end of a course.Further work should, therefore, also investigate ways to use the learning outcomes for reflection, to better prepare students for employability through the competencies they gain.
European Qualifications Framework for Lifelong Learning defines learning outcomes as a statement of what a learner knows, understands and can do when completing a learning process.What a student knows, understand and can do are defined in terms of knowledge, skills, and competence.(9 p. 17).The same definition is used in Defining, writing and applying learning outcomes: A European Handbook (5 p. 29).One of the basic theories in the European Handbook is Bloom's taxonomy.Bloom's taxonomy is an important theory that have influenced the thinking about learning outcomes and progression.A central focus in this taxonomy is action verb, and good learning outcomes will be described with action verbs that identifies what action the student should be able to perform.The action verbs are made solely associated with the cognitive dimension of learning, but it is important to avoid ambiguous verbs.

Table 2 .
Learning outcomes: Applied Programming for Teachers