Modelling Competency in the Field of OOP: From Investigating Computer Science Curricula to Developing Test Items

. In this paper, we describe the results of a thorough analysis of 44 K12 computer science curricula and standards documents conducted as part of an ongoing research project aiming at the development of a competency structure model and measurement instruments in the field of object-oriented programming (OOP). The curricula analysis builds upon a first model draft derived theoretically from a literature analysis in prior work. The model draft is 4-dimensional and consists of the four competency dimensions 1) OOP knowledge and skills, 2) Mastering representation, 3) Cognitive processes and 4) Metacognitive processes. We used these dimensions and the belonging sub-dimensions as a coding scheme and coded competency facets concerning OOP contained in the curricula and standards documents using the method of qualitative content analysis according to Mayring. This way, we could firstly successfully prove the curricular validity of our model draft and secondly, after a step of paraphrasing the identified competency facets, use these descriptions to initiate the process of item development to operationalize our competency model draft.


Introduction
One of the current trends in educational research is the defining and measuring of learning outcomes in terms of competencies [1,2].Besides various competency modelling approaches in different disciplines in Germany (e.g.[3,4,5]), on an international level a lot of research has been performed in the context of the Programme for International Student Assessment (PISA) by the Organisation for Economic Cooperation and Development (OECD) which -since the year 2000 -focusses every three years on competencies in the areas of mathematics, science and language.Formerly defined post-hoc (e.g.[6]), the assessed competencies now rely partly on empirically founded competency models [7].The results of these regular assessments are used to compare the educational levels of different countries or federal states.An expertise which analysed the "shocking" results (at least from the German perspective) of the first PISA studies under consideration of the different national educational systems came to the key conclusion that those countries performed best which oriented their educational systems at the intended learning outcomes [8] (and not at detailed curricula as before) and recommended (besides other aspects) the development of empirically founded competency models for all educational fields.
Concerning computer science education (CSE), the demand for competency models is still up to date.Although different normative models have been developed as educational standards in the past years (e.g.[9,10]), empirically founded competency models in the CSE field are still rare (e.g.[4]).So, we have started a project to develop such a competency structure model and appropriate measuring instruments for the field of object-oriented programming (OOP), which is widely included in many K12 computer science curricula worldwide.The long-term goal of this research is to develop instruments to compare the learning outcomes in the CSE field nationally and internationally between different student populations.

On the Concept of Competency
In the context of competency modelling and assessment the term competency is often used according to the definition of Weinert.In his expertise for the OECD [11] he defined competencies 2001 as "the cognitive abilities and skills possessed by or able to be learned by individuals that enable them to solve particular problems, as well as the motivational, volitional and social readiness and capacity to use the solutions successfully and responsibly in variable situations".Despite the awareness that metacognitive factors can have major impact on learning results, these aspects of competencies are quite difficult to assess.Hence, Klieme et al. used a more economic definition in a priority programme of the German research foundation on competency modelling by defining competencies as "context-specific, cognitive dispositions of achievement which are functionally related to situations and requirements of certain domains in the sense of specific learning and action fields" [12].In this context, competency models are used to describe the cognitive dispositions that learning individuals need to solve tasks and problems in a specific content or requirement area.Klieme et al. distinguish between modelling the structure, the level and the development of competencies in corresponding competency models [1].Schecker and Parchmann [13] distinguish between normative (which describe the behaviour learners should show after a specific period of education, usually found in educational standards) and descriptive competency structure models (which describe the cognitive processes that occur in specific learning situations).

The COMMOOP Project
Our project COMMOOP aims to develop a normative competency structure model [1,13] and appropriate measurement instruments for beginners, such as secondary school or undergraduate students, in the field of object-oriented programming (OOP).We started by reviewing existing literature on competency modelling in other subject areas (such as mathematics, science, humanities, arts, computer science and problem solving) regarding the development methodology as well as the model structures (for a detailed description of this process, see [14]).We identified the areas of knowledge structures, content representation, cognitive processes as well as metacognitive influences as common, recurring structural elements, and verified, expanded and refined these based on an extensive literature analysis on theoretical and empirical studies on teaching and learning as well as on psychological aspects in the field of OOP.As theoretically derived candidates for potential competency dimensions we identified: The proposed dimensions span a multidimensional competency space.Each competency candidate that is contained therein covers a certain subspace that is limited by fixed values on some dimensions that can be assigned to this competency.For example, the competency candidate "be able to program arrays in Java" might cover a subspace that is limited by the following fixed values: 1.1 Array, 2 Java, 3.1 Translating the solution in a computer program and 3.2.Understand and Apply.The remaining dimensions might be covered in total by this competency.It has to be noted that, due to national regulations of Java being one of the predominant languages in schools, our research will first focus on this language.
The theoretically derived model draft was validated based on various competency descriptions in terms of applicability and completeness.For this purpose, we identified competency descriptions related to OOP in 44 computer science curricula and standards from several countries and compared these with our model (see section 3).In 2015, an international working group at the ACM ITiCSE conference analysed 14 case studies on K12 computer science education in 12 different countries, which resulted in 247 extracted competency descriptions.Out of these, 119 were related to programming and could be fully integrated in our model.Finally, the structure model was aligned with the results of a survey among 59 computer science teachers and teacher students on learning difficulties.Their views on which skills and abilities someone needs, to become a competent programmer as well as their point of view on typical problems during that learning process could be also fully integrated in our model.
During this process, we had to adjust our theoretically derived model only in minor subsections (see section 3).

Learning Metrics, Psychometric Models and Item Development
The discussion around coding to be the new literacy is vivid and the way to a proficiency scale in computer literacy is in its first stages.In PISA the development of scales followed six steps: (1) Identifying possible scales, (2) Assigning items to scales, (3) Skills audit, (4) Analysing field trial data, (5) Defining the dimensions and ( 6) Revising and refining with main survey data.
In a first step, we identified OOP as an important subscale [14].Currently we're dealing with the construction of items as well as bringing them on their way to first field trials.For this stage of development, a more functional concept of competency is followed.This perspective is more interested in abilities to cope with challenges in specific situations rather than in the generative and cognitive systems behind [15].From this background items have a single achievement threshold; the derived information only distinguishes dichotomously between correct and incorrect answers.Thus, the Rasch model can be applied to the raw data, in PISA a generalized form of the Rasch model has been used [16].
The Rasch model is "the only model to date that provides the tools for approximating objective reproducible additive measures in the human sciences."[17, p. 8].It follows the concept that data conforms a reasonable hierarchy on a single continuum of interest.The most important fact is the measuring of a single variable with a map of persons (ξ v ) and items (σ i ) on the same scale, therefore the relationship between the participant and the item is probabilistic.The resulting model can be regarded as a model of competency levels.While those levels have already been regarded in the step of item construction, they will be precised by the step of skills audit and the analysis of the field trial data.At this stage numerical scores and terms of content, which describe what persons who achieve a specific level know and can do, can be reported.The final assessment of COMMOOP will be a computerized adaptive test (CAT), wherefore the Rasch model is essential [18].
Generative and structure models are more interested in the components behind the specific ability in a concrete item [1].Modelling latent variables, like working memory or intelligence, comes into the focus, which is mostly driven by cognitivism.These models are rather the result of small experimental designs than of large-scale assessments.Modelling the cognition behind competencies is nowadays a challenge.One of its fruits will be the improvement of intelligent tutorial systems (ITS) and the rise of cognitive tutors.A pioneer of this type of modelling is John R. Anderson and his ACT-R Theory [19,20].Anderson and his colleagues have realized different ITS considering the cognitive limitations of the learner, one specific ITS is about learning to code in LISP.At a later stage COMMOOP will regard cognitive models for code trainings.The prior testing will combine the most promising items with different working memory, intelligence and planning scales.

Analysis of K12 Computer Science Curricula and Standards
A key step in our model development process (see section 2.2) was a thorough analysis of K12 computer science curricula and educational standards.Altogether, we analysed 44 documents (German and international ones) and extracted competency facet descriptions concerning OOP from them.We performed this step because of two main reasons: first, we wanted to check the applicability of our theoretically derived normative competency model draft by trying to assign the competency facet descriptions to the proposed model categories to check and to ensure "curricular validity".This was an important step also in other competency modelling projects (cf.e.g.[3,4]).Second, we wanted to prepare the operationalization of our model draft by starting to develop test items suitable to measure the competency facets found in the curricula.
Since we plan to measure OOP competency first in German schools, we included all publically available computer science curricula of Germany's 16 federal states as well as national standards documents of the Board of Federal Ministers of Education and Cultural Affairs and the German Informatics Association in the analysis.If a federal state had different curricula e.g. for lower and upper secondary computer science education, we included both documents in the analysis.To ensure the applicability of our model also in the international context, we also investigated important documents published in English language by other nations, such as the "Computing at school" curriculum (UK), as well as recommendations of international organizations, such as the ACM/IEEE curriculum and the CSTA K12 standards.We included documents from secondary and higher education, because we aim to model and to measure competencies of programming novices regardless of the educational level.
Curricula and standards documents are usually highly structured documents, in which intended learning outcomes are ordered e.g. by age level or by fields of the discipline (such as "algorithms" or "impact on society").We used scientific definitions of the term "programming" taken from the literature (e.g.[21][22][23]), extracted possible search terms from them, such as "writing" and "implementing", and searched each of the documents for the sections, in which relevant OOP competency facets were located.Within these sections we performed a qualitative content analysis (QCA) according to Mayring [24] using the tool MAXQDA.We used the dimensions and subdimensions of our theoretically derived normative competency model draft as our category system and coded each OOP competency facet found in the documents with its belonging category.The QCA was performed deductively and inductively (cf.[25]), so the theoretically derived category system was extended inductively whenever a new category candidate was found in the documents.
Since we cannot present the whole coding process and its results in detail here, we give two examples.The text fragment "By the end of this course, students will […] use constants and variables, including integers, floating points, strings, and Boolean values, correctly in computer programs" (taken from the Ontario curriculum for Computer Studies in grades 10 to 12, Canada) was coded with the category OOP knowledge & skills, respectively with the subcategory data structure.The expected outcome that students should "know the basics of object-oriented programming, e.g.inheritance, polymorphism and encapsulation" (taken from the computer science curriculum of Saxony, Germany) was assigned to the subdimension class and object structure.
Overall, our model draft proved to be well applicable for the classification of most of the OOP competency facets found in the documents, which contributes to the curricular validity of our approach.Still there are three places, in which we modified it during the coding process: First, we found competency descriptions relating to the ability of being able to evaluate a different solution approach for a problem, so we added a subdimension "Evaluating different strategies" in the proposed competency dimension Problem solving stage.Second, it became obvious that syntax and semantics are almost inseparable and that third, there was a need to code competency facets relating to "Documenting and maintenance" (both: top-level dimension Mastering representation) to cover text fragments relating to the use of comments in the code as well as to writing clear and maintainable code.Especially because of the inseparability of syntax and semantics we decided to combine the subcategories belonging to Mastering representation in the top-level category and to analyse its inner structure in future work.Table 1 shows the distribution of codings at the end of the coding process.The next important step of the QCA according to Mayring [24] is the paraphrasing and summarizing of specific categories.The aim of this step is to combine similar competency facet formulations in fewer distinct joint formulations.Table 2 shows the result of this process within the dimension "OOP knowledge and skills".
It must be noted here that composite competency facet formulations like "Students are able to assign values to variables of primitive and compound data types as well as more complex data structures" were coded with more than one category.
It is not surprising, that fundamental algorithmic structures can be found most often in the documents, since neither does every country or state follow an object-oriented paradigm nor, in case they do, is the objects-first approach the most dominant one.

Developing Test Items
The next step on the way to an empirically validated competency structure model for OOP (see 2.2) is the operationalization of the competency facets of our theoretically derived model by describing specific situations that require a certain behaviour that persons show who are competent in this area.While competency formulations describe a subspace of skills and abilities limited by certain parameters (see 2.2), the items should only test for a distinct competency dimension in the model to prove that the assumed multidimensionality of the model can be shown empirically.Moreover, the items have to be constructed in a way that solving them correctly will indicate the underlying competency.Hence, many test items of varying difficulty are required.Furthermore, it is necessary to take into account the probands' previous knowledge to ensure the validity of the results.In Germany, students at the end of grade 12 should have the necessary competencies when they have gone through upper secondary computer science education or when they completed an "Introduction to OOP" course at university level.
As a result of the curricula analysis we have collected numerous formulations describing OOP competency facets and abstracted those in the paraphrasing step (see section 3).These descriptions are now used to construct test items, respectively to gain access to programming competencies and hence to validate the assumed theoretical competency structure on an empirical level.In the following, we give two examples of test items to be used at the end of an introduction to OOP.Item 1 is a considerably easy item addressing the understanding of classes as templates, objects as instances, attributes as characteristics and methods as modifiers of attribute values:

Table 1 .
Coding results of the curricula analysis.

Table 2 .
Categories formed in the subdimensions of the dimension OOP knowledge & skills by paraphrasing and summarizing the competency facet formulations