Evaluation of the learning path specification.
Flexible lifelong learning requires that learners can compare and select learning paths that best meet individual needs, not just in terms of learning goals, but also in terms of planning, costs etc. To this end a learning path specification was developed, which describes both the contents and the structure of any learning path, be it formal, non-formal, informal, or indeed a combination of these. This paper briefly explains the learning path specification and presents a framework for the evaluation of the specification based on theories of model quality. A study of learner choice processes (n=15) was carried out to investigate the specification's semantic and pragmatic quality (clarity, completeness and parsimony) with respect to the selection of a learning path. Results indicate that the specification does not contain any redundant information. Instead, the study has led to improvement of the specification's (feasible) completeness by further refinement of scheduling information.


Formal learning, Informal learning, Learning path specification, Learner choice

Article Type:
Learning (Evaluation)
Janssen, Jose
Berlanga, Adriana J.
Koper, Rob
Pub Date:
Name: Educational Technology & Society Publisher: International Forum of Educational Technology & Society Audience: Academic Format: Magazine/Journal Subject: Computers; Education Copyright: COPYRIGHT 2011 International Forum of Educational Technology & Society ISSN: 1436-4522
Date: July, 2011 Source Volume: 14 Source Issue: 3
Geographic Scope: United States Geographic Code: 1USA United States
Accession Number:
Full Text:

Learning paths can be roughly defined as sets of one or more learning activities leading to a particular learning goal. Learning paths can vary from a relatively small activity like reading a book or taking a course to following an entire programme or curriculum. Learning paths may vary also regarding the level of formality. In line with the Commission of the European Communities we distinguish formal, non-formal and informal learning (CEC, 2000). Whereas formal learning occurs in education and training institutions and leads to recognised diplomas and qualifications, informal learning is described as "a natural accompaniment to everyday life" which is not necessarily intentional learning (CEC, 2000, p. 8). Non-formal learning, finally, is learning that takes place alongside the mainstream systems of education and training, for instance at the workplace or in arts or sports, which does not necessarily lead to formalised certificates.

Lifelong learners' learning paths consist of a mixture of formal, informal and non-formal learning (Colardyn & Bjornavold, 2004; Colley, Hodkinson, & Malcolm, 2003; Livingstone, 1999). In order to support lifelong learners in comparing and selecting suitable learning paths, a uniform way to describe learning activities and learning paths has been developed, which covers these different ways in which people learn (Janssen, Berlanga, Vogten, & Koper, 2008).

The specification is envisaged to support several processes. Firstly, it is meant to be used by educational providers to describe formal and non-formal educational courses and programmes in order to make them available through specific search engines, thus enabling comparison across providers. We assume that educational providers will want to describe learning paths in a uniform, formalised way, because the benefits of transparency and opportunities for automated learner support outweigh the costs. Costs can be relatively low since educational providers already have to describe their offerings; it will merely be a matter of organising this information in a way that enables storage and update in one place and subsequent use in different contexts: printed catalogues, websites, and search engines.

A second process the learning path specification is meant to support was initially defined as follows: lifelong learners use the specification to describe their informal learning paths to make them available as an example to other learners with similar learning goals. However, a pilot-study revealed that it requires considerable efforts and skills on the part of the learner to identify activities that did or did not after all contribute to achieving those outcomes. To distil a learning path from one's own informal learning experiences and describe it in a way that is useful for others, is not an easy task (cf. Skule, 2004). Though we still maintain that the specification can be used to describe all kinds of learning (a point we later further elaborate), we believe that in the case of informal learning it is not likely going to happen on a large scale by lifelong learners themselves, because it requires learning design skills. It is not unreasonable though to expect employers and employment agencies to be willing to invest in these descriptions of informal learning paths as they can offer tried and tested alternatives to more costly formal and non-formal learning paths. Research indicates that people spend an average of 6 hours a week on employment related informal learning (Livingstone, 1999) and description of these informal learning paths is likely to enhance efficiency when they can offer guidance to learners rather than have them find things out through trial and error. In any case, the second process the learning path specification is meant to support eventually is defined as: description of informal learning paths in order to make these learning paths available for other learners with similar learning goals.

Finally, a third process the learning path specification is envisaged to support is selection of suitable learning paths. To this end the specification identifies main characteristics to be used in comparing and selecting a learning path (e.g. learning objectives, prerequisites, study load, costs, etcetera). Lifelong learners must be offered means to efficiently choose the learning path that best fits their needs. Taking a decision support perspective, we distinguish two stages in this process: screening and choice (Beach, 1997; Rundle-Thiele, Shao, & Lye, 2005). Screening involves selecting a number of options one wants to take into consideration, i.e. narrowing down the number of choice options to a number that can be "managed". Research shows that choice overload may occur due to the number of available options, as well as to the number of attributes related to these options (Fasolo, McClelland, & Todd, 2007; Malhotra, 1982). In other words: having to choose from a large number of learning paths is one thing, having to compare even a limited number of learning paths might lead to choice overload when a large number of attributes are related to these options. But even apart from these considerations regarding choice overload, lifelong learners will rather invest the scarce resources of time and attention in developing competences than in comparing all kinds of ways to do so. What is needed then is some tool for the learner to select a limited set of learning paths to take into account in the choice process.

There are quite a number of criteria that could be relevant to finding the most suitable learning path but not all criteria might be equally relevant to all learning paths or to all learners for that matter. The study of Fasolo, McClelland and Todd (2007, p. 23) shows that "it is possible for consumers to make good choices based on one or two attributes, when attributes are positively related or consumers care unequally about attributes and choose on the basis of the most important ones". To the extent that learners do not equally care about the learning path attributes included in the learning path specification progressive disclosure of functionality could contribute to help the learner focus on those criteria that are most relevant for her (Turbek, 2008). Progressive disclosure is a strategy for managing information complexity in which only necessary or requested information is displayed at any given time (Lidwell, Holden, & Butler, 2003).

Requirements for the specification have been derived from a review of literature on curriculum design and lifelong learning as well as observations of current practices to support learner choice (Janssen et al., 2008). This paper describes a study directed towards evaluation of the conceptual model of the learning path specification. It provides an outline of the specification and explains how the specification supports description and selection of learning paths. Subsequently a framework for the evaluation of model quality is presented, guiding the specific research questions. Finally the paper describes method and results of the evaluation.

Learning path conceptual model and specification

According to Moody (2005) conceptual modelling is a process of formally documenting a domain (a system or a problem) in order to enhance communication and understanding. He further points out that conceptual modelling may be used to describe requirements at different levels: functional and non-functional requirements at the level of an application, and information requirements at the level of an organisation or even an industry.

A formal specification can be considered a conceptual model as is illustrated by the following definition: "a formal specification is the expression, in some formal language and at some level of abstraction, of a collection of properties some system should satisfy" (Van Lamsweerde, 2000).

The learning path specification identifies information requirements for lifelong learners: generic elements of a learning path which are essential to selecting, planning and executing a learning path, such as learning goals, learning actions, delivery mode, etc. It describes fixed as well as optional elements; both contents and structure.

Like any other path a learning path has a finish and a start (i.e. learning goals and prerequisites). In order to get to the finish one or more learning actions have to be completed. Learning goals and prerequisites can be specified both at the level of the learning path and its constituent actions. They are preferably defined in terms of standardized competences so as to facilitate automated identification of parts of a learning path a learner may skip when these competences have already been attained through prior learning (Kickmeier-Rust, Albert, & Steiner, 2006). Learning actions can be grouped into clusters, for instance when they compose a set a learner can choose from (selection), or because they have to be studied in a particular order (sequence). Table 1 describes the information about the learning path and its constituent actions considered relevant in identifying and selecting suitable learning paths, and therefore included in the specification. (For a more detailed account of the attributes and metadata associated with each of the elements see Janssen, Hermans, Berlanga, & Koper (2008)).

Formal, non-formal and informal learning paths

The distinction between formal, non-formal and informal learning is not as clear-cut as the definitions provided in the introduction suggest. Schugurensky (2000) stresses the fact that informal learning can also take place inside formal and non-formal educational institutions: within these institutions some learning occurs independently of the intended goals of the curriculum. Using two categories (intentionality and consciousness) he goes on to identify three forms of informal learning: self-directed learning (intentional + conscious), incidental learning (unintentional + conscious) and socialization (unintentional + unconscious). The learning path specification is merely meant to enable description of informal learning with the aim to suggest informal ways to develop competences, drawing from other learners' personal informal learning experiences. This means the learning path specification is only meant to cover conscious informal learning. As to the intentionality of learning it is often stated that workplace learning and other informal learning have no formal curriculum or prescribed outcomes (Hager, 1998). Regarding unintentional conscious learning we maintain that this type of learning can be described in hindsight as a learning path, describing the previously unintentional learning outcomes as learning objectives, to present as an option to other learners interested in achieving these learning objectives.

Concerning the distinction between informal and non-formal learning, a major review of literature suggests there is no clear agreement: the terms are used interchangeably (Colley et al., 2003). Nor does it appear possible to distinguish formal learning from other learning in ways that have broad applicability or agreement. The authors conclude it is more sensible to consider attributes of informality and formality present in all learning situations. These attributes concern four aspects of learning:

* Process: informality and formality attributes relating to the learning process relate to questions like who's in control of the process (teacher controlled versus student led), whether and what kind of assessment is involved (formative or summative).

* Location/setting: where does the learning take place (e.g. in an educational institution, at the workplace, etc.) and does it involve certification?

* Purposes: is learning intended or does it happen unintentionally; are learning outcomes determined by the learner or designed to meet needs which are externally determined?

* Content: does the learning focus on acquisition of established knowledge or development of knowledge from experience?

Attributes relating to the process aspect of learning included in the specification are the metadata elements "guidance" and "assessment". The location/setting aspect is covered by the metadata "recognition", "delivery mode", and "location". Regarding the purpose aspect we conclude that the learning path specification only covers intentional learning: a learning path is directed towards learning goals. This does not mean that the learning path specification cannot be used to describe unintentional learning as well, but this would always be in hindsight: learning which has occurred unintentionally can be retrospectively described to serve as an example to other learners who can then embark on the same path purposefully. Attributes of formality and informality relating to the content aspect of learning can be described through the metadata element "description" of the learning path as well as of its constituent actions. Whether the learning goals of a learning path are achieved through "formal knowledge acquisition" or through "learning by doing" will be of interest to the learner, but whether it requires a separate metadata element remains to be seen.

Model evaluation: a framework

Seeking alignment with the ISO 9000 definition of quality Moody (2005) defines conceptual model quality as "The totality of features and characteristics of a conceptual model that bear on its ability to satisfy stated or implied needs" (p. 252). Based on a review of research in the field of conceptual model quality Moody concludes that there are no generally accepted guidelines for evaluating the quality of conceptual models. Nor do experts agree as to what makes a conceptual model a "good" model. One of the explanations given for this lack of consensus is that a conceptual model exists as a construction of the mind, and therefore quality of a conceptual model cannot be as easily assessed as the quality of a concrete product: "While the finished product (the software system) can be evaluated against the specification, a conceptual model can only be evaluated against people's (tacit) needs, desires and expectations. Thus the evaluation of conceptual models is by nature a social rather than a technical process, which is inherently subjective and difficult to formalise"(Moody, 2005, p. 245).

The learning path specification is a case in point: rather than a "finished product" it is a model to describe learning paths which can be used to develop tools to support lifelong learners in finding and navigating suitable learning paths. This implies a number of stakeholders:

* lifelong learners

* learning path designers

* providers

* software developers.

Someone interested in finding suitable learning paths is likely to focus on different aspects of the learning path specification than someone interested in designing learning paths or in developing tools to support these processes. Consequently, evaluation of the specification requires input from these different perspectives.

Addressing the lack of consensus in the field Moody (2005) proposes the ISO/IEC9126 software quality model as a template to structure conceptual model quality frameworks. This template identifies the following important features:

* hierarchical structure of quality characteristics (characteristics, sub-characteristics and metrics)

* familiar labels

* concise definitions

* measurement (characteristics are operationally defined)

* evaluation procedures (who should be involved how and when).

Concerning the hierarchical structure of quality characteristics, we will draw on a distinction which, despite the observed overall lack of consensus, several researchers in the field adhere to (albeit not exclusively): syntactic quality, semantic quality and pragmatic quality (Krogstie, 1998; Leung & Bolloju, 2005; Lindland, Sindre, & Solvberg, 1994; Moody, Sindre, Brasethvik, & Solvberg, 2002; Recker, 2006; Teeuw & Berg, 1997).

The framework for the evaluation of the learning path specification we developed is presented in Table 2.

Syntactic quality involves the extent to which the conceptual model adheres to the syntax rules of the language it is modelled in. In the case of the learning path conceptual model evaluating the question would be whether UML has been properly used (i.e., in accordance with UML syntax rules) to express what was meant to be expressed.

Semantic quality refers to the extent to which the model accurately represents the essential features of the phenomenon under study. Some of the differences in defining model quality revolve around the interpretation of what constitutes an accurate representation. Interpretations of accuracy vary, depending on whether or not the phenomenon under study is considered an "objective reality" (ontology), and whether or not it is possible to objectively know this reality (epistemology) (Recker, 2005). Regarding semantic quality several authors mention specific criteria like completeness, validity, clarity, consistency, etc. (Krogstie, 1998; Leung & Bolloju, 2005; Recker, 2005; Teeuw & Berg, 1997). However usage of these criteria is not consistent. Moody et al. (2002) for instance use the term validity to indicate a number of criteria (completeness, parsimony, and independence) which others use to define semantic quality. Interestingly, Krogstie (1998) introduces the notion of feasibility. Whereas completeness means that the model contains all the statements which are correct and relevant to the domain, feasible completeness means that there are no statements in the domain, and not in the model, which would be cost-efficient to include. Besides, this author distinguishes between semantic quality and perceived semantic quality. He argues that the primary goal for semantic quality is for the model to correspond with the domain. However, this correspondence can not be checked directly since:

"To build a model, one has to go through the participant's knowledge regarding the domain, and to check the model, one has to compare with the participant's interpretation of the externalized model. Hence, what we observe at quality control is not the actual semantic quality of the model, but a perceived semantic quality, based on comparisons of the two imperfect interpretations" (Krogstie, 1998, p. 87).

Pragmatic quality finally refers to the question whether/how easily the model is comprehended by the stakeholders in view of its purpose. The purposes of conceptual models can vary widely: enhance communication, document the current state of knowledge, guide system development, exploration, prediction, decision support (Beck, 2002; Moody, 2005). Pragmatic quality can be further split into technical pragmatic quality and social pragmatic quality (Nelson, Poels, Genero, & Piattini, 2005), indicating whether the model is easily interpreted by tools and human users respectively.

Research questions

Syntactic quality has been evaluated mainly through peer review and expert consultation. So far the model mainly has been used for communication purposes. Eventually the UML model will be transformed to an XML schema which requires greater refinement and detail, and further evaluation of syntactic quality. This evaluation will be reported about in a separate publication.

Semantic quality has been evaluated through collaboration with software developers and processes of peer review. However the elements and characteristics identified by the model have been derived from a review of literature and current practice, but are these really the elements and characteristics lifelong learners want to be informed about? Are these the elements and characteristics they take into account when considering different options?

Evaluation of pragmatic quality will focus on software developers and tools. However in our view it makes sense only to evaluate pragmatic quality after semantic quality has been sufficiently tested, because poor semantic quality will inevitably result in poor pragmatic quality. Still some aspects of pragmatic quality will be included in the present study as well, involving the question whether the learning path characteristics included in the specification are clear and easy to understand.

More particularly, the focus of the present study is on the following quality aspects relating to the purpose of enabling comparison and selection of learning paths:

1. Is the information provided by the model clear? (pragmatic quality)

2. Is the specification complete: does the model contain all essential information lifelong learners desire/need to select suitable learning paths? (semantic quality)

3. Is the specification minimal: does the model contain information which is not considered relevant by lifelong learners? (semantic quality)


Above research questions were addressed through a case study examining lifelong learners' decision making processes (Flyvbjerg, 2006; Yin, 2003). Data on decision making processes were gathered through semi-structured interviews with learners (n=15) who recently chose a learning path, having considered at least two different options. Participants for the study were recruited asking colleagues and acquaintances to propose candidates from their network of family and friends.

Typically sampling for multiple-case studies is guided by the research questions and conceptual framework. Our main sampling strategy was maximum variation of cases (Flyvbjerg, 2006; Miles & Huberman, 1994), meaning that we sought to include a broad variety of learning paths regarding domains of personal/professional development, and level of formality. Besides we aimed to have a broad variety of respondents regarding age, gender, employment status, and prior education. The number of cases to include was not pre-determined, but including over about 15 cases is acknowledged to make it harder to keep an overview without losing sight of necessary details (Miles & Huberman, 1994). Though essentially each case has unique properties and is therefore interesting in its own, in hindsight it appears that the last four interviews did not provide any new information regarding the characteristics taken into account in the decision making so that in this respect a point of saturation (Miles & Huberman, 1994) seems to have been reached. The risk of retrospective distortion due to inaccurate recall was reduced by requiring that the decision making process had come to a conclusion no longer than three months ago, and by using a technique of aided recall during the interviews (Coughlin, 1990).

Interview protocol

The interview protocol included four steps. First participants were asked to tell a bit more about their motives to learn. The second step focused on spontaneous recall: participants were asked to describe their search for ways to achieve these learning goals and how they "weighed" these different options, i.e., on which characteristics they compared them to arrive at a final choice. Any characteristics mentioned during the interview which were not part of the learning path specification were noted down by the interviewer. The third step involved aided or prompted recall: participants were invited to go through a set of cards, each card containing a label and description of a characteristic included in the specification as shown in Table 1, complemented with two additional cards for learning outcomes (knowledge and skills to be developed) and learning actions (things you have to do: study, investigate, write, present, etc.).

For each of the cards participants were asked to indicate whether the described characteristic was clear to them and whether it had played a role in the recent choice of a learning path. The fourth step required of participants that they shift from the most recent decision making process to deciding on a learning path more generally, and to consider whether in general they would want to take this information into account.


Figure 1 presents the learning paths included in the study classifying them along two dimensions: relation to career and "urgency", i.e., the question whether the learning path is considered a "must have". This second dimension emerged as a relevant distinction during the interviews: whether or not the learning path is conditional, i.e. whether it enables the learner to do things which will otherwise remain beyond her reach (e.g apply for another job). Though at face value one might expect conditional learning paths to exist mainly in the realm of professional development, there are several counter examples, such as learning to swim or drive a car. In the case of the career related learning paths the conditional learning paths were "must haves" either with respect to adequate job performance, or to a job or career switch. Other career related learning paths were merely meant to "look good on the CV", without an immediate urge to find another job.



The number of learning paths compared in depth in the decision making processes varied between 2 and 8, with an average of 4. In twelve cases Internet was used to search for suitable learning paths. Two cases involved a restricted choice between two options offered by the employer or educational institution. In a number of cases the process of screening had started about a year before. The distinction between screening and choice is not as clear-cut in practice as in theory: rather there exists a grey area of learning paths which are considered more closely but still get dropped long before the final choice is made. A clear distinction between screening and choice can be made only in those cases where one or two criteria stand out as initial selection criteria as was, for example, the case with the choice of a driving school, where a first selection (screening) took place on the base of reputation (pass/fail rates) and location.

An interesting general observation regarding the in-depth comparison leading up to the final choice is that in the case of the informal learning paths the choice process entailed some probing of different options. Of course this was possible because these options were freely available and did not require any formal subscription or enrolment. However they were nevertheless considered as clearly distinctive options: though there was a period of "trial" eventually a choice for a particular option was made, rather than for a mix.

Spontaneous recall

Figure 2 shows--in descending order--to what extent learning path characteristics played a role in the decision making process according to the spontaneous recall of participants. The characteristics "title" and "description" have been left out, as they are obvious. Characteristics which were mentioned during the interview and which were not included in the learning path specification are marked by (+).

Some caution is required regarding the interpretation of these results. All participants were more or less aware of the learning outcomes of the learning paths under consideration but they did not always play a role in the comparison, simply because the learning paths were more or less identical in this respect, or because the learning outcomes were less important than acquiring the associated diploma or certificate. Similarly, language was not mentioned as a criterion in the decision making process simply because all learning paths considered were in Dutch. In these cases the characteristic has played an (implicit) role in the process of screening.


Contact time, experience/advice, quality, and teacher were mentioned in addition to the characteristics included in the specification and merit closer inspection.

Experience/advice: six participants remarked they had been keen to acquire information on other peoples' experiences concerning the options they were considering. Preferably people they were acquainted with so that their judgement could be appraised, but otherwise in the form of Internet forums.

Teacher: three respondents compared information on the teacher involved, placing different accents: two were merely interested in teaching experience (number of years) and the third considered it very important that the teacher had practical work experience in the subject area (Law).

Contact time: contact time involves the question at what time of the week face-to-face meetings take place. Scheduling information is multi-faceted as is already expressed by a number of characteristics included in the specification: start/end date, delivery mode (contact: yes/no), and contact hours (amount of contact). Now additional information is called for regarding the time of the week contact takes place. The indication "part-time/full-time" which is sometimes used was not included in the specification because it is too general to be informative. This is confirmed by the specifications from participants in this study: "not on Wednesdays", "only evenings or weekends, depending on how far I have to travel", etc. What is required is a categorisation that is specific enough to be informative, yet general enough to be practical.

Quality: five respondents said they had taken into account the quality of learning paths. When asked how they had established quality, a variety of aspects was mentioned: pass/fail rates, "does the website look professional", and quality of learning materials (e.g., up-to-date content).

Aided recall

Despite the brief explanation offered on the cards the characteristics were not always clear and unambiguous. However, this seemed somewhat intrinsic to the domain as several characteristics included in the specification are closely related, nuances tended to get lost, for instance, regarding the concepts "assessment", "completion", and "recognition". Assessment describes the types of assessment(s) included in the learning path, and completion indicates whether there is a formal end to the learning path (set by an assessment or time limit for instance) or whether it is up to the learner to decide whether the learning goals have been reached. Though both concepts are clearly related to recognition, they are not identical: recognition is independent of types of assessment and does not necessarily mean deadlines.

Also, though characteristics themselves may be clear and unambiguous, the role of the characteristic in comparing and selecting learning paths may not be unambiguous. Indeed plain and simple characteristics like costs and time investment could lead respondents to ponder: of course, generally speaking, you would want to reduce costs as much as possible, but then again "quality comes with a price".

Figure 3 compares the results for spontaneous recall(s) with the results based on aided recall (a).


Clearly none of the characteristics included in the learning path specification can be considered superfluous. Apparently quite a number of characteristics are prone to be overlooked in spontaneous recall. In fact, only the results for outcomes and location appear remarkably stable. Figure 3 serves to illustrate how certain characteristics are more often taken into account in the process of selecting a learning path than reports based on spontaneous recall would suggest. Some of these characteristics were taken into account implicitly, without the learner being consciously aware of it (e.g., delivery mode). In other cases the characteristics had been consciously considered, and subsequently forgotten as they had not constituted an issue: "Yes, I do recall looking at guidance information, but it was ok...".

Several respondents commented that they had not seen any information regarding certain characteristics (e.g. assessment, actions, prior knowledge, and guidance). Thus results may to some extent reflect the availability of information. Figure 4 confirms that in general a majority of the learners want to be informed on each characteristic when deciding upon a learning path.



This section discusses the question whether the characteristics mentioned by respondents in addition to those included in the specification, should be added. Several participants started out selecting learning paths based on provider names with solid reputations. However, even in these cases additional information was sought on learner experiences with these learning paths. However, information on learner experiences can not be included in the specification, because the description of learning paths is made by the provider and the information on experiences is only of value when it is completely independent. Alternative solutions might be found in adding annotations or ratings provided by users, or in providing recommendations through collaborative filtering, e.g., "Your profile most closely matches the profile of learners choosing learning path X" (Drachsler, Hummel, & Koper, 2008). However, participants expressed a preference to hear about experiences from people they know so as to be able to appraise their judgement. Further research is needed to establish whether the proposed solutions are viable alternatives.

In three cases information was sought on the teacher (number of years in teaching or practical professional experience in the subject area). The question is whether this information should be provided through one or even two separate characteristics in the specification, or whether this is typical information a learner should be able to find through the link provided via "further information". Though teacher information can be decisive, it will hardly play a role at the stage of screening but rather towards the end of the process in the comparison of a limited set of options. This is not the case for the information regarding contact time, i.e., the scheduling of meetings associated with a learning path: this information will help to distinguish suitable learning paths at the very start of the decision making process. Including this element in the specification is therefore likely to contribute considerably to efficiency. So bearing in mind the notion of feasible completeness the element "contact time" will be added to the specification. Seeking a balance between the level of detail some participants described and considerations of what is practical, two dimensions will be distinguished: weekdays/weekend and daytime/evenings.

Finally, the aspect of "quality" was mentioned, referring to a variety of indicators: pass/fail rates, a probe of learning materials (up-to-date), or impressions of professionalism. This type of information can not be grasped simply by adding another learning path characteristic, but has to be sought in addition, through independent sources.


We investigated 15 choice processes involving a broad variety of learning paths, with the aim to evaluate semantic and pragmatic quality of the learning path specification: are characteristics included in the specification to support comparison and selection of a learning path clear, sufficient, and without redundancies?

Regarding clarity our study showed that related characteristics (e.g., delivery mode and contact hours) sometimes got mixed up. However, this can be solved by presenting them in combination and with possible values.

None of the characteristics included in the specification appeared redundant. Instead, several characteristics were mentioned in addition. However, upon closer inspection, only contact time appears an adequate improvement of the specification.

Following this investigation and adaptations made on the base of these results, a tool is being developed to describe learning paths in line with the specification. Subsequent tools can then be developed which use these descriptions to facilitate selection of suitable learning paths.

Though several participants hinted at information overload regarding the number of learning paths, one respondent specifically hinted at the risk of overload due to the number of criteria taken into account. She said her choice process had taken the shape of a funnel regarding the number of learning paths to compare, though not, unfortunately, regarding the number of criteria taken into account. Further quantitative research is required to investigate solutions aimed at reducing the risk of information overload by distinguishing between more and less important characteristics.


The work described here was sponsored by the TENCompetence Integrated Project, which has been funded by the European Commission's 6th Framework Programme, priority IST/Technology Enhanced Learning. Contract 027087 (http://www.tencompetence.org).


Beach, L. R. (1997). The Psychology of Decision Making: People in Organizations. Newbury Park, CA: Sage.

Beck, B. (2002). Model evaluation and performance. In A. H. El-Shaarawi & W. W. Piegorsch (Eds.), Encyclopedia of Environmetrics (Vol. 3, pp. 1275-1279). Chichester: John Wiley & Sons.

CEC. (2000). A Memorandum on Lifelong Learning. Brussels: Commission of the European Communities.

Colardyn, D., & Bjornavold, J. (2004). Validation of Formal, Non-formal and Informal Learning: policy and practices in EU Member States. European Journal of Education, 39(1), 69-89.

Colley, H., Hodkinson, P., & Malcolm, J. (2003). Informality and formality in learning: a report for the Learning and Skills Research Centre. London: Learning and Skills Research Centre.

Coughlin, S. S. (1990). Recall Bias in Epidemiologic Studies. Journal of Clinical Epidemiology, 43(1), 87-91.

Drachsler, H., Hummel, H., & Koper, R. (2008). Personal recommender systems for learners in lifelong learning: requirements, techniques and model. International Journal of Learning Technology, 3(4), 404-423.

Fasolo, B., McClelland, G. H., & Todd, P. M. (2007). Escaping the tyranny of choice: when fewer attributes make choice easier. Marketing Theory, 7(1), 13-26.

Flyvbjerg, B. (2006). Five Misunderstandings About Case-Study Research. Qualitative Inquiry, 12(2), 219-245.

Hager, P. (1998). Recognition of Informal Learning: challenges and issues. Journal of Vocational Education and Training, 50(4), 521-534.

Janssen, J., Hermans, H., Berlanga, A.J., & Koper, R. (2008). Learning Path Information Model. Retrieved November 9, 2008, from http://hdl.handle.net/1820/1620.

Janssen, J., Berlanga, A. J., Vogten, H., & Koper, R. (2008). Towards a learning path specification. International Journal of Continuing Engineering Education and Lifelong Learning, 18(1), 77-97.

Kickmeier-Rust, M. D., Albert, D., & Steiner, C. (2006). Lifelong Competence Development: On the Advantages of Formal Competence-Performance Modeling. Proceedings of the International Workshop in Learning Networks for Lifelong Competence Development, March 2007, Sofia, Bulgaria.

Krogstie, J. (1998). Integrating the Understanding of Quality in Requirements Specification and Conceptual Modeling. ACM SIGSOFT Software Engineering Notes, 23(1), 86-91.

Leung, F., & Bolloju, N. (2005). Analyzing the Quality of Domain Models Developed by Novice Systems Analysts. Paper presented at the 38th Annual Hawaii International Conference on System Sciences, Hawaii.

Lidwell, W., Holden, K., & Butler, J. (2003). Universal Principles of Design. Gloucester, Massachusetts: Rockport.

Lindland, O. I., Sindre, G., & Solvberg, A. (1994). Understanding quality in conceptual modeling. IEEE Software, 11, 42-49.

Livingstone, D. W. (1999). Exploring the Icebergs of Adult Learning: Findings of the First Canadian Survey of Informal Learning Practices. NALL Working Paper No. 10.

Malhotra, N. K. (1982). Information Load and Consumer Decision Making. The Journal of Consumer Research, 8, 419-430. Miles, M. B., & Huberman, A. M. (1994). Qualitative Data Analysis (2nd ed.). Thousand Oaks: Sage.

Moody, D. L. (2005). Theoretical and practical issues in evaluating the quality of conceptual models: current state and future directions. Data & Knowledge Engineering, 55, 243-276.

Moody, D. L., Sindre, G., Brasethvik, T., & Solvberg, A. (2002). Evaluating the Quality of Process Models: Empirical Analysis of a Quality Framework. Paper presented at the 21st International Conference on Conceptual Modeling, Tampere, Finland.

Nelson, H. J., Poels, G., Genero, M., & Piattini, M. (2005). Quality in conceptual modeling: five examples of the state of the art. Data & Knowledge Engineering, 55, 237-242.

Recker, J. (2005). Conceptual Model Evaluation. Towards more Paradigmatic Rigor. Paper presented at the EMMSAD 2005 Exploring Modeling Methods for Systems Analysis and Design.

Recker, J. (2006). Towards an Understanding of Process Model Quality. Methodological Considerations. Paper presented at the 14th European Conference on Information Systems, Goeteborg, Sweden.

Rundle-Thiele, S., Shao, W., & Lye, A. (2005). Computer Process Tracing Method: Revealing Insights Into Consumer Decision-Making. Paper presented at the Australian and New Zealand Marketing Academy Conference (ANZMAC).

Schugurensky, D. (2000). The forms of informal learning: towards a conceptualization of the field. NALL Working Paper No. 19.

Skule, S. (2004). Learning conditions at work: a framework to understand and assess informal learning in the workplace. International Journal of Training and Development, 8(1), 8-20.

Teeuw, W. B., & Berg, H. v. d. (1997). On the Quality of Conceptual Models. Paper presented at the ER'97 Workshop on Behavioral Models and Design Transformations: Issues and Opportunities in Conceptual Modeling, Los Angeles.

Turbek, S. (2008). Advancing Advanced Search. Boxes And Arrows: The Design Behind the Design. Retrieved February 7, 2008, from http://www.boxesandarrows.com/view/advancing-advanced.

Van Lamsweerde, A. (2000). Formal Specification: a Roadmap. In A. Finkelstein (Ed.), The Future of Software Engineering: 22nd international conference on software engineering: ACM Press.

Yin, R. K. (2003). Case Study Research. Design and Methods. Thousand Oaks, California: Sage.

Jose Janssen, Adriana J. Berlanga and Rob Koper

CELSTEC Open University of the Netherlands, Heerlen, The Netherlands // jja@ou.nl // abl@ou.nl // rkp@ou.nl
Table 1. Learning Path metadata

Characteristic         Explanation

Title                  Name of the program, course, workshop etc.

Description            Brief description of the program, course,
                       workshop, etc.

Prior knowledge        Competences which are expected to have been
                       acquired beforehand

Start conditions       Other conditions that must be met in order to
                       start: e.g., a minimum number of participants,
                       a special diploma, access to a computer,
                       microscope ...

Language               Languages used in the learning path

Diploma/certificate    Indicates whether completion of the learning
                       path results in an officially recognized
                       diploma or certificate

Time investment        Total number of hours it takes to complete

Delivery mode          Indicates whether the learning path involves
                       self-study, face-to-face meetings or a mixture
                       of these.

Guidance               Description of available guidance

Assessment             Description of assessments associated with the
                       learning path

Start date/end date    Start/end date

Costs                  Total costs for enrolment, materials, etc.

Number of contact      Indicates number of hours of required (virtual)
hours                  presence.

Location               Indicates where meetings take place

Completion             Indicates how and by whom it is decided whether
                       the learning path goals have been achieved.

Provider               Provider of the learning path

Further information    Link to a website for further details.

Table 2. Evaluation framework

Quality        Description    Sub-characteristics  Description

Syntactic      Does the       Proper notation
quality        model          of association,
               correctly      aggregation,
               express        generalization,
               what is        multiplicity etc.
               meant to be
               expressed in
               with UML

Semantic       Does the       adequate [1]         The model
quality        model          orthogonal/          adequately
               represent      independent          reflects the
               essential      [3, 5]               domain, i.e.,
               features of    valid [2, 4]         independent
               the                                 aspects are
               phenomenon                          captured by
               under study?                        different concepts
                                                   and relations are

                              complete [1, 3, 4,   The model
                              5] nothing missing   describes all
                              what is expected     essential
                              [2]                  features.

                              minimal [1]          The model does not
                              parsimonious [3,     contain irrelevant
                              5]] nothing          aspects and
                              unexpected           relations
                              presented [2]

Pragmatic      Is the model   unambiguous          Concepts and
quality        easy to        [1, 3]               relations have a
               understand?                         clear single
Social &                                           meaning

                              consistent [1,3]

                              general              Concepts should be
                                                   as independent as
                                                   possible from any

Quality        Evaluation           Metrics
dimensions     methods

Syntactic      --submit model to    --number and
quality        peer/expert review   type
                                    of errors,
               --validity checks    ambiguities,
               through software     etc.

Semantic       --explain the        --number and
quality        model to lifelong    type
               learners and         of issues open
               learning path        to
               providers to see     debate
               whether they
               find it adequate     --number of
               on points relevant   changes made
               to them              to
                                    the model
               --analyse lifelong
               learners' learning   --number and
               path choice          type
               processes to         of frictions in
               establish learning   mapping
               path                 learning
               characteristics      paths
               essential in this

               --map existing
               learning paths on

Pragmatic      --establish          --perceived
quality        whether the          ease of
               specification is     use
Social &       adequate to
Technic        develop tools.       --perceived
al                                  usefulness
               whether tools        --intention to
               developed are        use
               considered useful

               --map informal and
               paths from
               different domains

[1] van Lamsweerde (2000)

[2] Leung & Bolloju (2005)

[3] Teeuw & van den Berg (1997)

[4] Krogstie (1998)

[5] Moody et al. (2002)
Gale Copyright:
Copyright 2011 Gale, Cengage Learning. All rights reserved.