Learning paths can be roughly defined as sets of one or more
learning activities leading to a particular learning goal. Learning
paths can vary from a relatively small activity like reading a book or
taking a course to following an entire programme or curriculum. Learning
paths may vary also regarding the level of formality. In line with the
Commission of the European Communities we distinguish formal, non-formal
and informal learning (CEC, 2000). Whereas formal learning occurs in
education and training institutions and leads to recognised diplomas and
qualifications, informal learning is described as "a natural
accompaniment to everyday life" which is not necessarily
intentional learning (CEC, 2000, p. 8). Non-formal learning, finally, is
learning that takes place alongside the mainstream systems of education
and training, for instance at the workplace or in arts or sports, which
does not necessarily lead to formalised certificates.
Lifelong learners' learning paths consist of a mixture of
formal, informal and non-formal learning (Colardyn & Bjornavold,
2004; Colley, Hodkinson, & Malcolm, 2003; Livingstone, 1999). In
order to support lifelong learners in comparing and selecting suitable
learning paths, a uniform way to describe learning activities and
learning paths has been developed, which covers these different ways in
which people learn (Janssen, Berlanga, Vogten, & Koper, 2008).
The specification is envisaged to support several processes.
Firstly, it is meant to be used by educational providers to describe
formal and non-formal educational courses and programmes in order to
make them available through specific search engines, thus enabling
comparison across providers. We assume that educational providers will
want to describe learning paths in a uniform, formalised way, because
the benefits of transparency and opportunities for automated learner
support outweigh the costs. Costs can be relatively low since
educational providers already have to describe their offerings; it will
merely be a matter of organising this information in a way that enables
storage and update in one place and subsequent use in different
contexts: printed catalogues, websites, and search engines.
A second process the learning path specification is meant to
support was initially defined as follows: lifelong learners use the
specification to describe their informal learning paths to make them
available as an example to other learners with similar learning goals.
However, a pilot-study revealed that it requires considerable efforts
and skills on the part of the learner to identify activities that did or
did not after all contribute to achieving those outcomes. To distil a
learning path from one's own informal learning experiences and
describe it in a way that is useful for others, is not an easy task (cf.
Skule, 2004). Though we still maintain that the specification can be
used to describe all kinds of learning (a point we later further
elaborate), we believe that in the case of informal learning it is not
likely going to happen on a large scale by lifelong learners themselves,
because it requires learning design skills. It is not unreasonable
though to expect employers and employment agencies to be willing to
invest in these descriptions of informal learning paths as they can
offer tried and tested alternatives to more costly formal and non-formal
learning paths. Research indicates that people spend an average of 6
hours a week on employment related informal learning (Livingstone, 1999)
and description of these informal learning paths is likely to enhance
efficiency when they can offer guidance to learners rather than have
them find things out through trial and error. In any case, the second
process the learning path specification is meant to support eventually
is defined as: description of informal learning paths in order to make
these learning paths available for other learners with similar learning
Finally, a third process the learning path specification is
envisaged to support is selection of suitable learning paths. To this
end the specification identifies main characteristics to be used in
comparing and selecting a learning path (e.g. learning objectives,
prerequisites, study load, costs, etcetera). Lifelong learners must be
offered means to efficiently choose the learning path that best fits
their needs. Taking a decision support perspective, we distinguish two
stages in this process: screening and choice (Beach, 1997;
Rundle-Thiele, Shao, & Lye, 2005). Screening involves selecting a
number of options one wants to take into consideration, i.e. narrowing
down the number of choice options to a number that can be
"managed". Research shows that choice overload may occur due
to the number of available options, as well as to the number of
attributes related to these options (Fasolo, McClelland, & Todd,
2007; Malhotra, 1982). In other words: having to choose from a large
number of learning paths is one thing, having to compare even a limited
number of learning paths might lead to choice overload when a large
number of attributes are related to these options. But even apart from
these considerations regarding choice overload, lifelong learners will
rather invest the scarce resources of time and attention in developing
competences than in comparing all kinds of ways to do so. What is needed
then is some tool for the learner to select a limited set of learning
paths to take into account in the choice process.
There are quite a number of criteria that could be relevant to
finding the most suitable learning path but not all criteria might be
equally relevant to all learning paths or to all learners for that
matter. The study of Fasolo, McClelland and Todd (2007, p. 23) shows
that "it is possible for consumers to make good choices based on
one or two attributes, when attributes are positively related or
consumers care unequally about attributes and choose on the basis of the
most important ones". To the extent that learners do not equally
care about the learning path attributes included in the learning path
specification progressive disclosure of functionality could contribute
to help the learner focus on those criteria that are most relevant for
her (Turbek, 2008). Progressive disclosure is a strategy for managing
information complexity in which only necessary or requested information
is displayed at any given time (Lidwell, Holden, & Butler, 2003).
Requirements for the specification have been derived from a review
of literature on curriculum design and lifelong learning as well as
observations of current practices to support learner choice (Janssen et
al., 2008). This paper describes a study directed towards evaluation of
the conceptual model of the learning path specification. It provides an
outline of the specification and explains how the specification supports
description and selection of learning paths. Subsequently a framework
for the evaluation of model quality is presented, guiding the specific
research questions. Finally the paper describes method and results of
Learning path conceptual model and specification
According to Moody (2005) conceptual modelling is a process of
formally documenting a domain (a system or a problem) in order to
enhance communication and understanding. He further points out that
conceptual modelling may be used to describe requirements at different
levels: functional and non-functional requirements at the level of an
application, and information requirements at the level of an
organisation or even an industry.
A formal specification can be considered a conceptual model as is
illustrated by the following definition: "a formal specification is
the expression, in some formal language and at some level of
abstraction, of a collection of properties some system should
satisfy" (Van Lamsweerde, 2000).
The learning path specification identifies information requirements
for lifelong learners: generic elements of a learning path which are
essential to selecting, planning and executing a learning path, such as
learning goals, learning actions, delivery mode, etc. It describes fixed
as well as optional elements; both contents and structure.
Like any other path a learning path has a finish and a start (i.e.
learning goals and prerequisites). In order to get to the finish one or
more learning actions have to be completed. Learning goals and
prerequisites can be specified both at the level of the learning path
and its constituent actions. They are preferably defined in terms of
standardized competences so as to facilitate automated identification of
parts of a learning path a learner may skip when these competences have
already been attained through prior learning (Kickmeier-Rust, Albert,
& Steiner, 2006). Learning actions can be grouped into clusters, for
instance when they compose a set a learner can choose from (selection),
or because they have to be studied in a particular order (sequence).
Table 1 describes the information about the learning path and its
constituent actions considered relevant in identifying and selecting
suitable learning paths, and therefore included in the specification.
(For a more detailed account of the attributes and metadata associated
with each of the elements see Janssen, Hermans, Berlanga, & Koper
Formal, non-formal and informal learning paths
The distinction between formal, non-formal and informal learning is
not as clear-cut as the definitions provided in the introduction
suggest. Schugurensky (2000) stresses the fact that informal learning
can also take place inside formal and non-formal educational
institutions: within these institutions some learning occurs
independently of the intended goals of the curriculum. Using two
categories (intentionality and consciousness) he goes on to identify
three forms of informal learning: self-directed learning (intentional +
conscious), incidental learning (unintentional + conscious) and
socialization (unintentional + unconscious). The learning path
specification is merely meant to enable description of informal learning
with the aim to suggest informal ways to develop competences, drawing
from other learners' personal informal learning experiences. This
means the learning path specification is only meant to cover conscious
informal learning. As to the intentionality of learning it is often
stated that workplace learning and other informal learning have no
formal curriculum or prescribed outcomes (Hager, 1998). Regarding
unintentional conscious learning we maintain that this type of learning
can be described in hindsight as a learning path, describing the
previously unintentional learning outcomes as learning objectives, to
present as an option to other learners interested in achieving these
Concerning the distinction between informal and non-formal
learning, a major review of literature suggests there is no clear
agreement: the terms are used interchangeably (Colley et al., 2003). Nor
does it appear possible to distinguish formal learning from other
learning in ways that have broad applicability or agreement. The authors
conclude it is more sensible to consider attributes of informality and
formality present in all learning situations. These attributes concern
four aspects of learning:
* Process: informality and formality attributes relating to the
learning process relate to questions like who's in control of the
process (teacher controlled versus student led), whether and what kind
of assessment is involved (formative or summative).
* Location/setting: where does the learning take place (e.g. in an
educational institution, at the workplace, etc.) and does it involve
* Purposes: is learning intended or does it happen unintentionally;
are learning outcomes determined by the learner or designed to meet
needs which are externally determined?
* Content: does the learning focus on acquisition of established
knowledge or development of knowledge from experience?
Attributes relating to the process aspect of learning included in
the specification are the metadata elements "guidance" and
"assessment". The location/setting aspect is covered by the
metadata "recognition", "delivery mode", and
"location". Regarding the purpose aspect we conclude that the
learning path specification only covers intentional learning: a learning
path is directed towards learning goals. This does not mean that the
learning path specification cannot be used to describe unintentional
learning as well, but this would always be in hindsight: learning which
has occurred unintentionally can be retrospectively described to serve
as an example to other learners who can then embark on the same path
purposefully. Attributes of formality and informality relating to the
content aspect of learning can be described through the metadata element
"description" of the learning path as well as of its
constituent actions. Whether the learning goals of a learning path are
achieved through "formal knowledge acquisition" or through
"learning by doing" will be of interest to the learner, but
whether it requires a separate metadata element remains to be seen.
Model evaluation: a framework
Seeking alignment with the ISO 9000 definition of quality Moody
(2005) defines conceptual model quality as "The totality of
features and characteristics of a conceptual model that bear on its
ability to satisfy stated or implied needs" (p. 252). Based on a
review of research in the field of conceptual model quality Moody
concludes that there are no generally accepted guidelines for evaluating
the quality of conceptual models. Nor do experts agree as to what makes
a conceptual model a "good" model. One of the explanations
given for this lack of consensus is that a conceptual model exists as a
construction of the mind, and therefore quality of a conceptual model
cannot be as easily assessed as the quality of a concrete product:
"While the finished product (the software system) can be evaluated
against the specification, a conceptual model can only be evaluated
against people's (tacit) needs, desires and expectations. Thus the
evaluation of conceptual models is by nature a social rather than a
technical process, which is inherently subjective and difficult to
formalise"(Moody, 2005, p. 245).
The learning path specification is a case in point: rather than a
"finished product" it is a model to describe learning paths
which can be used to develop tools to support lifelong learners in
finding and navigating suitable learning paths. This implies a number of
* lifelong learners
* learning path designers
* software developers.
Someone interested in finding suitable learning paths is likely to
focus on different aspects of the learning path specification than
someone interested in designing learning paths or in developing tools to
support these processes. Consequently, evaluation of the specification
requires input from these different perspectives.
Addressing the lack of consensus in the field Moody (2005) proposes
the ISO/IEC9126 software quality model as a template to structure
conceptual model quality frameworks. This template identifies the
following important features:
* hierarchical structure of quality characteristics
(characteristics, sub-characteristics and metrics)
* familiar labels
* concise definitions
* measurement (characteristics are operationally defined)
* evaluation procedures (who should be involved how and when).
Concerning the hierarchical structure of quality characteristics,
we will draw on a distinction which, despite the observed overall lack
of consensus, several researchers in the field adhere to (albeit not
exclusively): syntactic quality, semantic quality and pragmatic quality
(Krogstie, 1998; Leung & Bolloju, 2005; Lindland, Sindre, &
Solvberg, 1994; Moody, Sindre, Brasethvik, & Solvberg, 2002; Recker,
2006; Teeuw & Berg, 1997).
The framework for the evaluation of the learning path specification
we developed is presented in Table 2.
Syntactic quality involves the extent to which the conceptual model
adheres to the syntax rules of the language it is modelled in. In the
case of the learning path conceptual model evaluating the question would
be whether UML has been properly used (i.e., in accordance with UML
syntax rules) to express what was meant to be expressed.
Semantic quality refers to the extent to which the model accurately
represents the essential features of the phenomenon under study. Some of
the differences in defining model quality revolve around the
interpretation of what constitutes an accurate representation.
Interpretations of accuracy vary, depending on whether or not the
phenomenon under study is considered an "objective reality"
(ontology), and whether or not it is possible to objectively know this
reality (epistemology) (Recker, 2005). Regarding semantic quality
several authors mention specific criteria like completeness, validity,
clarity, consistency, etc. (Krogstie, 1998; Leung & Bolloju, 2005;
Recker, 2005; Teeuw & Berg, 1997). However usage of these criteria
is not consistent. Moody et al. (2002) for instance use the term
validity to indicate a number of criteria (completeness, parsimony, and
independence) which others use to define semantic quality.
Interestingly, Krogstie (1998) introduces the notion of feasibility.
Whereas completeness means that the model contains all the statements
which are correct and relevant to the domain, feasible completeness
means that there are no statements in the domain, and not in the model,
which would be cost-efficient to include. Besides, this author
distinguishes between semantic quality and perceived semantic quality.
He argues that the primary goal for semantic quality is for the model to
correspond with the domain. However, this correspondence can not be
checked directly since:
"To build a model, one has to go through the
participant's knowledge regarding the domain, and to check the
model, one has to compare with the participant's interpretation of
the externalized model. Hence, what we observe at quality control is not
the actual semantic quality of the model, but a perceived semantic
quality, based on comparisons of the two imperfect interpretations"
(Krogstie, 1998, p. 87).
Pragmatic quality finally refers to the question whether/how easily
the model is comprehended by the stakeholders in view of its purpose.
The purposes of conceptual models can vary widely: enhance
communication, document the current state of knowledge, guide system
development, exploration, prediction, decision support (Beck, 2002;
Moody, 2005). Pragmatic quality can be further split into technical
pragmatic quality and social pragmatic quality (Nelson, Poels, Genero,
& Piattini, 2005), indicating whether the model is easily
interpreted by tools and human users respectively.
Syntactic quality has been evaluated mainly through peer review and
expert consultation. So far the model mainly has been used for
communication purposes. Eventually the UML model will be transformed to
an XML schema which requires greater refinement and detail, and further
evaluation of syntactic quality. This evaluation will be reported about
in a separate publication.
Semantic quality has been evaluated through collaboration with
software developers and processes of peer review. However the elements
and characteristics identified by the model have been derived from a
review of literature and current practice, but are these really the
elements and characteristics lifelong learners want to be informed
about? Are these the elements and characteristics they take into account
when considering different options?
Evaluation of pragmatic quality will focus on software developers
and tools. However in our view it makes sense only to evaluate pragmatic
quality after semantic quality has been sufficiently tested, because
poor semantic quality will inevitably result in poor pragmatic quality.
Still some aspects of pragmatic quality will be included in the present
study as well, involving the question whether the learning path
characteristics included in the specification are clear and easy to
More particularly, the focus of the present study is on the
following quality aspects relating to the purpose of enabling comparison
and selection of learning paths:
1. Is the information provided by the model clear? (pragmatic
2. Is the specification complete: does the model contain all
essential information lifelong learners desire/need to select suitable
learning paths? (semantic quality)
3. Is the specification minimal: does the model contain information
which is not considered relevant by lifelong learners? (semantic
Above research questions were addressed through a case study
examining lifelong learners' decision making processes (Flyvbjerg,
2006; Yin, 2003). Data on decision making processes were gathered
through semi-structured interviews with learners (n=15) who recently
chose a learning path, having considered at least two different options.
Participants for the study were recruited asking colleagues and
acquaintances to propose candidates from their network of family and
Typically sampling for multiple-case studies is guided by the
research questions and conceptual framework. Our main sampling strategy
was maximum variation of cases (Flyvbjerg, 2006; Miles & Huberman,
1994), meaning that we sought to include a broad variety of learning
paths regarding domains of personal/professional development, and level
of formality. Besides we aimed to have a broad variety of respondents
regarding age, gender, employment status, and prior education. The
number of cases to include was not pre-determined, but including over
about 15 cases is acknowledged to make it harder to keep an overview
without losing sight of necessary details (Miles & Huberman, 1994).
Though essentially each case has unique properties and is therefore
interesting in its own, in hindsight it appears that the last four
interviews did not provide any new information regarding the
characteristics taken into account in the decision making so that in
this respect a point of saturation (Miles & Huberman, 1994) seems to
have been reached. The risk of retrospective distortion due to
inaccurate recall was reduced by requiring that the decision making
process had come to a conclusion no longer than three months ago, and by
using a technique of aided recall during the interviews (Coughlin,
The interview protocol included four steps. First participants were
asked to tell a bit more about their motives to learn. The second step
focused on spontaneous recall: participants were asked to describe their
search for ways to achieve these learning goals and how they
"weighed" these different options, i.e., on which
characteristics they compared them to arrive at a final choice. Any
characteristics mentioned during the interview which were not part of
the learning path specification were noted down by the interviewer. The
third step involved aided or prompted recall: participants were invited
to go through a set of cards, each card containing a label and
description of a characteristic included in the specification as shown
in Table 1, complemented with two additional cards for learning outcomes
(knowledge and skills to be developed) and learning actions (things you
have to do: study, investigate, write, present, etc.).
For each of the cards participants were asked to indicate whether
the described characteristic was clear to them and whether it had played
a role in the recent choice of a learning path. The fourth step required
of participants that they shift from the most recent decision making
process to deciding on a learning path more generally, and to consider
whether in general they would want to take this information into
Figure 1 presents the learning paths included in the study
classifying them along two dimensions: relation to career and
"urgency", i.e., the question whether the learning path is
considered a "must have". This second dimension emerged as a
relevant distinction during the interviews: whether or not the learning
path is conditional, i.e. whether it enables the learner to do things
which will otherwise remain beyond her reach (e.g apply for another
job). Though at face value one might expect conditional learning paths
to exist mainly in the realm of professional development, there are
several counter examples, such as learning to swim or drive a car. In
the case of the career related learning paths the conditional learning
paths were "must haves" either with respect to adequate job
performance, or to a job or career switch. Other career related learning
paths were merely meant to "look good on the CV", without an
immediate urge to find another job.
[FIGURE 1 OMITTED]
The number of learning paths compared in depth in the decision
making processes varied between 2 and 8, with an average of 4. In twelve
cases Internet was used to search for suitable learning paths. Two cases
involved a restricted choice between two options offered by the employer
or educational institution. In a number of cases the process of
screening had started about a year before. The distinction between
screening and choice is not as clear-cut in practice as in theory:
rather there exists a grey area of learning paths which are considered
more closely but still get dropped long before the final choice is made.
A clear distinction between screening and choice can be made only in
those cases where one or two criteria stand out as initial selection
criteria as was, for example, the case with the choice of a driving
school, where a first selection (screening) took place on the base of
reputation (pass/fail rates) and location.
An interesting general observation regarding the in-depth
comparison leading up to the final choice is that in the case of the
informal learning paths the choice process entailed some probing of
different options. Of course this was possible because these options
were freely available and did not require any formal subscription or
enrolment. However they were nevertheless considered as clearly
distinctive options: though there was a period of "trial"
eventually a choice for a particular option was made, rather than for a
Figure 2 shows--in descending order--to what extent learning path
characteristics played a role in the decision making process according
to the spontaneous recall of participants. The characteristics
"title" and "description" have been left out, as
they are obvious. Characteristics which were mentioned during the
interview and which were not included in the learning path specification
are marked by (+).
Some caution is required regarding the interpretation of these
results. All participants were more or less aware of the learning
outcomes of the learning paths under consideration but they did not
always play a role in the comparison, simply because the learning paths
were more or less identical in this respect, or because the learning
outcomes were less important than acquiring the associated diploma or
certificate. Similarly, language was not mentioned as a criterion in the
decision making process simply because all learning paths considered
were in Dutch. In these cases the characteristic has played an
(implicit) role in the process of screening.
[FIGURE 2 OMITTED]
Contact time, experience/advice, quality, and teacher were
mentioned in addition to the characteristics included in the
specification and merit closer inspection.
Experience/advice: six participants remarked they had been keen to
acquire information on other peoples' experiences concerning the
options they were considering. Preferably people they were acquainted
with so that their judgement could be appraised, but otherwise in the
form of Internet forums.
Teacher: three respondents compared information on the teacher
involved, placing different accents: two were merely interested in
teaching experience (number of years) and the third considered it very
important that the teacher had practical work experience in the subject
Contact time: contact time involves the question at what time of
the week face-to-face meetings take place. Scheduling information is
multi-faceted as is already expressed by a number of characteristics
included in the specification: start/end date, delivery mode (contact:
yes/no), and contact hours (amount of contact). Now additional
information is called for regarding the time of the week contact takes
place. The indication "part-time/full-time" which is sometimes
used was not included in the specification because it is too general to
be informative. This is confirmed by the specifications from
participants in this study: "not on Wednesdays", "only
evenings or weekends, depending on how far I have to travel", etc.
What is required is a categorisation that is specific enough to be
informative, yet general enough to be practical.
Quality: five respondents said they had taken into account the
quality of learning paths. When asked how they had established quality,
a variety of aspects was mentioned: pass/fail rates, "does the
website look professional", and quality of learning materials
(e.g., up-to-date content).
Despite the brief explanation offered on the cards the
characteristics were not always clear and unambiguous. However, this
seemed somewhat intrinsic to the domain as several characteristics
included in the specification are closely related, nuances tended to get
lost, for instance, regarding the concepts "assessment",
"completion", and "recognition". Assessment
describes the types of assessment(s) included in the learning path, and
completion indicates whether there is a formal end to the learning path
(set by an assessment or time limit for instance) or whether it is up to
the learner to decide whether the learning goals have been reached.
Though both concepts are clearly related to recognition, they are not
identical: recognition is independent of types of assessment and does
not necessarily mean deadlines.
Also, though characteristics themselves may be clear and
unambiguous, the role of the characteristic in comparing and selecting
learning paths may not be unambiguous. Indeed plain and simple
characteristics like costs and time investment could lead respondents to
ponder: of course, generally speaking, you would want to reduce costs as
much as possible, but then again "quality comes with a price".
Figure 3 compares the results for spontaneous recall(s) with the
results based on aided recall (a).
[FIGURE 3 OMITTED]
Clearly none of the characteristics included in the learning path
specification can be considered superfluous. Apparently quite a number
of characteristics are prone to be overlooked in spontaneous recall. In
fact, only the results for outcomes and location appear remarkably
stable. Figure 3 serves to illustrate how certain characteristics are
more often taken into account in the process of selecting a learning
path than reports based on spontaneous recall would suggest. Some of
these characteristics were taken into account implicitly, without the
learner being consciously aware of it (e.g., delivery mode). In other
cases the characteristics had been consciously considered, and
subsequently forgotten as they had not constituted an issue: "Yes,
I do recall looking at guidance information, but it was ok...".
Several respondents commented that they had not seen any
information regarding certain characteristics (e.g. assessment, actions,
prior knowledge, and guidance). Thus results may to some extent reflect
the availability of information. Figure 4 confirms that in general a
majority of the learners want to be informed on each characteristic when
deciding upon a learning path.
[FIGURE 4 OMITTED]
This section discusses the question whether the characteristics
mentioned by respondents in addition to those included in the
specification, should be added. Several participants started out
selecting learning paths based on provider names with solid reputations.
However, even in these cases additional information was sought on
learner experiences with these learning paths. However, information on
learner experiences can not be included in the specification, because
the description of learning paths is made by the provider and the
information on experiences is only of value when it is completely
independent. Alternative solutions might be found in adding annotations
or ratings provided by users, or in providing recommendations through
collaborative filtering, e.g., "Your profile most closely matches
the profile of learners choosing learning path X" (Drachsler,
Hummel, & Koper, 2008). However, participants expressed a preference
to hear about experiences from people they know so as to be able to
appraise their judgement. Further research is needed to establish
whether the proposed solutions are viable alternatives.
In three cases information was sought on the teacher (number of
years in teaching or practical professional experience in the subject
area). The question is whether this information should be provided
through one or even two separate characteristics in the specification,
or whether this is typical information a learner should be able to find
through the link provided via "further information". Though
teacher information can be decisive, it will hardly play a role at the
stage of screening but rather towards the end of the process in the
comparison of a limited set of options. This is not the case for the
information regarding contact time, i.e., the scheduling of meetings
associated with a learning path: this information will help to
distinguish suitable learning paths at the very start of the decision
making process. Including this element in the specification is therefore
likely to contribute considerably to efficiency. So bearing in mind the
notion of feasible completeness the element "contact time"
will be added to the specification. Seeking a balance between the level
of detail some participants described and considerations of what is
practical, two dimensions will be distinguished: weekdays/weekend and
Finally, the aspect of "quality" was mentioned, referring
to a variety of indicators: pass/fail rates, a probe of learning
materials (up-to-date), or impressions of professionalism. This type of
information can not be grasped simply by adding another learning path
characteristic, but has to be sought in addition, through independent
We investigated 15 choice processes involving a broad variety of
learning paths, with the aim to evaluate semantic and pragmatic quality
of the learning path specification: are characteristics included in the
specification to support comparison and selection of a learning path
clear, sufficient, and without redundancies?
Regarding clarity our study showed that related characteristics
(e.g., delivery mode and contact hours) sometimes got mixed up. However,
this can be solved by presenting them in combination and with possible
None of the characteristics included in the specification appeared
redundant. Instead, several characteristics were mentioned in addition.
However, upon closer inspection, only contact time appears an adequate
improvement of the specification.
Following this investigation and adaptations made on the base of
these results, a tool is being developed to describe learning paths in
line with the specification. Subsequent tools can then be developed
which use these descriptions to facilitate selection of suitable
Though several participants hinted at information overload
regarding the number of learning paths, one respondent specifically
hinted at the risk of overload due to the number of criteria taken into
account. She said her choice process had taken the shape of a funnel
regarding the number of learning paths to compare, though not,
unfortunately, regarding the number of criteria taken into account.
Further quantitative research is required to investigate solutions aimed
at reducing the risk of information overload by distinguishing between
more and less important characteristics.
The work described here was sponsored by the TENCompetence
Integrated Project, which has been funded by the European
Commission's 6th Framework Programme, priority IST/Technology
Enhanced Learning. Contract 027087 (http://www.tencompetence.org).
Beach, L. R. (1997). The Psychology of Decision Making: People in
Organizations. Newbury Park, CA: Sage.
Beck, B. (2002). Model evaluation and performance. In A. H.
El-Shaarawi & W. W. Piegorsch (Eds.), Encyclopedia of Environmetrics
(Vol. 3, pp. 1275-1279). Chichester: John Wiley & Sons.
CEC. (2000). A Memorandum on Lifelong Learning. Brussels:
Commission of the European Communities.
Colardyn, D., & Bjornavold, J. (2004). Validation of Formal,
Non-formal and Informal Learning: policy and practices in EU Member
States. European Journal of Education, 39(1), 69-89.
Colley, H., Hodkinson, P., & Malcolm, J. (2003). Informality
and formality in learning: a report for the Learning and Skills Research
Centre. London: Learning and Skills Research Centre.
Coughlin, S. S. (1990). Recall Bias in Epidemiologic Studies.
Journal of Clinical Epidemiology, 43(1), 87-91.
Drachsler, H., Hummel, H., & Koper, R. (2008). Personal
recommender systems for learners in lifelong learning: requirements,
techniques and model. International Journal of Learning Technology,
Fasolo, B., McClelland, G. H., & Todd, P. M. (2007). Escaping
the tyranny of choice: when fewer attributes make choice easier.
Marketing Theory, 7(1), 13-26.
Flyvbjerg, B. (2006). Five Misunderstandings About Case-Study
Research. Qualitative Inquiry, 12(2), 219-245.
Hager, P. (1998). Recognition of Informal Learning: challenges and
issues. Journal of Vocational Education and Training, 50(4), 521-534.
Janssen, J., Hermans, H., Berlanga, A.J., & Koper, R. (2008).
Learning Path Information Model. Retrieved November 9, 2008, from
Janssen, J., Berlanga, A. J., Vogten, H., & Koper, R. (2008).
Towards a learning path specification. International Journal of
Continuing Engineering Education and Lifelong Learning, 18(1), 77-97.
Kickmeier-Rust, M. D., Albert, D., & Steiner, C. (2006).
Lifelong Competence Development: On the Advantages of Formal
Competence-Performance Modeling. Proceedings of the International
Workshop in Learning Networks for Lifelong Competence Development, March
2007, Sofia, Bulgaria.
Krogstie, J. (1998). Integrating the Understanding of Quality in
Requirements Specification and Conceptual Modeling. ACM SIGSOFT Software
Engineering Notes, 23(1), 86-91.
Leung, F., & Bolloju, N. (2005). Analyzing the Quality of
Domain Models Developed by Novice Systems Analysts. Paper presented at
the 38th Annual Hawaii International Conference on System Sciences,
Lidwell, W., Holden, K., & Butler, J. (2003). Universal
Principles of Design. Gloucester, Massachusetts: Rockport.
Lindland, O. I., Sindre, G., & Solvberg, A. (1994).
Understanding quality in conceptual modeling. IEEE Software, 11, 42-49.
Livingstone, D. W. (1999). Exploring the Icebergs of Adult
Learning: Findings of the First Canadian Survey of Informal Learning
Practices. NALL Working Paper No. 10.
Malhotra, N. K. (1982). Information Load and Consumer Decision
Making. The Journal of Consumer Research, 8, 419-430. Miles, M. B.,
& Huberman, A. M. (1994). Qualitative Data Analysis (2nd ed.).
Thousand Oaks: Sage.
Moody, D. L. (2005). Theoretical and practical issues in evaluating
the quality of conceptual models: current state and future directions.
Data & Knowledge Engineering, 55, 243-276.
Moody, D. L., Sindre, G., Brasethvik, T., & Solvberg, A.
(2002). Evaluating the Quality of Process Models: Empirical Analysis of
a Quality Framework. Paper presented at the 21st International
Conference on Conceptual Modeling, Tampere, Finland.
Nelson, H. J., Poels, G., Genero, M., & Piattini, M. (2005).
Quality in conceptual modeling: five examples of the state of the art.
Data & Knowledge Engineering, 55, 237-242.
Recker, J. (2005). Conceptual Model Evaluation. Towards more
Paradigmatic Rigor. Paper presented at the EMMSAD 2005 Exploring
Modeling Methods for Systems Analysis and Design.
Recker, J. (2006). Towards an Understanding of Process Model
Quality. Methodological Considerations. Paper presented at the 14th
European Conference on Information Systems, Goeteborg, Sweden.
Rundle-Thiele, S., Shao, W., & Lye, A. (2005). Computer Process
Tracing Method: Revealing Insights Into Consumer Decision-Making. Paper
presented at the Australian and New Zealand Marketing Academy Conference
Schugurensky, D. (2000). The forms of informal learning: towards a
conceptualization of the field. NALL Working Paper No. 19.
Skule, S. (2004). Learning conditions at work: a framework to
understand and assess informal learning in the workplace. International
Journal of Training and Development, 8(1), 8-20.
Teeuw, W. B., & Berg, H. v. d. (1997). On the Quality of
Conceptual Models. Paper presented at the ER'97 Workshop on
Behavioral Models and Design Transformations: Issues and Opportunities
in Conceptual Modeling, Los Angeles.
Turbek, S. (2008). Advancing Advanced Search. Boxes And Arrows: The
Design Behind the Design. Retrieved February 7, 2008, from
Van Lamsweerde, A. (2000). Formal Specification: a Roadmap. In A.
Finkelstein (Ed.), The Future of Software Engineering: 22nd
international conference on software engineering: ACM Press.
Yin, R. K. (2003). Case Study Research. Design and Methods.
Thousand Oaks, California: Sage.
Jose Janssen, Adriana J. Berlanga and Rob Koper
CELSTEC Open University of the Netherlands, Heerlen, The
Netherlands // firstname.lastname@example.org // email@example.com // firstname.lastname@example.org
Table 1. Learning Path metadata
Title Name of the program, course, workshop etc.
Description Brief description of the program, course,
Prior knowledge Competences which are expected to have been
Start conditions Other conditions that must be met in order to
start: e.g., a minimum number of participants,
a special diploma, access to a computer,
Language Languages used in the learning path
Diploma/certificate Indicates whether completion of the learning
path results in an officially recognized
diploma or certificate
Time investment Total number of hours it takes to complete
Delivery mode Indicates whether the learning path involves
self-study, face-to-face meetings or a mixture
Guidance Description of available guidance
Assessment Description of assessments associated with the
Start date/end date Start/end date
Costs Total costs for enrolment, materials, etc.
Number of contact Indicates number of hours of required (virtual)
Location Indicates where meetings take place
Completion Indicates how and by whom it is decided whether
the learning path goals have been achieved.
Provider Provider of the learning path
Further information Link to a website for further details.
Table 2. Evaluation framework
Quality Description Sub-characteristics Description
Syntactic Does the Proper notation
quality model of association,
what is multiplicity etc.
meant to be
Semantic Does the adequate  The model
quality model orthogonal/ adequately
represent independent reflects the
essential [3, 5] domain, i.e.,
features of valid [2, 4] independent
the aspects are
phenomenon captured by
under study? different concepts
and relations are
complete [1, 3, 4, The model
5] nothing missing describes all
what is expected essential
minimal  The model does not
parsimonious [3, contain irrelevant
5]] nothing aspects and
Pragmatic Is the model unambiguous Concepts and
quality easy to [1, 3] relations have a
understand? clear single
Social & meaning
general Concepts should be
as independent as
possible from any
Quality Evaluation Metrics
Syntactic --submit model to --number and
quality peer/expert review type
--validity checks ambiguities,
through software etc.
Semantic --explain the --number and
quality model to lifelong type
learners and of issues open
learning path to
providers to see debate
find it adequate --number of
on points relevant changes made
to them to
learners' learning --number and
path choice type
processes to of frictions in
establish learning mapping
essential in this
learning paths on
Pragmatic --establish --perceived
quality whether the ease of
specification is use
Social & adequate to
Technic develop tools. --perceived
whether tools --intention to
developed are use
--map informal and
 van Lamsweerde (2000)
 Leung & Bolloju (2005)
 Teeuw & van den Berg (1997)
 Krogstie (1998)
 Moody et al. (2002)