The design and the formative evaluation of an adaptive

educational system based on cognitive styles

Evangelos Triantallou*, Andreas Pomportsis, Stavros Demetriadis

Computer Science Department, Aristotle University of Thessaloniki, PO Box 888, 540 06 Thessaloniki, Greece
Received 18 September 2002; accepted 28 January 2003
Abstract
Adaptive Hypermedia Systems (AHS) can be developed to accommodate a variety of individual dier-
ences, including learning style and cognitive style. The current research is an attempt to examine some of
the critical variables, which may be important in the design of an Adaptive Educational System (AES)
based on students cognitive style. Moreover, this paper describes the design issues that were considered for
the development of the system that are reported in the relevant literature Throughout the development of
the system, formative evaluation was an integral part of the design methodology. The results of the for-
mative evaluation were used to improve our system in order to make the instruction more eective and
ecient. Furthermore, the recommendations resulted from the formative evaluation could be seen as some
points worth considering from designers of AHS.
# 2003 Elsevier Science Ltd. All rights reserved.
Keywords: Evaluation methodologies; Multimedia/hypermedia systems; Teaching/learning strategies
1. Introduction
This article describes the design, the development and the formative evaluation of an Adaptive
Educational System (AES) that includes accommodations for cognitive styles in order to improve
students interactions and learning outcomes. Although, cognitive styles are one of the several
0360-1315/03/$ - see front matter # 2003 Elsevier Science Ltd. All rights reserved.
doi:10.1016/S0360-1315(03)00031-9
Computers & Education 41 (2003) 87103
www.elsevier.com/locate/compedu

The system interface is developed in the Greek Language, however for the purpose of this paper, Figs. 4 and 5 that
illustrate screen-shots of the interface, are partially translated into the English Language. Part of this work (System
Architecture and Implementation) was presented at the AH2002 Workshop on Adaptive Systems for Web-based
Education in Malaga, Spain, May 2002.
* Corresponding author. Tel.: +30-2310-998443; fax: +30-2310-998419.
E-mail address: [email protected] (E. Triantallou).
important factors to be considered from designers and instructors of hypermedia-based course-
ware, little research has been done regarding the adaptation of hypermedia system to students
cognitive styles (Liu & Ginther, 1999). The current research is an attempt to examine some of the
critical variables, which may be important in the design of an adaptive hypermedia system based
on students cognitive style. As a case study a Higher Education module was developed, called
AES-CS (Adaptive Educational System based on Cognitive Styles).
A formative evaluation of AES-CS was carried out in order to investigate the relationship of
learning behavior and cognitive style in an adaptive hypermedia environment. The purpose of the
evaluation was to investigate how to improve our system in order to make the instruction more
eective and ecient and also to assess its usability and appealing. This paper rst discusses
adaptive hypermedia technology and cognitive styles as considerable confusion appears in the
literature regarding the terms cognitive style and learning style. Next, it describes the design issues
that were considered for the development of the system that are reported in the relevant literature
and should be taking into account from instructional designers of adaptive hypermedia. Finally,
it presents the description and the results of the formative evaluation.
2. Theoretical framework
2.1. Adaptive hypermedia
The phenomenal growth of the Internet and the Web over recent years has led to an increasing
interest in creating Web-based learning tools and learning environments. Hypermedia seems to be
suitable for supporting the new constructivist way of active and self regulated learning. However,
empirical studies have shown contradictory results about the eciency and eectiveness of
learning with hypermedia. Some studies indicate that hypermedia-based learning may contribute
to enhance learning and promote cognitive exibility when the learning environment is designed
task appropriately (Spiro, Feltovich, Jacobson, & Coulson, 1991). At the same time, other studies
have revealed problems for hypermedia-based learning with regards to cognitive overload and
disorientation (Marchionini, 1998).
In order to overcome the problems identied, a hypermedia system should be designed in a way
that can identify the users interests, preferences and needs and give appropriate guidance
throughout the learning process. Adaptive Hypermedia (AH) was introduced as one possible
solution. Adaptivity is especially important for Web-based educational hypermedia, as these
systems are expected to be used by several learners without assistance from a physical tutor, who
usually can provide adaptivity in an actual educational environment, i.e. classroom.
Adaptive Hypermedia Systems (AHS) build a user model of the goals, preferences and knowl-
edge of the individual user and use this model to adapt the content of pages and the links between
them to the needs of that user. Since the users goals, preferences and needs may change over
time, AHS observe these changes in order to update the users model (Brusilovsky, 1996).
AHS can be developed to accommodate various learner needs; is the ideal way to accommodate
a variety of individual dierences, including learning style and cognitive style (Ayersman &
Minden, 1995). Numerous Adaptive Educational Systems have been implemented over the last
fteen years. INSPIRE (Papanikolaou, Grigoriadou, & Kornilakis, 2001) and CS383 (Carver,
88 E. Triantallou et al. / Computers & Education 41 (2003) 87103
Howard, & Lavelle, 1996) are good examples of Adaptive Educational Systems with regards to
learning style. Although, cognitive styles are one of the several important factors to be considered
from designers and instructors of hypermedia-based courseware, little research has been done
regarding the adaptation of hypermedia system to students cognitive styles (Liu & Ginther, 1999)
and this is the focus of our research.
2.2. Cognitive style and learning
There is a technical dierence between the use of the terms cognitive style and learning style,
although numerous authors use the terms interchangeably. Cognitive style deals with the form
of cognitive activity (i.e. thinking, perceiving, remembering), not its content. Learning style, on
the other hand, is seen as a broader construct, which includes cognitive along with aective and
physiological styles.
Cognitive style is usually described as a personality dimension, which inuences attitudes,
values, and social interaction. It refers to the preferred way an individual processes information.
There are many dierent denitions of cognitive styles as dierent researchers emphasize on dif-
ferent aspects (Riding & Cheema, 1991). Witkin (1962) and Witkin, Moore, Goodenough, & Cox
(1977) has done extensive work in this area and has dened two basic styles, eld dependent (FD)
and eld independent (FI). Field dependence/independence (FD/FI) is probably the most well
known division of cognitive styles (Witkin et al., 1977).
According to Witkin, eld dependenceindependence has important implications for an indivi-
duals cognitive behavior and for his/her interpersonal behavior. While most learners fall on a
continuum between these two cognitive processing approaches, each style is dened by certain
characteristics. Specically, eld independent people tend to be more autonomous in relation to
the development of cognitive restructuring skills and less autonomous in relation to the develop-
ment of interpersonal skills. Conversely, eld dependent people tend to be more autonomous in
relation to the development of high interpersonal skills and less autonomous in relation to the
development of cognitive restructuring skills.
Furthermore, FD/FI dimension refers to a tendency to approach the environment in an analy-
tical, as opposed to global, way. FI learners generally are analytical in their approach while FD
learners are more global in their perceptions. Furthermore, FD learners have diculty separating
the part from the complex organization of the whole. In other words, FD individuals see things in
the entire perceptual eld (the forest than the trees). Additionally, FI individuals tend to be
intrinsically motivated and enjoy individualized learning, while FD ones tend to be extrinsically
motivated and enjoy cooperative learning.
The Group Embedded Figures Test (GEFT) (Witkin, Ottman, Raskin, & Karp, 1971) usually is
used to identify the FD/FI cognitive style. In this test, subjects perceived the information, which
is a series of simple gures, independently from the larger complex gure, in which the simple
gures are embedded. This test describes those who tend to rely on external cues and are less able
to dierentiate an embedded gure from an organized eld as being eld dependent; and those
who tent to rely on internal cues and are more able to dierentiate an embedded gure from an
organized eld as being eld independent.
Many experimental studies have showed the impact of eld dependence/independence on the
learning process and academic achievement. Studies have identied a number of relationships
E. Triantallou et al. / Computers & Education 41 (2003) 87103 89
between FD/FI cognitive style and learning, including the ability to learn from social environ-
ments, types of educational reinforcement needed to enhance learning and amount of structure
preferred in an educational environment (Summerville, 1999).
Studies have shown that FD are holistic and require external help while FI people are serialistic
and possess internal cues to help them solve problems. FD learners are more likely to require
externally dened goals and reinforcements while FI tend to develop self-dened goals and rein-
forcements (Witkin et al., 1977). Furthermore, FD learners appeared to benet most from illus-
trative advance organizers, while FI learners preferred illustrative post organizers (Meng & Patty,
1991). An advance organizer is a bridging strategy that provides a connection between one unit
and another. It also acts as a schema for the learner to make sense out of the new concept. A post
organizer serves as a synopsis and supports the reconstruction of knowledge. Usually, it is avail-
able after the presentation of new information.
The amount of learner control seems to be a central variable when integrating adaptive meth-
ods in educational settings. There are several arguments in the literature for and against learner
control. On the one hand, learners motivation is increased when they control the navigation of a
hypermedia environment. On the other hand, research seems to indicate that the amount of
learner control depends on the pre-skills and the knowledge state of a learner (Williams, 1993).
Furthermore, many studies have demonstrated student preference and improved performance
using a linear structure. With regards to cognitive styles, there is evidence that FD individuals
perform better using program control while FI ones prefer more learner control (Yoon, 1993).
Several problems of learning in a hypermedia environment arise from the structure of the
environment itself. In an ideal Web site, the structure is evident to the user and the information is
organized coherently and meaningfully. Navigational tools are essential in order to assist learners
to organize the structure of the Web site as well as the connections of the various components. A
coherent resource collection will allow the user to construct an accurate mental model of the
topic. Research has indicated that FD learners are less likely to impose a meaningful organization
on a eld that lacks structure and are less able to learn conceptual material when cues are not
available (Witkin et al., 1977). Furthermore, Jonassen and Wang (1993) argue that the FI lear-
ners generally prefer to impose their own structure on information rather than accommodate the
structure that is implicit in the learning materials.
3. Design of AES-CS
3.1. System architecture and implementation
The earlier studies illustrate some aspects of the interaction between cognitive styles and learn-
ing strategies. Moreover, they demonstrate the importance of cognitive style on students per-
formance in hypermedia learning environments. This study is an attempt to examine some of the
critical variables, which may be important in the design of an adaptive hypermedia system based
on students cognitive style. As a case study a Higher Education module was developed, called
AES-CS (Adaptive Educational System based on Cognitive Styles), to support the course Mul-
timedia Technology Systems which is typically oered to fourth year undergraduate students in
Computer Science Department at the Aristotle University of Thessaloniki, Greece.
90 E. Triantallou et al. / Computers & Education 41 (2003) 87103
The main characteristic of AES-CS is that it can be adapted to the cognitive style and to the
level of knowledge acquired by the student. The system is organized in the form of three basic
modules: the domain model, the student model, and the adaptation module (Fig. 1). These three
components interact to adapt dierent aspects of the instructional process, i.e. adapting the con-
tent according to users prior knowledge; adapting the presentation of contents through selection
and combination of appropriate media; adapting the teaching strategies; modifying the selection
of examples and links; and recommending appropriate hyperlinks.
3.1.1. Domain model
The domain model is a set of domain concepts. It serves as a basis for structuring the content of
AES-CS. Each concept is structured into a set of topics. Topics represent basic pieces of knowl-
edge for the given domain and their size depends on the domain. Topics are linked to each other
thus forming a kind of semantic network. This network is actually the structure of the knowledge
domain. In AES-CS each hypermedia page actually corresponds to one topic only.
3.1.2. Student model
The student model needs to be easy to construct and modify and should accurately reect the
characteristics of dierent students. Three dierent categories of information are built-in in the
student model: personal prole (which includes static data, e.g. name and password), cognitive
prole (which includes adaptable data like cognitive style preferences), and an overlay student
knowledge prole (which illustrates students knowledge on a subject). Table 1 shows the
description of some attributes included in each category, and provides information on the possi-
ble values types of the attributes and how these can be acquired.
3.1.3. Adaptation module
To support adaptivity, AES-CS uses the adaptive presentation technique (Brusilovsky, 1996)
that aims to adapt the information presented to the user according to his/her cognitive style and
knowledge state. Conditional text and page variants representations are used to accomplish
adaptive presentation. With the conditional text technique, a page is divided into chunks. Each
chunk of information is associated with a condition indicating which type of user should be pre-
sented with it. With page variants technique, two variants of the pages associated with a concept
Fig. 1. System architecture.
E. Triantallou et al. / Computers & Education 41 (2003) 87103 91
are prepared. Each variant of the page presents information in a dierent style according to FD/
FI dimension.
Adaptive navigation support is a specic adaptive hypermedia technology that aims to help
users to nd an appropriate path in a hypermedia-learning environment (Brusilovsky, 1996).
AES-CS provides adaptive navigation support by manipulating the selection and the presentation
of links through adaptive annotation and direct guidance. Adaptive annotation is the most pop-
ular form of adaptive navigation support. It was rst used in ELM-ART and since it is applied in
several other systems such as Interbook, KBS Hyperbook (Brusilovsky, 1999). Adaptive anno-
tation of hyperlinks supplies the user with additional information about the content behind a
hyperlink. The selection and the color of hyperlinks are adapted to the individual student by
taking into account information about the learners knowledge state and the instructional strat-
egy. Within AES-CS, blue color is used for recommended and gray color for not ready to be
learned. With the direct guidance, the system suggests to the student the next part of the learning
material. This technique can be seen as a generalization of curriculum sequencing but within the
hypermedia context it oers more options for direct guidance. Students prior knowledge is used
by the system in order to provide him/her the most suitable sequence of knowledge units to learn
and to work with.
3.2. Adapting instructional strategies
In the ideal educational environment, a tutor with instructional experience on a learning
domain can identify students individual dierences, with regards to cognitive styles and acquired
knowledge, and thus can provide them with learning material individually selected and struc-
tured. Moreover, the interaction that takes place in a physical classroom allows tutors to experi-
ence and understand students personal goals and preferences and thus to promote their skills. In
Table 1
The student model
Type Item Value How acquired
Personal prole Name Free text User
Password Free text User
Cognitive prole Cognitive style FD or FI User or system
Program Control Yes or No User or system
Learner Control Yes or No User or system
Instructions Yes or No User or system
Feedback Yes or No User or system
Graphics Path Indicator Yes or No User or system
Knowledge prole Concept 1 Unknown
Know
Learned
Well- Learned
User or system
Concept 2 -//- -//-
92 E. Triantallou et al. / Computers & Education 41 (2003) 87103
order to simulate in a sense an ideal educational environment, an adaptive hypermedia system
should provide learners the ability to use dierent instructional modes in order to accommodate
their individual needs and to improve their performance. Therefore, it has to include in its design
both issues of cognitive style and teaching strategy. Teaching strategy refers to the instructional
material and the instructional strategy. Table 2 presents the instructional strategies adopted in
AES-CS that support students according to their cognitive style, and follows a brief description
of these strategies and the way they were implemented in the design of AES-CS.
Global versus analytical approach: AES-CS uses adaptive presentation techniques to provide
global or analytical approach since FI learners tend to approach things analytically while FD
learners tend to approach task in a global way. More specically, conditional text and page
variants representations are used to provide information from specic to general for FI lear-
ners. On the other hand, the system provides information from general to specic for FD
learners.
Program control versus learner control: AES-CS provides both program and learner control
option, according to theoretical assumptions in FD/FI dimension. In the case of learner control
option, AES-CS provides a menu from which learner can choose to proceed the course in any
order (Fig. 2). In the program control option there is no menu, but the system guides the user
through the learning material via adaptive navigation support (see Fig. 3).
The use of contextual organizer: another feature that is embedded in AES-CS is the use of
contextual organizers. FD learners appeared to benet most from illustrative advance orga-
nizers, while FI learners preferred illustrative post organizers (Meng & Patty, 1991). AES-CS
uses adaptive navigational support by manipulating the selection and the presentation of links
in order to present and support the learner with advance or post organizer according to his/her
cognitive style.
Instructions: Jonassen and Grabowski (1993) in their study summarized the research on the
implications of the individual dierences based on FD/FI dimension. These implications of
style characteristics are considered in order to design the instructional support and the
instructional environment of AES-CS. As a result, an additional frame at the bottom part of the
screen is used to provide clear, explicit directions and the maximum amount of guidance to FD
learner. On the other hand, the system provides minimal guidance and direction to FI learner.
Specically, the only instructions that are provided to the FI learners are those in the initial
page of the course.
Table 2
Instructional strategies
Field-Dependent learners Field-Independent learners
Provide global approach Provide analytical approach
Provide information from general to specic Provide information from specic to general
Program control Learner control
Provide advance organizer Provide post organizer
Provide maximum instructions Provide minimal instructions
Provide maximum feedback Provide minimal feedback
Provide structured lessons Allow learners to develop their own structure
E. Triantallou et al. / Computers & Education 41 (2003) 87103 93
Feedback: According to Jonassen and Grabowski (1993), FD learners are more likely to require
extensive feedback (especially informative), while FI tend to prefer only to know that an error
had occurred. Therefore, the self-assessment unit provides extensive feedback to FD learner by
presenting additional information on concept, relations to previous knowledge etc. On the
contrary it provides minimal feedback to FI learner.
Fig. 2. System screen with the initial adaptation for Field Independence (FI) learners.
Fig. 3. System screen with the initial adaptation for Field Independence (FD) learners.
94 E. Triantallou et al. / Computers & Education 41 (2003) 87103
Structure: AES-CS allows FI learners to impose their own structure on information since they
perform better with material that requires structure and organization. On the other hand, AES-
CS provides structured lessons to FD learners since they prefer material that is organized and
structured. Additionally the system supports FD learners with two navigational tools in order
to help them organize the structure of the knowledge domain: concept map and graphic path
indicator (Fig. 3). Concept map is a visual representation of a knowledge domain and consists
of nodes representing concepts connected by directional links that dene the relationships of the
nodes. Concept maps may act as tools to aid study and assist to the comprehension of a
domain. In AES-CS concept map is used to help FD learners understand the big picture and
place detail in perspective. The graphic path indicator can orient users to the surrounding
hyperspace and to the content organization, aecting both cognitive overhead and coherence.
The graphic path indicator is dynamically created and presents the current, the previous and the
next topic. The graphic path indicator appears at the bottom of each page and illustrates clearly
the local neighborhood of a topic.
Table 3 summarizes the earlier instructional strategies that have been adopted in AES-CS to
support students according to their cognitive style and outlines their implementation
Table 3
Instructional strategies and their implementation
Instructional Strategy Implementation
Approach: Global Approach (FD)
From general to specic
Analytical approach (FI)
From specic to general
Control option: Program control (FD)
Adaptive navigational support
Learner control (FI)
A content menu
Contextual organizer: Advance organizer (FD)
Post Organizer (FI)
Study Instructions: Provide maximum instructions (FD)
Additional frame with instructions
Provide minimal instructions (FI)
Only the initial instructions
Feedback: Provide maximum feedback (FD)
Additional information
Provide minimal feedback (FI)
Basic information
Structure: Provide structured lessons (FD)
Graphics Path Indicator and Concept map
Allow learners to develop their own structure (FI)
E. Triantallou et al. / Computers & Education 41 (2003) 87103 95
3.3. Adaptation exibility
The initial adaptation of AES-CS to FD/FI learners was based on research results (Jonassen &
Wang, 1993; Meng & Patty, 1991; Yoon, 1993) and theoretical assumptions in FD/FI dimension
(Jonassen & Grabowski, 1993; Witkin et al., 1977). According to them, the system AES-CS
initially supports FD/FI learners with the respectively instructional strategies (see Table 2).
However, users are not just novice, intermediate or expert but range a scale of many intermediate
values. Furthermore, users are not simply FD or FI learners but instead are some combination of
both characteristics. Thus, a learner should use a combination of instructional strategies in order
to accommodate his/her individual needs and to improve his/her performance. According to
Carver, Hill, and Pooch (1999) adaptive hypermedia systems should not only model multiple
dimensions of the user, but also each dimension should have as much delineation as necessary to
truly model the user.
AES-CS allows users to change the initial adaptation based on their individual needs. More
specically, users can modify the status of four instructional strategies without necessary modify
the status of their cognitive style. Initially, the system provides FD learners with program control,
maximum instructions, maximum feedback, and structure lessons. On the other hand, the system
supports FI learners with learner control, minimum instructions, minimum feedback and allows
the learners to develop their own structure. Learners have the ability to change the initial stage
through the student model and/or appropriate interactive features (Fig. 4). The learners may
modify the control options between learner and program control, may choose minimal or max-
imum feedback, may request instructions and so on.
The aim of the exibility provided is the optimization of the adaptation under basic assumption
that adaptive systems need to be controllable by the user because they cannot be intelligent
enough to appropriately adapt in all possible cases.
4. Formative evaluation
Throughout the development of AES-CS, formative evaluation was an integral part of the
design methodology. Formative evaluation is the judgments of the strengths and weakness of
instruction in its developing stages, for the purpose of revising the instruction. The major goal of
formative evaluation is to improve the eectiveness and eciency of the instruction. In AES-CS
case, three types of formative evaluation were used: expert review, one-to-one evaluation and
small group evaluation. Tessmer (1993) denes these evaluations types as follows:
Expert reviewexperts review the instruction with or without the evaluator present. The
experts can be content experts, designers or instructors.
Fig. 4. The Toolbar.
96 E. Triantallou et al. / Computers & Education 41 (2003) 87103
One-to-one evaluationone learner at a time reviews the instruction with the evaluator and
comments upon it.
Small group evaluationthe evaluator tries out the instruction with a group of learners and
records their performance and comments.
The current interface of AES-CS and its functionalities are the result of revisions based on the
analysis of the data collected during the formative evaluation. The design of the AES-CS was
revised twice: (1) after the expert review and the one-to-one evaluation and (2) after the small
group evaluation. Next we will discuss the process of the formative evaluation, i.e. subjects, pro-
cedure, and results.
5. Discussion
5.1. Expert review and one-to-one evaluation
Five experts acted as evaluators in the expert review: a teaching/training expert, an instructional
design expert, a subject-matter expert, an educational technologist and a subject sophisticates (i.e.
a student who has successfully completed the course). In this phase, a semi-structured interview
aimed at determining the reactions of experts and a debrieng session were used. During the
semi-structured interview, although the subjects were prompted with questions, the main aim was
to get their subjective reactions to the clarity, completeness and ease of use of the prototype
courseware. The debrieng session was used to conclude the evaluation with general questions
about the instruction and the design of the prototype courseware and to prompt subjects sug-
gestions for the improvement of the courseware.
The expert review evaluation was followed by the one-to-one evaluation. Ten subjects partici-
pated in the one-to-one evaluation. They were fourth year undergraduate students studying the
course Multimedia Technology Systems in Computer Science Department at the Aristotle Uni-
versity of Thessaloniki, Greece. In this phase, a semi-structured interview and debrieng session
were used as well. Most of the subjects agreed on the user friendliness of the Web courseware.
They found that it was easy to modify the initial adaptation through the student model or the
appropriate interaction buttons. Additionally, they agreed that the structure of the courseware
was clear and easy to understand. Although, most comments were positive, they pointed out
some weakness of the software. These comments from one-to-one evaluation were compared and
processed together with experts suggestions. Some of the more signicant recommendations that
were implemented during the revision phase are as follows:
Subjects agreed on the usefulness of the instructional guidance at the bottom of the screen,
but they suggested that it should be included in the minimum amount of space possible with
regards to the overall appearance of the screen. So, a dynamic technique has implemented
which resize this frame according to the mouse position (Fig. 5).
As it has been explained earlier, in program control option AES-CS provides adaptive
navigation support by manipulating the selection and the color of hyperlinks (blue color for
recommended and gray color for not ready to be learned). However, the user was limited
E. Triantallou et al. / Computers & Education 41 (2003) 87103 97
to hyperlinks recommended by the system and could not select the gray colour hyperlinks.
The majority of the students did not agree with this method since they felt that they were
restricted by the system. After the revision, AES-CS kept the same adaptive annotation but
all the links are available to be selected by the student.
Originally, the system uses white as background color and black for the font. However,
most of the subjects suggested that they would prefer to have an option regarding the
ability to adapt the background of the HTML pages. After the revision, these suggestions
were implemented.
5.2. Small group evaluation
After these revisions, the nal stage of formative evaluation took place, consisted of the small
group evaluation in a real world environment. Ten subjects took part in the small group eval-
uation. A lecture on Digital image was used as the example learning material. The aim of this
lecture is to introduce the role of digital image into the development of a multimedia application.
The evaluation was conducted by collecting data about the instruction from a variety of sources,
using a variety of data gathering methods and tools. The following sub-sections will describe
these tools and methods used to conduct the small group evaluation.
5.2.1. Subjects
Fourth year undergraduate students studying the course Multimedia Technology Systems were
asked to volunteer for the small group evaluation. The GEFT (Witkin et al., 1971) was used to
classify the participants (N=68) into Field Dependent (FD) and Field Independent (FI) groups.
The score of the test ranged from 0 to 18. The mean score was 12.76 and the standard deviation
was 4.35. According to the Gauss theory, in a normal distribution, the range from one S.D. below
the mean to one S.D. above the mean contains the 68.27% of the case. Therefore, in the small
Fig. 5. The frame with instructional guidance before and after the revision.
98 E. Triantallou et al. / Computers & Education 41 (2003) 87103
group evaluation, the subjects were selected carefully in order to represent the target population.
Ten subjects were selected according to their score: eight students with their score into the range
[mean score S.D., mean score + S.D.] that is 817, one student from them with the highest
score (18) and one student from them with lowest score (3).
5.2.2. Procedure
A user account was created for each student to login into the system AES-CS. The initial
adaptation of the system was according to the cognitive style of the students. Thus, the students
with their scores ranged from 0 to 9 were classied as FD (n=4) and these with their scores
ranged from 10 to 18 were classied as FI (n=6).
Before proceeding with the instruction, a pre-test was used to determine subjects prior knowl-
edge on Digital image. Then, the students received a short introduction on how to use the sys-
tem and were prompted to use its various tools and instructional strategies. After that, students
were login into the system by using their user account. All students were advised to work at their
own pace without being given any time limit. During the instruction, the students used comment
logs in order to note specic strengths or weaknesses of the system.
When all the students have nished the instruction, a post-test was used in order to measure the
learning gain from the instruction. Moreover, an attitude questionnaire was given to the students
aiming to determine their experience in using the system. Finally, a debrieng session was used to
assess the subjective satisfaction of subjects on the instructional and interface design of AES-CS.
5.2.3. Results
The purpose of this phase of the evaluation was to investigate how to improve our system in
order to make the instruction more eective and ecient. Pre-tests and post-tests having the same
content in the form of open-ended items were used to measure the learning gains from the
instruction. Eight open-ended items were included in each test. Since each correct answer was
graded with two points, the total score for the test was within a range of 0 (08, none right
answer) to 16 (82, eight right answers). Table 2 presents the means of the pre- and post-test
performance scores for both the FD and FI group.
The results of the performance test indicated that subjects performance was increased after the
instruction. Furthermore, FI subjects had better results than FD subjects. However, as Table 2
illustrates, the dierence between the means of the pre-test and post-test showed that FD subjects
were improved more than FI students. Table 3 presents the pre and post-test performance scores
for every student. An interesting point here is that students with lower prior knowledge scores,
demonstrated a better performance in the post-test, than students with higher prior knowledge.
Learning eectiveness and eciency are interrelated evaluation goals, since learning and
learning time are intertwined measures of instructional worth. The eciency of the instruction is
related to the time required for learners to master the objectives (Tessmer, 1993). The example
learning material for the small group evaluation was prepared based on one typical lecture hour.
Although the students spent enough minutes to see and to use the variety of instructional strate-
gies and tools (concept map, graphics path indicator), they needed less than 60 min in order to
complete the courseware. So, regarding the eciency, the instruction was consider successful.
Furthermore, small group evaluation can oer useful implementation information concerning
the usability and the appealing of the courseware. Usability is a measure of the ease with which a
E. Triantallou et al. / Computers & Education 41 (2003) 87103 99
system can be learned and used. Because users principally know a system by its interface,
designers eorts at improving usability are primarily directed at improving interfaces. The
usability and the appealing of the system were investigated in this study through the attitude
questionnaires and the debrieng session. An analysis of the data collected showed that the sub-
jects were satised with the system. In addition, they felt that the system was clear and easy to
understand and after working with it they had a better understanding of the area studied. Fur-
thermore, they felt challenged by the instruction and they stated that they wanted to use the
courseware again. However, in the debrieng session they made suggestions for the improvement
of the system. Some of the more signicant recommendations are summarized into the following
points:
The FD subjects stated that concept map and graphic path indicator were very useful in
order to organize the structure of the knowledge domain. Furthermore, they suggested that
these tools should be active so to be used as an extra navigation tool. Although FI students
did not use these tools in order to organize the structure of the knowledge domain, they
agreed with the earlier suggestion and they felt that these tools could help them to easily
navigate through the learning material thereby avoiding disorientation.
Two of the FD students had changed many times the control option between program
and learner control. In the case of learner control option, AES-CS provides a menu
from which learner can choose to proceed the course in any order (see Fig. 2), while in
the program control option there is no menu, but the system guides the user through
the learning material via adaptive navigation support (see Fig. 3). In the debrieng session
these two FD students explained that on the one hand they preferred the program
control option, but on the other hand the menu helped them to understand the
structure of the courseware. Moreover, they suggested that the content menu should
always appear at the left frame. Therefore, the new version of AES-CS supports the
users with the ability to display or not the content menu irrespectively from program or
learner control.
Another interesting nding concerns the instructions provided by the system. As it has been
explained earlier, the system provides clear, explicit directions and the maximum amount of
guidance to FD learner, while it provides minimal guidance and direction to FI learner. The
results of the small group evaluation showed that FD students made entire use of the
maximum amount of guidance while FI ask often for guidance and direction. In fact, one
FI student had changed the option provided by the system so as to receive the maximum
amount of guidance. Finally, in the debrieng session all students agreed on the usefulness
of the exibility to change between maximum and minimum amount of guidance and
direction according to their individual needs.
Summarizing, the majority of the students were satised with the initial adaptation based on
their cognitive style and they stated that the ability to change the initial stage through the student
model or appropriate interactive features was very useful. The students indicated as very impor-
tant the system exibility to select and to use dierent instructional modes in order to accom-
modate their individual needs. Moreover, they felt challenged by the system exibility and they
were satised with the fact that the system was completely controllable by them.
100 E. Triantallou et al. / Computers & Education 41 (2003) 87103
As mentioned earlier, most of the above suggestions were implemented during the revision of
AES-CS. However, they will be further processed in order to develop more the adaptive features
of AES-CS and to fully complete the design of the system. After the nal revision, summative
evaluation will follow to assess the eectiveness of the system with reference to other educational
material used for the instruction of the particular module.
6. Conclusions and recommendations
This article describes the design and development of an Adaptive Educational System based on
Cognitive Styles (AES-CS), a prototype that includes accommodations for cognitive styles in
order to improve student interactions and learning outcomes. Currently, the basic architecture of
the system has been implemented. The AES-CS (http://mlab.csd.auth.gr/adapt, guest for login
and password) has two main components: the client running in the Web browser and the server
running in a PC workstation. The client is implemented as Web pages, residing in a number of
linked frames. The development platform used is ASP technology using Dynamic HTML and
JavaScript language that made possible to overcome HTML limitations.
Throughout the development of the AES-CS, formative evaluation was an integral part of the
design methodology. The design process was driven by continuous formative evaluation.
Recommendations from expert reviews, suggestions from the students and results of the small
group evaluation are summarized later:
Table 5
Performance on pre- and post-test
User Cognitive style Pre-test Post-test
User1 FI 11 13
User2 FI 2 15
User3 FI 7 13
User4 FI 0 13
User5 FI 0 15
User6 FI 2 12
User7 FD 0 11
User8 FD 4 13
User9 FD 0 12
User10 FD 0 11
Table 4
Performance means on pre- and post-test
Pre-test Post-test Dierence
Mean S.D. Mean S.D.
FD 1.00 2.00 11.75 0.95 10.75
FI 3.67 4.41 13.50 1.22 9.83
E. Triantallou et al. / Computers & Education 41 (2003) 87103 101
Design: the design and the development of an AES that includes accommodations for cognitive
styles were eective and ecient.
Initial adaptation: the initial adaptation based on research results and theoretical assumptions
in FD/FI dimension was consider successful. However, adaptive systems need to be con-
trollable by the user because they cannot be intelligent enough to appropriately adapt in all
possible cases.
Instructions: students should be able to access the maximum amount of guidance and
instructions whenever they need.
Structure: the system should provide tools such as the concept map and the graphic path
indicator, in order to help learners organize the structure of the knowledge domain. Further-
more, these tools should be active so to be used as an extra navigation tool.
These recommendations resulting from an empirical study, although they do not have a uni-
versal value as the design and development of a courseware always depends on the target popu-
lation and the subject matter, could be seen as some points worth considering from designers of
adaptive hypermedia systems. However, they will be further rened the summative evaluation of
the AES-CS.
Moreover, additional research direction is the evaluation of the educational eectiveness of
systems adaptation by investigating the hypothesis that the adaptivity based on students cogni-
tive style could be benecial for the observed learning outcomes. An experiment by Jonassen and
Wang (1993) showed that FI learners are better hypermedia processors, especially as the form of
the hypermedia becomes more referential and less overtly structured. In that term, part of our
further research is to examine whether or not FD learners will reach the same level of perfor-
mance as FI ones when studying in AES-CS environment.
Uncited tables
Tables 4 and 5
References
Ayersman, D. J., & Minden, A. V. (1995). Individual dierences, computers, and instruction. Computers in Human
Behavior, 11(3-4), 371390.
Brusilovsky, P. (1996). Methods and techniques of adaptive hypermedia. User Modeling and User-adapted Interaction,
6, 87129.
Brusilovsky, P. (1999) Adaptive and intelligent technologies for Web-based education. In C. Rollinger, & C. Peylo
(Eds.), Special issue on intelligent systems and teleteaching (vol. 4; pp. 1925). Kunstliche Intelligenz.
Carver, C., Hill, M., & Pooch, U. (1999). Third generation adaptive hypermedia systems. Honolulu, Hawaii: WebNet 99.
Carver, C., Howard, R., & Lavelle, E. (1996). Enhancing student learning by incorporating learning styles into adaptive
hypermedia. Boston, MA: EDMEDIA 96.
Jonassen, D. H., & Grabowski, B. L. (1993). Handbook of individual dierences, learning and Instruction. Hillsdale, NJ:
Lawrence Erlbaum Associates.
Jonassen, D., & Wang, S. (1993). Acquiring structural knowledge from semantically structured hypertext. Journal of
Computer-based Instruction, 20(1), 18.
102 E. Triantallou et al. / Computers & Education 41 (2003) 87103
Liu, Y., & Ginther, D. (1999). Cognitive styles and distance education. On-line Journal of Distance Learning Adminis-
tration, 2(3).
Marchionini, G. (1998). Hypermedia and learning: freedom and chaos. Educational Technology, 28, 812.
Meng, K., & Patty, D. (1991). Field-dependence and contextual organizers. Journal of Educational Research, 84(3),
183189.
Papanikolaou, K., Grigoriadou, M., & Kornilakis, H. (2001). Instructional and interface design in an adaptive edu-
cational hypermedia system. In Panhellenic Conference in HumanComputer Interaction (PC-HCI 2001). Patra,
Greece.
Riding, R., & Cheema, I. (1991). Cognitive stylesan overview and integration. Educational Psychology, 11(3-4), 193
215.
Spiro, R. J., Feltovich, R. J., Jacobson, M. J., & Coulson, R. L. (1991). Cognitive Flexibility, constructivism and
hypertext: random access instruction for advanced knowledge acquisition in ill structured domains. Educational
Technology, 31, 2433.
Summerville, J. (1999). Role of awareness of cognitive style in hypermedia. International Journal of Educational Tech-
nology, 1(1).
Tessmer M. (1993). Panning and conducting formative evaluations. Kogan Page Limited.
Williams, M. D. (1993). A comprehensive review of learner-control: the role of learner characteristics. Paper presented at
the Annual Meeting of the Association for Educational Communications and Technology, New Orleans, LA.
Witkin, H. A. (1962). Psychological Dierentiation. Studies of Development.
Witkin, H. A., Moore, C. A., Goodenough, D. R., & Cox, P. W. (1977). Field-dependent and eld-independent cog-
nitive styles and their educational implications. Review of Educational Research, 47(1), 164.
Witkin, H. A., Ottman, P. K., Raskin, E., & Karp, S. A. (1971). A manual for the embedded gures tests. Palo Alto,
CA: Consulting Psychologists.
Yoon, G. S. (1993). The eects of instructional control, cognitive style and prior knowledge on learning of computer-
assisted instruction. Journal of Educational Technology Systems, 22(4), 357370.
E. Triantallou et al. / Computers & Education 41 (2003) 87103 103