Sign up

Using content analysis in the textbook selection process.
Abstract:
The authors show how the traditional textbook selection process can be made more robust and quantified by the use of Content Analysis tools. These Content Analysis tools include measures of grade level, reading ease, white space, passive voice, and word count. The application of traditional textbook selection tools and content analysis tools is demonstrated with an analysis of twelve Java textbooks. The results show a wide range of reading level, total words, and arrangement of topics.

Article Type:
Report
Subject:
Textbooks (Evaluation)
Content analysis (Communication) (Usage)
Content analysis (Communication) (Educational aspects)
Educational evaluation (Methods)
Authors:
Harris, Mark
Fleck, Robert A.
Loughman, Thomas P., Jr.
Pub Date:
05/01/2000
Publication:
Name: Academy of Educational Leadership Journal Publisher: The DreamCatchers Group, LLC Audience: Academic Format: Magazine/Journal Subject: Education Copyright: COPYRIGHT 2000 The DreamCatchers Group, LLC ISSN: 1095-6328
Issue:
Date: May, 2000 Source Volume: 4 Source Issue: 2
Product:
Product Code: 2731100 Textbooks; 9105111 Educational Quality Assessment NAICS Code: 51113 Book Publishers; 92311 Administration of Education Programs
Geographic:
Geographic Scope: United States Geographic Code: 1USA United States
Accession Number:
208747732
Full Text:
INTRODUCTION

Content analysis is a systematic approach to describing and understanding the meanings and potential effects of messages by gathering, categorizing, and analyzing message components (see Riffe et al., 1998, pp. 18-20, for a review of content analysis definitions). As a research technique content analysis casts a wide net and can be observed in a variety of fields including communications, psychology, sociology, medicine, political science, organizational behavior, and information technology, to name a few.

A provocative recent example of content analysis is Kathleen Hall Jamieson's Everything You Think You Know About Politics ... and Why You're Wrong (2000). This book measures the amount of media coverage devoted to recent political campaigns in print and broadcast media. It also reviews the idea of "media bias" by assessing the degree of favorability expressed by reporters towards various candidates and issues. The general intent of the analyses is to explore and explode some common misperceptions about the role of print and broadcast media in political contests and to provide insights into why some political campaigns succeed while others fail.

Other recent examples involving popular media include analyses of profanity in rap songs, violence episodes in movies such as The Matrix, sexual situations in movies and television programs, and overt as well as subliminal messages in advertising. Content analysis can also be useful in providing insights into the design and potential impact of web sites (Loughman & Fleck, 1999), annual reports (Fleck, et al., 1995), and computer system documentation (Fleck & Loughman, 1996).

In addition to its usefulness as a standalone analysis tool, content analysis also functions well as part of larger methodologies such as sociotechnical analysis, which attempts to integrate various organizational assessment tools from a range of disciplines. For example, in a recent study (Loughman et al., 1999), content analysis of communication behaviors was linked with system analysis measures to provide bank management with a robust source of information to help them decide how to resolve a customer attrition problem.

The present study uses content analysis to provide educators in information systems with advanced tools to help them decide which textbooks to choose based upon their assessments of various content criteria contained in the books. The process is illustrated using several well-known Java textbooks and occurs in two stages: (1) a description of traditional text evaluation procedures, and (2) the application of additional tools from content analysis.

TEXTBOOK SELECTION PROCESS

The textbook selection process can be viewed as an application of the systems development life cycle. The first step in the process is to determine the course objectives and then to build the content from those objectives. Course objectives are influenced by market demands, course level, prerequisites, and other environmental variables. Many of these variables are outside the purview of the person or committee making the textbook selection and are not addressed in this study. Instead, our focus is on the tools that can be used to assist in the objective evaluation of available course materials.

After course objectives have been established, the second step is to search for textbooks that appear to match course content. Desk copies of the textbooks are ordered in the third step and textbook content is compared with course objectives. One way to compare textbooks with each other and with course objectives is to list both mandatory and desirable text features. Text content by chapter could also be used in this features/objectives list. The first column of Tables 1a and 1b lists some of the features of twelve Java textbooks. Users of Java will recognize the terms and may even think of additional criteria that should be included. Those not familiar with Java should focus on the process rather than the terms. The row labeled "Chapter 15" contains the last row of textbook information that might traditionally be gathered by a textbook evaluator. The rows following the "Chapter 15" row contain our suggestions for additional evaluation criteria made available through content analysis and are discussed more fully below.

In comparing textbooks with course objectives, texts that do not include mandatory materials (e.g., a student web site) could be eliminated from further consideration. In cases where none of the textbooks can supply mandatory content, the course objectives and/or selection criteria could be changed. The texts that do contain mandatory material are then further evaluated based on desirable characteristics such as color, degree of text saturation on the page, and writing style that could affect readability.

Even in the best of textbook selection processes, those participating in the decision-making may find that text selections are difficult to read, use awkward or obfuscatory writing styles, or provide less than expected value to readers. Tools from content analysis can help evaluators by supplying a set of quantitative measures that can be used in conjunction with traditional textbook evaluation criteria. These measures, Passive Voice, Flesch Reading Ease, and Flesch-Kincaid Grade level, are described below and then applied to each of the twelve Java texts. Other listed measures are discussed in the Conclusion.

Passive Voice

In the active voice the action is in the verb and is performed by a clearly identified agent, whereas in the passive voice the action is in a form of the verb "to be" and a past participle, often with no agent clearly identified. For example, in sentence 1 below, Maria is the agent who performs the action, but in sentence 2 the agent responsible for sending the report is no longer identifiable:

1. After the report was finished, Maria sent it to the mailroom.

2. After the report was finished, it was sent to the mailroom.

Generally speaking, teachers of writing and authors of textbooks on writing tend to focus on the negative aspects of passive voice. Varner (1991), for example, examines how the active voice usually is more direct, stronger, and more concise than passive voice constructions (p. 45). She does state that passive voice is preferable in some cases, such as when a company does not wish to directly accuse a customer of doing something wrong, or when the action is more important than the agent performing the action (p. 46). However, she maintains (p. 46) that it is good practice "to avoid slipping routinely or accidentally into the passive voice."

Another viewpoint more favorable to the use of passive constructions appears in Williams' The Teacher's Grammar Book (1999). He maintains (p. 171) that in science and social science the "passive is a well established and quite reasonable convention" because it is based on the "worthwhile goal of providing an objective account of procedures." The passive also can be used to avoid assessing blame. Nevertheless, it is easier, says Williams, for people to "process sentences in the active voice with a readily identifiable subject." (p. 172)

Despite the sometimes conflicting opinions over the relative value of passive versus active voice constructions, most commentators on the subject agree that context plays a pivotal role in deciding which one to use in given situations. The present study presents data on passive voice from a sample textbook page but does not specify whether more or less usage is desirable. That decision is left up to the educator who needs to establish the objectives the textbook is intended to achieve, which in part depend upon the contexts in which the text will be used.

The values of the "counted" measures such as Passive Voice, Grade Level, and Characters per Page provided in the tables were determined as follows:

1 The first page in the first chapter that did not have any graphics was selected, copied, and scanned. The scanned document was converted to text using OCR and saved as a Word 2000 file. Any small icon on the page was eliminated.

2 Headers and footers were excluded from the analysis. If the page included a sentence fragment at the top or bottom of the selected page, that fragment was eliminated before "counting" or determining readability measures.

3 The number of characters and words per page as well as the reading statistics were determined by applying the "Word Count" and "Spelling and Grammar" Word 2000 Tools.

4 The percentage of page with text was calculated by measuring the page size and then measuring the rectangular area that contained all the text. Header and footer areas were excluded in the text area calculation. Paragraph headings intruding into the white space beyond the body text margins did not expand the computed text area. Text height is the vertical distance from the first to last line of text on the page.

Countable measures were based on one page. While the authors believe that this approach is solid, based on consistent sampling rules, they also recognize the potential for variability within any text. Those who wish to apply our methodology should first establish their own course objectives and independently assess each text's qualities. If the content analysis methodology is to be used as a deciding factor, additional page samples should be taken. Furthermore, while all data in the table are believed to be accurate, they should be used as a process guide rather than for the direct evaluation of a specific Java text.

Flesch Reading Ease

The help function in MS Word 2000 defines this measure as follows: Rates text on a 100-point scale; the higher the score, the easier it is to understand the document. For most standard documents, aim for a score of approximately 60 to 70.

The formula for the Flesch Reading Ease score is:

206.835 - (1.015 x ASL) - (84.6 x ASW)

where:

ASL = average sentence length (the number of words divided by the number of sentences)

ASW = average number of syllables per word (the number of syllables divided by the number of words)

Flesch-Kincaid Grade Level

The help function in MS Word 2000 defines this measure as follows:

Rates text on a U.S. grade-school level. For example, a score of 8.0 means that an eighth grader can understand the document. For most standard documents, aim for a score of approximately 7.0 to 8.0.

The formula for the Flesch-Kincaid Grade Level score is:

(.39 x ASL) + (11.8 x ASW) - 15.59

where:

ASL = average sentence length (the number of words divided by the number of sentences)

ASW = average number of syllables per word (the number of syllables divided by the number of words)

Other Content Analysis Measures

Tables 1a and 1b also include the several other measures that can affect readability as well as depth of coverage, including percentage of text per page, number of words on the sample page, the estimated number of words in the textbook, and the number of pages in the text.

The percentage of text per page is an indicator of saturation, how much of the page is used for text and how much is "white space." Parker (1988) maintains that white space generally makes text more appealing and easier to read. However, large amounts of white space can be coupled with larger font sizes to expand the thickness of a text, giving the false impression of depth. The percentage-of-text measurement should be used in conjunction with the estimated words-per-text measure, which for this study was calculated by multiplying the estimated words per page by the publisher's page count.

The number of pages in a text can be an indirect indicator of depth of coverage: the more pages, the more detailed the coverage. The number of pages in the table is based on the publisher's numbering system and represents the last page number in the book. Hence, introductory sections are not included in this measure.

CONCLUSION

Combined with traditional evaluation criteria, the content analysis measures suggested in this paper can give evaluators a more sophisticated and useful set of tools to help in choosing textbooks. Once texts have been winnowed using traditional methods based on topical coverage, tools such as Flesch Reading Ease and Flesch Kincaid Grade Level can be used to further assist in the process. A textbook with a lower grade level, for example, might be more appropriate for use in a freshman-level course than a text with a higher grade level. Similarly, a professor who stresses the use of the active voice in documentation may prefer to select a text with lower passive voice scores. Scores on characters per square inch of text and square inch of page can give an evaluator a sense of how "dark" or saturated the page is, an indirect indicator of readability (Parker, 1988). Total word count can be used to measure cost effectiveness: thicker books do not necessarily have more content, they may only have more pages.

REFERENCES

Fleck, R. A. Jr. & Loughman, T. P. (1996). Content analysis of computer system documentation, Academy of Managerial Communication Proceedings (pp. 6-9). Maui, Hawaii.

Fleck, R. A. Jr., Loughman, T. P., Garcia, P. & Hortman, S. (1995). Content analysis of annual reports from several English-speaking nations, Pan-Pacific Conference XII Proceedings (pp. 230-232). Dunedin and Queenstown, New Zealand.

Jamieson, K. H. (2000). Everything you think you know about politics...and why you're wrong. Basic Books.

Loughman, T. P. & Fleck, R. A., Jr. (1999). Content analysis of the best web pages, Fall Conference Guide, Mid-Southeast Chapter of ACM (p. 28). Gatlinburg, Tennessee.

Loughman, T. P., Fleck, R. A., & Snipes, L. S. A cross-disciplinary model for improved information systems analysis. Industrial Management & Data Systems, In Press.

Parker, R. C. (1988). Looking good in print: A guide to basic design for desktop publishing. Chapel Hill, North Carolina: Ventana Press.

Riffe, D., Lacy, S., & Fico, F. G. (1998). Analyzing media messages: Using quantitative content analysis in research. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Varner, I. L. (1991). Contemporary business report writing (2nd ed.). Chicago: The Dryden Press.

Williams, J. D. The teacher's grammar book (1999). Mahwah, New Jersey: Lawrence Erlbaum Associates.

Mark Harris, Appalachian State University

Robert A. Fleck, Jr., Columbus State University

Thomas P. Loughman, Columbus State University
Table 1a: Content Analysis Criteria and Measures for
Selected Java Textbooks

Author                  Horstmann            Gittleman

                        Computing            Computing
                      Concepts with          with Java:
                    Java 2 Essentials,   Programs, Objects,
Title                    2nd ed.             Graphics

Publisher                 Wiley             Scott/Jones
ISBN                  0-471-34609-8        1-57676-023-5
Publication                1999                 1998
Printing                    1                    7
1st application             21                   10
First applet               145                   11
Swing?                     yes                   No
Input                 Custom console       Custom console
                          class                class
OOAD                  Ch. 14, p 563       Getting Started
                                              with OOP
                                          section in each
                                              chapter
Pages                      762                  528
Accompanying                No                  Disk
  media
Code available              No                  Disk
Development                 No                   No
  environment
  included
Internet site               No            www.engr.csulb.
                                           edu/~artg/cwj
Instructor's                No                   No
  manual
# colors                    2                    1
Comment
Ch. 1                     Intro                Intro
Ch. 2                      Data            Data, methods
Ch. 3                    Classes              Control
                                             structures
Ch. 4                    Graphics             Control
                                             structures
Ch. 5                    Control                OOP
                        structures
Ch. 6                    Control               Arrays
                        structures
Ch. 7                    Methods              Graphics
Ch. 8                    Testing             Components
Ch. 9                  Inheritance             Events
                       /interfaces
Ch. 10                    Events
Ch. 11                    Arrays
Ch. 12
Ch. 13
Ch. 14
Ch. 15
Page width               7  1/8               7  3/8
Page height                 9                 9  1/8
Text width                  5                    5
Text height              7  7/16              7  3/8
% of page with            57.99%               54.79%
  text
# of characters           2,112                2,276
Words                      427                  450
Characters per            56.79                61.72
  sq. inch of
  text
Characters per            32.94                33.82
  sq. inch of
  page
Estimated                325,374              237,600
  words in text
Passive                    14%                   0%
  sentences
Flesch Reading             58.9                 45.0
  Ease
Flesch-Kincaid             8.3                  11.4
  Grade Level

Author                    Holmes         Garside & Mariani

                       Programming          Java: First
Title                   with Java             Contact

Publisher            Jones & Bartlett        Technology
ISBN                  0-7637-0707-4        1-85032-316-X
Publication                1998                 1998
Printing                    1                    3
1st application             8                    21
First applet               431                  499
Swing?                      No                   No
Input                  Brute force         Custom console
                                               class
OOAD                     Ch. 3 &
                        throughout          Ch. 2, p 535
Pages                      586                  660
Accompanying               Disk                  No
  media
Code available         Online, disk            Online
Development                 No                   No
  environment
  included
Internet site         www.jbpub.com/       www.course.com
                          disks
Instructor's              Online                 No
  manual
# colors                    2                    1
Comment
Ch. 1                     Intro                Intro
Ch. 2                   Data, I/O             Classes
Ch. 3                     Design          Classes, methods

Ch. 4                    Control         Control structures
                        structures
Ch. 5                    Control         Control structures
                        structures
Ch. 6                     Arrays                Data
Ch. 7                    Classes              Classes
Ch. 8                    Classes              Classes
Ch. 9                    Classes               Arrays
Ch. 10                  Exception             Classes
                         handling
Ch. 11                 Components &        priority queue
                         graphics
Ch. 12                   Applets              Classes
Ch. 13                                        Classes
Ch. 14                                       Exception
                                              handling
Ch. 15                                       Graphics,
                                             components
Page width                  7                    7
Page height              8  7/8               9  1/16
Text width               5 11/16                 5
Text height              6 11/16              6 15/16
% of page with            61.22%               54.68%
  text
# of characters           2,333                2,694
Words                      479                  595
Characters per            61.34                77.66
  sq. inch of
  text
Characters per            37.55                42.47
  sq. inch of
  page
Estimated                280,694              392,700
  words in text
Passive                    21%                  30%
  sentences
Flesch Reading             45.2                 52.0
  Ease
Flesch-Kincaid
  Grade Level               12                   12

Author                Stoker & Plew        Lewis & Loftus

                     An Introduction
                         to Java
                      Programming:        Java Softtware
                    Developing Applets       Solutions:
                     Using Microsoft      Foundations of
Title                  Visual J++         Program Design

Publisher               Technology         Addison-Wesley
ISBN                  0-7600-5043-0        0-201-57164-1
Publication                1998                 1998
Printing                    1                    4
1st application            299                   34
First applet                31                   65
Swing?                      No                   No
Input                     Applet            Brute force
OOAD                        No             Ch. 11, p 407
Pages                      400                  857
Accompanying                CD                   No
  media
Code available            Online               Online
Development            Visual J++,               No
  environment         Trial Edition
  included
Internet site          www.course.         www.awl.com/c
                           com            seng/author/lewi
                                               s/java
Instructor's            Online, CD              Yes
  manual
# colors                    2                    2
Comment                Bundled with
                     student edition
                      of Visual J++
Ch. 1                     Intro                Intro
Ch. 2                    Classes               Intro
Ch. 3                      Data                 Data
Ch. 4                    Control              Classes
                        structures
Ch. 5                    Control              Control
                        structures           structures
Ch. 6                     Arrays               Arrays
Ch. 7                   Components            Graphics
Ch. 8                                         Classes
Ch. 9                                         Classes
Ch. 10                                       Components
Ch. 11
Ch. 12
Ch. 13
Ch. 14
Ch. 15
Page width               8  1/2               6 15/16
Page height              10  3/4              8  7/8
Text width                  5                    5
Text height              8 13/16              7  5/16
% of page with            48.22%               59.38%
  text
# of characters           2,652                1,632
Words                      570                  306
Characters per            60.19                44.64
  sq. inch of
  text
Characters per            29.02                26.51
  sq. inch of
  page
Estimated                228,000              262,242
  words in text
Passive                    25%                  22%
  sentences
Flesch Reading             47.9                 47.2
  Ease
Flesch-Kincaid             11.5                 10.4
  Grade Level

Table 1b: Content Analysis Criteria and Measures for
Selected Java Textbooks

                          Hume &             Lambert &
Author                  Stephenson            Osborne

                       Programming
                         Concepts        Java: A Framework
                    in Java, 2nd ed.      for Programming
                       (Visual Age          and Problem
Title                   edition)              Solving

Publisher             Holt Software             PSW
                        Associates
ISBN                  0-921598-32-7        0-534-95116-3
Publication date           1999                 1999
Printing                    1                    1
First                       16                   15
  application
First applet               215                  419
Swing?                      No                   No
Input                 Custom console       Custom console
                          class                class
OOAD                        No              Very little
Pages                      555                  533
Accompanying                CD                   CD
  media
Code available              CD                   CD
Development             Visual Age            JBuilder
  environment
  included
Internet site               No             www.wlu.edu/~l
                                           ambertk/java/
Instructor's                No                   No
  manual
# colors                    1                    3
Comment                Bundled with
                       IBM's Visual
                           Age
Ch. 1                     Intro                Intro
Ch. 2                   Visual Age             Intro
Ch. 3                      Data                 Data
Ch. 4                  standard I/o           Control
                                             structures
                         Control              Methods
Ch. 5                   structures
Ch. 6                    Strings              Control
                                             structures
                                              Control
Ch. 7                    Methods             structures
Ch. 8                    Classes              Classes
Ch. 9               Applets, graphics          arrays
Ch. 10                    Arrays              Classes
Ch. 11                                       Recursion
Ch. 12                                        Graphics
Ch. 13
Ch. 14
Ch. 15
Page width               6 10/16                 7
Page height              8 14/16               9 2/16
Text width               4 13/16               5 3/16
Text height              6 11/16               7 8/16
% of page with            54.74%               60.91%
  text
# of characters           1,495                1,740
Words                      269                  317
Characters per            46.45                44.72
  sq. inch of
  text
Characters per            25.43                27.24
  sq. inch of
  page
Estimated                149,295              168,961
  words in text
Passive                    33%                  17%
  sentences
Flesch Reading
  Ease                     30.9                 48.7
Flesch-Kincaid              12                   10
  Grade Level

Author                   Savitch              Farrell

                        Java: An
                       Introduction
                       to Computer
                       Science and        Java Programming
Title                  Programming        (Comprehensive)

Publisher             Prentice Hall            Course
                                             Technology
ISBN                  0-13-287426-1        0-7600-1070-6
Publication date           1999                 1999
Printing                    1                    1
First                       17                   6
  application
First applet               682                  271
Swing?                      No              Very little
Input                 Custom console      in.read(): char
                          class              input only
OOAD                        No                Not much
Pages                      725                  687
Accompanying                CD                   CD
  media
Code available          Online, CD           Online, CD
Development            Code Warrior             JDK
  environment           Lite (very
  included            limited), Code
                         Warrior
                    Professional with
                     course adoption
Internet site              www-            www.course.com
                    cse.ucsd.edu/user
                    s/savitch/books/c
                         s1.java/
Instructor's         manual, web site          online
  manual
# colors                    2                    1
Comment                                   Shorter version
                                          available: just
                                           the first 319
                                               pages
Ch. 1                     Intro                Intro
Ch. 2                      Data           Classes, methods
Ch. 3                    Control              Classes
                        structures
Ch. 4                    Classes              Control
                                             structures
Ch. 5                    Methods               arrays
Ch. 6                    Classes              Applets
Ch. 7                     Events              Graphics
                        Exception
Ch. 8                    handling             Classes
Ch. 9                    File I/O             Classes
Ch. 10                    arrays             Components
Ch. 11                                         Events
Ch. 12
Ch. 13
Ch. 14
Ch. 15
Page width                7 6/16               7 4/16
Page height                 9                  9 2/16
Text width                  5                 4 15/16
Text height              7 10/16              7 13/16
% of page with            57.44%               58.31%
  text
# of characters           2,979                2,798
Words                      656                  489
Characters per            78.14                72.54
  sq. inch of
  text
Characters per            44.88                42.29
  sq. inch of
  page
Estimated                475,600              335,943
  words in text
Passive                    41%                  14%
  sentences
Flesch Reading
  Ease                     59.7                 23.2
Flesch-Kincaid             9.4
  Grade Level
                                             Gilbert &
Author               Deitel & Deitel          McCarty

                                          Object-Oriented
                       Java: How to         Programming
Title               Program, 3rd ed.          in Java

Publisher             Prentice Hall            Waite

ISBN                  0-13-012507-5        1-57169-086-7
Publication date           1999                 1997
Printing                    1                    1
First                       36                  239
  application
First applet                79                   75
Swing?                     Yes                   No
                        Components
                      until streams       in.read(): char
Input                     (late)             input only
OOAD                Software
Engineering         No
                        throughout
Pages                      1355                 953
Accompanying                CD                   CD
  media
Code available          Online, CD               CD
Development           JDK, Jbuilder             JDK
  environment
  included
Internet site         www.deitel.com             No
Instructor's               yes                   No
  manual
# colors                    2                    2
Comment              suitable for 2nd
                          course
Ch. 1                     Intro                Intro
Ch. 2                     Intro               Classes
Ch. 3                    Applets              Classes
Ch. 4                    Control              methods
                        structures
                         Control              Control
Ch. 5                   structures           structures
Ch. 6                    Methods              Control
                                             structures
Ch. 7                     Arrays              Testing
Ch. 8                    Classes               Arrays
Ch. 9                    Classes              Classes
Ch. 10                   Strings              Classes
Ch. 11                   Graphics             Graphics
Ch. 12                  Components           Components
Ch. 13                  Components
Ch. 14
Ch. 15
Page width               6 14/16               7 8/16
Page height                 9                 8 15/16
Text width                  5                  4 7/8
Text height              7 11/16               7 1/4
% of page with            62.12%               52.73%
  text
# of characters           3,162                1,917
Words                      589                  400
Characters per            82.26                54.24
  sq. inch of
  text
Characters per            51.10                28.60
  sq. inch of
  page
Estimated                798,095              381,200
  words in text
Passive                     6%                   4%
  sentences
Flesch Reading
  Ease                     44.8                 65.6
Flesch-Kincaid             11.5                 7.1
  Grade Level
Gale Copyright:
Copyright 2000 Gale, Cengage Learning. All rights reserved.