What instructor qualities do students reward?
Article Type:
Report
Subject:
College teachers (Evaluation)
College teachers (Practice)
Student evaluation of teachers (Standards)
Student evaluation of teachers (Analysis)
Education (Methods)
Education (Evaluation)
Authors:
Pepe, Julie W.
Wang, Morgan C.
Pub Date:
09/01/2012
Publication:
Name: College Student Journal Publisher: Project Innovation (Alabama) Audience: Academic Format: Magazine/Journal Subject: Education Copyright: COPYRIGHT 2012 Project Innovation (Alabama) ISSN: 0146-3934
Issue:
Date: Sept, 2012 Source Volume: 46 Source Issue: 3
Topic:
Event Code: 350 Product standards, safety, & recalls; 200 Management dynamics
Geographic:
Geographic Scope: United States Geographic Code: 1USA United States

Accession Number:
302464023
Full Text:
Most higher education institutions have a policy regarding instructor evaluation and students play a dominant role in evaluation of classroom instruction. A standardized course/instructor evaluation form was used to understand the relationship of item responses on the student evaluation form, to the overall instructor score given by students taking general education program (GEP) courses. All student evaluation information from all GEP courses at a large public metropolitan university in the southeast United States for fall 2002 through spring 2009 semesters was used for data analysis.

Results suggest that students reward, with higher evaluation scores, instructors who they perceive as organized and strive to clearly communicate content. Additionally, instructors of GEP courses need to be informed that students connect the level of respect and concern shown by the instructor and having an interest in student learning with the overall score they give the instructor.

Course characteristics were related to the relative starting point for the scale chosen by students, but instructor qualities were consistent when considering class size, class mode and course area (English, mathematics, communication, etc ...). Individual instructor characteristics were not considered in this study.

Keywords: student evaluation, instructor qualities, general education program, core curriculum

Introduction

In higher education a current focus on the educational consumer is one driver of the accountability movement that pervades American learning culture. In 2008, the federal government provided $52.1 billion in financial aid support to students attending a college or university (National Center for Education Statistics, 2009). Because of the amount of money being spent on education, this trend is gaining momentum (Gravestock & Gregor-Greenleaf, 2008; Landrum & Braitman, 2008). This emphasis is further supported by accreditation requirements for assessing student learning (Association of American Colleges and Universities, 2008; Council for Higher Education Accreditation, 2008). Seldin (1999) reported that, "student ratings are now the most widely used source of information on effective teaching" (p. 15).

There is no denying the importance of the student's role in evaluating instruction in higher education. D'Apollonia and Abrami (1997) reported that 98% of higher education institutions in the United States use some form of student evaluation and that an increasing percentage of international institutions are doing so as well (Moore & Kuol, 2005). The fact is, evaluation information is being used to make policy and personnel decisions and will continue to be utilized by students, department heads and administration for an even broader array of purposes (Kulik, 2001).

One problem with analysis of student evaluation scores is the inconsistency of exactly what the scores are measuring. As for the meaning of evaluation scores, some researchers agree they measure several aspects of effective teaching while others believe they measure student satisfaction (Abrami, d'Apollonia, & Cohen, 1990; Beyers, 2008; Centra, 1993; Marsh & Roche, 1997). To further complicate matters, a generally accepted definition of effective teaching has not been determined (Trout, 2000; Paulsen, 2002). McKeachie (1997) notes the meaning of effective teaching has not been defined and depends on the goals for instruction. Qualities of an effective teacher are much easier to define. Kolitch and Dean (1999) say that an effective teacher will be able to communicate clearly, be organized and interact well with students via examples, and relevant questions.

Data mining analysis techniques are a relatively new collection of statistical methods that apply to analyzing very large data sets to maximize extraction of information (Hand, Mannila, & Smyth, 2001). Data mining methodology and associated tools, such as decision tree analysis, allows all responses to be utilized, which in this study comprises several hundred thousand observations. Data mining tools have no strict assumptions for the functional form of the model, are robust against the presence of outliers and are resistant to the curse of dimensionality. Decision tree analysis is a flexible modeling tool that is an efficient method for studying large data sets. Results from a decision tree analysis take the form of a set of if-then rules and associated probabilities. For example a rule would be, "if the instructor gets an "Excellent" score for Organization, then the probability of getting an overall score of "Excellent" is .75."

Methodology

The purpose of this study was to determine, for GEP courses, what individual items on the student evaluation form and course variables were predictive of the overall instructor rating value. Decision tree analysis was used to find a model for determining an "Excellent" overall rating or a "Poor" overall rating, the highest and lowest options. In order to provide an honest estimate of the model results, the data set was divided into a subset (approximately 70%) to produce association rules and a separate subset (approximately 30%) to validate the model results. There was no sampling procedure applied as part of the process; data mining is specifically suited for searching in large data sets with a large number of variables and missing values (Breiman et al., 1984).

Limitations of the study are based on data collection procedures. Student information was collected prior to the study and not specifically for the purposes of this study. Procedural conditions were not controlled and direct contact with participants was not possible. The data set available was not a random sample of students or courses. Information was collected from students willing to complete the questionnaire; this group of students may represent a biased sample. Under the current data collection procedures, it was not possible to obtain random samples or complete information from all students enrolled in the course. There may be important components of the variation in evaluation scores that were not being considered as part of this study. Individual student information was not obtained at the time of data collection because the forms are anonymous. Faculty information could be obtained, but for reasons of confidentiality, instructor information was not used in this study.

Information on how students interpret the items on the student evaluation form and what specific instructor actions were being applied to that item were not available at the present time. Thus, an item such as organization of the course is based on the individual student perspective of that particular item. This study assumes that students were answering truthfully and accurately to the individual items presented on the form. Results could be invalidated if students did not fill out the form seriously, felt pressured to slant the scores, or there was a violation of procedure protocol. No provisions are currently in place to identify any of these possible threats. It is also assumed that procedural guidelines were followed or that any deviations would not be related to changes in evaluation scores.

The advantage of data mining techniques is that all data can be used including the missing category, where traditional statistics procedures typically eliminate missing data (Hand, Mannila, & Smyth, 2001). Missing values, which may be substantial, do not need to be imputed. Software options allow missing data values to be included as a separate category in initial modeling stages and surrogate rules can be implemented for analyzing additional data sets (SAS User's Manual, 2009). The disadvantages of decision tree analysis is determining the right sized tree is not always easily decided and it is possible that due to software limitations some differences will exist between alternate models. Depending on the data structure (how many variables and of what type), all possible combinations may not be checked. Linear relationships between dependent (target) and independent variables are difficult for decision trees to model efficiently due to the subdivision strategy used in the analysis.

Results could be generalized to other student populations at similar universities or to future semesters at the same university. Results should be most closely related to future semesters at the same institution, as the students, instructors, and courses will continue to be very closely related to the information used in the analysis. The novel methodology developed here however, should transfer to other institutions with similar student evaluation information.

Data Analysis

All student evaluation information from all GEP courses at a large public metropolitan university in the southeast United States for fall 2002 through spring 2009 semesters was used for data analysis. The data set contained all sixteen individual item responses from the student evaluation form completed by students at the end of a course and additional variables generated from course characteristics. The evaluation form used a 1 to 5 Likert scale (e.g., Excellent, Very Good, Good, Fair, Poor) for the sixteen items listed on the form. In addition to the item responses available from each form, information on class size, GEP theme, foundational area and teaching mode (e.g., face-to-face, web assisted, and web based) was considered in the analysis. Individual courses were grouped by foundational area; this imposed data structure was initiated in order to be as confident as possible that no individual course, instructor, or student was identified.

Analysis data set contained 23 variables, 8,065 course sections and 294,692 student responses. Final tree size was determined by incorporating two subgroups from the original data set into the model building process. The first data set was used to grow the tree, to a larger than optimum size, while the second data set pruned or cut back the tree. In order to provide an honest estimate of the modeling results, the data set was divided into a subset (approximately 70%) to produce association rules and a separate subset (approximately 30%) to adjust the model complexity. The final model identifies association rules for describing variable relationships with the target variable (overall instructor evaluation score).

Decision tree models are built by segmenting the observations into smaller and smaller groups. Model results are in the form of multiple if-then statements. For each node or group the probability of correct classification is supplied via software options; higher probabilities indicating homogeneity of observations in regard to the target variable. In order to determine the overall tree model results, a global misclassification rate was used as the criterion of model homogeneity.

Decision tree analysis was used to answer research questions regarding what rules can be identified to understand the determination of an "Excellent" overall rating or a "Poor" overall rating on the student evaluation form. The target variable was the overall instructor score, item 16 on the form. Item 1 to item 15, instructional mode, foundational area, percent responding, and GEP theme were the independent variables used to find splits to create homogenous groups. Rating categories for all items on the student evaluation form were: Excellent, Very Good, Good, Fair, and Poor. Course characteristic information of instructional mode, foundational area, percent responding and GEP theme were not chosen as significant variables for the model. These variables when considered with individual items on the form were not altering the student scoring procedure. Although the student scoring approach did not change, the percent of "Excellent" scores did change based on course characteristics.

Results

The final model had a misclassification rate of 25.5% for the training data set and 26.0% for the validation data set. Interpretation of the misclassification rate is situation specific, thus no general rules for misclassification values are appropriate. Six rules, three for "Excellent" and three for "Poor" were selected based on the highest percentage of response conformity. All resulting roles based their classifications on some combination of the following items: communication of ideas and information; facilitation of learning; respect and concern for students; instructor's overall organization of the course; instructor's assessment of your progress in the course; instructor's interest in your learning; stimulation of interest in the course.

Excellent overall score rules

The final tree model resulted in rules determining the probability of obtaining an overall instructor score of "Excellent" when certain conditions held within the complete data set. The top three rules have the highest probability based on scores from other items on the student evaluation form. All the rules for "Excellent" used only items on the evaluation form and eliminated foundational area, class size, percent responding, instructional mode, and GEP theme information. The top three rules for predicting an "Excellent" overall evaluation rating are summarized in Table 1. Rules for an "Excellent" overall score based their classifications on some combination of the following items: communication of ideas and information; facilitation of learning; respect and concern for students; instructor's overall organization of the course; instructor's assessment of your progress in the course. Of the 109,759 responses that conformed to one of the rules, the largest percentage (81.6%) belonged to rule one, followed by rule two (10.9%) and rule three (7.5%).

Rule one produced the highest rating for an "Excellent" overall score with an associated probability of 0.94; indicating that if the conditions for those variables in the rule hold true, then for those responses 94% of the responses would have received an "Excellent" overall score. Rule one states that when communication of ideas and information was "Excellent" or "Very good" and facilitation of learning and respect and concern for students items were rated as "Excellent," then the result was generally an "Excellent" overall score. In the data set as a whole, the percentage of "Excellent" overall scores was 43.82% thus making the odds ratio of rule one equal to 2.15; odds ratios indicate the odds of getting an overall score of "Excellent" if you receive the above scores represented in the rule. Odds ratios are calculated by dividing the specific rule probability for the target variable by the probability for the entire data set of having that same target variable value. For example, odds of 2.15 results from using the rule probability (0.94) divided by the overall probability of receiving an "Excellent" overall score (0.4382).

Rule two produced a rating for an "Excellent" overall score with an associated probability of 0.78. Rule two combines information from the following five individual items: communication of ideas and information as "Excellent" or "Very good"; facilitation of learning as "Very good," "Good," or "Fair"; respect and concern for students as "Excellent" or "Very good"; instructor's interest in your learning as "Excellent"; instructor's overall organization of the course as "Excellent". In the data set as a whole, the percentage of "Excellent" overall scores was 43.82% thus making the odds ratio of rule two equal to 1.78.

Rule three has an associated probability of 0.71. Rule three contains four items, three of which were in Rule one. Individual items for Rule 3 consisted of: communication of ideas and information as "Excellent' or "Very good" ; facilitation of learning as "Excellent"; respect and concern for students as "Very good" or "Good"; instructor's assessment of your progress as "Excellent" or "Very good". The odds ratio for rule three was 1.62; meaning these instructors are more than one and a half times as likely to receive an "Excellent" overall rating as one drawn at random.

When "Excellent" and "Very good" overall scores were combined, all three rules (one, two, and three) had predictive probabilities of 99%. In other words, if the rule identified a student response, 99% of those responses had an overall score of "Excellent" or "Very good."

Poor overall score rules

The final tree model resulted in rules determining the probability of obtaining an overall instructor score of "Poor" when certain conditions held within the complete data set. The top three rules have the highest probability based on scores from other items on the student evaluation form. All the rules for "Poor" used only items on the form and eliminated foundational area, class size, percent responding, instructional mode, and GEP theme information. The top three rules for predicting a "Poor" overall evaluation rating are summarized in Table 2. Rules for a "Poor" overall score based their classifications on some combination of the following items: communication of ideas and information; facilitation of learning; stimulation of interest in the course; respect and concern for students; instructor's overall organization of the course; instructor's interest in your learning; expression of expectations for performance. Of the 3,821 responses that conformed to one of the rules used for rating the overall instructor as "Poor", the largest percentage (56.1%) belonged to rule four, followed by rule five (27.7%), and rule six (16.2%).

Rule four produced the highest rating for a "Poor" overall score with an associated probability of 0.89; indicating that if the conditions for those variables in the rule hold true, then for those responses 89% of the responses would have received a "Poor" overall score. Rule four reveals that when communication of ideas and information was "Good," "Fair," or "Poor" and facilitation of learning was "Fair" or "Poor" along with respect and concern for student and overall organization items are rated as "Poor," then the result was generally an "Poor" overall score. In the data set as a whole, the percentage of "Poor" overall scores was only 2.26% thus making the odds ratio of rule four equal to 39.2; odds ratios indicate the odds of getting an overall score of "Poor" if you receive the above scores represented in the rule. Because the percentage of instructors rated as "Poor" was so low (2.26%), the odds ratio may overestimate the magnitude of this likelihood.

Rule five produced a probability of 0.68 for a "Poor" overall score. Rule five combines information from the following five individual items: communication of ideas and information was "Good," "Fair," or "Poor" ; facilitation of learning was "Fair" or "Poor"; respect and concern for students was "Poor"; overall organization was "Good" or "Fair"; instructor's interest in your learning was "Poor". The odds ratio for rule five was 29.96; meaning these instructors are much more likely to receive a "Poor" overall rating as one drawn at random.

Rule six has an associated probability of 0.60. Rule six contains six items, four of which were in Rule four. Individual items for Rule six were: communication of ideas and information at "Good," "Fair," or "Poor"; facilitation of learning at "Fair" or "Poor"; respect and concern for students at "Excellent," "Very good," "Good," or "Fair"; overall organization at "Fair" or "Poor"; stimulation of interest in the course at "Poor"; expression of expectations for performance at "Poor". The odds ratio for rule six was 26.55; meaning these instructors are more than twenty-six times as likely to receive a "Poor" overall rating as one drawn at random.

When "Poor" and "Fair" overall scores were combined, all three rules (four, five, and six) had predictive probabilities of 99%. In other words, if the rule identified a student response, 99% of these responses had an overall score of "Poor" or "Fair."

In order to check the validity of the data mining results, each of the six rules was applied to each foundational group separately. A valid model would show consistency of rule conformity percentages across all data subgroups. Results demonstrated the consistency of the overall model. Adjusted percentages using all three rules, for "Excellent" were similar across foundational areas.

Interpretation of Results

Students consistently rated communication of ideas and information as the most important item on the form in relation to the overall evaluation score. Communication was present in all six rules and suggests that the ability to communicate effectively is essential to being viewed positively by students. Study results agree with Wang et al., (2009) that communication is an important factor in the modeling of overall score. The authors found the communication item to be important in all rules cited. Their study included both undergraduate and graduate course for years 1996-2001. Wang et al., (2009) state that "communication" has been considered a standard for effective teaching. Moore, Moore, and McDonald (2008) used a qualitative study to ask questions of 271 college students. The authors found that students cited "learning the material" as the most frequently stated expectation for college courses.

Facilitation of learning also appeared in all rules and suggests related aspects of communication. In their book on facilitative teaching, Wittmer and Myrick (1974) provided instructor characteristics for promotion of learning as: good listeners; empathetic; caring; concerned; genuine; warm; interested; knowledgeable; trusting; friendly with a sense of humor; dynamic; and able to communicate. Seldin (1984) believes that being interested in teaching students and motivating students are two of the five characteristics of an effective teacher.

Empathy for students was reflected in all rules used to describe "Excellent" or "Poor" instructors. Caring instructors were rated higher than instructors rated low on the respect and concern item. In contrast, Wang et al. (2009) found respect and concern to be a consideration in only one rule out of six when using information from both undergraduate and graduate students. Seldin (1984) refers to "fair and reasonable management" while Chickering and Gamson (1987) use the statement "respects diverse talents and ways of learning." A supportive climate as defined by consideration and respect was found to be important by Kim et al. (2000).

Organization showed up as an important quality in three of the six rules; matching the results of Wang et al. (2009). Rule two reveals that a well organized instructor can overcome lower ratings for respect and concern and still achieve an "Excellent" overall score. Conversely an instructor that is not organized along with not caring about students will generally receive a low overall score. Marsh (1982) and Seldin (1984) both have a measure for organization listed in the important dimensions of effective teaching. Course organization is clearly under the control of the instructor and can be enhanced through professional development.

Interest in student learning is related to respect and concern and the associated item on the student evaluation form showed up in rules two, five, and six. Chickering and Gamson (1987) use the statement "respects diverse talents and ways of learning" which speaks to a supportive course climate.

Assessment of progress only showed up in rule three and only after communication, facilitation of learning, and respect and concern for students. In this rule if an instructor was rated slightly lower on the respect and concern scale but received an "Excellent" or "Very good" for assessment of progress they were more likely to get an overall "Excellent" score. Seldin (1984) refers to fair and reasonable management as a factor for effective teaching. Chickering and Gamson (1987) use "encourages contacts between students and faculty" and "gives prompt feedback" as important principles for undergraduate instruction. Abrami and d' Apollonia (1991) identified "feedback" as an important dimension of effective instruction.

Expectations of performance and stimulation of interest were only present in rule six which describes a "Poor" overall score. These two items are only important after taking into account scores for communication, facilitation, respect and concern, and interest in student learning. In this rule if an instructor was rated "Poor" for both expectation of student performance and stimulation of interest, in addition to low scores for communication, facilitation, and interest in student learning, they had an elevated chance of getting an overall "Poor" score. Wang et al. (2009) did not identify these items as important when using undergraduate and graduate student responses. This difference between this study and Wang et al. (2009) study results may be related to the student populations used, undergraduates only compared to undergraduate and graduate students. Upper level undergraduate and graduate students would have more college experience and therefore having the instructor spell out what is necessary for performance in the course may not be critical to their rating of the instructor or possibly, they are able to recognize when instructors are presenting performance expectations. Seldin (1984) refers to fair and reasonable management and motivating students as two important components of effective teaching. Marsh (1982) refers to enthusiasm as a dimension used to evaluate effective teaching. Abrami and d'Apollonia (1991) designate "rapport" as a dimension of effective instruction. Remedios and Lieberman (2005) found student perceived quality of teaching was related to how much students enjoyed or felt stimulated by the course content.

Summary

Instructor teaching qualities, as rated by students, is an important aspect of evaluation of instruction. Individual item responses on a standardized student evaluation form and additional course information variables were used to investigate the connection to the overall instructor score given by students taking general education program (GEP) courses at a large metropolitan university in the southeastern U.S. for fall 2002 through spring 2009 semesters. Analysis data set contained 23 variables, 8,065 course sections and 294,692 student responses.

The final model had a classification rate of approximately 75% when the overall rating was a single option ("Excellent"), but improved to approximately 99% when the rule was used to predict a rating of either "Excellent" or "Very good" for the overall instructor rating. The model results present six if then rules indicating how students rate individual instructor qualities to the overall score given to the instructor.

Items related to communication, facilitation, organization, respect and concern, instructor's assessment of your progress in the course, instructor's interest in your learning, and stimulation of interest in the course were found to characterize aspects important to students in regard to evaluating instructors. Course variables of class size, foundational area, GEP theme, and percent responding did not change student approach to evaluation. Research findings reflect that students had a consistent approach to completion of the evaluation forms and specific individual items on the form were related to important components in the students' perception of instruction. Even though the student approach was consistent across these variable groups associated with course characteristics, the unadjusted percentage of "Excellent" scores did change for certain variables such as foundational area and GEP theme.

The ability of the instructor to effectively communicate information was consistently related to the overall score given by students. Excellent communication was rewarded with an "Excellent" or "Very good" overall evaluation rating while conversely poor or fair communication typically resulted in a "Poor" overall evaluation score. If instructors were rated "Excellent" for communication, they could still achieve an excellent overall score even with lower ratings for facilitation of learning and respect and concern for students. Similar results were found by Wang et al. (2009) based on data from all graduate and undergraduate courses for years 1996-2001 at the same institution and using the same student evaluation form. The major difference was the inclusion of the respect and concern item as an important variable for GEP courses, but not in the previous study using all graduate and undergraduate courses.

Instructors need to be aware of the emphasis students place on communication, so that they can focus their efforts toward increasing this quality. Institutions could provide workshops to assist instructors with their communication skills.

References

Abrami, P. C., d'Apollonia, S., & Cohen, P. A. (1990). Validity of student ratings of instruction: What we know and what we do not. Journal of Educational Psychology, 82(2), 219-231.

Abrami, P. C., & d'Apollonia, S. (1991). Multidimensional students' evaluations of teaching effectiveness--generalizability of" N= 1" research: Comment on Marsh (1991). Journal of Educational Psychology, 83(3), 411-415.

Association of American Colleges and Universities. (2008). Project on accreditation and assessment. Retrieved from the Association of American Colleges and Universities Web site: http://www.aacu.org/resources/liberaleducation/accreditation.cfm

Beyers, C. (2008). The hermeneutics of student evaluations. College Teaching, 56(2), 102-106.

Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1998). Classification and regression trees. Boca Raton, Fla.: Chapman & Hall/CRC.

Centra, J. A. (1993). Reflective faculty evaluation: Enhancing teaching and determining faculty effectiveness. San Francisco: Jossey-Bass.

Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), 3-7.

Council for Higher Education Accreditation. (2008). An overview of U.S. accreditation, May 2009. Retrieved from the Council for Higher Education Accreditation Web site http://www.chea.org/

d'Apollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52(11), 1198-1208.

Gravestock, P., & Gregor-Greenleaf, E. (2008). Student course evaluations: Research, models and trends. Higher Education Quality Council of Ontario. Ontario: Queens Printer.

Hand, D. J., Mannila, H., & Smyth, P. (2001). Principles of data mining. Cambridge, Mass.: MIT Press.

Kim, C., Damewood, E., & Hodge, N. (2000). Professor attitude: Its effect on teaching evaluations. Journal of Management Education, 24(4), 458-473.

Kolitch, E., & Dean, A. (1999). Student ratings of instruction in the USA: Hidden assumptions and missing conceptions about good teaching. Studies in Higher Education, 24(1), 27-42.

Kulik, J. A. (2001). Student ratings: Validity, utility, and controversy. In P. Abrami, M. Theall, & L. Mets (Eds.), The student ratings debate: Are they valid? How can we best use them? New Directions for Institutional Research, no. 109, (pp. 9-25). San Francisco: Jossey-Bass.

Landrum, R. E., & Braitman, K. A. (2008). The effect of decreasing response options on students' evaluation of instruction. College Teaching, 56(4), 215-218.

Marsh, H. W. (1982). Validity of students' evaluations of college teaching: A multitrait-multimethod analysis. Journal of Educational Psychology, 74(2), 264-279.

Marsh, H. W., & Roche, L. A. (1997). Making students' evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187-1197.

Moore, M. L., Moore, R. S., & McDonald, R. (2008). Student characteristics and expectations of university classes: A free elicitation approach. College Student Journal, 42(1), 82-89.

Moore, S., & Kuol, N. (2005). Students evaluating teachers: Exploring the importance of faculty reaction to feedback on teaching. Teaching in Higher Education, 10(1), 57-73.

National Center for Educational Statistics, U.S. Department of Education. (2009). Federal support and estimated federal tax expenditures for education, by category. In Digest of educational statistics 2009 (table 373). Retrieved October 21, 2009, from the National Center for Educational Statistics Web site: http://nces.ed.gov/programs/digest/d08/tables/dt08_373.asp/

Paulsen, M. B. (2002). Evaluating teaching performance. New Directions for Institutional Research, no. 114, (pp. 5-18). San Francisco: Jossey-Bass.

Remedios, R., & Lieberman, D. A. (2008). I liked your course because you taught me well: The influence of grades, workload, expectations and goals on students' evaluations of teaching. British Educational Research Journal, 34(1), 91-115.

SAS Institute. (2009). SAS Users Guide, version 8 [available via software]. Cary, NC: Author.

Seldin, P. (1999). Changing Practices in Evaluating Teaching: A Practical Guide to Improved Faculty Performance and promotion/tenure Decisions. Bolton, MA: Anker.

Seldin, P. (1984). Faculty evaluation: Surveying policy and practices. Change 28-33.

Trout, P. (2000). Flunking the test: The dismal record of student evaluations. Academe, 86(4), 58-61.

Wang, M. C., Dziuban, C. D., Cook, I. J., & Moskal, P. D. (2009). Dr. fox rocks: Using data-mining techniques to examine student ratings of instruction. Quality Research in Literacy and Science Education, 383-398.

Wittmer, J., & Myrick, R. D. (1974). Facilitative teaching: Theory and practice. Pacific Palisades, CA: Goodyear Publishing.

JULIE W. PEPE

Florida Hospital

MORGAN C. WANG

University of Central Florida
Table 1
Decision Rules that Lead to an Overall Instructor Rating of "Excellent"

                                                  Rating
                                         E   VG   G   F   P

Question

Rule 1 (n = 89,592)
Communication of Ideas and Information   *   *
Facilitation of Learning                 *
Respect and Concern for Students         *

Rule 2 (n =11,947)
Communication of Ideas and Information   *   *
Facilitation of Learning                     *    *   *
Respect and Concern for Students         *   *
Interest in Student Learning             *
Overall Organization of the Course       *

Rule 3 (n = 8,220)
Communication of Ideas and Information   *   *
Facilitation of Learning                 *
Respect and Concern for Students             *    *
Assessment of Progress                   *   *

                                           Excellent
                                          proportion
                                          (proportion
                                         for Excellent
                                           and Very
Question                                     Good)

Rule 1 (n = 89,592)
Communication of Ideas and Information       .94
Facilitation of Learning                    (.99)
Respect and Concern for Students

Rule 2 (n =11,947)
Communication of Ideas and Information
Facilitation of Learning                     .78
Respect and Concern for Students            (.99)
Interest in Student Learning
Overall Organization of the Course

Rule 3 (n = 8,220)
Communication of Ideas and Information
Facilitation of Learning                     .71
Respect and Concern for Students            (.99)
Assessment of Progress

Table 2

Decision Rides that Lead to an Overall Instructor Rating; of "Poor"

                                            Rating
                                     E   VG    G    F   P

Question

Rule 4 (n = 2,143)
Communication of Ideas                         *    *   *
and Information
Facilitation of Learning                       *        *
Respect and Concern for Students                        *
Overall Organization of the Course                      *

Rule 5 (n =1,057)
Communication of Ideas                         *    *   *
and Information
Facilitation of Learning                            *   *
Respect and Concern for Students                        *
Overall Organization of the Course                      *
Interest in Student Learning                            *

Rule 6 (n = 621)
Communication of Ideas                         *    *   *
and Information
Facilitation of Learning             0              *   *
Respect and Concern for Students     *   *     *    *
Interest in Student Learning                        *   *
Expectations for Student                                *
Performance
Stimulation and Interest                                *
in the Course

                                         Poor
                                      proportion
                                     (proportion
                                     for Poor and
Question                                Fair)

Rule 4 (n = 2,143)
Communication of Ideas                   .89
and Information
Facilitation of Learning                (.99)
Respect and Concern for Students
Overall Organization of the Course

Rule 5 (n =1,057)
Communication of Ideas
and Information
Facilitation of Learning                 .68
Respect and Concern for Students        (.99)
Overall Organization of the Course
Interest in Student Learning

Rule 6 (n = 621)
Communication of Ideas
and Information
Facilitation of Learning                 .60
Respect and Concern for Students        (.99)
Interest in Student Learning
Expectations for Student
Performance
Stimulation and Interest
in the Course
Gale Copyright:
Copyright 2012 Gale, Cengage Learning. All rights reserved.