Assessment of tutoring laboratories in a learning assistance center.
Article Type:
Report
Subject:
Laboratories (Educational aspects)
Tutors and tutoring (Management)
School buildings (Educational aspects)
Author:
Fullmer, Patricia
Pub Date:
03/22/2012
Publication:
Name: Journal of College Reading and Learning Publisher: College Reading and Learning Association Audience: Academic Format: Magazine/Journal Subject: Education Copyright: COPYRIGHT 2012 College Reading and Learning Association ISSN: 1079-0195
Issue:
Date: Spring, 2012 Source Volume: 42 Source Issue: 2
Topic:
Event Code: 200 Management dynamics Computer Subject: Company business management
Product:
Product Code: 8516000 R&D Facilities NAICS Code: 5417 Scientific Research and Development Services
Geographic:
Geographic Scope: United States Geographic Code: 1USA United States

Accession Number:
290733177
Full Text:
The Learning Resource Center at Lincoln University, Pennsylvania, provides tutoring laboratories that are required for developmental reading, writing, and math courses. This article reviews the processes used to plan and determine the effectiveness of the tutoring laboratories, including logic models, student learning outcomes, and the results of an analysis of the differences between pretest and posttest scores using a t test and [eta.sup.2] (measure of effect). The pretest and posttest scores of students who attended at least six lab sessions were examined. The t test results indicated a high level of significance and the [eta.sup.2] score indicated a moderate to strong effect. Students in the tutoring laboratories showed gains in academic skills from the pretest to the posttest, supporting the effectiveness of the tutoring laboratories, and the assessment process provided accessible information that was used to improve the tutoring laboratory program.

**********

Lincoln University, Pennsylvania, the first Historically Black College and University (HBCU) in the United States, is situated on the East Coast near Philadelphia. The University's Learning Resource Center (LRC) provides individual tutoring, student workshops, and tutoring laboratories required for success (developmental) courses in education, English, and mathematics. In 2008, the LRC was enhanced through the addition of computers and updated technology, and its administrators began to search for an appropriate online tutoring program. The goal was to improve the tutoring laboratories, foster the engagement of students, and effectively utilize the available professional tutors.

Online tutoring programs, with available professional tutors to guide students, provide an efficient way to reach larger numbers of students and still retain individualized instruction and practice. An exemplary online tutoring program can also provide reliable service to students 24 hours per day, seven days per week, when professional or peer tutors are unavailable. Gillen, Lye, and Vaughn (2004) found that online tutoring provided an effective use of class time as well as opportunities for practice for students. With these considerations in mind, the staff of the LRC reviewed online tutoring programs for use in the tutoring laboratories, and began the process of implementing online tutoring.

This article examines the incorporation of online tutoring into required tutoring laboratories for developmental courses in reading, writing, and mathematics, including the evaluation of the effectiveness of the laboratories. As part of the self-assessment process, the online tutoring programs made available information that was crucial in the evaluation of the tutoring laboratories. Students' gains in academic skills from a pretest to a posttest were examined and analyzed, leading to modifications and improvements in the tutoring laboratory programs.

Tutoring Laboratories

Online tutoring programs proved crucial in the enhancement of the tutoring laboratories in the LRC. The year prior to fall 2008 had been spent researching online programs in order to make an informed decision about which program to implement. At that time, the reading and writing tutoring laboratories used paper-and-pencil exercises, and no mathematics lab existed. The Reading Lab used binders of reading selections with multiple-choice questions, and the Writing Lab required students to write one paragraph per lab session on prompts devised by the tutors.

At the time, there were two basic types of online tutoring available: online tutoring (which could be either synchronous or asynchronous, and involves a live tutor who is matched with a student), and web-based tutorials (which consist of content that a student works through independently). After reviewing input from the university's academic departments and information on a number of online tutoring programs, content-type programs were chosen. The nature of the required tutoring laboratories and the need for diagnostic assessments were the main factors in this decision. The goal of the required tutoring laboratories was to provide review and practice of the content areas covered in the students' courses.

In collaboration with the Education and English Departments, the LRC staff chose to implement My Reading Lab (2011a) and My Writing Lab (2011) from Pearson Publishing. The tutors customized the content of both programs in the areas requested by the Education and English Departments. The Education Department, which provides the developmental reading courses, requested that the Reading Lab concentrate on the topics that are covered in the syllabus and on the department's reading comprehension assessment. The English Department requested that the Writing Lab focus on grammar while the professors would focus on the writing process in the classroom. The Mathematics Department recommended the purchase of ALEKS[R], an online mathematics program published by McGraw-Hill.

ALEKS[R], Assessment and LEarning in Knowledge Spaces, utilizes artificial intelligence programming to individualize math problems by determining what the student knows and what the student is prepared to learn. Software engineers, cognitive scientists, and mathematicians assembled by Professor Jean-Claude Falmagne of New York University and the University of California (Irvine) developed ALEKS[R] with the support of a National Science Foundation grant. The basis of ALEKS[R] is Knowledge Space Theory developed through the research of Professor Falmagne and Professor Jean-Claude Doignon at the University of Brussels in the area of mathematical cognitive science (ALEKS[R], 2011).

The online tutoring programs used in the LRC in the reading, writing, and math labs provided a computer-scored diagnostic pretest and post-test, a direct measurement of skills learned, and a solid foundation on which to base student learning outcomes and an analysis of effectiveness. Students entered the online tutoring programs using access codes, and professors and tutors received a username and password for free access. Each program was tailored to the student's study plan and to his or her needs. A management module allowed professors and tutors to review students' work.

Beginning in fall 2008, students accessed online tutoring programs to take a diagnostic assessment after which an individualized study program was generated for each student. At the end of the semester, each student completed a postdiagnostic assessment to determine the amount of improvement from pretest to posttest. The results of the diagnostic testing were examined, and student learning outcomes were developed.

Laboratory Tutoring Sessions

The Reading Lab and Writing Lab weekly tutoring sessions, each lasting 50 minutes, began with a 15-minute review mini lesson presented by the tutors, after which the students worked on the online program in that week's topic area. Each week's designated topic was chosen in collaboration with the respective academic departments. The Reading Lab topics included main idea, supporting details, patterns of organization, inference, critical thinking, vocabulary, and purpose and tone. The sequence of topics for the Writing Lab included thesis statement, point of view, revision/concise language, proofreading and parallel structure, sentence fragments, run-ons/comma splices, commas, pronouns, verb agreement, and parts of speech. The Math Lab mini review topics included fractions, operations, decimals, percents, introduction to algebra, real numbers, order of operations, and linear equations.

The online tutoring programs provided an efficient way to assess the effectiveness of the tutoring laboratories. Prior to the online programs, students completed paper-and-pencil pretests and posttests that were scored by hand, with scoring taking up valuable tutor time that could have been best spent with students. Also, the scoring of the pretests under the previous system was not completed until near the end of the semester, negating any formative assessment value. With the instantaneous scoring of the pretests and posttests with the online system, students immediately saw their areas of strengths and weaknesses, and the tutors adapted the tutoring laboratories to proactively assist the students in their areas of need.

The online tutoring programs also provided individual progress reports for each student, as well as class progress reports, that fulfilled grant-reporting requirements. At the end of each semester information was available in the online tutoring programs for a summative evaluation and an assessment of effectiveness of the tutoring laboratories.

Assessment

The Council for the Advancement of Standards in Higher Education (CAS) (2010) provides evaluation guidelines for learning assistance programs. The guidelines detail how exemplary learning assistance programs systematically and periodically gather data from a variety of sources, analyze the data, and use the results to make improvements in the program. The CAS self-evaluation process and guidelines made available parameters of a high-quality program, in essence a blueprint for exemplary services.

In 2008-2009, the LRC conducted a self-evaluation using the CAS guidelines and the resulting recommendations were the basis of the present assessment process, including the development and modification of student learning outcomes and the use of a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis and action plan. The LRC is currently progressing through a self-assessment in preparation for applying for certification of the tutoring laboratories from the National Association for Developmental Education.

In the LRC, qualitative and quantitative methods that included information from all stakeholders were used to evaluate and improve the program and to enhance services. The LRC examined quantitative data from the online tutoring programs and, additionally, reviewed qualitative information from self-report surveys completed by students and professors. After the analysis of the data, changes were implemented to the tutoring laboratories: review topics were dropped or added, handouts were developed, and additional practice on certain topics was included in the laboratories.

In addition, the establishment of student learning outcomes was recommended as a yardstick for measuring the effectiveness of the program. The online tutoring programs provided diagnostic pretests and posttests on which measureable student learning outcomes were based. Each semester, data on student learning outcomes were reviewed and the student learning outcomes revised.

Moreover, accreditation from professional organizations such as the Middle States Commission on Higher Education (MSCHE) required direct evidence of student learning, including improvement between a pretest and a posttest, in order to meet official standards (MSCHE, 2009; Suskie, 2009). The online assessments conducted in the LRC fulfilled the requirements of the CAS's self-assessment and the standards of the MSCHE.

Student Learning Outcomes After completing a self-assessment based on the CAS's standards and guidelines, the LRC staff established student learning outcomes. These outcomes were then modified each semester after a review and analysis of data from a pretest and posttest, and after an analysis of responses from an end-of-semester student survey and a professor survey. A graphic organizer, in the form of a table, was created to facilitate the development of the student learning outcomes.

The student learning outcomes were based primarily on an analysis of the results of the comparison of the pretest and the posttest that shows the increase, or lack thereof, of the skills of the participating students. Each semester the student learning outcomes were revised after determining the percent improvement from pretest to posttest. The lab team then created an action plan consisting of the strategies to be used in the following semester to assist students in increasing their skills. Table 1 illustrates the fall 2010 student learning outcomes for the LRC Writing Lab.

SWOT Analysis and Action Plan

In addition to student learning outcomes, the staff conducted a SWOT analysis for each program and developed an action plan based on published evidence of best practices. This assessment helped the lab team develop prescriptions for improving services and increasing the effectiveness of those services. For a comprehensive review of the use of a SWOT analysis and action plan in a learning assistance center, see Fullmer (2009).

Logic Models

Logic models--depictions of how a program or organization works--facilitate reflection, planning, and communication among an organization's employees and stakeholders, leading to a high-quality evaluation of the program. The staff of the LRC developed a logic model for the LRC using an outcomes approach, developing the short-, medium-, and long-term outcomes and impact first, and then proceeded to complete the model working backwards. After the outcomes and impact were developed, the outputs (activities and participants) were listed, and then the inputs (resources) were determined. In addition, each tutoring laboratory team developed a logic model for their lab. Professional tutors developed a logic model for their respective labs in order to discover any gaps in services and to gain a comprehensive understanding of the program for planning purposes. This information was used to complete a self-evaluation, analyze services, and generate strategies for improvements in the effectiveness of services. In this manner, significant informed decisions were made concerning goals, objectives, and student learning outcomes.

The creation of logic models for the LRC and the tutoring labs revealed gaps in services and programs, and focused attention on the relationship between actions and results. With this information, funds were able to be efficiently allocated and resources effectively apportioned. The LRC professional tutors reported that the realization of the long-term outcomes and impact led to increased self-esteem and an awareness of the significance of their contribution to the university. In addition, the development of the logic models built a shared understanding of the LRC and supported cohesiveness of the staff through the engagement of all in the design and formation of LRC's logic model. The LRC logic model is depicted in Figure 1.

Self-Report Surveys

The professional tutors created end-of-semester surveys for both the students and the professors teaching the developmental courses. After a review of the surveys completed by the students and the professors, the resulting information was used in the assessment of the tutoring laboratories. A comprehensive report, completed after each semester, provided an opportunity to conduct a self-evaluation and develop ways to improve services for students and the Lincoln community. The LRC used multiple sources of data and a variety of methods of collection, including direct sources of evidence, such as a pretest and posttest, in order to develop accurate results. Quantitative and qualitative data were collected through the self-report surveys, pretest and posttest scores, and course grades. The semester reports can be viewed on the LRC website at http://www.lincoln.edu/lrc/index.html.

Effectiveness of Tutoring

Students who used the tutoring laboratories and online tutoring programs were given an opportunity to spend further time on task and practice their skills. Brophy (1988) found that additional time on task relates to increased learning if the activities are successfully implemented and effectively designed, and if the instruction is proficient. Increased practice results in increased knowledge, critical thinking skills, and student satisfaction and confidence (Cant & Cooper, 2010). Additionally, the tutoring laboratories provided students with a short mini lesson to review the material presented in the classroom and to provide an opportunity for practice at the end of the mini lesson in a brief activity and in the online tutoring program.

Rheinheimer, Grace-Odeleye, Francois, and Kusorgbor (2010) found that tutoring significantly improved academic performance, including increased persistence, increased retention, and increased graduation rates. Bloom (1984) found that the most effective method of instruction was one-on-one (or small groups of two or three) tutoring using mastery learning. Bloom's study defined mastery learning as including formative testing, feedback, and corrective procedures and found that higher order thinking skills, such as analysis, evaluation, and synthesis, significantly increased. Additionally, Bloom (1984) found that at the beginning of a course, if intense individualized review and relearning took place, students' learning increased compared to instruction without intensive review and relearning. Using enhanced cues and explanations in instruction was also found to increase students' learning.

Bonham and Boylan (2011) reviewed and identified initiatives that were related to improved success rates in developmental mathematics. They found that utilizing all the following strategies were linked to improved student success: mastery learning, self-paced learning, mentoring, active learning, learning strategies and study skills instruction specifically in math, supplemental instruction, contextual learning, problem solving, modeling, incorporating coursework with labs and learning assistance centers, use of online tutoring programs such as ALEKS[R], computerized tutorials, individual tutoring in math, small-group instruction, reviews for tests, classes held on consecutive days, training and professional development for professors and tutors, and assistance with math anxiety and attitudes toward math.

The tutoring laboratories made available to students opportunities to review, relearn material, and receive further explanations of concepts. This study found that students' academic achievement increased during the time period from pretest to posttest. Academic achievement is related to the persistence and retention of students in higher education. Hodges and White (2001) found that tutoring is a contributing factor to the academic success of students. The goal of the LRC is to support the persistence and retention of students in their pursuit of a college degree by helping students improve their academic skills, which can result in higher GPAs as well as increased persistence and retention.

Online tutorial programs can be effective tools in assisting students academically. Online tutoring programs give immediate feedback, and immediate feedback has been linked to "a positive effect on both metacognitive and cognitive gains [in the use of an intelligent tutoring system]" (Saadawi et al., 2010, p. 10). The online tutoring programs also generate pretest and posttest assessments and progress reports, keys to formative assessment and the summative evaluation of program effectiveness. Computerizing the scoring of the pretest and posttest frees up the tutors' time so that more individual interaction with students can take place and allows tutors to assist more students during a lab session. Such technology engages today's students, provides hands-on activities, and fits in with students' perceptions of up-to-date higher education. All of the above can support students in their progress toward a college degree.

The training of tutors has been shown to correlate with an increase in students' academic skills. Boylan, Bliss, and Bonham (1997) found that tutoring programs in postsecondary education that included the training of tutors were significantly related to higher first term GPA, higher cumulative GPA, and the retention of students in developmental courses. In consideration of that evidence, the LRC participates in the International Tutoring Program Certification (ITPC) of the College Reading and Learning Association and has earned certification at all three levels of the ITPC. The increase in the skills of tutors through training has contributed to the high caliber of service to students.

Purpose of the Study

This study was part of a self-assessment process of the LRC to ascertain the effectiveness of the required tutoring laboratories through determining whether the students who completed both the pretest and posttest, usually the students who spent six or more sessions in the laboratories, significantly increased their skills in reading, writing, and mathematics. Students who completed both the pretest and posttest usually were the students who spent six or more sessions in the lab because each laboratory spends up to five sessions conducting an orientation, a pretest, and a posttest.

Method

Students who were required to participate in reading, writing, and/ or mathematics tutoring laboratories as part of their developmental coursework completed a pretest and posttest in an online tutoring program: My Reading Lab, My Writing Lab, and/or ALEKS[R] (mathematics). Students were placed into the developmental courses according to their math and verbal reasoning SAT scores, as well as a locally administered reading comprehension assessment. Some students attended more than one lab because they placed into more than one developmental course. At this point in time, the respective academic departments are reviewing the placement process, and the Mathematics Department piloted a hand-scored departmental placement test in 2011-2012.

Participants

The participants in the study were first-year students who had been placed in developmental courses because they did not meet the admission requirements of the university. Sixty percent of the students were female and 40% were male. One hundred percent (408) of the students were African American and in their first year of college. One hundred and eighty-five students attended the Reading Lab in fall 2010, 313 students attended the Writing Lab, and 155 attended the Math Lab. Students attended the tutoring laboratories in addition to their developmental courses: Education 097 (Reading and Study Skills), Education 098 (Critical Reading Skills), English 098 (Basic Writing Skills I), English 099 (Basic Writing Skills II), MAT 098 (Algebra I), and MAT 099 (Algebra and Applications).

Students in this study entered the university with differing levels of motivation, college preparedness, and perceptions and expectations of college life. The professional tutors of the LRC had been trained to use strategies to motivate students to devote time and effort to their studies in order to increase the students' persistence in their postsecondary education. Students' motivation may have been a key factor in their increase in academic skills, with students of higher motivation attending more labs and classes and putting more time and effort into their coursework. Thus, higher motivated students may have gained greater benefits from attending class and the labs, spending more time studying, and successfully completing more assignments.

To increase student motivation, the tutors greeted the students by name, made the students feel comfortable in the lab, and exhibited a positive attitude. Tutors were trained to incorporate positive and encouraging verbal motivators and practice active listening skills, such as questioning, reflection, and summarization. Students' feelings were acknowledged and the importance of social relationships, including the relationship between the tutor and the student, were recognized. Visual and hands-on activities were planned, and colorful pictures of lab students and posters, including posters identifying star students, graced the walls of the lab.

The qualifications of the professional tutors included bachelor's degrees and master's degrees. All tutors have earned level one certification by the College Reading and Learning Association's International Tutoring Program Certification Program, and 50% have earned all three levels of certification.

Measurements

My Writing Lab included a diagnostic pretest and posttest, exercises, and an online grade book in which to view students' progress. Students attended the Writing Lab for one 50-minute session per week. My Writing Lab, which was customized in collaboration with the English Department, focused on grammar. The mini review lessons presented by the professional tutors also focused on grammar at the request of the English Department. The professional tutors encouraged and guided students through the online tutoring program during the lab session, answering questions and providing additional examples. The difference in scores between the pretest and the posttest was examined to determine the percent of improvement.

Professional reading tutors conducted a mini review lesson on the topic of the week at the beginning of each lab session. Students attended one 50-minute Reading Lab session per week. The topics of the week in the Reading Lab were aligned with the course syllabi. The students then read and worked in the online reading tutoring program, My Reading Lab, which included reading selections with comprehension questions. The professional reading tutors facilitated the students' reading in the online tutoring program, providing further explanations of concepts and supporting students in their reading. The reading selection component provided an assessment of each student's reading level by means of Lexiles[R], which can be converted into reading grade levels such as the Flesch Reading Grade Level (My Reading Lab, 2011b). In this study, the Flesch Reading Grade Levels were examined at the beginning of the semester and at the end of the semester to determine the improvement in reading grade level.

Professional math tutors presented a mini review lesson, aligned with the math course syllabi, at the beginning of each lab. Students attended two 50-minute Math Lab sessions per week. After the mini review lessons, students worked at their own pace in the online tutoring program, ALEKS[R]. (Due to the nature of the online program, with students working at their own pace, students' work in the online program may not be congruent with the topic being covered in the classroom.)

Professional math tutors assisted students in their progress through the online program, providing encouragement, detailed examples, and additional explanations. A diagnostic pretest, conducted in the first two lab sessions, determined the beginning point for each student, and the program then presented mathematical problems for each student at the level for which he or she was ready to learn. A comprehensive posttest assessment, similar to the diagnostic pretest, was conducted during the last two sessions of the laboratory. The differences in the scores between the pretest and the posttest were examined in this study.

Procedure

At the beginning of the semester, students completed an online diagnostic pretest in their respective online tutoring programs, and at the end of the semester the students completed an online posttest. In the Writing Lab and the Math Lab these tests consisted of appropriate questions based on the material covered. In the Reading Lab, the pretest and posttest consisted of a reading selection and comprehension questions.

Math Lab

Students in a math-intensive major such as in the sciences, psychology, or business were registered in one of two developmental courses that required a math tutoring laboratory: MAT 098 (Algebra I) and MAT 099 (Algebra and Applications). Students were placed according to their math SAT scores (less than 400 for MAT 098, and 400-450 for MAT 099). In the fall of 2010, 38 students in MAT 098 and 65 students in MAT 099 completed both the pretest and posttest and attended at least six sessions. Math Lab students were required to attend two lab sessions per week for a total of 25 sessions. At the discretion of the professor, Math Lab attendance counted for about 10% of a student's grade. A Math Lab session consisted of a 15- to 20-minute mini review lesson on topics in the course syllabi and 30 to 35 minutes of practice in the online math tutoring program, ALEKS[R]. Students progressed in the online tutoring program at their own pace. Students completed a comprehensive assessment (pretest) in the first two lab sessions and the last two lab sessions (posttest).

Reading Lab

The Education Department administered a reading comprehension assessment, created by a professor, and the students' scores determined their placement in one of two levels of developmental courses, EDU 097 (Reading and Study Skills) and EDU 098 (Critical Reading Skills). Each course level of developmental reading required students to attend the Reading Lab. In fall semester, 2010, there were 45 students in EDU 097 and 79 students in EDU 098 who completed both the pretest and posttest and attended at least six lab sessions. Students in both courses were required to attend 11 weekly lab sessions (including an orientation session), and attendance counted as 10% of their grade. Each laboratory session was 50 minutes long and consisted of approximately a 15-minute mini review lesson and 35 minutes in the online tutoring program, My Reading Lab. The review mini lessons focused on topics covered in the course syllabi. At the completion of each reading selection and comprehension questions, the online program determined each student's Lexile[R] reading level. The Lexile[R] reading level may be translated into a reading grade level using a conversion chart provided by My Reading Lab.

Writing Lab

Students in the Writing Lab were registered in one of two developmental courses, ENG 098 (Basic Writing I) and ENG 099 (Basic Writing II), depending on their verbal SAT scores (200-380 for ENG 098, and 390-470 for ENG 099). In fall semester, 2010, there were 30 ENG 098 students and 70 ENG 099 students who completed both the pretest and posttest and attended at least six sessions. Students in both courses were required to attend 12 weekly sessions, including one orientation session, and attendance counted as 10% of their course grade. A laboratory session was 50 minutes in length and consisted of an approximately 15-minute mini review lesson presented by professional tutors and 35 minutes of practice in the online tutoring program, My Writing Lab. The mini review lessons and online practice covered various grammar topics based on the syllabi of the courses. Students completed the pretest in the first two lab sessions and the posttest in the last two lab sessions.

Data Analysis and Results

t test of Dependent Samples and [Eta.sup.2]

The paired sample t test examines the differences in means between groups that are related, and tests the effect of a treatment over time (Kiess, 2002, p. 211-215; Ravid, 2000, p. 190). The t test examines the difference between the pretest and posttest to determine if the difference is statistically significant or due to random variation. Although the t test determines if the differences between two variables are significant, [eta.sup.2] measures the strength of the relationship between the two variables (Ferguson, 2009). The difference between the pretest and the posttest may be statistically significant and not due to random variation (t test), but may be of a varying effect size ([eta.sup.2]). The statistical program SSPS was used to determine [eta.sup.2], and there has been some discussion that SPSS actually determines partial [eta.sup.2] rather than [eta.sup.2]. Partial [eta.sup.2] ([etap.sup.2]) may overestimate the effect size (Levine & Hullett, 2002; Olejnik & Algina, 2003).

Ferguson (2009) suggests that a moderate effect size for squared associations, such as [eta.sup.2], is .25, and a strong effect size is .64. In this study, the effect size ranged from approximately 0.60 to 0.89, indicating that about 60% to 89% of the differences between variables could be attributed to the treatment. Although a result may be statistically significant, it may or may not be practically significant, and practical significance must be determined in the context of the study (Ferguson, 2009; Gelman & Stern, 2006). In the context of this study, the results were both statistically and practically significant, as well as moderate to strong in effect size, because the goal of the treatment was to increase students' academic skills in order to positively influence the persistence and retention of the students in their pursuit of a college degree.

The difference between the pretest and posttest scores for students who attended six or more laboratory sessions was examined. Data were gathered from the entire population of students who attended six or more tutoring laboratory sessions, eliminating any sampling error. Scores from a pretest and posttest for students in the Writing Lab and Math Lab as well as the initial and highest reading grade levels for students in the Reading Lab were reviewed.

Table 2 indicates that there was a high level of statistical significance for the differences in the pretest and posttest, and a moderate to strong effect. For the time interval of the fall 2010 semester, those students who attended at least six tutoring lab sessions and took both the pretest and posttest improved significantly in skills. Both levels of developmental math, Math 098 and Math 099, showed a strong effect size; both levels of developmental reading, EDU 097 and EDU 098, showed a moderate effect size; and both levels of developmental English, ENG 098 and ENG 099, showed a strong effect size.

The t test cannot determine the cause of any significant difference, and several factors may have influenced the students' performance between the pretest and posttest. The factors that may have influenced students' gains in academic achievement include students' work and effort, the professors' facilitation of the course, the tutors' facilitation of the lab, and time on task in the tutoring laboratories. All of the above factors may have contributed to a significant increase in the students' academic skills.

Limitations

This study was confined to the fall 2010 semester at a Historically Black College or University (HBCU) where the majority of the students are African American, most from urban areas on the East Coast and nearly all receiving financial aid to attend college. This study needs to be repeated over time, in differing semesters, and in other milieus in order to assess whether or not the findings of this study can be generalized to other situations.

This study included several treatments over the time interval: students' commitment to completing the work and studying, professors' teaching and management of the course, and time on task and practice, including the tutoring laboratories. It may be that the above factors created a synergistic effect that is more powerful than each individual treatment. Future studies with an experimental design may be able to break down the treatments into components and determine the contribution of each.

Future Research

Many opportunities exist for further research in this area. A study with an experimental design may be able to determine causation and separate out the contribution of the individual components of the treatment factors examined in this study. Studies that can evaluate the effectiveness of the separate factors within the treatment time interval (student motivation and effort, professors' teaching and management of the course, and tutoring laboratories) may be able to determine the contribution of each and to discover whether there is a synergistic effect.

This study involved a particular student population and context. However, future studies could involve different student populations, large and small colleges and universities, and urban and rural milieus. Time of year may be a factor in the motivation of students, specifically if there are distracting factors at a particular time of year. Students' motivation may be greater in the first semester of a school year versus the second semester. The processes examined in this study could be applied to different types of tutoring situations--such as required and not required, one-on-one and group, small group and large group, online versus not online, etc.--to evaluate the methods used to determine the effectiveness of these types of tutoring.

Further research in assessment of tutoring programs is needed to serve as a guide for making informed decisions concerning the most effective programs and strategies, and the allocation of funds and resources. Effective programs increase students' skills, which are linked to students' increased persistence and retention rates that are correlated to the graduation rates of colleges and universities (Rheinheimer et al., 2010).

References

ALEKS[R] (2011). Research behind ALEKS[R]. Retrieved from: http://www.aleks.com/about_ aleks/research_behind.html

Bloom, B. (1984). The 2-Sigma problem: The search for methods of group instruction as effective as one-on-one tutoring. Educational Researcher, 13(6), 4-16.

Bonham, B.S., & Boylan, H.R. (2011). Developmental mathematics: Challenges, promising practices, and recent initiatives. Journal of Developmental Education, 34(3), 2-10.

Boylan, H., Bliss, L., & Bonham, B. (1997). Program components and their relationship to student performance. Journal of Developmental Education, 20(3), 2-8.

Brophy, L. (1988). Educating teachers about managing classrooms and students. Teaching and Teacher Education, 4(1), 1-18.

Cant, R. R, & Cooper, S.J. (2010). Simulation-based learning in nursing education: Systematic review. Journal of Advanced Nursing, 66(1), 3-15.

Council for the Advancement of Standards in Higher Education (2010). CAS Professional Standards for Higher Education (7th ed.). Washington, DC: The Council for the Advancement of Standards in Higher Education.

Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532-538.

Fullmer, R (2009). The assessment of a tutoring program to meet CAS standards using a SWOT analysis and action plan. Journal of College Reading and Learning, 40(1), 51-76.

Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60(4), 328-331.

Gillen, C. M., Lye, B. R., & Vaughn, J. (2004). An online tutorial for helping non-science majors read primary research literature in biology. Advances in Physiology Education, 28(4), 95-99.

Hodges, R., & White, W. (2001). Encouraging high-risk student participation in tutoring and Supplemental Instruction. Journal of Developmental Education, 24(3), 2-11.

Kiess, H. O. (2002). Statistical concepts for the behavioral sciences (3rd ed.). Allyn & Bacon: Boston, MA.

Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28(4), 612-625.

Middle States Commission on Higher Education. (2009). Characteristics of excellence in higher education: Requirements of affiliation and standards for accreditation. Retrieved from http://www.msche.org/publications/CHX06_Aug08REVMarch09.pdf

My Reading Lab (2011a). Retrieved from http://www.myreadinglab.com

My Reading Lab (2011b). The Lexile framework for reading. Retrieved from http://www. myreadinglab.com/lexile.html

My Writing Lab (2011). Retrieved from: http://www.mywritinglab.com

Olejnik, S., & Algina, J. (2003). Genrealized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8(4), 434-447.

Ravid, R. (2000). Practical statistics for educators (2nd ed.). Lanham, MD: University Press of America.

Rheinheimer, D. C., Grace-Odeleye, B., Francois, G. E., & Kusorgbor, C. (2010). Tutoring: A support strategy for at-risk students. Learning Assistance Review, 15(1), 23-34.

Saadawi, G., Azevedo, R., Castine, M., Payme, V., Medvedeva, O., Tseytlin, E., Legowski, E., Jukic, D., & Crowley, R. (2010). Factors affecting feeling-of-knowing in a medical intelligent tutoring system: The role of immediate feedback as a metacognitive scaffold. Advances in Health Science Education, 15, 9-30.

Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco, CA: Jossey Bass.

University of Wisconsin-Extension. (2003). Enhancing program performance with logic models. Retrieved from http://www.uwex.edu/ces/lmcourse

Patricia Fullmer is Director of the Learning Resource Center at Lincoln University, Pennsylvania. She received her doctorate degree from Wilmington University, and her master's and bachelor's degrees from Temple University.
Table 1
Writing Lab Student Learning Outcomes

Student Learning
Outcomes                Assessment Method       Findings

Students will           Data from the           For students who
increase their skills   diagnostic pretest      attended six or more
in the area of          and posttest of My      sessions:
sentences.              Writing Lab.
                                                English 098 students
English 098 students                            showed a 54%
will show a 50%                                 improvement in
improvement from the                            sentence grammar.
pretest to the
posttest in sentence                            English 099 students
grammar for those who                           showed a 48%
attend six or more                              improvement in
sessions.                                       sentence grammar.

English 099 students                            The significant
will show a 25%                                 improvement of the
improvement from the                            students in the area
pretest to posttest                             of sentence grammar
in sentence grammar                             results from the
for those who attend                            following reasons:
six or more
sessions.                                       The sentence grammar
                                                portion of the test
                                                covers eight
                                                different areas of
                                                sentence grammar and
                                                is the largest part
                                                of the test.

                                                Tutors aggressively
                                                addressed the
                                                deficient areas early
                                                in the semester.

                                                This semester's
                                                student population
                                                includes more
                                                motivated learners.

Students will           Data from the           For students who
increase their skills   diagnostic pretest      attended six or more
in the area of usage.   and posttest of My      sessions:
                        Writing Lab.
English 098 students                            English 098 students
will show a 45%                                 showed a 21%
improvement from the                            improvement in usage
pretest to the                                  and style.
posttest in usage and
style for those who                             English 099 students
attend six or more                              showed an 18%
sessions.                                       improvement in usage
                                                and style.
English 099 students
will show a 30%                                 Although many
improvement from the                            students did improve
pretest to the                                  in usage, our
posttest in usage and                           benchmark was not
style for those who                             met; however, this
attend six or more                              part of the test is
sessions.                                       the smallest part
                                                with only 15
                                                questions which
                                                address only two
                                                areas--parallelism
                                                (which was taught
                                                in lab) and easily
                                                confused words
                                                (which was not
                                                taught in lab).

Students will           Data from the           For students who
increase their skills   diagnostic pretest      attended six or
in the area of          and posttest of My      more sessions:
punctuation and         Writing Lab.
mechanics.                                      English 098 students
                                                show a 6% improvement
English 098 students                            in punctuation and
will show a 15%                                 mechanics.
improvement from the
pretest to the                                  English 099 students
posttest in                                     show a 7% improvement
punctuation and                                 in punctuation and
mechanics for those                             mechanics.
who attend six or
more sessions.

English 099 students
who attend six or
more sessions will
show a 15%
improvement.

Students will           Data from the           For students who
increase their skills   diagnostic pretest      attended six or more
in the area of          and posttest of My      sessions:
grammar.                Writing Lab.
                                                English 098 students
English 098 students                            show a 16%
will show a 15%                                 improvement in basic
improvement from the                            grammar.
pretest to the
posttest in basic                               English 099 students
grammar for those who                           show a 21%
attend six or more                              improvement in basic
sessions.                                       grammar.

English 099 students
will show a 20%
improvement from the
pretest to the
posttest in basic
grammar for those who
attend six or more
sessions.

Student Learning        Modified Student
Outcomes                Learning Outcomes       Action Plan

Students will           Students will           Students will improve
increase their skills   increase their skills   sentence grammar by:
in the area of          in the area of
sentences.              sentences.              Working on individual
                                                study plans in My
English 098 students    English 098 students    Writing Lab.
will show a 50%         will show a 50%
improvement from the    improvement from the    Writing sentences
pretest to the          pretest to the          and paragraphs.
posttest in sentence    posttest in sentence
grammar for those who   grammar for those who   Using authentic
attend six or more      attend six or more      student writing from
sessions.               sessions.               class assignments.

English 099 students    English 099 students    Receiving tutor-
will show a 25%         will show a 33%         directed feedback.
improvement from the    improvement from the
pretest to posttest     pretest to the
in sentence grammar     posttest in sentence
for those who attend    grammar for those who
six or more             attend six or more
sessions.               sessions.

Students will           Students will           Students will improve
increase their skills   increase their skills   usage by:
in the area of usage.   in the area of usage
                        and style.              Working on individual
English 098 students                            study plans in My
will show a 45%         English 098 students    Writing Lab.
improvement from the    will show a 30%
pretest to the          improvement  from the   Writing sentences
posttest in usage and   pretest to the          and paragraphs.
style for those who     posttest in usage and
attend six or more      style for those who     Using authentic
sessions.               attend six or more      student writing from
                        sessions.               class assignments.
English 099 students
will show a 30%         English 099 students    Receiving tutor-
improvement from the    will show a 25%         directed feedback.
pretest to the          improvement from the
posttest in usage and   pretest to the          Direct instruction
style for those who     posttest in usage and   in both areas usage
attend six or more      style for those who     from tutor of
sessions.               attend six or more
                        sessions.

Students will           Students who attend     Students will improve
increase their skills   six or more sessions    punctuation and
in the area of          will increase their     mechanics by:
punctuation and         skills in the area of
mechanics.              punctuation and         Working on individual
                        mechanics.              study plans in My
English 098 students                            Writing Lab.
will show a 15%         English 098 students
improvement from the    will show a 10%         Writing sentences and
pretest to the          improvement             paragraphs.
posttest in             from the pretest to
punctuation and         the posttest.           Using authentic
mechanics for those                             student writing
who attend six or       English 099 students    from their classes.
more sessions.          will show a 10%
                        improvement from the    Receiving tutor-
English 099 students    pretest to the          directed feedback.
who attend six or       posttest.
more sessions will                              Direct instruction on
show a 15%                                      punctuation and
improvement.                                    mechanics from tutors.

Students will           Students will           Students will improve
increase their skills   increase their skills   basic grammar by:
in the area of          in the area of basic
grammar.                grammar.                Working on individual
                                                study plans in My
English 098 students    English 098 students    Writing Lab.
will show a 15%         show a 20%
improvement from the    improvement in basic    Writing sentences and
pretest to the          grammar.                paragraphs.
posttest in basic
grammar for those who   English 099 students    Using authentic
attend six or more      show a 25%              student writing from
sessions.               improvement in basic    class assignments.
                        grammar.
English 099 students                            Receiving tutor-
will show a 20%                                 directed feedback.
improvement from the
pretest to the                                  Direct instruction
posttest in basic                               from tutors on verb
grammar for those who                           tense and use of
attend six or more                              present tense when
sessions.                                       writing essays.

Table 2
Paired Samples t test and Measure of Effect, Eta (2)

             Number of   Average Pre-   Average Post-   Degrees of
Lab/Course   Students     test Score     test Score     Freedom df

Math 098        38          17.63           57.27           37

Math 099        65          17.43           30.59           64

Reading         45           7.40           8.78            44
(EDU 097)

Reading         79           7.38           8.97            78
(EDU 098)

Writing         30          64.30           77.67           29
(ENG 098)

Writing         70          67.76           81.30           69
(ENG 099)

             Statistical
              Inference                Measure of
Lab/Course     (alpha)     t-value   Effect Eta (2)

Math 098        p<.01      -16.45         0.88

Math 099        P<.01      -12.23         0.70

Reading         P<.01       -8.25         0.61
(EDU 097)

Reading         P<.01      -11.62         0.63
(EDU 098)

Writing         P<.01      -14.41         0.88
(ENG 098)

Writing         P<.01      -20.86         0.86
(ENG 099)

Figure 1
Learning Resource Center Logic Model Adapted from University
of Wisconsin-Extension (2003)

Inputs

What we invest:

University funds
Title III funds
Act 101 funds
Staff: Tutors, Peer Tutors, Secretary, Administrators
Training for staff
Online programs and access codes
Mini lesson plans and exercises Computers
Rooms and offices
Collaboration with academic departments and administrators

Outputs

Activities                               Participation

What we do:                              Who we reach:

Tutoring laboratories                    Students
Online tutoring programs                 Students' families
Review mini lessons                      Faculty
Individual math tutoring                 Administrators
Individual writing tutoring              Alumni
Writing portfolio assistance
Revision and proof-reading assistance
Workshops
Summer bridge labs and tutoring

Assumptions:

Following best practices in our services will help students
increase their skills.

Increased student skills will improve retention and graduation
rates.

Increased graduation rates will help attract students and
increase Lincoln's reputation.

Outcomes--Impact

        Short                  Medium                   Long

This semester's         Next semester/next      Results in 4/6 years
results:                year's results:         and more:

Students increase       Students increase       Students successfully
their skills to         their GPA               graduate Lincoln with
college level                                   skills for graduate
                        Students persist and    school or the fast
Students earn a B or    return the next         track in a
above in their          semester/next year      corporation
course
                        Students become         Lincoln's graduation
Students increase       successful              rate increases
life skills and         academically,
coping skills           socially, and           Lincoln's reputation
                        emotionally at          increases
Students adjust to      Lincoln
college life                                    More and better
                        Lincoln's               prepared students are
                        persistence/            attracted to. Lincoln
                        retention rate
                        increases               Growth of Lincoln

External Factors:

Location

Economy

State and Federal funding

Perception of LRC by Faculty, Administrations, and Students
Gale Copyright:
Copyright 2012 Gale, Cengage Learning. All rights reserved.