A Mixed-Methods Assessment of Using an Online Commercial Tutoring System to Teach Introductory Statistics

Yonghong Jade Xu
Katrina A. Meyer
Dianne D. Morgan
The University of Memphis

Journal of Statistics Education Volume 17, Number 2 (2009), jse.amstat.org/v17n2/xu.html

Copyright © 2009 by Yonghong Jade Xu, Katrina A. Meyer, and Dianne D. Morgan, all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor.

Key Words: ALEKS; Hybrid course; Statistics education; Teaching software packages.

Abstract

This study used a mixed-methods approach to evaluate a hybrid teaching format that incorporated an online tutoring system, ALEKS, to address students’ learning needs in a graduate-level introductory statistics course. Student performance in the hybrid course with ALEKS was found to be no different from that in a course taught in a traditional face-to-face format. Survey and focus group interviews revealed that students’ experience with ALEKS and learning of statistics varied systematically across performance levels. Both quantitative and qualitative data suggest that 1) class format may not be as important as students’ mathematical ability and skills for their success in introductory statistical courses, and 2) a teaching approach that addresses the underlying determinants of learning behaviors would be more effective than simply delivering the material in a different format.

1. Introduction

In the past decade, an increasing number of studies have investigated and discussed how integration of technology in statistical courses influences the performance and attitude of students (e.g., Gal & Ginsburg, 1994; Grabowski & Harkness, 1996; Symanzik & Vukasinovic, 2006; Tudor, 2006; Utts et al., 2003; Ward, 2004). Although the majority of the studies concluded there was no significant difference between student performance in technology-enhanced format and in traditional face-to-face teaching (Russell, 1999; Ward, 2004), researchers and instructors recognize that teaching software packages (or "teachware") have valuable features, including hands-on exercises, real-life examples, self-assessments, and instant feedback (Symanzik & Vukasinovic, 2006).

Integration of technology in statistical courses varies in three ways: the degree of technology use, the type of technology incorporated, and the interaction with the learners. First, in terms of the degree of technology integration, a technology-enhanced class can be either hybrid or completely online without face-to-face contact (Utts et al., 2003). Even in hybrid classes, the use of software could be anywhere from occasional in-class demonstrations to online learning blended with occasional class meetings (Ward, 2004).

Second, the teachware packages that have been experimented with and reported on range from expert systems tailor-made by the instructor for his or her students (e.g., Grabowski & Harkness, 1996), an innovative use of a class management platform (e.g., Zhang, 2002), to web-based statistics courses that offers text, practice problems, and assessments (e.g., Symanzik & Vukasinovic, 2006; Utts et al., 2003). With rapid development in Internet and computing technology, commercial teachware packages are gaining popularity. Symanzik and Vukasinovic (2006) introduced some statistical teachware packages and electronic textbooks available on the Web and compared three popular electronic textbooks: ActivStats, CyberStats, and MM*Stat (Symanzik & Vukasinovic, 2003).

Last, the teachware packages often emphasize different interactive activities with learners. Some packages are designed to encourage learners’ active involvement in the content, while others are tools to assist instructors in classroom teaching (Ferrall, 1995; Grabowski & Harkness, 1996; Symanzik & Vukasinovic, 2006). Some commercial packages (e.g., CyberStats) are virtual classrooms that offer a stand-alone online course with text instruction, practice problems, assessments, and a message board or chat room for students and instructor.

The availability of different teachware packages provides instructors with a variety of options for incorporating technology in teaching statistics. However, very limited information is available on how the features of those packages meet different learning needs. Although the literature is growing with studies that evaluate particular teachware packages and/or discuss briefly challenges students face in hybrid courses (e.g., Symanzik & Vukasinovic, 2006; Tudor, 2006), few studies have discussed the aspect of choosing a computer-assisted or Web-based learning environment that fits the characteristics of the learners. The importance of learner characteristics was discussed by Mayer (1989) in his learning process model, which suggests learner characteristics, along with material to be learned and presentation method, determines a successful learning outcome. Mayer’s model suggests that, for teaching statistics, if an instructor plans to make changes to the presentation method in order to assist and improve student learning, it is important to have a fair understanding of students’ ability, skill, and motivation. Additional studies also found that demographic variables of learners, included gender, age, race, existing knowledge in mathematics, need to be considered when examining student performance in statistics (e.g., Anderson, Benjamin & Fuss, 1994; Tay, 1994).

This study is motivated by the desire of a statistics professor to test a hybrid approach to teaching graduate-level introductory statistics and how a teachware package, called Assessment and LEarning in Knowledge Spaces or ALEKS (http://www.aleks.com), meets the specific characteristics of the students in the course. In order to understand and evaluate the dynamics of the hybrid course, a mixed-methods approach was used to collect quantitative and in-depth qualitative data on student performance, students’ perception of the class format, and their attitude towards statistics. This study is different from other studies in two perspectives. First, the course is offered to graduate students in a college of education. Graduate students may be different from undergraduate students in terms of their learning preferences. For example, there is evidence that graduate students prefer to study in their own time when they are free of work and family responsibilities (MacGillivray, 2003). Second, students’ perceptions of the class format and the teachware package were evaluated at different performance levels. The findings show that performance level captures some specific characteristics of the learners and shed light on the importance of selecting a teachware package that meets the needs of students.

2. Course Description

In the College of Education of a mid-south urban university, all graduate students are required to take Statistical Methods Applied to Education I, known as Stats I, that covers the introductory statistical topics, including descriptive statistics, Pearson correlation, simple linear regression, t tests, other one-sample and two-sample tests, and X² test. The course is designed for students to understand the basic principles of quantitative research methods and statistical data analysis in an educational setting and also to be prepared for advanced topics in statistics. In a regular 14-week semester, two or three sections of Stats I are usually offered, with the class size in each section limited to 25 students. The classes are scheduled to meet once every week for 3 hours in the evening or weekend to accommodate the high percentage of part-time students.

Students enrolled in Stats I often have a diverse background academically, professionally, and demographically. They are pursuing either a master’s or doctoral degree; their major could be in one of the college’s 15 degree programs that include teaching, instruction and curriculum, instructional design and technology, special education, counseling psychology, educational psychology, higher and adult education,  human movement science, and educational research. The 2005-06 enrollment data show that more than half of the graduate students in the College attended part-time and about 40% were 35 years or older. Students also vary widely in terms of their readiness for Stats I and knowledge in mathematics. Some students may have had statistics classes in their undergraduate training not long ago, while a few others go back decades for their most recent experience in mathematics or statistics. Many students report anxiety when working with the basic mathematical symbols and functions used in statistics.

3. Teaching Stats I with ALEKS

Having taught Stats I in a traditional face-to-face format since 2004, the instructor of the course – the first author – realized that a couple of major obstacles must be overcome in order to improve student performance. First, students’ substantial difference in terms of their readiness for introductory statistics posed severe challenges to the effective delivery of the class material. For instance, at the beginning of every semester, an initial quiz of student knowledge almost always showed that a few students lacked basic mathematical skills, such as the order of operations or finding a square-root of a number, while a few others understood the basic statistical concepts. With a relatively large class size (about 20-25 students), the pace of instruction had to be adjusted toward the median level of the class and personal attention to individual students was limited. Therefore, students with disadvantages were often frustrated and those with strong preparation bored. And second, a large percentage of students have full-time work commitments and family responsibilities. Limited by their schedule, they are more likely to feel challenged, but less likely to have time to visit the instructor for additional help.

A hybrid class with web-based components may be the answer to more individualized assistance and providing students with flexible learning. Thus, the instructor redesigned the course in Fall 2006 by adding a Web-based tutoring system to traditional classroom teaching and making attendance to some class meetings optional. Several different online systems were evaluated and a teachware called ALEKS was the final choice.

3.1 ALEKS: The Course Tool

ALEKS is a web-based learning environment similar to CyberStats discussed by Symanzik and Vukasinovic (2006) with regards to its design and purposes in statistical teaching. Rather than serving as an electronic textbook, however, ALEKS is designed as a tutoring system supported by artificial intelligence. The operational foundation of ALEKS is Knowledge Space Theory (KST), a framework that provides a particular way to map the elements of knowledge (e.g., concepts in statistics) into distinct knowledge states for individuals. ALEKS teaches a student on selected topics with textbook explanations, examples, and practice questions, and assesses the student’s learning individually and continuously. The assessments adapt to the knowledge structure of the student and can quickly yield an analysis of the knowledge state of the individual by asking 15 to 25 questions. According to its website, ALEKS has been used by over 250,000 students (e.g., students in K-12 schools and higher education, and private users) worldwide in their learning of mathematics and statistics and significant learning gains have been observed (Hagerty & Smith, 2005; Stillson & Alsup, 2003).

ALEKS seems to have several advantages that meet the needs of the Stats I instructor. First, it runs in a Web browser with a free plug-in that requires only 32 MB RAM, and the cost is only $30 for an 18-week student account license. The affordable price and minimal system requirement means it is cost-effective and accessible. Second, ALEKS allows the instructor to select topics and customize the online material to match the textbook content. Such flexibility is critical for instructors who choose to use commercial software to meet some existing teaching requirements. Third, ALEKS is designed as an individualized learning assistant that can identify what each individual already knows and what s/he is now ready to learn. Students learn through self-paced practice and can get automated help in ALEKS or email questions to the instructor. Last, the Web-based system is accessible to students 24 x 7. Its availability to students who have different schedules is desirable. The decision to use ALEKS is also supported by a professor in the Department of Psychology at the same University who had used the system to teach Behavioral Statistics to over 230 undergraduate students (see Hu et al., 2007).

3.2 Class Structure with ALEKS

In Fall 2006, the instructor redesigned Stats I into a hybrid class by integrating ALEKS into the syllabus. Students were required to buy the textbook, Applied Statistics for the Behavioral Sciences (5^th Ed.) by Hinkle, Wiersma, and Jurs (2003), but the ALEKS licenses were purchased with the support of a University fund at no additional cost to students. To begin a semester, the instructor needed to initiate a course in ALEKS first. Using about three workdays, the instructor learned to manage and maneuver the ALEKS system, selected 95 topics (including 19 on arithmetic and algebraic readiness) from the 142 system-default concepts, and organized them into 11 weekly objectives (no objectives were scheduled for the three weeks during which in-class exams were scheduled) to match the 14 textbook chapters. Even though the instructor could not add or edit the given topics, she was able to select the topics that were needed, and organize them into periodical objectives for students with flexibility.

The students signed on to ALEKS through a web browser and each kept a username and password to a personal account, in which he or she could access the weekly topics. At first login, ALEKS evaluated a student’s "knowledge state" with an initial assessment. Then each student began learning from the topics ALEKS had determined they were ready for according to the assessment result. For instance, if students were underprepared, they had to start with some or all the arithmetic and algebraic concepts that help them to get ready for learning statistics. If a student knew measures of central tendency but not standard deviation, s/he began with topics on measures of variation. Related chapters in three electronic textbooks were linked to each topic. If the student had trouble, the "tutor" would ask the student to read more on the topic and come back for the practice questions later. The student could log off any time and come back to resume the learning at a time of their choosing. After learning 20 topics (i.e., progress) and/or logging in for 15 hours (i.e., log-in time), the student had to take an assessment before they could continue. The assessment determined whether a student had mastered the learned topics. A topic will come back for the student to re-learn if s/he consistently misses questions on that particular topic. Students will take a final assessment at the end of the semester after learning all the required topics. Throughout the semester, students can track their own progress and time spent in the system.

The instructor can communicate with students using a built-in message system and oversee student accounts to monitor their progress. Rather than assigning homework to be done and submitted on paper, the instructor asked students to complete the weekly objectives in ALEKS and read the corresponding textbook chapters before the scheduled class meeting. Furthermore, the instructor made attendance optional for six of the 11 lecture classes. That is, the instructor was available in class every week; however, except for the five required classes, students could choose not to attend class if they felt confident they had learned the current chapters and ALEKS topics. In the optional classes, the instructor attended to questions and problems from students rather than having regular lectures. All students were required to come to take the in-class exams (two mid-terms and one final; the final was not comprehensive) on the textbook materials, given that learning of topics in ALEKS are regularly assessed online.

Overall, the amount of effort put into preparing and managing the hybrid course by the instructor was comparable to, if not less than, a class taught using more traditional methods. The instructor relied on the information tracked by the ALEKS system to evaluate student learning progress. Rather than spending time grading homework, the instructor was available as additional support via email to students.

4. Objective of the Study

ALEKS was added not only because it was expected to give individualized help to students, but it was hoped that the hybrid class format with optional classes would provide students with more flexibility in their learning and allow the instructor to individualize instruction according to students’ needs. Because it was the first time the instructor taught Stats I in a hybrid format using ALEKS, a study was designed to answer the following questions:

1. Does the integration of ALEKS improve student performance in Stats I?
2. How does the hybrid class with an online commercial tutoring system impact students’ learning and attitude about statistics?

5. Design and Methodology

A mixed-methods approach was taken to collect both quantitative and qualitative data to answer the research questions. Because it was impractical for the instructor to teach different sections of Stats I in the same semester with different methods, the quantitative data were collected from students in Stats I classes in Fall 2005 (face-to-face teaching format) and Fall 2006 (hybrid class).

The sample size was 45 and 41 in Fall 2005 and Fall 2006, respectively. Samples taken from the two semesters were comparable because they were taught by the same instructor and tested with the same exams. In addition, students’ age, gender, race, and GRE-quantitative score were obtained from the Office of Institutional Research after this study was approved by the Institutional Review Board. Thus, the quantitative information includes student age, gender, race, GRE-quantitative score, and their total exam score in Stats I.

In addition, qualitative data were collected by a qualitative researcher from the Fall 2006 hybrid class with a paper-based survey and three focus group interviews at the end of the semester. The data were shielded from the instructor until final course grades were assigned to students. Twenty-eight out of 41 students (response rate = 68%) completed the anonymous survey that included questions about their background, 15 Likert-scale questions and three open-ended questions about their experience with ALEKS, the instructor, and the class format. Answers to the Likert-scale questions were coded as: 1 – strongly disagree, 2 – disagree, 3 – neutral, 4 – agree, and 5 – strongly agree, and negatively worded questions were coded in reverse before the analysis. Last, three focus group interviews were conducted and videotaped: one group had two low-performance students, a second group had two medium-performing students, and the last group had three high-performing students. The semi-structured interview questions were derived mainly from issues raised from the survey to probe further into students’ experience with using ALEKS, how the ALEKS system contributed to their learning, how the hybrid class facilitated learning statistics, how ALEKS influenced their thoughts, feelings, actions, and abilities as a learner and regarding leaning statistics. Transcripts from the interviews were analyzed using standard qualitative methods (Maxwell, 1996) for development of themes and consistency or inconsistency in themes across individuals and groups.

6. Data Analysis

6.1 Student Performance

Analysis of the exam scores and related quantitative data answers the first research question: Does the integration of ALEKS improve student performance in Stats I?  The descriptive information of the two classes is provided in Table 1 and Table 2.

Table 1. GRE-Quantitative information (Median) of students in Stats I

Table 2. Descriptive information of exam scores of students in Stats I

The total exam score is the sum of the three exams (two midterms and one final; they coved different segments of the class content and the scores were equally weighted). On a scale from 0 to 160, the difference between the average scores of the hybrid class (Fall 2006) and the traditional class (Fall 2005) is nonsignificant in a t-test (t = 1.30, p =.20, df = 84). This result is confirmed by an ANCOVA test in which GRE-quantitative score and age are the covariates, and class type (hybrid vs. traditional) and race are the independent variables. Age is included as an indirect indicator of how long ago was a student’s previous exposure to statistics/mathematics given that older students often cite how old their math training is and their years without practicing mathematics as reasons for experiencing difficulty in the class. Race has been identified as a relevant factor to academic performance (Tay, 1994), but due to the small number of Asian students, only Black and White students could be included (a t-test showed that there was significant difference in the total exam score between the two groups at t = -4.73, p <.01, df = 76).

The underlying assumptions of ANCOVA were met in the analysis. The homogeneity of regression was confirmed by the nonsignificant interactions between the independent variables and the two covariates. Linearity is also apparent between the dependent variable (i.e., the total exam score) and the covariate GRE score. A scatterplot showed that the negative relationship between the total exam score and age is weak, but age is included in the analysis to confirm the relationship, or the lack of it. As shown in Table 3, the only factor that significantly explains the class performance is the incoming students’ GRE quantitative score. Race and class type make no statistical difference. This result adds another study to the "no significant difference phenomenon," even though the instructor was originally hoping for improved student performance in the hybrid class by trying to meet the needs of the students.

Table 3. ANCOVA test of total exam score

Notes.     1. R Squared = .508 (Adjusted R Squared = .471)

                2. The dependent variable is total exam score.

                3. Asian students are excluded due to small sample size

4. Gender is not included because it leads to cell size smaller than 5. A separate t-test shows that no significant performance difference is found between male and female students.

6.2 Student Learning Experience and Attitudes

Survey and focus group data were analyzed to answer the second research question: how does the hybrid class with ALEKS impact students’ learning and attitude about statistics? 

6.2.1 Survey Data

Although the response rate to the survey is not very high (68%), answers to the Likert-scale questions (see Table 4) suggest that students on average had balanced responses to the majority of the questions regarding how ALEKS and the class format affected their learning of statistics, but they had strong concerns with three aspects of the ALEKS system: ALEKS is time consuming (Question 2), ALEKS does not match the textbook (Question 5), and assessments in ALEKS cause frustration (Question 8).

Table 4. Questions on the Fall 2006 Survey

At the end of the survey three open-ended questions were also asked:

1)                  What changes do you want to see in the class format to make the teaching and learning more effective?

2)                  How did the class format influence your attitude toward statistics?

3)                  How did the class format influence your attitude toward online learning?

These questions were explored with the focus groups, which will be presented in the next section.

6.2.2 Focus Group Interviews

As previously mentioned, Mayer’s learning process model emphasizes the importance of learner characteristics, including ability, skill, and motivation, in determining successful learning outcomes. In the focus group analysis, Mayer’s model provided the theoretical framework for understanding student learning experiences with the course as experienced by students at different performance levels. The three focus groups were created by grouping students into three different performance levels: low, medium, and high performance. In the section that follows, qualitative findings are presented in three major groups or themes: class format, ALEKS, and student.

6.2.3 Class Format

In the focus group interviews, students appreciated the flexibility of the hybrid class. Students at all performance levels commented positively about the optional classes. The flexibility seems to have benefited students for three reasons. First, students are in charge of their work schedules. One student said, "I was able to chose whenever I want to work. I thought it was great to be able to, you know, every time day or night you want to work on it…and you don’t have to show up to class if you are familiar enough. I think that… really gave me the feeling that I was controlling that class." Along with the confirmation of freedom, students also acknowledged that self-discipline is critical when flexibility increases. For instance, one student in the mid-level performance group said, "You definitely need to be self aware – of yourself and your ability. I mean, it takes some self-discipline in that aspect to decide that yeah, I do need to come to [the optional] class or I don’t need to go to [the optional] class." And one student in the low-performance group admitted "Now, had the class been required the whole time, and I had been smart enough to come to all the…classes…I’d probably know more than I know now, but that’s my fault, though. That has nothing to do with how she set it up…that falls on my shoulders." The comments seem to be consistent with the instructor’s observation that students at higher performance levels attended the optional classes more often than those at lower levels.

Second, the class format better accommodates different learning styles. In every focus group, at least one student expressed the need to interact with the teacher and fellow students. They understand that "for the person that said they liked the traditional format…they just have to come to optional classes." Some students felt empowered by the self-paced learning in ALEKS, and were comfortable not coming to optional classes. Isolation was not a concern, because, "she [the instructor] is pretty much connected online 24 hours a day, you know it seems to me she spends a lot of time online so if you have a question and you send her an email she would get right back to you. So you never really feel totally alone," stated by one student in the high-performance group. Another student made an interesting remark, "I’m not sure I would feel very comfortable just doing stuff online. . . [however] versus the traditional classes I would rather do the hybrid."

Last, the optional classes were an advantage to those who attended them. One student in the high-performance group said she came to the optional classes because the classes "just felt like reinforcement for me" after she completed the weekly objectives in ALEKS. Students who chose to come were prepared with the questions they wanted answered. They liked the open discussion format and "the fact that it was a smaller group compared to our class…we could actually ask more questions and it just seems like we had more time because it was so small."

6.2.4 ALEKS

Students had mixed feelings about ALEKS and it appears that perceptions varied systematically among the three performance levels. Three sub-themes are identified with regards to how ALEKS fits the class and student needs: the assessments, the time-demanding nature, and the match between textbook and the ALEKS.

6.2.5 Assessments

Students in the high-performance level seem to view the periodical assessments in ALEKS positively. One student thought it was great because "being able to always kind of get some feedback, you don’t have to wait for the midterm to know how you are doing." Another said,

I know that the assessments… [were] very frustrating for a lot of my …fellow students. And I thought it was frustrating too. But then I thought it was very helpful. I’m really big on reinforcement: that’s how I learn. Because it really forces you to learn objectives. And, because when you take an assessment, and you miss one thing, it’s going to kind of add that on to your objective, and you’ve got to master that again. So, even though that was frustrating for a lot of people, I think it’s really good.

For students at the middle and lower performance levels, comments about assessments in ALEKS were mixed. Some clearly understood the purpose of regular assessment, as one student put it, "I use the word accountability because the assessments test you frequently to make sure you are learning those objectives."  However, frustration often outweighed the perceived benefits. Frustration mainly came from the fact that ALEKS would add back topics that student failed to show "mastery" of in the assessment. The number of topics repeated seemed to generate different levels of frustration. Two students shared similar feelings. One described that,

Let’s say in that 5 hours, I’ve completed 9 topics or something like that…so, then at the end it gives me an assessment… so I’ve gone 9 topics ahead, but [the assessment] knocks me back 19…so, it’s just sort of like…you know, you just want to throw your pencil down and rip your hair out…you’re just like, you know, I just kind of wasted 5 hours.

And another said that, "I had…at one point I was down to like 3 or 4 [topics]. Then I had an assessment, it jumped up to like 27 or something like that. …when it keeps adding on stuff that you think you know…but then you just miss one thing because you accidentally multiplied wrong instead of adding or something." The comments, such as considering time "wasted" and having "accidentally multiplied wrong instead of adding or something," indicated many students at the relatively low performance level might have a different motivation than those in high-performance groups: their learning was more likely to be driven by external factors, such as meeting class requirements, where as those in high-performance group emphasized internal motivation such as self-realization and self-reinforcement. This is a speculation worth further investigation.

6.2.6 Time Demands

The instructor, at the beginning of Fall 2006, warned the students in Stats I that they were expected to spend about 9 hours per week out of class studying. Even though it is an ordinary expectation at the graduate level, students (except for those at the high performance level) complained strongly about how time demanding it was to meet the weekly objectives in ALEKS. The system tracks individual students’ time spent and the number of topics they have learned, but with assessments adding back "un-mastered" topics, students sensed increasing pressure and felt more frustrated when the assessments "messed up" their progress: "It’s kind of like we were always fighting getting our objective done," said one student in the mid-level performance group.

One potential cause for the time pressure may be from the use of a textbook and a "stand-alone" online tutoring system. ALEKS is designed to offer online courses. Although the textbook and ALEKS cover the same topics, the textbook had a stronger emphasis on mathematical reasoning while ALEKS uses more examples to apply statistical concepts. Maneuvering between the two could be challenging. As one student observed,

Because ALEKS, to me, would be a sufficient, stand-alone on-line statistics class because it’s equivalent: quizzes, and a final exam, and topic-wise, everything that we covered in class really is in ALEKS – if not even some other more detailed things. . . So, it’s kind of a struggle to keep them balanced at points.

6.2.7 Match/Mismatch between ALEKS and the Textbook

The textbook used in Stats I was not designed to accompany ALEKS, nor is the reverse true as well. Different approaches taken by the text and ALEKS in teaching the same topics also caused mixed feeling among students. Some students "Really enjoyed the online portion." One student stated that "I like ALEKS more. I get it more because the book is more like the formulas and the math and with ALEKS it’s like the examples…and it’s like – if I see the examples without the formula – and if they’re telling me the steps, I got it." But to some others, "I don’t feel ALEKS helped me too much." The mid-level group was especially strong in feeling "The book and ALEKS don’t compliment each other. Like, they teach stuff in different ways or sometimes they’ll use different terms for things…and so…it was kind of different." And another said, "They don’t supplement each other… I wouldn’t have a problem with ALEKS if it supplemented the book."

In comparison, a much more positive opinion was given by the high-performance group. They felt that "ALEKS and the textbook really worked well together…enforced the concepts" because "there was a very clean correlation between even how ALEKS worked a problem out compared to what we were expected to know on the test or when we came in and [the instructor] worked some out on the board and it’s like it mirrored exactly what we had to do in ALEKS." The varying perceptions about the match or mismatch between ALEKS and the textbook across performance levels are actually a good indication of levels of knowing and understanding. Those who master a topic are no longer confused by different presentations of the same content.

6.2.8 Student Attitudes

The analysis on student attitudes draws heavily from the focus groups, which included in-depth information about students’ perception of ALEKS and the class format, and the open-ended questions on the survey, which asked more specifically about students’ attitude towards statistics and the class format.

Eight of the 28 students responding to the open-ended questions commented positively on the influence of ALEKS in their learning of and attitude towards statistics. The advantages of ALEKS include the assessment providing instant feedback, applying statistical concepts in examples, and feeling progress when meeting weekly objectives. In their words, ALEKS is "simple and enjoyable," "makes the learning easier," and "helps me adapt positively to it [statistics]." However, five students voiced their strong negative feelings about ALEKS and felt that it is "anxiety-ridden" and "does little in helping me to learn stats." Others offered more neutral comments, such as, "I don’t think it’s hurt my attitude toward statistics or anything like that," or, "overall, it hasn’t really turned me off to math or given me that much more excitement either way."  

Nine students believe that the hybrid class format helped their learning and impacted their attitude positively. For example, one wrote "I was very worried about my ability to do well in the course, but the format has allowed me to do well, despite the workload." Another student had similar feelings, "I actually don’t like any math-based classes but this class format has enabled me to carry out the semester with a positive attitude." Negative feeling about the class format is rare, although a couple of students thought all classes should be required, or suggested more structure in the optional classes.

It is not surprising that the findings of the survey and the focus group interviews are consistent and reinforce each other because the focus groups followed up the concerns and questions raised in the survey. Even though the comments on the survey appeared more positive about ALEKS than discussions in the focus groups, many of the comments contained a "but:" They liked ALEKS, but it would be better if there were less frequent "assessments," no repeated topics, and a better alignment between the text and ALEKS.

6.3 Limitations

A mixed-methods approach was taken to gain better understanding about how the use of a commercial system in a hybrid introductory statistics course contributed to students’ learning and attitude. During the process, there were some circumstances that complicated this study and created limitations. First, the performance of the Stats I classes of Fall 2005 and Fall 2006 was compared to examine the effect of ALEKS to meet students’ needs. Classes in the two semesters were comparable with regards to instructor, exams, and textbook, but one could make the argument that it would be better if classes in the same semester were taught in two formats and compared accordingly. Second, the instructor allowed several students to give up on ALEKS and concentrate on the textbook materials after the second midterm exam because they were so behind with an overwhelming number of topics to "relearn." This change shed doubt on the consistency of the study design, but it was done in the interest of the students. In addition, the survey response rate is not ideal; fortunately, it does not appear to threaten the study’s validity because the results of survey and focus groups are consistent and reinforcing findings.

Readers should also be reminded that the hybrid class was different from the traditional format in two ways: the adoption of the ALEKS system and fewer required class meetings. Thus, the result is about the effectiveness of different teaching formats and should not be simply interpreted as whether ALEKS is an effective online system. From a different perspective, the experience of the instructor may be another factor to consider when interpreting the findings of this study. That is, the instructor had taught Stats I in the traditional format a few times, but only the first time in the hybrid format. The instructor may know how to better meet students’ needs in the traditional class, but needs to learn how to optimize the use of ALEKS and instructional approach in the hybrid class. Finally, only graduate students were included in this study. Generalizations should not be made to college or high school students.

7. Conclusions

This study concludes that students in the hybrid class using ALEKS, the online tutoring system, and students in traditional face-to-face class format performed comparably in Stats I. Although there are many other studies that have reached similar "no significant difference" conclusions, this study suggests effective teaching of statistics in different class formats should be examined in association with learner characteristics: their ability, skill, and motivation (Merisotis & Phipps, 1999). Student’s mathematical ability and skill are critical for their success in the introductory statistical course, as shown by the significant effect of GRE-quantitative score in the ANCOVA test of class performance and by students’ comments in the focus group interviews. This confirms Johnson and Kuennen’s finding (Johnson & Kuennen, 2006) and Sowey’s observation that (Sowey, 1998) student’s mathematical readiness is an important determinant of student success in statistical learning. Lack of an appropriate standard of prerequisite knowledge is one of the causes for student poor performance as well as their negative reactions to statistics.

The systematic variations in the perception of the class format and ALEKS across performance levels also indicate that student’s performance is associated with learning motivation. Motivation, therefore, is another influential factor in the statistics education and may be strengthened if individuals’ complex needs are met. As Ward (2004) argued, the hybrid class with optional meetings seems to offer both convenience and social involvement (Ward, 2004; Young, 2002). With more flexibility, students’ sense of self-control is heightened. However, for individuals lacking self-discipline, flexibility could lead to students misusing their time or choose not to attend classes because this was allowed. This study suggests that the range of student motivations, attitudes, and expectations may complicate the claim that combining online learning with traditional face-to-face instruction comprises the "best of both worlds" for teaching statistics (see Gal & Ginsburg, 1994). In other words, the process of combining online and face-to-face learning may be more complex than is implied by simply adding some online to the face-to-face class, but must incorporate an analysis of learner characteristics, appropriate pedagogies, and necessary statistical knowledge.

Mayer’s learning process model (Mayer, 1989) emphasized both presentation method and learner characteristics for successful learning outcomes. The large number of "non-significant difference" studies suggests that no matter what teaching format is used, not all students will learn the material (Utts et al., 2003), but it does not mean that we no longer need to explore more effective approaches to better meet student needs. Perhaps in future research, more emphasis can be placed on the intersection of learner characteristics and the range of instructional designs in order to design a class for more effective teaching and learning.

Statistics instructors may want to keep in mind cost effectiveness as well as learning effectiveness when choosing a teachware (Ferrall, 1995). For example, ALEKS might serve students better in a hybrid class if a companion textbook were available. A teachware package designed to use good teaching pedagogies could help to ease students’ anxiety and enhance their motivation. All commercial packages have advantages and disadvantages. Continuing research will help vendors improve their products and serve the educational community better.

At the time this paper was written, the instructor started teaching Stats I in a new semester. Based on the lessons learned during the initial hybrid offering, the instructor decided to make three major changes in the use of ALEKS system. First, nine topics that were not tightly related to the book chapters were removed. Second, the students were asked to complete the weekly objectives after attending the class lectures. This change reduced the pressure of self-teaching and eased their anxiety level. Finally, the optional attendance policy was removed. All students were required to come to class. The authors will continue to evaluate this class and report the findings in future papers.

References

Anderson, G., Benjamin, D., and Fuss M.A. (1994). "The determinants of success in university introductory economics courses," Journal of Economic Education, 25, 99–119.

Ferrall, C. (1995), "Interactive Statistics Tutorials in Stata," Journal of Statistics Education [On line] 3(3). jse.amstat.org/v3n3/ferrall.html

Gal, I., and Ginsburg, L. (1994). "The Role of Beliefs and Attitudes in Learning Statistics: Towards an Assessment Framework," Journal of Statistics Education [On line] 2(2). jse.amstat.org/v2n2/gal.html

Grabowski, B. L. , and Harkness, W. L. (1996), "Enhancing Statistics Education with Expert Systems: More than an Advisory System," Journal of Statistics Education [On line] 4(3). jse.amstat.org/v4n3/grabowski.html

Hagerty, G. & Smith, S. (2005). "Using the Web-Based Interactive Software ALEKS to Enhance College Algebra," Mathematics and Computer Education, 39(3), 183-194.

Hinkle (2003)- need reference

Hu, X., Luellen, J. K., Okwumabua, T. M., Xu, Y., & Mo, L. (2007), "Observational findings from a web-based intelligent tutoring system: Eliminiation of racial disparities in an undergraduate behavioral statistics course," Paper presented at the 2007 Annual Meeting of the American Educational Research Association (AERA), Chicago, IL.

Johnson, M., and Kuennen, E. (2006). Basic Math Skills and Performance in an Introductory Statistics course," Journal of Statistics Education [On line] 14(2). jse.amstat.org/v14n2/johnson.html

Maxwell, J. A. (1996). Qualitative research design: An interactive approach. Thousand Oaks, CA: Sage Publications.

Mayer, R. E. (1989), "Models for understanding," Review of Educational Research, 59(1), 43-64.

MacGillivray, H. L. (2003). "Making statistics significant in a short course for graduates with widely-varying non-statistical backgrounds," Journal Applied Mathematics and Decision Sciences, 7(2), 105-113.

Merisotis, J. P., and Phipps, R. A. (1999). "What’s the Difference? A Review of Contemporary Research on the Effectiveness of Distance Learning in Higher Education," Washington, D. C.: The Institute for Higher Education Policy, 31.

Russell, T. L. (1999), No Significant Difference Phenomenon [Online]. www.nosignificantdifference.org/

Sowey, E. R. (1998), "Statistical Vistas: Perspectives on Purpose and Structure," Journal of Statistics Education [On line] 6(2). jse.amstat.org/v6n2/sowey.html

Stillson, H. & Alsup, J. (2003). "Smart ALEKS. . . Or Not? Teaching Basic Algebra Using an Online Interactive Learning System," Mathematics and Computer Education, 37(3), 329-340.

Symanzik, J. and Vukasinovic, N. (2003), "Comparative Review of ActivStats, CyberStats, and MM*Stat," MSOR Connections, 3(1), 37-42.
ltsn.mathstore.ac.uk/newsletter/feb2003/pdf/activstatscyberstatsmmstat.pdf

Symanzik, J. and Vukasinovic, N. (2006), "Teaching an Introductory Statistics Course with CyberStats, an Electronic Textbook," Journal of Statistics Education [On line] 14(1). jse.amstat.org/v14n1/symanzik.html

Tay, R.S. (1994). "Students’ performance in economics: does the norm hold across cultural and institutional settings?" Journal of Economic Education, 25, 291–301.

Tudor, G. (2006), "Teaching Introductory Statistics Online – Satisfying the Students," Journal of Statistics Education [On line] 14(3). jse.amstat.org/v14n3/tudor.html

Utts, J., Sommer, B., Acredolo, C., Maher, M., and Matthews, H. (2003). "A Study Comparing Traditional and Hybrid Internet-Based Instruction in Introductory Statistics Classes," Journal of Statistics Education [On line] 11(3).
jse.amstat.org/v11n3/utts.html

Ward, B. (2004), "The Best of Both Worlds: A Hybrid Statistics Course," Journal of Statistics Education [On line] 12(3).
jse.amstat.org/v12n3/ward.html

Young, J. R. (2002), "’Hybrid’ Teaching Seeks to End the Divide Between Traditional and Online Instruction," The Chronicle of Higher Education, 48(28), A33-A34.

Zhang, J. (2002), "Teaching Statistics On-Line: Our Experiences and Thoughts," Proceedings of the Sixth International Conference on Teaching Statistics, ed. B. Phillips, Voorburg, The Netherlands: International Statistical Institute.

Dr. Yonghong Jade Xu
Assistant Professor of Educational Research
The University of Memphis
300 Ball Hall
Memphis, TN 38152-3340
(901) 678-5026
yxu@memphis.edu

Dr. Katrina A. Meyer
Associate Professor of Higher and Adult Education
The University of Memphis
310 Browning Hall
Memphis, TN  38152-3340
(901) 678-2466
kmeyer@memphis.edu

Dr. Dianne D. Morgan
Assistant Professor of Educational Psychology
The University of Memphis
100 Ball Hall
Memphis, TN 38152-3340
(901) 678-2841
ddmorgan@memphis.edu