Stanley A. Taylor, Manfred W. Hopfe, and Thomas E. Hebert
California State University, Sacramento
Journal of Statistics Education v.8, n.1 (2000)
Copyright (c) 2000 by Stanley A. Taylor, Manfred W. Hopfe, and Thomas E. Hebert, all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor.
Key Words: Examinations; Microcomputer task based testing; Student performance evaluation.
Given the emphasis on utilizing the computer in many statistics courses, we discuss how we have implemented microcomputer task based testing in our courses. Background information is provided about a required, undergraduate, multiple section course, and why we believe computer-based testing is an effective evaluation instrument. Issues of examination design, administration, and evaluation are presented. Examples of problems used in computer-based exams are also included.
1 At the California State University, Sacramento (CSUS), College of Business Administration, the analysis and model building phases of our data analysis courses are done with microcomputer software. This is the standard practice in most universities and industry as well. In class, the instructor uses a microcomputer with projection equipment to interactively build models and to explain the theory and practice of model building. After class, the students use microcomputers to work on homework assignments and projects. Because students are exposed to interactive model building with microcomputers in lecture classes, use microcomputers to interactively build models for homework and projects, and will be expected to do the same in their careers, it is a natural extension to allow students to use the microcomputer hardware and software to aid their work during exams. This paper presents the current status and thinking at CSUS College of Business Administration on the experience of allowing students to use microcomputers and statistical software to aid their analysis on exams.
2 We wish to note that the term "microcomputer task based testing" is used in this paper to indicate that a computer and software are used interactively to aid a student's analysis by performing routine statistical calculations. The exam is written by the instructor and is provided to the student in printed paper form. The data for the exam are on a disk in a suitable file. The student must supply the intelligence, knowledge, and creativity to perform the analysis and write the answers. The microcomputer hardware and software are tools used to execute standard statistical calculations. The instructor must still grade the exam, based upon the student's written analysis. Students turn in their disks in case there is a question about their data file(s) once the exams are returned to the students.
3 Although this paper deals primarily with "microcomputer task based testing" in a data analysis course, the same approach can be and is used in other academic fields that require extensive calculation, model building, or software-aided data manipulation. Examples of other academic fields that are currently using this testing approach at the CSUS College of Business Administration are management science, investment portfolio management, computer literacy, database, and programming courses.
4 In this paper we initially discuss the goals of the course, Data Analysis for Managers, in which we use microcomputer task based testing. The following sections discuss the rationale for our testing procedures, the design of the examinations, the administration of the examinations, and the grading procedures. In the final two sections we provide examples of the problems we have used and a conclusion.
5 Using many of the recommendations cited in "Making Statistics More Effective in Schools of Business" (Easton, Roberts, and Tiao 1988), we have designed and implemented a course that places a heavy emphasis on students' learning to think intuitively about statistics and being able to communicate their research findings to colleagues who have strong statistical backgrounds. The stated course objectives are for students to (1) develop an intuitive feel for statistical techniques; (2) learn to use statistical software and interpret computer output; and (3) learn to apply and convey the results of advanced business-applied statistical techniques in managerial decision-making situations. Our first objective implies that we teach sound statistical theory and reasoning without rigorous mathematical proofs or notation.
6 The foundation for our approach is based on the recommendations of Roberts (1991). We develop the course by introducing the concepts of specific and common variation, discussing control charts, and then progressing into the major component of model building. The primary vehicle used is multiple regression with the major emphasis placed on economic time series data obtained via the Internet.
7 To evaluate the students' comprehension of the class material, we use a number of instruments, including microcomputer task based testing, case analysis homework, and a research project. We will discuss our computer-based analysis testing procedures.
8 Traditional methods of evaluation in a data analysis course include
9 Homework scores may be a real indication of knowledge and skill. Ideally, homework assignments can be a very valid way to measure achievement. If students can use a computer, software, statistical theory, heuristic rules, and modest amounts of judgment to analyze and model data in the support of business decisions, then they have satisfied the goals of our course. Realistically, there are several flaws in this type of measurement. First, there is the issue of academic honesty. It is very easy to copy all or part of an assignment requiring computer analysis and models. It is hard to detect anything but obvious, unmodified duplication. It is also difficult to detect collaboration and ensure that each student can and did perform the required analysis and modeling. Besides, collaboration and cooperation are valid learning strategies for homework and should be encouraged to some extent. Second, there is the issue of time, because homework is not restricted in terms of time and there is some expectation of productivity in addition to correctness. Third, extensive homework assignments may be expensive in terms of instructor time. For these reasons, homework has a marginal role in measuring performance in our data analysis course.
10 Papers and projects are very valuable academic assignments but suffer the same benefits and drawbacks as homework assignments. Ideally, they are very good tools and are required in our course, but realistically they are not practical as the sole measures for large numbers of undergraduate students.
11 Conventional exams using the above mentioned methods are acceptable for measuring knowledge of concepts and facts. But the key elements in our data analysis course are analysis and modeling of datasets. Conventional exams with paper and pencil or calculator problems are impossible due to the extensive calculation burden for any realistic problem(s). In addition, our emphasis on analysis and modeling instead of number crunching makes this type of problem almost irrelevant.
12 Conventional exams that present realistic and complex datasets with computer-generated output are useful and valid testing methods for our course. However, these are static situations with no feedback mechanism for student decisions. They are essentially cross-sectional snapshots of an analysis and modeling process. A student may be able to correctly answer many cross-sectional situations but lack the ability to deal with earlier or later stages of the analysis. In other words, he or she may not be able to put the entire process together into a complete analysis and modeling process and may be incapable of actually completing an analysis. While valuable, cross-sectional, static snapshots of analysis and modeling issues are not completely valid for measuring the skills and knowledge that are the key goals of our course.
13 We have come to realize that what is needed is a longitudinal instrument capable of measuring the entire analysis and modeling process, including the decision feedback mechanisms, heuristic judgment issues, and the syntactical computer software skills. We believe that the data analysis and modeling process is analogous to skills like driving a car and flying a plane. Testing for concepts, facts, and isolated skills is not enough to infer that an individual can drive a car or fly a plane. A real life demonstration of skill in driving or flying is required to issue a license. To complete our analogy, we believe that students should actually analyze and model limited, but realistic, datasets during an exam.
14 One of the testing methods used in our data analysis course is a microcomputer task based test. The students are given a printed exam and a data disk that includes the required datasets. The exams are held in a computer lab so that each student has a computer and the required software (StatGraphics). The printed exams give a brief scenario for the datasets, questions that have to be answered, and space for the students to write their answers. We believe that this gives us a longitudinal instrument that measures how well the student responds throughout the problem analysis. The entire modeling process is documented in the student's write-up, including heuristic judgment issues and diagnostics.
15 One of the course elements that make this type of testing possible is the Electronic Lecture Classroom in which the course is held. In this classroom there is an instructor's workstation and projector that shows a screen image on a large motion picture screen on the left front of the classroom. This allows simultaneous use of the computer, transparencies on another screen, and the whiteboard. The instructors step through the data analysis and modeling process live, with the computer, software, and instructor's interaction all visible to the students. The students see the analysis, the syntactical software details, and the computer output as the instructor does the analysis. Because the software license allows all students to have a copy of the software, this is the same process that they can work at home or at most of the university's personal computers. Little instruction on the syntactical details of the software is necessary because they are demonstrated in class throughout the semester. The exams are held in another room, a computer lab with one machine per student. On the exam, the students are asked to perform similar analysis and modeling tasks with the same software and computers.
16 For those instructors who do not have computer demonstration equipment in their classrooms, but can give exams in a computer lab, microcomputer task based testing is still possible. Computer screens can be printed on overhead projector transparencies and the entire analysis and modeling process shown to the class as the instructor lectures. This approach can work, but it does not show the syntactical aspects as well. Nor does it allow for some spontaneous explorations in response to a student question or request. However, transparencies do avoid the necessity of the instructor's lecturing and using the keyboard and mouse at the same time. Instructors who are not accustomed to doing live software demonstrations may find that transparencies are a faster and more error-free approach. However, our informal feedback from instructors and students is that live software demonstrations are strongly preferred.
17 The design of the examination requires that the students demonstrate the ability to analyze data using a computer and software. Each problem will need to use either real data or simulated data. Because of many practical constraints, it has been our experience that using simulated data on exams has some advantages over using real data. The College of Business Administration offers 10 to 15 sections of the Data Analysis course each semester, and a single instructor may have sections separated by most of a day. This is enough time for an early section's students to brief a later section's students. The usual approach to frustrate a student who has been thoroughly briefed and who intends to take the exam by rote playback of his or her briefing is to have different versions of the same exam. It is difficult to do this with real data. In addition, the exam is limited in time, usually only 75 minutes, and the problem or problems must be such that they can be reasonably analyzed and written up in this time period. Real data must be carefully selected to meet such a time limit. Further, because exams are given at intervals in the semester, not all topics are covered before each exam. Care must be taken that the datasets reflect only topics covered in lectures before the exam and not have specific variations that cannot be explained or modeled with the students' current set of tools or techniques. We have used both real and simulated data for exams, and reasonable, fair, and challenging exams can be designed either way. One approach is to use simulated data for earlier, more specific, exams and real data for later, more comprehensive, exams.
18 The generation of datasets is a fairly simple process. The statistical software package we use is StatGraphics Plus for Windows (version 3.1); however, most statistical packages have similar capabilities. In particular, we generate the data using a random number generator for different distributions. The most common distributions we use to generate data are the normal and discrete uniform distributions. Then, through arithmetic operations, we are able to construct conditions that can be used to approximate situations business practitioners may encounter.
19 One of the nice features of using simulated data is that one can use the same scenario in different versions of the exam, modifying the simulation model (changing numbers, switching variable names, ...) so that there are different problems (and solutions). We have done this even from one semester to the next, while giving out the previous semester's exam as a study guide.
20 In contrast to multiple choice exams, the administration of microcomputer task based testing is much more difficult and critical to the success of the examination itself. There are two major elements -- the computer and the data files. It sounds simple enough, but when students sit down to analyze data on the computer for an exam, their anxiety levels increase so that even the simplest computer malfunction becomes magnified. To help reduce these kinds of problems, we have found it helpful for the instructor and/or assistants to go around prior to the exam to make sure that the computers are working properly and that the required software is set for each station. This setting up involves making sure that the computers are stand-alone, i.e., not networked together, to prevent e-mail messages from being transmitted during the examinations. Because we want to prevent "networking" environments, the data files for the exams need to be distributed in the form of diskettes.
21 Even with the precautions set out above, problems will occur from time to time. Hence we have found it helpful to keep a few computers in reserve and to include a couple of non-computer questions on the exams so that a student who experiences some "down time" can still work on the exam. In the event of a computer problem, the test administrator can resolve the problem, minimizing the likelihood that the student will blame the computer or software for his or her poor performance.
22 Another useful technique is to give a practice exam. We have done this, and the student feedback is very positive. We have given a previous semester's exam in the same room and under the same conditions as the real exam, except that we answer questions, assist students, and let students help each other. The noise level goes up, and the instructor may end up running all over helping students. The students are busy explaining things and showing syntactical tips to each other and learning. This practice helps reduce the anxiety that some students have when dealing with this type of exam.
23 One final administrative detail is that we require the students to turn in their data disks with their exams. Each disk has an unique number that the student records on the written exam. This, along with using stand-alone (non-networked) computers with write-protected hard drives, prevents the dissemination of the data to additional sections of students. Collecting the disks can also aid in the grading of examinations. A student may come up with an unusual model, and the grader may need to examine the residuals to evaluate the model or check to see if the student entered data correctly. For example, a student may have made a data entry error entering time, seasonal, or intervention variables. These errors can be identified to ensure that proper credit is awarded.
24 The type of exam being discussed here is much more difficult and time consuming to grade than exams where there is only one correct answer per problem. It is possible for students to have various results on a particular problem and still obtain full credit because we are more interested in the students' thought processes than whether their answers are "correct."
25 To implement the evaluation process, the instructor frequently needs to replicate a student's analysis on a computer. For example, if the students are provided with a dataset and asked to construct a regression model (see Examples 1-3 below), there are a number of different approaches that may be taken, each resulting in equivalent models. To ascertain whether a particular model is equivalent (to the instructor's), it may be necessary for the instructor (evaluator) to work through the student's analysis. Did the student evaluate the residuals? Are all of the variables in the model significant?
26 Of course, being able to replicate a student's analysis requires that the student carefully document his or her work. Hence, all of our exams declare in the instructions that the student needs to SHOW ALL OF YOUR WORK. We stress ahead of time, when going over exams from previous semesters (practice exams), that what we mean by SHOW ALL OF YOUR WORK is that we want to see written on the exam the thought process and not the individual key strokes.
27 In this section, we provide three sample exam questions, each based on simulated data. The datasets for the three examples are provided in two formats. The file examples.xls is an Excel file containing all three datasets. cd.txt, tz.txt, and zi.txt are text files containing the three individual datasets.
28 Herr Andes Luthi owns a bank in Bern, Switzerland. One of Herr Luthi's requirements of his employees is that they must continually solicit unnumbered accounts from foreign investors. Herr Luthi prefers to call such accounts "CD" because they have time limits similar to certificates of deposits used in the United States.
29 Being very computer literate, Herr Luthi created a file, cd.txt, to store his data. In his historical file, he maintains monthly data on volume (the sales volume of CDs), call (the number of cold calls Herr Luthi's employees made each month during the period November 1992 through April 1998), rate (the average rate for a CD), and mail (the number of mailings Herr Luthi sent out to potential customers). Because of the excellent services provided by the bank, it is the norm for customers to roll over their CDs into new CDs when their original CDs expire.
30 It should be noted that several years ago Herr Luthi took many of his employees on an extended ski vacation. Records show that the ski vacation lasted from December 1994 through the end of March 1995. The few non-skiers, who opted to take their holidays in Spain, continued soliciting CDs. They were, of course, credited with any walk-in traffic and any roll-over accounts.
31 You were recently offered a position at Herr Luthi's bank. As part of your responsibilities, you are to construct a regression model that can be used to analyze the bank's performance with regard to selling CDs. When the Board of Directors last met, they projected the following for May 1998:
|Number of Cold Calls||900|
|Average Rate for a CD||3.5|
|Number of mailings||4,500|
32 The marketing department for TZ Inc. has been struggling with how they will spend their advertising budget in the coming year. As an outside consultant, you have been requested to make a recommendation. You have been provided the monthly values for net sales (sales) and advertising expenditures for radio (radio), paper (paper), and television (tv), along with their sum (total = radio + paper + tv), in the file tz.txt, covering the months from April 1990 through December 1997. Your assignment is to evaluate the effectiveness of each of the advertising components (radio, paper, and television) with regard to net sales.
33 In the file zi.txt reside variables that represent monthly financial data for ZI's Pizza, a restaurant located in South Natomas. The data cover the months from June 1994 through April 1998. The variables represented are
|nnews:||amount of advertising spent in The Natomas News|
|bee:||amount of advertising spent in The Sacramento Bee|
34 When questioned about significant events that affected their sales, Kas Ziegler, the manager, stated that a competitor opened up business in the same general location starting in December 1995, but stayed open only until November 1996.
35 We believe that for computer-based statistics courses, any examination designed to determine whether a student has the ability to think statistically should include a component that allows students to interact with a computer. In discussions with colleagues at other institutions, we have been surprised by the fact that very few instructors use computer-based testing like that described here. We hope that this article provides enough information for those who want to try this approach to get started.
Easton, G., Roberts, H. V., and Tiao, G. C. (1988), "Making Statistics More Effective in Schools of Business," Journal of Business and Economic Statistics, 6, 247-260.
Roberts, H. V. (1991), Data Analysis for Managers with MINITAB (2nd ed.), South San Francisco: The Scientific Press.
Stanley A. Taylor email@example.com
Manfred W. Hopfe firstname.lastname@example.org
Thomas E. Hebert email@example.com
College of Business Administration
California State University, Sacramento
Sacramento, CA 95819-6088
JSE Homepage | Subscription Information | Current Issue | JSE Archive (1993-1998) | Data Archive | Index | Search JSE | JSE Information Service | Editorial Board | Information for Authors | Contact JSE | ASA Publications