Training Doctoral Students in Educational Statistics in the United States: A National Survey

Deborah A. Curtis
San Francisco State University

Michael Harwell
University of Pittsburgh

Journal of Statistics Education v.6, n.1 (1998)

Copyright (c) 1998 by Deborah A. Curtis and Michael Harwell, all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor.


Key Words: Doctoral student preparation; Statistics education; Survey research.

Abstract

Although numerous research studies have focused on issues related to the teaching of statistics, few studies have focused on the training of people who may become statistics teachers. The purpose of this study was to examine doctoral students' preparation in statistics in the field of education. A national survey was conducted of twenty-seven quantitative methods (QM) programs. One QM professor from each program was identified and asked to describe and evaluate the training of QM and non-QM doctoral students at his or her institution. The vast majority of professors indicated that most or all of the students in their QM programs received training in the "old standard" procedures -- ANOVA, multiple regression, and traditional multivariate procedures, whereas fewer than half of the professors indicated that most or all of their QM students received training in more recent procedures such as bootstrapping and multilevel models. Professors were also asked to rate the skills of their QM students in areas such as mathematical statistics and computing on a scale from "Weak" to "Strong." Most professors gave high ratings to their QM students' skills with statistical packages, but gave much more mixed ratings of their QM students' training in mathematical statistics. Nearly half of the professors thought that most of their QM students could have benefited from one or two additional statistics courses. Results are discussed in terms of training future doctoral students.

1. Introduction

1 Much of the literature on statistics education focuses on basic issues of designing and delivering statistics courses. Thus, a good deal of attention has been paid to issues such as what topics to teach in statistics courses, how to teach specific topics in statistics, how to organize classrooms for effective instruction, which textbooks to use, which computer programs to use, and what kinds of examples to use. Substantially less attention has been paid to examining the graduate school preparation of the statistics educators themselves. This seems unfortunate, because the knowledge and skills of these statistics educators will likely have a direct impact on their effectiveness as classroom instructors, statistical consultants, and mentors. The purpose of this paper is to address this topic by taking a closer look at doctoral training in statistics in one particular academic discipline, namely the field of education.

2 Consider doctoral students who are enrolled in quantitative methods (QM) programs in the U.S. Most doctoral programs in education have QM programs, loosely defined here to include doctoral students interested in research design, educational measurement, and educational statistics. Many of the graduates of these QM doctoral programs will go on to teach introductory and advanced statistics, measurement, and research design courses in colleges and universities across the nation. Moreover, these graduates may serve as statistical educators in ways that are not related to direct classroom instruction, including serving as consultants with colleagues; serving on thesis and doctoral committees; serving on editorial boards of research journals; and training and supervising future doctoral students. Although there may be disagreement among faculty members over what should be the specific content of doctoral training programs in quantitative methods, there is substantial agreement on the importance of adequately preparing students for their future roles as teachers, researchers, and consultants. As faculty members who are responsible for training QM doctoral students in research methods and statistics and who recognize the variety of skills our graduates need, we have a special responsibility to train students in techniques that will allow them to expertly discharge their professional responsibilities.

3 Next consider doctoral students in education in the U.S. who are not enrolled in QM programs. These non-QM doctoral students in education vary widely in their areas of specialization. Typical specializations offered in U.S. universities include educational leadership, educational psychology, curriculum and instruction, special education, mathematics education, science education, bilingual education, language and literacy, and counseling. When studying the statistics preparation of doctoral students in education, we considered these non-QM students to be an important group to study because non-QM graduates who take post-secondary faculty positions in education may also play a part in the statistical education of students at their institutions, either directly, as classroom instructors, or indirectly, as advisors, mentors, and consultants.

4 This paper presents the results of a survey addressing one particular aspect of doctoral student preparation, namely, their skills and knowledge in the area of statistics. A national survey was conducted of twenty-seven quantitative methods (QM) programs. One QM professor from each program was identified and given a questionnaire to complete. These professors were asked to describe and evaluate the training of both QM and non-QM doctoral students at their institutions.

2. Background and Literature Review

5 The literature on training doctoral students in educational statistics is quite sparse. A survey of statistics course content for doctoral students has not been conducted in the field of education, although studies of statistical preparation have been conducted in psychology (see Aiken, West, Sechrest, and Reno 1990) and medicine (see Dawson-Saunders, Azen, Greenberg, and Reed 1987). The lack of systematic research on what doctoral students are learning in their statistics classes is unfortunate because this information could help facilitate discussions of whether this training should be restructured and, if so, how it should be restructured. This is not to suggest that discussions of what should be taught do not take place. For example, Tukey (1980), among others, has argued that students need much more experience with exploratory data analysis in their statistics training. Similarly, Noether (1980) argued for teaching nonparametric methods in introductory statistics classes because nonparametrics can provide a strong conceptual framework for students. Brogan (1986) also suggested that nonparametrics should be taught, especially in social science and nursing disciplines. Brogan further recommended that in a required two-course statistics sequence, graduate students should learn about general linear models, including multiple linear regression, ANOVA, and possibly categorical data analysis, and should be trained to use statistical computer packages.

6 Content analyses of the statistical procedures and research designs currently being used in published research can serve as a foundation for discussions of what we should be teaching. A number of such content analyses have been performed over the years. Elmore and Woehlke (1988, 1996), for example, conducted a content analysis of statistical methods used in the American Educational Research Journal (AERJ), Educational Researcher, and Review of Educational Research for all articles published. In their 1988 publication, these researchers focused on articles published between 1978 and 1987, and in their 1996 paper, they focused on articles published between 1988 and 1995. Consider their findings for AERJ articles published between 1988 and 1995. Here, Elmore and Woehlke (1996) found that the predominant statistical method used was ANOVA/ANCOVA, which was used in 64 out of 245 articles that they coded. The next most frequent procedures were multivariate procedures, which were used in 12% of the articles, followed by bivariate correlation (12%), multiple regression/correlation (11%), t-tests (9%), structural equation modeling (9%), and nonparametrics (8%).

7 Aiken, West, Sechrest, and Reno (1990) examined the adequacy of doctoral student preparation in statistics in psychology programs. Because our research is similar to theirs, we will describe their study in detail. Aiken et al. surveyed all 222 psychology departments or schools identified by the American Psychological Association as granting doctoral degrees. From these 222 units, 186 responded, for a response rate of 84%. The goals of their research were (a) to describe the current content of the statistics, measurement, and research design courses; (b) to look at differences in requirements between sub-disciplines in psychology; and (c) to gather data on professors' perceptions of students' abilities to apply statistical and measurement techniques in their own research. In addition, these researchers distinguished between "elite" and "other" institutions, and presented data comparing the two types of institutions.

8 In terms of demographics, Aiken et al. (1990) found that 89% of the departments offered an introductory graduate statistics sequence, and of the departments offering this sequence, 77% were one year in length. A minority of institutions -- 17% -- offered doctorates in a quantitative area of specialization, although one-third of these did not have any first-year students. In terms of curriculum offerings (including optional and required courses), they found that the majority of programs offered at least a partial course (defined as at least a half a semester or full quarter) in ANOVA (88% of the programs), multiple regression (58%), and MANOVA (53%). More recent statistical topics such as structural equation models and time series models were less frequently covered (18% and 6%, respectively).

9 In terms of the content of the required introductory statistics course sequence, the authors distinguished between "old standards of statistics" and "more advanced statistical considerations," and collected information on the percent of institutions offering in-depth coverage of specific procedures. ("In-depth" was defined as coverage to the point that students could perform the analysis in question themselves). In-depth coverage of the "old standards" was provided in the majority of institutions: multifactor ANOVA, multiple comparison procedures, repeated measures via traditional factorial ANOVA, and multiple regression were covered in-depth in the required course sequence in 73%, 69%, 73% and 63% of the institutions, respectively. Topics considered more advanced were covered in-depth in required courses less often. For example, 39% of the institutions provided in-depth coverage of ANCOVA in their required course sequence, 38% provided in-depth coverage of ANOVA as a special case of regression, 20% provided in-depth coverage of exploratory data analysis, 21% provided in-depth coverage of multivariate procedures, and 18% provided in-depth coverage of statistical power analysis.

10 Aiken et al. (1990) interpreted these findings to mean that the statistical and methodological curriculum in psychology has changed little in 20 years. They also concluded that typical first-year courses only serve those students who undertake traditional laboratory research. They summarized their work by stating that new Ph.D. recipients are competent to handle traditional statistical techniques but not newer and often more useful procedures (p. 721).

11 Guo and Nitko (1996) surveyed 80 programs that offer graduate training in educational measurement. Among the 48 programs that responded and award doctorates, 19 offer training in measurement alone, 16 offer degrees in the broader area of research methodology, and 13 offer degrees in other areas. Guo and Nitko's focus was on surveying the content of courses offered to QM students and assessing agreement on what is taught. Although their focus was on educational measurement, they also presented aggregated information that reflected the frequency of coverage of various statistical topics. At the doctoral level, students in the 19 programs in measurement take 21 credits of statistics on average, and those in research methodology programs take 24 credits of statistics. The statistics courses include regression, factor analysis, multivariate analysis of variance (MANOVA), and other unspecified advanced topics.

12 Like Aiken et al. (1990), our general purpose was to survey the course-related training of doctoral students in statistics. One goal was to assess the statistical methods and procedures used in training QM and non-QM students in education. Another goal was to learn about the perceptions of QM professors regarding the competence of QM and non-QM doctoral students with various statistical techniques. This information should add to the statistical education literature on training doctoral students in QM.

3. Methods

3.1 Sampling Procedure

13 The universities we initially targeted were educational research institutions identified in a University of Illinois study as constituting the top thirty-one Schools and Colleges of Education, ranked in terms of academic productivity and prestige (West and Rhee 1995). A list of these institutions is provided in Table 1. Our target population consisted of those universities that had QM programs, and our plan was to identify one QM professor from each institution as a contact person who would complete a questionnaire. The use of the 31 schools listed in West and Rhee (1995) as our initial sample necessarily narrows our inferences to these or similar institutions. In order to identify these contacts, several strategies were used. First, we simply identified people that we knew personally at specific universities, with the belief that these people would be most likely to complete questionnaires. Second, we used the membership roster of the American Educational Research Association (AERA) Educational Statisticians Special Interest Group. Because the initial results of this study would be presented at an AERA conference, we believed that using this membership roster would increase our chance of having the questionnaires completed and returned. Third, we used a book entitled Graduate Study in Educational and Psychological Measurement, Quantitative Psychology, and Related Fields, published by the Pennsylvania State University, which lists quantitatively-oriented programs in the United States and provides names of appropriate contact persons. Finally, in a few instances, we used personal contacts to identify a non-quantitative faculty member at a particular university who then gave us the name of a QM professor at that university. There were only two instances where we could not identify a contact person at a given university, leaving an effective sample of 29 universities.


Table 1. Universities Initially Contacted for this Study

1. University of Wisconsin -- Madison
2. University of Illinois
3. Ohio State University
4. Stanford University
5. University of Minnesota
6. Indiana University -- Bloomington
7. Michigan State University
8. Columbia -- Teachers College
9. University of Georgia
10. Pennsylvania State University
11. University of Maryland
12. University of Texas -- Austin
13. University of Michigan
14. Arizona State University
15. University of California -- Los Angeles
16. University of Washington
17. University of California -- Berkeley
18. University of Chicago
19. Harvard University
20. University of Virginia
21. Vanderbilt University -- Peabody
22. University of North Carolina
23. University of Florida
24. Florida State University
25. Syracuse University
26. University of Arizona
27. University of Nebraska
28. Virginia Polytechnic
29. SUNY-Buffalo
30. University of Missouri
31. University of Pittsburgh


14 Because our goal was to survey universities with QM programs, the next step was to determine whether each university had a QM program. To do this, QM programs were broadly defined -- thus, some programs might place a heavy emphasis on measurement, whereas others might place a heavy emphasis on educational statistics. To figure out which universities had a QM program, we again used our personal knowledge of various universities. In cases where we were not sure, we contacted someone at that university. Of the 29 universities, all but two had QM programs. Thus, our final sample consisted of 27 universities. (The two Schools/Colleges without QM programs still offered doctoral training in statistics.)

15 In November 1995, all participants received a copy of the questionnaire (described below) to complete. A second mailing was conducted approximately one month later, in December 1995. In January 1996, this second mailing was followed by e-mail and/or phone calls asking that participants return their questionnaires. All questionnaires received within three months of the initial mailing were then analyzed. Twenty-three of the 27 universities with QM programs responded for a response rate of 85%. This rate is consistent with that reported by Aiken et al. (1990) and Guo and Nitko (1996). We believe that this response rate is large enough to generate credible results, although it is possible that there is some unknown but systematic difference between QM faculty members who responded to the survey and those who did not respond.

3.2 Instrumentation

16 A questionnaire (see Appendix) was used to collect information about the statistics courses offered in QM programs and the perceptions of professors of QM. The questionnaire was divided into two sections, one directed toward students majoring in QM and one for non-QM students.

3.2.1 Questionnaire Items Pertaining to QM Programs and QM Students

17 On the portion of the questionnaire pertaining to QM programs and training of QM doctoral students, we collected the following data:

  1. Demographic information, including information on the number of full-time faculty in the QM program, the number of lecturers in the QM program, the number of students in the QM program (full-time or part-time), and where students took their required statistics courses (e.g., in the School/College of Education, statistics department, etc.),

  2. The extent to which QM doctoral students are trained in specific statistical topics or methods (assessed using a checklist),

  3. Professors' perceptions about the adequacy of QM doctoral students' computer skills and mathematical training, and

  4. Professors' perceptions on whether QM doctoral students could have benefited from additional statistics coursework.

3.2.2 Questionnaire Items Pertaining to the Training of Non-QM Students

18 The development of the questionnaire addressing the training of non-QM students was more challenging because it needed to be sufficiently flexible to accommodate the various ways in which Schools and Colleges of Education are organized. Specifically, a School/College of Education might consist of a series of schools, divisions, departments, or other organizational units, each with separate programs. Each organizational unit could conceivably have its own statistics requirement and its own statistics course sequence. A non-QM student with one area of specialization at a particular institution could, therefore, take one required statistics course taught in one particular organizational unit, whereas another student with a different area of specialization could take a different required statistics course taught in a different organizational unit.

19 Considering these complexities, it became apparent that completion of the survey of what non-QM students learn in their statistics courses would require input from all of the statistics instructors in each organizational unit at each institution. We thought that such a survey would be too unwieldy, and decided to shift the focus for the non-QM students away from a survey of course content and course requirements and towards a survey of quantitative professors' perceptions of non-QM doctoral students' preparation in educational statistics. We asked one QM professor at each institution to evaluate non-QM doctoral students' training in educational statistics at his or her institution, and to frame this evaluation in terms of doctoral students with whom he or she had direct contact over the past few years -- through committees, research assistantships, classes, and so forth. In this way, we hoped to circumvent potential problems of having professors speculate about what was taught in statistics classes they might not have thorough knowledge of and about the statistical competence of doctoral students whom they might not know well.

20 For the non-QM students, we collected the following data:

  1. Information on statistics requirements for these students,

  2. Whether statistics requirements have changed over the past five years,

  3. Professors' perceptions of doctoral students' ability to critically read and interpret substantive research articles utilizing specific statistical procedures, and

  4. Professors' opinions on whether the non-QM doctoral students could have benefited from more statistics coursework.

21 In designing the questionnaire, we were initially concerned about QM faculty members' ability to provide accurate information regarding the preparation of non-QM students. First, we were concerned that some QM professors might not be familiar with the statistics requirements of non-QM students. These concerns led us to conduct informal conversations with some of our colleagues at other institutions, who did, in fact, appear to be familiar with the statistics requirements at their universities. Our concerns were further alleviated by the results of the survey itself: Of the 23 QM professors we surveyed, only one person was unfamiliar with the statistics requirements of the non-QM students at the university. Second, we were concerned that the QM professors might not have adequate familiarity with the non-QM students to accurately evaluate their competence with statistics. To address this problem, we decided to ask the professors to evaluate only the non-QM doctoral students with whom they had direct contact over the previous five years -- through dissertation committees, classes, consulting, and so forth. While this strategy is still problematic from a design perspective, we believe that the results provide a solid first step in looking at the adequacy of non-QM doctoral students' training in educational statistics.

3.3 Limitations of the Current Research

22 It is important to point out three limitations of the current research. First, sampling the schools listed in the West and Rhee (1995) paper limits the generalizability of the results, as does the use of only one QM faculty member per institution in the survey. Second, because of time and resource constraints we did not collect survey data on doctoral students' preparation in the areas of educational research methods or measurement (see Guo and Nitko 1996), thus limiting the picture we can paint about doctoral student training. Third, no thorough reliability or validity information was obtained for the questionnaire items. On the positive side, we believe that the survey has several advantages, among them, that it is not a time-consuming research method and it is not particularly intrusive.

4. Results

4.1 Statistics Preparation of QM Students in Education

4.1.1 General Information for QM Programs

23 General information for QM programs and students is presented in Table 2.1 and Table 2.2. The median number of full-time tenured and tenure-track faculty in these QM programs is four, with three full-time faculty regularly teaching statistics during the academic year. The distribution of new doctoral students enrolling in QM programs was positively skewed, with a median of four students admitted annually during the previous five years. All of the institutions required students to take at least one statistics course in the School or College of Education; nearly half required students to take one or more courses in the Mathematics or Statistics Departments; and two (9%) required students to take one or more courses in the Psychology Department.


Table 2.1. Demographics: Program Size


Mean Median SD N
Number of full-time tenured or tenure-track faculty in QM program 5.1 4 2.2 23
Number of full-time faculty who regularly teach statistics during the academic year 3.6 3 1.7 23
Number of lecturers and adjunct faculty members who regularly teach statistics in QM program 1.8 1 2.8 23
Approximate number of students admitted to QM program each year over past five years 7.2 4 8.8 22

Table 2.2. Location of Required Statistics Courses in QM Program (N = 22)

School/College of Education100%
Mathematics/Statistics Department46%
Psychology Department9%
Other5%


4.1.2 Training Doctoral Students in QM Programs in Specific Procedures

24 To assess the extent to which doctoral students in QM programs received training in specific procedures, we developed a list that we believe reflects a breadth of statistical procedures, including those considered to be "recent," such as meta-analysis, bootstrapping/jackknifing, multilevel models, and structural equation models. We asked faculty to indicate those procedures and methods in which most or all of their QM students received training. Results are presented in Table 3.


Table 3. Preparation of Quantitative Methods Doctoral Students (Percent of Universities Where Most or All QM Program Graduates Receive Training in Topic/Method, N = 23)

ANOVA
Covariance analysis 100%
Repeated measures designs 96%
Power/sample size calculations 96%
Mixed-effects models 91%
Random-effects models 87%
Nonorthogonal designs 83%
Thorough coverage of multiple comparison procedures 74%
Cell means models 57%
Complex designs (e.g., fractional factorial) 39%
Multiple Regression
Ordinary least squares estimation 100%
Weighted least squares estimation 70%
Nonlinear-in-the-predictors models 70%
Logistic regression 65%
Nonlinear-in-the-parameters models 39%
Traditional Multivariate Procedures
Canonical correlation 91%
MANOVA 91%
Discriminant analysis 91%
Factor analysis 87%
MANCOVA 83%
Principal components analysis78%
Log-linear models 56%
Nonparametric Procedures
Kruskal-Wallis test 48%
Exact tests 44%
Jackknifing/Bootstrapping 39%
Repeated measures tests (e.g., Friedman, Cochran) 30%
Asymptotic relative efficiency 26%
Rank-transform tests 22%
Other Topics and Procedures
Matrix algebra 83%
Meta-analysis 74%
Structural equation models 74%
Multilevel models/Hierarchical linear models 35%
Times series models 4%


25 The data suggest that the vast majority of QM doctoral students receive training in the "old standards" -- ANOVA, multiple regression, and traditional multivariate procedures. The exceptions to this appear to be logistic regression and log-linear models, where more than half of the respondents indicated that their QM students receive training. Nonparametric procedures, on the other hand, appear to be less prevalent. Although a more detailed list of nonparametric procedures was not provided, it seems reasonable to use the Kruskal-Wallis test as a benchmark and to note that only half of the respondents indicated that their QM students received training in this procedure. Finally, in terms of more recent procedures, we see that the majority of programs train their students in meta-analysis and structural equation models (74%), and over one-third of the programs train their students in jackknifing/bootstrapping (39%) and multilevel models (35%).

4.1.3 Mathematical Statistics Training and Computer Skills of QM Students

26 We also asked faculty to rank the mathematical statistics training and computer skills of most of their QM doctoral graduates using a 4-point scale with "1" indicating "weak" skills and "4" indicating "strong" skills. Results are presented in Table 4. Faculty ratings were reasonably symmetrically distributed, with most of the faculty rating their students midway on the scale, either as a "2" (38% of faculty) or as a "3" (48% of faculty). In terms of computer skills, the vast majority (91%) thought their students had "strong" skills in standard data analysis programs such as SAS and SPSS. Faculty perceptions of student competence in other computer skills -- database management and programming -- were more varied.


Table 4. Faculty Members' Perceptions of the Mathematical/Statistical Training and Computer Skills of Quantitative Methods Program Graduates


Weak

Strong


(1)(2)(3) (4) M SD N
1. Mathematical statistics training of most QM graduates 181022.6 .7421
(5%) (38%)(48%)(10%)


2. Computer skills: Standard data analysis programs (e.g., SAS, SPSS, SYSTAT) 0 0221 3.9.2823
(0%) (0%) (9%) (91%)


3. Computer skills: Database management 1 6 14 2 2.7 .69 23
(4%) (26%) (61%) (9%)


4: Computer skills: Programming (e.g., FORTRAN, PASCAL, C) 75 9 2 2.3 1.01 23
(30%) (22%) (39%) (9%)


NOTE: For this question, the anchors were 4 = Strong, perhaps took multiple courses in mathematical statistics and probability theory, would probably be well acquainted with topics such as quadratic forms, Likelihood Ratio Test Principle, Gauss-Markoff Theorem, etc.; and 1 = Weak, little or no coursework in mathematical statistics or probability theory.


4.1.4 Comparing the Statistical Training of Past and Present QM Graduates

27 Colleagues have occasionally remarked that they believe earlier graduates of QM programs were better trained than more recent graduates. We investigated these kinds of comments by asking faculty in the survey if they thought that doctoral students who had graduated in the last year or two were better prepared or less well prepared to do publishable statistical work, as compared to students who graduated five to six years earlier. Data from Table 5 do not support the claim that earlier graduates are perceived to be better trained. In fact, none of the faculty indicated that they thought their recent QM students were less well prepared than graduates from five to six years prior. Over half of the faculty (56%) thought that there was no difference between past and present graduates, and the remaining 44% believed that recent graduates are better prepared.


Table 5. Faculty Members' Perceptions of QM Program Graduates' Ability to Do Publishable Statistical Work (Percent of Faculty Selecting Each Option, N = 23)

Faculty who believe that recent graduates are better prepared than graduates from 5-6 years ago 44%
Faculty who believe that recent graduates are less well prepared than graduates from 5-6 years ago 0%
Faculty who believe that there is no difference between recent graduates and graduates from 5-6 years ago 56%

NOTE: "Recent graduates" were defined as students who graduated within the last two years.


4.1.5 Faculty Perceptions of QM Students' Needs for Additional Statistical Methods Coursework

28 Finally, we asked faculty members to indicate whether they thought their QM students could benefit from additional (probably specialized) statistics coursework. Specifically, they were asked to assess (a) whether more than half or fewer than half would have benefited from one or two additional courses, and (b) whether more than half or fewer than half would have benefited from three or more additional statistics courses. Data are presented in Table 6. Nearly half of the respondents (48%) indicated that they thought more than half of their students could have benefited from one or two additional courses in statistics. Moreover, 20% indicated that they thought more than half of their students could benefit from three or more additional courses.


Table 6. Faculty Members' Perceptions of Doctoral Students' Need for Additional Statistics Coursework for Quantitative Methods Program (Percent of Faculty Selecting Each Option)

Statistics Coursework <50% >50% Can't Judge
1-2 additional (probably specialized) statistics courses (N = 21) 43% 48% 10%
3 or more additional (probably specialized) statistics courses (N = 20) 65% 20% 15%

NOTE: Data are in response to the following question: In hindsight, of the quantitative methods program doctoral students who graduated from your program in the last 5 years, what percent do you think could have profited from (the following additional coursework)?


4.2 Statistics Preparation of Non-QM Students in Education

4.2.1 Statistics Requirements

29 For the non-QM doctoral students, we began by determining whether each institution had a minimal statistics requirement for all doctoral students in education. Descriptive statistics are presented in Table 7. In nearly half of the institutions (44%), students in some programs can graduate without taking a statistics course, whereas in 39% of the institutions, students in all programs are required to take at least one statistics course. Only 13% of the institutions had a uniform two-course statistics requirement. (Note that this statistic does not estimate how many students actually graduate without a statistics course in a given institution.)


Table 7. Coursework Requirements for Doctoral Students in Education Who Are Not in Quantitative Methods Programs (Percent of Faculty Selecting Each Option, N = 23)

Students in some programs can graduate without taking a statistics course 44%
Students in all programs are required to take at least one statistics course 39%
Students in all programs are required to take at least two statistics courses 13%
Don't know 4%


30 We speculated that students might be taking fewer statistics courses currently than in previous years, due to the increased prevalence of qualitative research methods. The data in Table 8, however, do not indicate a decreasing trend in the number of students taking statistics over the past five years. In approximately one-third of the institutions surveyed, students are taking fewer statistics classes, but in a roughly equal number of institutions, they are taking more statistics classes.


Table 8. Changes in Statistics Enrollment over the Past Five Years for Doctoral Students in Education Who Are Not in Quantitative Methods Programs (Percent of Faculty Selecting Each Option, N = 22)

Doctoral students have tended to take fewer statistics courses 27%
Doctoral students have tended to take more statistics courses 36%
Doctoral students have tended to take about the same number of statistics courses 27%
Don't know/No Response 9%


4.2.2 Perceived Competence with Specific Statistical Procedures

31 We asked faculty to rate non-QM doctoral students in terms of their ability to critically read and interpret research articles utilizing specific statistical procedures (see Table 9). In particular, faculty were asked to think about the non-QM doctoral students with whom they had contact over the past five years, and to indicate whether they thought more than half or fewer than half of these students would be competent with a particular statistical method. Notice that this question requires a lower threshold of "competence" than a question that asks about doctoral students' abilities to carry out and analyze research utilizing these procedures. The procedures listed -- including ANOVA, ordinary least squares (OLS) regression, MANOVA, and nonparametric rank tests -- were not meant to be an exhaustive list of procedures, but rather were selected because they appeared to represent a cross-section of "common" and "advanced" statistical methods found in published educational research.

32 Results indicate that the faculty perceived the students to be most competent with ANOVA: 63% of the faculty thought that more than half of their graduates could critically read an article using ANOVA. Only 41% of the faculty thought that more than half of their graduates could critically read an article using OLS regression. Only a few of the 22 QM professors thought that more than half of their students could critically read and interpret substantive articles that utilized more advanced procedures, namely MANOVA, log-linear models, and structural equation models.


Table 9. Perception of Students' Competence to Critically Read and Interpret Research Articles Utilizing Specific Statistical Procedures for Doctoral Students Who Are Not in Quantitative Methods Programs (Percent of Faculty Selecting Each Option, N = 22)


<50% >50%

N (Pct) N (Pct)
ANOVA 8 (36%) 14 (63%)
OLS regression 13 (59%) 9 (41%)
Nonparametric rank tests 19 (86%) 3 (14%)
MANOVA 20 (91%) 1 ( 5%)
Log-linear models 21 (95%) 1 ( 5%)
Causal models 20 (91%) 1 ( 5%)

NOTE: Data are based on responses to the following question: Think about the doctoral students in your School/College of Education who are NOT in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting, and so forth. Of these non-quantitative students, what percent do you think are competent to critically read and interpret research articles that utilize the following procedures? (1) Less than half, (2) More than half, (3) Can't judge.


4.2.3 Faculty Members' Perceptions of Graduates' Need for Additional Statistical Coursework

33 Faculty were asked to think about non-QM students with whom they had contact over the past five years, and who had graduated, and to assess whether more than half or fewer than half would have benefited from additional statistics coursework. Results are shown in Table 10. Virtually all of the faculty thought that more than half of the graduates could have benefited from additional (probably specialized) statistics coursework: 44% of the faculty thought that over half of the graduates could have benefited from one or two additional courses, and 43% of the faculty thought that at least half of the graduates could have benefited from three or more courses.


Table 10. Faculty Members' Perceptions of Graduates' Need for Additional Statistics Coursework for Doctoral Students Who Are Not in Quantitative Methods Programs (Percent of Faculty Selecting Each Option)

Statistics Coursework <50% >50% Can't Judge
1-2 additional (probably specialized) statistics courses (N = 23) 52% 44% 4%
3 or more additional (probably specialized) statistics courses (N = 21) 48% 43% 10%

NOTE: Data are in response to the following question: Think about the doctoral students in your School/College of Education who are NOT in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting and so forth. Of these non-quantitative students, what percent do you think could have profited from (the following additional coursework)?


5. Discussion

34 Has this survey accomplished anything? Despite some methodological shortcomings, we think so. For the portion of the survey addressing the preparation of QM students, we offer three general findings. First, we now have empirical evidence of the nature of the curriculum offered in a sample of QM programs. We found much of this information encouraging because it indicated widespread professional agreement in the way that doctoral students majoring in quantitative methods are trained in statistics. For example, almost all of the surveyed programs emphasized traditional data-analytic procedures, a result that agrees with the work of Elmore and Woehlke (1988, 1996). There was also evidence that most respondents thought recent graduates were as well or better trained than earlier graduates. The fact that a number of QM programs also offered instruction in more recent or advanced procedures such as meta-analysis, structural equation models, and jackknifing/bootstrapping implies that a number of faculty believe this kind of training is important enough to incorporate into the curriculum. A second implication of our findings is their relationship to those reported by Aiken et al. (1990). In general, the two sets of findings are similar in terms of what is being taught in QM programs in education and psychology, and in terms of the perceptions of QM professors of students. One area of modest disagreement is that our findings suggest that newer statistical techniques are being taught more frequently in Schools and Colleges of Education than in Departments of Psychology. Still, the results suggest that much of the information provided by the two studies is similar and can be combined to draw stronger conclusions and guide future research.

35 Perhaps the most important implication of our research on QM programs is that the findings offer a platform from which we can ask (and in some cases provide tentative answers to) some important questions about the training of doctoral students in quantitative methods. For example, about half of the quantitative methods programs surveyed have incorporated newer techniques into their curricula. In our view, this places our findings in a somewhat less positive light. What accounts for the failure to offer training in these newer techniques? Is it simply the result of constraints on credit hours, too few faculty, or that training students who are not majoring in quantitative methods is now the primary business of many quantitative methods programs, and that this has a severe effect on the curriculum?

36 We believe the questions raised above (and others) are important and thus provide directions for additional work in the area of preparing doctoral students in quantitative methods. One obvious need in future work is to survey at least two faculty per program to allow within- and between-institution variability to be examined; another is to collect information on additional variables that should controlled, such as rank of the school and required statistics training for QM and non-QM students. It is also important to widen the participation of QM programs and to obtain information about the kinds of positions taken by graduates. As the statistical education literature has argued, curricula must be responsive to the needs of the institutions and industries hiring graduates. We believe that the emergence of information about the training of students in quantitative methods programs should encourage a long-overdue dialogue among educational statisticians.

37 For the portion of the survey that addressed the preparation of non-QM students, the research findings suggest that a number of QM faculty think many non-QM doctoral students are under-prepared to be critical consumers of quantitatively-oriented research articles. Specifically, we found that 36% of the faculty thought that fewer than half of the non-QM doctoral students could critically read and interpret research articles utilizing ANOVA, and 59% of the faculty thought that fewer than half of the non-QM doctoral students could critically read and interpret research articles utilizing OLS regression. Moreover, the vast majority of faculty thought that fewer than half of the non-QM doctoral students could critically read articles utilizing more advanced procedures, namely MANOVA, log-linear models, nonparametric rank tests, and structural equation models. These research findings provide preliminary data suggesting that we may have cause for concern about non-QM doctoral students' preparation in statistics.

38 Although these findings on the preparation of non-QM doctoral students are only tentative, they do suggest a direction for future research and discussion. One line of research suggested by this investigation involves studying faculty members' opinions about why some non-QM doctoral students at their universities are under-prepared in educational statistics, and what they think can be done to improve these students' preparation. Another line of research suggested by this investigation involves studying editors of educational research journals. Because these editors review so many manuscripts (both publishable and unpublishable), they are in a good position to evaluate the statistical and methodological training of today's educational researchers. In addition to journal editors, it might prove fruitful to study the opinions of the doctoral students themselves regarding the adequacy of their own training. Lastly, the perceptions of non-QM faculty would help inform a discussion on the statistical preparation of non-QM doctoral students.


Appendix

Instructions

This questionnaire is part of a national survey of statistics training in doctoral programs in education. The first part of the questionnaire addresses training of doctoral students in quantitative methods (QM) programs in education. The second part of the questionnaire addresses statistical training of doctoral students in education who are not in a quantitative methods program. Your responses to this questionnaire are anonymous and confidential.

I. Program Description

  1. Does your university operate on the quarter, trimester, or semester system?
    _____ quarter
    _____ trimester
    _____ semester

  2. How many full-time faculty tenured or tenure-track are there in your quantitative methods program? ________

  3. How many full-time faculty members regularly teach statistics during the academic year? ________

  4. How many lecturers and adjunct faculty members teach statistics in your quantitative methods program during the academic year? ________

  5. Over the past five years, approximately how many doctoral students were admitted to your quantitative methods program each year? ________

  6. On average, how much time do the full-time tenure/tenure track faculty in your quantitative methods program spend on research?
    _____ >80%
    _____ 70%-79%
    _____ 60%-69%
    _____ 50%-59%
    _____ <50%

  7. Over the last 8-10 years, what types of careers have the majority of graduates of your doctoral program in quantitative methods entered into? (Check all that apply).
    _____ Positions in the private sector (i.e., small businesses, corporations)
    _____ University faculty positions
    _____ Positions in the public sector (e.g., school districts; non-profit educational organizations, including testing companies; non-profit research institutions; non-faculty university positions)
    _____ Other (please explain):

  8. Where do the doctoral students in your quantitative methods program take their required statistic(s) courses? (Check all that apply).
    _____ In the School/College of Education
    _____ In the Mathematics/Statistics Department
    _____ In the Psychology Department
    _____ Other (please explain):

II. Preparation of Quantitative Methods Students

  1. Please check each of the following topics or methods that most or all of the graduates of your doctoral program in the quantitative methods have received training in:

    ANOVA

    _____ Nonorthogonal designs
    _____ Covariance analysis
    _____ Repeated measures designs
    _____ Cell means models
    _____ Random-effects models
    _____ Mixed-effects models
    _____ Complex designs (e.g., fractional factorial)
    _____ Thorough coverage of multiple comparison procedures
    _____ Power/sample size calculations
    _____ Other: Please describe

    Multiple Regression

    _____ Ordinary least squares estimation
    _____ Weighted least squares estimation
    _____ Nonlinear-in-the-predictors models
    _____ Nonlinear-in-the-parameters models
    _____ Logistic regression
    _____ Other: Please describe

    Traditional Multivariate Procedures

    _____ Canonical correlation
    _____ MANOVA
    _____ MANCOVA
    _____ Discriminant analysis
    _____ Principle Components Analysis
    _____ Factor analysis
    _____ Log-linear models
    _____ Other: Please describe

    Nonparametric Procedures

    _____ Kruskal-Wallis test
    _____ Repeated measures tests (e.g., Friedman, Cochran, Hodges-Lehmann block test)
    _____ Asymptotic relative efficiency
    _____ Exact tests
    _____ Rank-transform tests
    _____ Jackknifing/Bootstrapping
    _____ Other: Please describe

    Other Topics and Procedures

    _____ Causal models
    _____ Meta-analysis
    _____ Time-series models
    _____ Multilevel models/hierarchical linear models
    _____ Matrix algebra

III: Other Training and Preparation Issues

  1. How would you rate the mathematical/statistical training of most of your graduates from the quantitative methods program? Use the following scale:

    4 = Strong, perhaps took multiple courses in mathematical statistics, probability theory, would probably be well-acquainted with topics such as quadratic forms, Likelihood Ratio Test Principle, Gauss-Markoff Theorem, etc).

    1 = Weak, little or no coursework in mathematical statistics or probability theory, probably not familiar with Gauss-Markoff Theorem, etc.).

          Weak             Strong
            1     2     3     4
    

  2. Please rate the computer skills of the graduates from your quantitative methods program.

                                         Weak             Strong
    Standard data analysis programs
    (e.g., SAS; SPSS; SYSTAT; BMDP)         1     2     3     4
    
    Database management                     1     2     3     4
    
    Computer programming
    (e.g., FORTRAN; PASCAL; C)              1     2     3     4
    

  3. Do you think that students who graduated from your quantitative methods program in the last year or two are better prepared to do publishable statistical work than students who graduated 5 or 6 years ago?
    _____ I think they are better prepared now
    _____ I think they are less-well prepared now
    _____ I think they are the same

  4. In hindsight, of the quantitative methods program doctoral students who graduated from your program in the last 5 years, what percent do you think could have profited from:

                                   Less       More    Can't
                                than Half  than Half  Judge
    1-2 additional (probably       <50%>50%      CJ
    specialized) statistics
    courses
    
    3 or more additional           <50%>50%      CJ
    (probably specialized)
    statistics courses
    
    1-2 additional qualitative     <50%>50%      CJ
    methods courses
    

  5. Has the increased prevalence of qualitative research methods had an impact on training doctoral students in your quantitative methods program?
    _____ yes
    _____ no

    If yes, please explain.

IV. Preparation of Non-Quantitative Methods Program Doctoral Students

Directions: We are now interested in finding out about methodological training for doctoral students in your School or College of Education who are NOT in your Quantitative Methods program.

  1. Which of the following statements best describes the statistics requirement in your School/College of Education for doctoral students who are NOT in the Quantitative Methods Program?:
    _____ Students in some programs can graduate without taking a statistics course
    _____ Students in all programs are required to take at least one statistics course
    _____ Students in all programs are required to take at least two statistics courses
    _____ Don't know

  2. Which of the following statements best describes the qualitative research requirement in your School/College of Education for doctoral students who are NOT in the Quantitative Methods Program?:
    _____ Students in some programs can graduate without taking a qualitative methods course
    _____ Students in all programs are required to take at least one qualitative methods course
    _____ Don't know

  3. Over the past five years, what changes, if any, have you seen in terms of enrollment in statistics courses for doctoral students in your School/College of Education for doctoral students who are NOT in the Quantitative Methods program?
    _____ In the past 5 years, doctoral students have tended to take fewer statistics courses
    _____ In the past 5 years, doctoral students have tended to take more statistics courses
    _____ In the past 5 years, doctoral students have tended to take about the same number of statistics courses
    _____ Don't know

  4. Think about the doctoral students in your School/College of Education who are NOT in your quantitative methods program that you have had contact with over the past 5 years or so -- through dissertation committees, classes, consulting, and so forth.

    *Of these non-quantitative students, what percent do you think are competent to critically read and interpret research articles that utilize the following procedures? (circle your answers)

                                 Less       More    Can't
                              than Half  than Half  Judge
    ANOVA                        <50%>50%      CJ
    OLS regression               <50%>50%      CJ
    MANOVA                       <50%>50%      CJ
    Log-linear models            <50%>50%      CJ
    Nonparametric rank tests     <50%>50%      CJ
    Causal models                <50%>50%      CJ
    

    *Of these non-quantitative students, what percent do you think could have profited from:

                                        Less       More    Can't
                                     than Half  than Half  Judge
    1-2 additional (probably            <50%>50%      CJ
    specialized) statistics courses
    
    3 or more additional (probably      <50%>50%      CJ
    specialized) statistics course
    
    1-2 additional qualitative          <50%>50%      CJ
    methods courses
    


References

Aiken, L. S., West, S. G., Sechrest, L., and Reno, R. (1990), "Graduate Training in Statistics, Methodology, and Measurement In Psychology: A Survey Of Ph.D. Programs In North America," American Psychologist, 45, 721-734.

Anonymous (1995), Graduate Study in Educational and Psychological Measurement, Quantitative Psychology, and Related Fields, College of Health and Human Development, Pennsylvania State University.

Brogan, D., and Kutner, M. H. (1986), "Graduate Statistics Service Courses," The American Statistician, 40, 252-254.

Dawson-Sanders, B., Azen, S., Greenberg, R. S., and Reed, A. H. (1987), "The Instruction of Biostatistics in Medical School," The American Statistician, 41, 263-266.

Elmore, P. B., and Woehlke, P. L. (1988), "Statistical Methods Employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research from 1978 To 1987," Educational Researcher, 19-20.

----- (1996), "Research Methods Employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research From 1978 To 1995," paper presented at the annual meeting of the American Educational Research Association, New York.

Guo, F., and Nitko, A. J. (1996), "Graduate Programs that Prepare Educational Measurement Specialists," Educational Measurement: Issues and Practice, 15, 28-31.

Noether, G. E. (1980), "The Role Of Nonparametrics in Introductory Statistics Courses," The American Statistician, 34, 22-23.

Tukey, J. W. (1980), "We Need Both Exploratory and Confirmatory," The American Statistician, 34, 23-25.

West, C. K. and Rhee, Y. (1995), "Ranking Departments or Sites within Colleges of Education Using Multiple Standards: Departmental and Individual Productivity," Contemporary Educational Psychology, 20, 151-171.


Deborah A. Curtis
DAIS/College of Education
San Francisco State University
1600 Holloway Avenue
San Francisco, CA 94132

curtis@sfsu.edu

Michael Harwell
Department of Psychology in Education
5B27 Forbes Quad
University of Pittsburgh
Pittsburgh, PA 15260

mrh@fs1.sched.pitt.edu


Return to Table of Contents | Return to the JSE Home Page