Journal of Statistics Education v.3, n.2 (1995)
Copyright (c) 1995 by Thomas H. Short, Helene Moriarty, and Mary E. Cooley, all rights reserved. This text may be freely shared among individuals, but it may not be republished in any medium without express written consent from the authors and advance notification of the editor.
Key Words: Ordinal data; Means; Medians; Histograms.
The American Cancer Society and the National Cancer Institute both develop pamphlets and booklets to inform patients with cancer and their families about the nature and treatment of the illness. Written materials are often given to patients to reinforce verbal instructions, or in some cases, given in place of verbal instructions. Unfortunately, published materials may be written at a reading level that is difficult for many patients to understand.
The data presented here represent the readabilities of 30 booklets about cancer and the reading levels of 63 patients with cancer. A number of elementary but important statistical issues must be resolved before conclusions can be drawn. To analyze the data, students must be familiar with the notions of scales of measurement, data reduction, measuring center, constructing and interpreting displays, and reaching conclusions in real problems.
1 In this age of shortened hospital stays and a shift toward ambulatory care, innovative methods for informing patients about their diseases and treatments are critical. One common vehicle for patient self-learning is published material in the form of pamphlets and booklets.
2 Several studies have indicated that published health care materials are written at reading levels much higher than that of the general population. For example, Zion and Arman (1989) analyzed educational materials published by the American College of Obstetricians and Gynecology Committee, and Swanson et al. (1990) assessed the readability of contraception materials.
3 Meade, Diekmann, and Thornhill (1992) evaluated the readability of 51 booklets developed by the American Cancer Society, but did not address the reading levels of the relevant patient population. Our dataset compares the readability of written materials about cancer to the reading ability levels of patients with cancer. More details about the data collection, a more thorough review of the literature, and information about other analyses we performed may be found in Cooley, Moriarty, Berger, Selm-Orr, Coyle, and Short (1995). Measures of readability determine the reading comprehension level a person must have to understand written information. These measures "indicate if a printed piece is written at a level which can be understood by its target audience" (National Cancer Institute 1981).
4 The readabilities of 14 pamphlets published by the American Cancer Society and 16 pamphlets published by the National Cancer Institute were measured using the Flesch Reading Ease and Grade Level Index. One page from each pamphlet was randomly chosen and was analyzed for readability. The Reading Ease procedure assigned a score between 0 and 100, with higher scores indicating more difficult material, to one page sampled from each brochure. The Flesch Index then assigned a grade level equivalent to each of the Reading Ease scores, in order to compare the results to the reading levels of patients.
5 To determine reading levels of the patient population, a convenience sample was acquired from outpatient oncology clinics at the Philadelphia Veterans Affairs Medical Center. To be eligible for the study, patients had to be 18 years of age or older and use English as their primary language. Patients with severe pain, brain metastasis, central nervous system disorders, dementia, or metabolic problems that would interfere with mental status were excluded from the study. Eligible patients who were willing to participate were verbally interviewed by data collectors.
6 Reading ability was assessed using the Wide Range Achievement Test Revised Level (WRAT R2), which assigned a reading grade level achieved for each patient. An assumption was made that the grade levels reported by the WRAT test are comparable to the Flesch Index grade levels measured for the brochures. The assumption of equivalence of grade levels over different instruments such as the WRAT R2 and Flesch Index has apparently not been tested in health care research. Curious students may wish to develop their own experiments to investigate the definition, measurement, and comparison of grade level abilities.
7 One approach to comparing the readability of the literature to the reading levels of the patients is to identify the "typical" brochure and the "typical" patient. If the chosen representatives are at or near the same grade level, one might be tempted to conclude that the written materials are appropriate for the reading levels of the patients.
8 Students who have been introduced to statistical inference should recognize that a two-sample t-test is one tool for comparing "typical" or average values. The instructor may wish to encourage students to perform the test and interpret the results. After students have completed the t-test, the instructor should encourage the students to consider the scales of measurement of the data.
9 The grade levels for Flesch scores, which measure the readabilities of the pamphlets, could be viewed as interval or ordinal data, since the grade levels are equally spaced in time. On the WRAT R2 scale for patient reading levels, however, the grade level values "2" and "13" actually represent larger classes of reading levels, namely "Below Third Grade" and "College and Above." Numerically, it is possible to compute the mean grade level for both readability and reading level, but the ordinal scale of measurement for at least the reading levels prohibits a sensible interpretation of the mean.
10 As the students discover that the mean of ordinal data is not meaningful, some will probably suggest comparing the medians. The instructor should stress that the median of ordinal data is a sensible measure of center, and might make connections to nonparametric procedures. Interestingly, the median readability of the 30 cancer information pamphlets is ninth grade, which is the same as the median reading level of the 63 patients in the sample.
11 The students should realize that some thought must be put into an appropriate measure of center for ordinal data; they should also note that any test for the equivalence of the medians will obviously not yield a significant difference.
12 After discussing the interpretation of "typical" readabilities of the pamphlets and reading levels of the patients, the students may be guided to return to the original data and construct a graphical display. Although students often perceive histograms to be trivial to construct and interpret (see Short 1993), in this example histograms or similar graphical displays convey the information necessary to reach important conclusions.
13 Histograms of both readabilities of pamphlets and reading levels of patients appear below. The instructor should highlight the fact that "typical values" or centers are usually not appropriate measures for data that are not unimodal. In this case neither display is bell-shaped, and the reading level histogram clearly represents a non-normal distribution.
Grade | Brochure Readability Grade | Patient Reading Level
Level | Flesch Score Level | WRAT R2 Score
------+--------------------- ---------+----------------------
2 | Below 3 | ******
3 | 3 | ****
4 | 4 | ****
5 | 5 | ***
6 | *** 6 | ***
7 | *** 7 | **
8 | ******** 8 | ******
9 | **** 9 | *****
10 | * 10 | ****
11 | * 11 | *******
12 | **** 12 | **
13 | ** College and Above | *****************
14 | *
15 | **
16 | *
14 Interpretation of the histograms leads to the conclusion that the readabilities of the brochures are not appropriate for the patients with cancer in this study. The conclusion can be quantified using percentages. For example, 17 of the 63 patients (27%) would not be able to fully understand any of the pamphlets represented in the data. Although basic statistical tests fail to produce a significant difference, the data seem to indicate that there is a problem with the readabilities and perhaps some corrective action should be considered. As a related writing exercise, an instructor might ask students to write mock letters to the American Cancer Society or the National Cancer Institute describing the conclusions they have reached from the data with recommendations to pursue alternative forms of communication if appropriate.
15 It is also possible to begin the analysis of these data with the construction of a display, which is a starting point that many practicing statisticians advocate. Beginning with the display may "spoil the fun" of thinking about the appropriateness of measuring and testing centers. We have found that constructing the display only after discussing the numerical measures of center highlights the importance of simple displays that can be easily interpreted and that may provide the best analysis for a particular problem.
16 Although the dataset in this example is small, it illustrates many important statistical concepts. Students discussing or analyzing the data must grapple with scales of measurement, data reduction, measuring center, constructing and interpreting displays, and reaching logical conclusions in a practical and important problem.
17 We have used this example in Introductory Statistics classes at Villanova University both in settings where the students are exploring data through descriptive methods and also after they have encountered the notion of statistical inference and t-tests.
18 The nature of the problem also lends itself to questions about data collection. Students may wish to consider how readability and reading level could be measured, how much of a brochure must be analyzed to determine its readability, or how stable human reading level measurements remain in repeated trials on the same subject. Examining readability and reading levels is a concept that most students find easy to relate to, yet they often find the statistical concepts challenging.
19 The authors thank the section editor and Allan Rossman for insightful comments that improved the quality of this article.
20 The file readability.dat.txt contains the raw data. The file readability.txt is a documentation file containing a brief description of the dataset.
Columns
1 - 2 Grade Level
4 - 5 Frequency of occurrence for brochure readabilities
7 - 8 Frequency of occurrence for patient reading levels
Values are aligned and delimited by blanks.
Cooley, M.E., Moriarty, H., Berger, M.S., Selm-Orr, D., Coyle, B., and Short, T. (1995), "Patient Literacy and the Readability of Written Educational Materials for Patients with Cancer," to appear in Oncology Nursing Forum, October 1995.
Meade, C., Diekmann, J., and Thornhill, D. (1992), "Readability of American Cancer Society Patient Education Literature," Oncology Nursing Forum, 19, 51-62.
National Cancer Institute (1981), "Readability Testing in Cancer Communications," DHEW Publication No. 79-1689, Bethesda, MD.
Short, T. (1993), "Getting More Mileage Out of Histograms," The Statistics Teacher Network, 33, 4-5.
Swanson, J., Forrest, K., Led Bettes, C., Hall, S., Holstine, E., and Shafer, M. (1990), "Readability of Commercial and Generic Contraceptive Instructions," Image, 22, 96-100.
Zion, A., and Arman, J. (1989), "Level of Reading Difficulty in the American College of Obstetricians and Gynecologists' Patient Education Pamphlets," Obstetrics and Gynecology, 74, 955-960.
Thomas H. Short
Department of Mathematical Sciences
Villanova University
Villanova, PA 19085