Navigation

Return to the front page.

Return to the overview for this issue of CCC.

Access a full text PDF of this article. (Note: PDFs are available through NCTE, or JSTOR for issues prior to Volume 48. Site subscription may be required.)

Delicious Bookmark this entry on Delicious

CiteULike Bookmark this entry on CiteULike.

Print this entry.

[need to do a plaintext style sheet for entries]

Then below here, we'll add the lower navbar stuff from the original version, inc. site search, tag search, categories, issue bar, etc.)

Brian Huot. Reliability, Validity, and Holistic Scoring: What We Know and What We Need to Know.

Huot, Brian. "Reliability, Validity, and Holistic Scoring: What We Know and What We Need to Know." CCC 41.2 (1990): 201-213.

Abstract:

The author's purpose in this essay is "to outline the present state of holistic writing evaluation, the inflated position of reliability and the neglected status of validity, and to consider what we know and what we need to know in order to establish the theoretic soundness of holistic scoring procedures." Holistic, rubric-based scoring emphasizes the reliability of scores, but the author warns that these holistic scoring procedures change the natural relationship between the reader and the text, forcing the scorer to look narrowly at a piece of writing instead of valuing a personal, subjective reaction to the text. The author argues that the field needs to further develop holistic scoring procedures that will be more accurate and valid in assessing the effectiveness of student writing.

Categories:

Works Citing

Listed below are links to articles that reference Brian Huot. Reliability, Validity, and Holistic Scoring: What We Know and What We Need to Know.

» Richard Haswell and Susan Wyche-Smith. Adventuring into Writing Assessment. from CCC Online

» Liz Hamp-Lyons and William Condon. Questioning Assumptions about Portfolio-Based Assessment. from CCC Online

Works Cited
Anastasi, Anne. Psychological Testing. 4th ed. New York: Macmillan, 1976.
Anderson, Richard c., and P. David Pearson. "A Schema-Theoretic View of Basic Processes in Reading Comprehension." Handbook of Reading Research. Ed. P. David Pearson. New York: Longman, 1984. 225-92.
Baurer, Barbara A. A Study of the Reliabilities and Cost Efficiencies of Three Methods of Assessment for Writing Ability. ERIC, 1981. ED 216 357.
Bleich, David. Readings and Feelings: An Introduction to Subjective Criticism. Urbana: NCTE, 1975.
---. Subjective Criticism. Baltimore: Johns Hopkins UP, 1978.
Braddock, Richard, Richard Lloyd-Jones, and Lowell Schoer. Research in Written Composition. Champaign: NCTE, 1963.
Breland, Hunter M. "Can Multiple-Choice Tests Measure Writing Skills?" College Board Review 103 (Spring 1977): 11-13, 23-33.
Breland, Hunter M., and Robert J. Jones. "Perceptions of Writing Skills." Written Communication 1 (Jan. 1984): 10 1-09.
Charney, Davida A. "The Validity of Using Holistic Scoring to Evaluate Writing: A Critical Overview." Research in the Teaching of English 18 (Feb. 1984): 65-81.
Collins, James L., and Michael M. Williamson. "Spoken Language and Semantic Abbreviation in Writing." Research in the Teaching of English 15 (February 1981): 23-35.
Cooper, Charles R. "Holistic Evaluation of Writing." Evaluating Writing: Describing, Measuring, Judging. Ed. Charles R. Cooper and Lee Odell. Urbana: NCTE, 1977. 3-32.
Crowhurst, Marion. "Syntactic Complexity and Teachers' Ratings of Narratives and Arguments." Research in the Teaching of English 14 (Oct. 1980): 223-32.
Davis, Barbara G., Michael Scriven, and Susan Thomas. The Evaluation of Composition Instruction. Inverness: Edgepress, 1981.
Diederich, Paul B. Measuring Growth in English. Urbana: NCTE, 1974.
Diederich, Paul B., John W. French, and Sydell T. Carlton. Factors in the judgment of Writing Quality. Princeton: Educational Testing Service, 1961. ETS RB NO 61-15.
Faigley, Lester, Roger D. Cherry, David A. Jolliffe, and Anna M. Skinner. Assessing Writers' Knowledge and Processes of Composing. Norwood: Ablex, 1985.
Fish, Stanley. Is There a Text in This Class? Cambridge: Harvard UP, 1980.
Freedman, Sarah W. "How Characteristics of Students' Essays Influence Teachers' Evaluation." Journal of Educational Psychology 71 (June 1979): 328-38.
---. "Influences of Evaluation of Expository Essays: Beyond the Text." Research in the Teaching of English 15 (Oct. 1981): 245-55.
---. "Influences on the Evaluators of Student Writing." DAI 37 (1977): 5306A. Stanford U.
---. "Why Do Teachers Give the Grades They Do?" CCC30 (May 1979): 161-64.
Freedman, Sarah W., and Robert C. Calfee. "Holistic Assessment of Writing: Experimental Design and Cognitive Theory." Research on Writing. Ed. Peter Mosenthal, Lynne Tamor, and Sean A. Walmsley. New York: Longman, 1983. 75-98.
Gebhard, Anne O. "Writing Quality and Syntax: A Transformational Analysis of Three Prose Samples." Research in the Teaching of English 12 (Oct. 1978): 211-31.
Gere, Anne R. "Written Composition: Toward a Theory of Evaluation." College English 42 (Sept. 1980): 44-48.
Godshalk, Fred I., Frances Swineford, and William E. Coffman. The Measurement of Writing Ability. Research Monograph 6. New York: College Entrance Examination Board, 1966.
Goodman, Kenneth S. Language and Literacy: The Selected Writings of Kenneth S. Goodman. London: Rouc1edge and Kegan Paul, 1982.
Greenberg, Karen. The Effects of Variations in Essay Questions on the Writing Performance of CUNY Freshmen. New York: CUNY Instructional Resource Center, 1981.
Grobe, Cary. "Syntactic Maturity., Mechanics and Vocabulary as Predictors of Quality Ratings." Research in the Teaching of English 15 (Feb. 1981): 75-88.
Harris, Winfred H. "Teacher Response to Student Writing: A Study of the Response Patterns of High School English Teachers to Determine the Basis for Teacher Judgment of Student Writing." Research in the Teaching of English 11 (May 1977): 175-85.
Hoetker, James. "Essay Examination Topics and Student Writing." CCC 33 (Dec. 1982): 377-92.
Holland, Norman N. The Dynamics of Literary Response. New York: Oxford UP, 1968.
Huot, Brian. "The Validity of Holistic Scoring: A Comparison of the Talk-Aloud Protocols of Novice and Expert Holistic Raters." Diss. Indiana U of Pennsylvania, 1988.
Iser, Wolfgang. The Act of Reading: A Theory of Aesthetic Response. Baltimore: Johns Hopkins UP, 1978.
Jones, Bennie E. "Marking of Student Writing by High School Teachers in Virginia During 1976." DAI 38 (1978): 3911A. U of Virginia.
Lyman, Howard B. Test Scores and What They Mean. 3rd ed. Englewood Cliffs: Prentice, 1978.
Markham, Lynda R. "Influences of Handwriting Quality on Teacher Evaluation of Student Work." American Educational Research Journal 13 (Fall 1976): 277-83.
McColly, William. "What Does Educational Research Say About the Judging of Writing?" Journal of Educational Research 64 (December 1970): 148-56.
Myers, Miles. A Procedure for Writing Assessment and Holistic Scoring. Urbana: NCTE, 1980.
Neilsen, Lorraine, and Gene Piche. "The Influence of Headed Nominal Complexity and Lexical Choice on Teachers' Evaluation of Writing." Research. in the Teaching of English 15 (Feb. 1981): 65-74.
Nold, Ellen W. The Basics of Research: The Evaluation of Writing. ERIC, 1978. ED 166 713.
Nold, Ellen W., and Sarah W. Freedman. "An Analysis of Readers' Responses to Essays." Research in the Teaching of English 11 (May 1977): 164-74.
Odell, Lee, and Charles R. Cooper. "Procedures for Evaluating Writing: Assumptions and Research." College English 42 (Sept. 1980): 35-43.
Popham, James W. Modern Educational Measurement. Englewood: Prentice, 1981.
Puma, Vincent D. "The Effects of the Degree of Audience Intimacy on Linguistic Features and Quality in the Audience Specified Essays of First Year College Students." Diss. Indiana U of Pennsylvania, 1986.
Scherer, Darlene L. Measuring the Measurements: A Study of the Evaluation of Writing-An Annotated Bibliography. ERIC, 1985. ED 260 455.
Sloan, Charles A., and Iris McGinnis. The Effects of Handwriting on Teachers' Grading of High School Essays. ERIC, 1978. ED 220 836.
Smith, Frank. Understanding Reading. 3rd ed. New York: Holt, 1983.
---. "The Writer as Reader." Language Arts 60 (1983): 558-67.
Smith, William L. Personal Correspondence. U of Pittsburgh, 1988.
Spandel, Vicki, and Richard J. Stiggins. Direct Measures of Writing Skill: Issues and Applications. Portland: Northwest Regional Laboratory, 1980.
Standards for Educational and Psychological Tests. Washington: American Psychological Association, 1974.
Stewart, Murray F., and Cary H. Grobe. "Syntactic Maturity, Mechanics, Vocabulary and Teachers' Quality Ratings." Research in the Teaching of English 13 (Oct. 1979): 207-15.
Stock, Patricia L., and Jay L. Robinson. "Taking on Testing." English Education 19 (May 1987): 93-121.
Tierney, Robert J., and P. David Pearson. "Toward a Composing Model of Reading." Language Arts 60 (May 1983): 568-80.
Vaughan, Carolyn. "What Affects Raters' Judgments)" CCCC Convention. Atlanta, Mar. 1987.
Veal, L. Ramon, and Sally A. Hudson. "Direct and Indirect Measures for Large-Scale Evaluation of Writing." Research in the Teaching of English 17 (Oct. 1983): 290-96.
White, Edward M. Teaching and Assessing Writing. San Francisco: Jossey-Bass, 1985.
White, Edward M., and Linda G. Polin. Research in Effective Teaching of Writing: Volumes I and II. Final Project Report to California State U Foundation. ERIC, 1986. ED 275007.
____________
Tags (?)

Works Citing