Significant Questions Raised regarding the Setting of Achievement Level Cut Scores
On September 4, 2018, DESE released an administrative memorandum regarding the process for establishing achievement level cutoff scores for assessments. The memorandum outlines the process that is “not new” and requires substantial input from Missouri educators. DESE has turned control of this process over to private, third party vendors who have limited the input from Missouri educators. Missouri teachers and administrators involved in the process at the bookmarking and policy committee levels have raised substantial issues with the integrity of the process. At this point, school administrators should consider what actions, if any, should be taken before DESE leadership establishes achievement level cutoff scores for ELA or Math.
The process requires three phases prior to a recommendation to DESE leadership: (1) bookmarking by Missouri teachers to set cutoff scores; (2) a DESE performance level “policy committee;” and (3) review by an assessment technical advisory committee. Significant concerns have been presented regarding each phase as identified below.
I. Bookmarking: Concerns Raised by Teachers Participating in Bookmarking were Ignored.
As DESE’s memo points out, the first phase of the process for setting achievement level cutoff scores for assessment requires bookmarking by Missouri educators. In this process, Missouri educators review actual test questions arranged sequentially from least to most difficult, then recommend the questions that represent the cutoff scores that separate one level from the next.
For ELA and Math this year, DESE turned control of the bookmarking process over to a private Minnesota corporation, Data Recognition Corporation (DRC). DRC paid Missouri teachers to participate in the process. DRC representatives made clear they did not support collaboration among the teachers. When teachers raised concerns about the process, the teachers were told it was not their job to concern themselves with anything other than the placement of cutoffs based upon the sequencing of the questions provided. For example, for Math and ELA, teachers raised concerns that the ordering item booklets (OIB’s) contained test items that were not in order from the least to the most difficult and as a result, would skew the placement of the cutoff scores. DRC personnel informed teachers that they should not concern themselves with anything other than to simply identify the cutoffs at the question they believed appropriate.
Notably, for the bookmarking process to be valid, the rationale and procedures used for establishing cutoff scores must be clearly documented and the “judgmental process” for participants setting cutoff scores must “be designed so that participants providing the judgments can bring their knowledge and experience to bear in a reasonable way.” See Standard Setting Process and Methodology, 8.1.1, DRC Technical Report 2017 to DESE. Here, teachers raised concerns yet they were not permitted to bring their knowledge and experience to bear in a way that allowed cutoff scores to be appropriately set. Teachers have reported that DRC disregarded and dismissed these concerns without examination.
II. Policy Committee: Questions Raised by Educators at the “Policy Committee” Remain Unanswered.
Like the bookmarking process, DESE turned control of its “Policy Committee” to a private, third party vendor. The Missouri Assessment Program Performance Level Policy Committee was chaired by Dr. Karla Egan of EdMetric, LLC, a psychometrist. The committee is composed of representatives from private industry, higher education, two superintendents, three principals and a data director from a public school district. The DESE policy committee meeting was considered a “private meeting” run by a private contractor and participants were given little to no notice of the information to be discussed.
At the policy committee, the private contractor provided her conclusion that Missouri achievement levels are inadequate when compared to ACT data and there must be greater consistency with the ACT. When school administrators questioned this conclusion, and asked for historical or trend data supporting her conclusions, the private contractor refused to provide any such supporting data. The private contractor also refused to provide any supporting data tying the 9th Grade EOC scores to the ACT scores that students would obtain years later. It was clear to administrators on the committee that the private contractor, who is a member of the next level of review (Assessment Technical Advisory Committee), sought a particular result, regardless of the input from Missouri K-12 educators.
III. Assessment Technical Advisory Committee (TAC): The Assessment Technical Advisory Committee Should Investigate the Concerns Raised by Teachers and Administrators.
As the DESE memo explains, the next level of review for setting cutoff scores is conducted by the Assessment Technical Advisory Committee, a “group of nationally recognized experts in large-scale assessment, measurement, and related issues” that “meets three to four times each year.” Their charge from DESE is to “review all aspects of our assessment content and results. They review recommendations from the bookmark work sessions and policy committee discussion and offer guidance for establishing achievement level cutoff scores on new assessments.”
Although this Committee is to meet three to four times each year, the DESE website reflects that the “last TAC Meeting Update” was the agenda for the August 17-18, 2017 meeting. Requests for all notices and agendas for 2018 meetings of this committee have been met with delay and objections from DESE. The policy committee that met on August 21, 2018, was informed at the meeting that final cutoff scores for Math and ELA would be set by September 18, 2018. One item listed for discussion at the September 18, 2018, State Board of Education meeting is a “Report on 2018 Missouri Assessment Program (MAP) Grade Level and End-of-Course Performance Levels and Preliminary Statewide Results in English Language Arts and Mathematics.”
Proposed Action Steps:
Given the high stakes implications of this process, school administrators should carefully review the process employed by DESE in establishing cutoff scores for achievement levels. As deemed appropriate, school administrators should also consider:
- Engaging the teachers who participated in the bookmarking process to determine any concerns they had regarding the process and making any such concerns known to DESE.
- Engaging the administrators who participated in the policy committee to determine any concerns they had regarding the process and making any such concerns known to DESE.
- Requesting that the Assessment Technical Advisory Committee consider all concerns raised by Missouri educators as part of the bookmarking and policy committee processes.
- Requesting reduced direct control over the process by third party, private contractors, who should be provided open, written and clear direction from DESE in their work.
- Calling for an independent investigation that carefully reviews the process and recommendations provided at each level of review (bookmarking, policy committee, TAC, DESE leadership) and determine the manner to make the process of setting standards an open, fair and credible process in the future.
It is critical that public school administrators engage at this point. Without increased scrutiny, the course we are currently on will be followed and all should understand it is a path that leads to a particular result, but one that may not be truly justified. If it is justifiable, clear justification, and the details of the process used to arrive at that justification, should be provided for all to see. If it is not justified, we are all going to experience a substantial adverse impact to our schools in a way that doesn’t really help kids at all.