How does Quantum provide a valid and reliable assessment?
Ok, ready? Here comes some psychometric chatter that might be a tad overwhelming…
Quantum’s validity evidence spans eight facets of construct validity: content, substantive, structural, generalizability, external, consequential, responsiveness and interpretability (Messick, 1989, 1995; Medical Outcomes Trust, 1995). Regarding reliability and standard error, Quantum provides a direct estimate of the modeled error variance for each estimate of a learner’s ability, an item’s difficulty, and an instructor’s severity (Wright & Masters, 1982; Wright & Stone, 1979). These standard errors provide a quantification of the precision of individual person measure, item difficulty, and instructor severity, and can be used to describe the range (i.e., confidence intervals) within which each learner’s true ability, item’s true difficulty or instructor’s true severity falls. Reference: Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measurement (3rd ed. pp.13-103), New York: American Council on Education and Macmillan. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749. Medical Outcomes Trust Scientific Advisory Committee. (1995). Instrument Review Criteria. Medical Outcomes Trust Bulletin, 1-4. Wright, B.D. & Masters, G.N. (1982). Rating scale analysis. Chicago: MESA Press. Wright, B.D. & Stone, M.H. (1979). Best test design. Chicago: MESA Press.
How does Quantum provide fair and objective ratings to measure my learners' competence?
Ok, so you clicked on it… here are more psychometrics for you!
For example, given a three dimensional matrix of data represented by: learners X items X instructors, the difficulty of item i is estimated based on the sum of all ratings on that item across all learners and all instructors. Likewise, the severity of instructors j is estimated based on the sum of all ratings comprising all items across all learners scored by that instructor. The person measure positions learners relative to each other after the additive effects of items and instructors specified in the model have been subtracted from the calculation. The resulting person measures are linear and also objective because they have been freed from the differences due to variations in items and instructors (Linacre, 1994). Reference: Linacre, J.M. (1994). Many-faceted Rasch measurement (2nd ed.). Chicago: MESA Press.
What is a scale score?
Why scale the scores?
How is passing determined?
Can I find out how many question I answered correctly?
Does scaling the scores affect who passes or fails an assessment?
Why is the scale 100 to 500 when I didn't see 400 questions?
What is included in Quantum?
The system consists of:
- Wireless devices (e.g., iPads or Tablets): The number of wireless devices will be contingent upon the number of concurrent courses run each semester or quarter based upon information provided by the organization.
- Test administration schedule: This feature meets the linkage requirement so every element can be compared directly and unambiguously with every other element, providing precise and accurate measures of all parameters in a shared frame of reference.
- Course specific scenarios for RN and PN programs: Learning objectives listed for each scenario with flexible scenarios that can reflect three difficulty levels.
- Rating sheets: Based on the scenario’s content domain of reference, each rating category is listed with information about specific criterion behaviors appropriate for the scenario in use. From a legal aspect, each rating sheet reflects expected outcomes, recorded outcomes, and reflects the differences between expected and actual outcomes based on the learner’s performance.
- Online registration site: Institutions may add an additional fee for online purchases. Any additional money collected will be returned to the institution at the conclusion of each semester or quarter.
- Remediation: This is based on each scenario’s content domain of reference and leveled by scenario difficulty.
- Set-up: A trainer may be requested to assist with the set-up of Quantum. A fee is associated with this service.
- Psychometrics: A method to gather and construct validity and reliability evidence to support the use of your organization’s scenarios. A fee is associated with this psychometric service.