Our Data & FAQs

Our Data


How does Quantum provide a valid and reliable assessment?

Ok, ready? Here comes some psychometric chatter that might be a tad overwhelming…

Quantum’s validity evidence spans eight facets of construct validity: content, substantive, structural, generalizability, external, consequential, responsiveness and interpretability (Messick, 1989, 1995; Medical Outcomes Trust, 1995). Regarding reliability and standard error, Quantum provides a direct estimate of the modeled error variance for each estimate of a learner’s ability, an item’s difficulty, and an instructor’s severity (Wright & Masters, 1982; Wright & Stone, 1979). These standard errors provide a quantification of the precision of individual person measure, item difficulty, and instructor severity, and can be used to describe the range (i.e., confidence intervals) within which each learner’s true ability, item’s true difficulty or instructor’s true severity falls.   Reference: Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measurement (3rd ed. pp.13-103), New York: American Council on Education and Macmillan.  Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.  Medical Outcomes Trust Scientific Advisory Committee. (1995). Instrument Review Criteria. Medical Outcomes Trust Bulletin, 1-4.  Wright, B.D. & Masters, G.N. (1982). Rating scale analysis. Chicago: MESA Press.  Wright, B.D. & Stone, M.H. (1979). Best test design. Chicago: MESA Press.

How does Quantum provide fair and objective ratings to measure my learners' competence?

Ok, so you clicked on it… here are more psychometrics for you!

For example, given a three dimensional matrix of data represented by: learners X items X instructors, the difficulty of item i is estimated based on the sum of all ratings on that item across all learners and all instructors. Likewise, the severity of instructors j is estimated based on the sum of all ratings comprising all items across all learners scored by that instructor. The person measure positions learners relative to each other after the additive effects of items and instructors specified in the model have been subtracted from the calculation. The resulting person measures are linear and also objective because they have been freed from the differences due to variations in items and instructors (Linacre, 1994). Reference: Linacre, J.M. (1994). Many-faceted Rasch measurement (2nd ed.). Chicago: MESA Press.

FAQs


What is a scale score?
A scale score is created when the number of questions the learner performs correctly is mathematically transformed. The scale score ranges from 100 to 500. This transformation is very similar to converting inches to centimeters; for example a 10 inch ribbon will be 25.4 centimeters long. The length of the ribbon has not changed, only the units of measurement used to describe it.  
Why scale the scores?
The use of scale scores allows for direct comparison of the scores from one assessment to another because the standard will always be the same. This scale score was selected to represent a mean and standard deviation of 300 and 50. In the long run, this process will make the scoring of Quantum assessments easier to understand for all concerned. It will not matter which Quantum assessment a learner takes, the score range will remain constant.  
How is passing determined?
The final passing score is established by the organization’s Examination Committee using a criterion-referenced passing score process. During this process, the Examination Committee members discuss the acceptable level of performance required for passing the assessment and assigned stake. This discussion focuses attention on specific knowledge, skills, and attitudes the learner should possess at the conclusion of each course. For example, a School of Nursing may assign stakes to an acceptable level of performance as a requirement for promotion to the next class. The Examination Committee then systematically evaluates each scenario and corresponding rating sheet and rates its difficulty levels. These ratings are combined to arrive at the passing score for that specific scenario. No explicit stakes will be assigned by Objectivity Plus to individual learner results because stakes are determined by the organization, local districts, or states; not by test publishers.  
Can I find out how many question I answered correctly?
Objectivity Plus will not report the number of items or overall percentage of items performed correctly. However, the learner will be provided with scale scores for each subdomain of the assessment to help target areas of further study. It is important to note as the number of items contained in each subdomain category of the assessment varies, adding or averaging scale scores will not be an accurate reflection of your overall scale score.  
Does scaling the scores affect who passes or fails an assessment?
No. The use of scale scores does not affect if a learner passes or fails the assessment.  
Why is the scale 100 to 500 when I didn't see 400 questions?
The scale used by Quantum is similar to scales used by other testing programs such as the SAT, ACT, or GRE. The upper and lower bounds were selected because an interval of plus or minus three standard deviations contains more than 99% of a population. The scale has more points on it than there are questions on the assessment.  
What is included in Quantum?

 

The system consists of:

  • Wireless devices (e.g., iPads or Tablets): The number of wireless devices will be contingent upon the number of concurrent courses run each semester or quarter based upon information provided by the organization.
  • Test administration schedule: This feature meets the linkage requirement so every element can be compared directly and unambiguously with every other element, providing precise and accurate measures of all parameters in a shared frame of reference.
  • Course specific scenarios for RN and PN programs: Learning objectives listed for each scenario with flexible scenarios that can reflect three difficulty levels.
  • Rating sheets: Based on the scenario’s content domain of reference, each rating category is listed with information about specific criterion behaviors appropriate for the scenario in use. From a legal aspect, each rating sheet reflects expected outcomes, recorded outcomes, and reflects the differences between expected and actual outcomes based on the learner’s performance.
  • Online registration site: Institutions may add an additional fee for online purchases. Any additional money collected will be returned to the institution at the conclusion of each semester or quarter.
  • Remediation: This is based on each scenario’s content domain of reference and leveled by scenario difficulty.

Optional inclusions:

  • Set-up: A trainer may be requested to assist with the set-up of Quantum. A fee is associated with this service.
  • Psychometrics: A method to gather and construct validity and reliability evidence to support the use of your organization’s scenarios. A fee is associated with this psychometric service.