Interview With Dr. H

Interview with Dr. Stephen Hetherman

In July and August of 2017, two articles were published about Quantum, a new performance assessment system; the first in the Journal of Nursing and Healthcare Development of Quantum, the second in Medical Training Magazine Bridging Data.  We had a chance to sit with Dr. Hetherman, founder and lead psychometrician for Objectivity Plus, to ask him a few questions about Quantum.

Dr. Hetherman, you’ve recently had two articles published about your performance assessment system, Quantum. Knowing there are other methods to assess performance, why is this assessment tool so significant, and how do these recent papers address that?
Both articles speak to the construct validity evidence that was collected, to support the intended interpretation of test scores for specific uses. This offers a solution to obtaining error-adjusted test scores, since every test score, regardless of how it is measured, includes some level of fallibility, which can come from a variety of sources.
Quantum’s capability to estimate and adjust for instructor bias and test item variations is a significant step forward to objectively measuring an individual’s competence. This is a distinct advantage over other performance assessments offered in the industry or talked about in the literature to measure competence. Other tools do not model all the variables of the performance assessment and do not account for their effects on learner’s test scores.
It sounds as though a performance assessment tool is valuable and greatly needed. Why aren’t more schools and/or hospitals using an assessment tool or methods to measure the performance of students and staff?
Objectively evaluating an individual’s competence can be a very challenging task, whether it is in a clinical setting, the simulation lab, or for that matter, practically anywhere. The subjective nature of clinical evaluation in education and clinical practice stems from having instructors or clinical experts rate an individual’s performance. The variation from one rater to another significantly complicates the measurement process. A review of the literature shows that, until now, there were no psychometrically-sound performance assessment tests available to measure a learner’s clinical competence across all three domains of learning; knowledge, skill and attitude.
We developed Quantum to specifically address the many challenges and issues associated with a performance assessment that measures an individual’s competence. Quantum makes it much easier for faculty or facilitators who are responsible to evaluate a learner’s performance, to differentiate competency, and then to assign grades. Quantum ensures that the rater will have confidence that he or she is providing a consistent and objective evaluation, ensuring that students or staff are safe and competent.
It seems as though there could be other applications for using this system. Where else can this performance assessment tool be applied?
Healthcare has many possible future applications and Objectivity Plus is nimble enough to work with various groups, including Schools of Medicine, Inter-Professional Education (IPE), and Doctor of Nursing Practice (DNP), and Family Nurse Practitioner (FNP) programs. IPE is very interesting. This year Objectivity Plus is collaborating with academia and visionaries to improve patient safety given the gaps in inter-professional communication which account for a large percentage of medical errors.
Outside of healthcare, we have begun work in the sports industry. Consistency among officials has eroded and nothing is more aggravating to a coach than inconsistent officiating. We have a proprietary algorithm to quantify each official’s overall consistency, and his or her consistency over any given season.  A second venture is to measure a player’s football intelligence. Objectivity Plus believes it can assist a team’s coaching staff’s by gathering additional data points on their recruits, draftees, or free agents to ensure a proper fit to the coaching staff’s offensive or defensive schemes.  In short, get that player on the field.  In theory, this can be applied to any sport.
Many sim labs are using simulation mostly as a teaching tool, in order to provide a safe environment for their participants, and not necessarily as a place for summative or “high stakes” assessments. How would Quantum work in those settings?
When simulation is used as a teaching tool, Quantum can be used as a formative assessment to meet this approach. Quantum can provide formative feedback to the learner with the intent to improve learning and future performance. The research proves that the development of competence is dependent upon learners receiving formal feedback as part of a continuous assessment cycle.  Objectivity Plus is passionate about continuous learning, and Quantum was developed with that in mind. This tool goes a step further than just providing the assessment. We provide each learner with customized remediation which enables them to identify their strengths and weaknesses, and then show them how to improve on items where they are weak or need to build upon what they do best.  Ultimately, Quantum provides a holistic view of the learner in either formative or summative experiences.

“Quantum’s capability to estimate and adjust for instructor bias and test item variations is a significant step forward to objectively measuring an individual’s competence.”

The healthcare industry is always looking for ways to achieve optimal patient outcomes. Is there an opportunity for this assessment system to be used to accomplish this task?
Yes, absolutely!  The Joint Commission (TJC) noted that healthcare organizations have historically relied on education and experience to support competence, but an increasing number of healthcare organizations are seeking objective measures of a nurse’s knowledge that is required for safe practice.  We believe there is more to the practice of nursing than just knowledge… it IS a “hands on” vocation! Since Quantum measures all three domains of learning, the educator can obtain a holistic view of the individual.  The anticipated benefit of using Quantum’s objective performance assessment’s data is that the healthcare organization will work to improve what it measures; it simply can NOT manage what it doesn’t.  Whether the goal is to drive an educational intervention to improve safe practice or to safely decrease the length of orientation for new employees, a healthcare organization that uses simulation for teaching and learning reflects their presumption that learning is taking place and that the teaching is effective.  How can you be assured that learning is taking place if you don’t measure it objectively?  In healthcare, Quantum operates along several verticals: (1) educators like it because it promotes learning and provides a basis of competency-based education; (2) H.R. likes it because it evaluates individuals and identifies exceptional (future leaders) or poor performers; and (3) Risk Management likes it because it permits the documentation of competencies.