Using Psychometrics and Learning Science Analysis to Enhance Nursing Education

by John V. Harnisher  Ph.D., Vice President, Psychometrics and Learning Analytics | October 27, 2020

After spending the last 7 years working in the world of learning science, analytics, and psychometrics at Kaplan, lately I have begun to think about the true reason that investing the time and energy on improving our products based on best practice is so crucial to learning―and the field of nurse education specifically. At the heart of learning is the question of trust. In order to progress toward any degree, one needs to trust their learning environment. The instructor, written materials, and digital assets all need to convey a deep level of credibility in order to allow for an optimal learning experience. 

Reliability & Validity

As you know, the most basic requirement of an assessment is that it be reliable and valid, and we rely upon Kaplan’s team of psychometric experts to ensure this is the case. 

In order to feel secure when teaching material, most programs rely on benchmark exams and quizzes. Programs need to know if their students have mastered the material presented to them, and assessments are the most standard tool for that purpose. 

Assessments that indicate mastery of material should yield results that are similar within certain time intervals in order to be reliable. For example, if a student has mastered a set of material related to patient care and received a high score on an assessment of that material on a Monday, she should get the same high score on that material on a Tuesday. This notion of reliability is critical in order for learners and educators to trust the results of assessments and feel secure that the material has been learned.

The other critical component of any good assessment is its validity. The items on the exam must be related to the topic at hand, and the scores on the assessment should be related to other measures of mastery in that topic area. We need to be sure we are measuring what we intend to. It is only when both reliability and validity exist for a measure that trust will be felt by users of that measure.

Benchmarking

The Nursing program at Kaplan has some of the most critically important assessments in our company portfolio. These Integrated Exams function in a variety of ways within a nursing school’s curriculum, including being used as summative exams to check student progress within a subject area. 

Since it is critical to understand how well a nursing student is gaining mastery of material, as well as to compare their performance against other students, the items on these exams need to be field tested and reviewed for item quality. The field testing process as performed by the Kaplan team is detailed and specific. Multiple data are collected and analyzed on every test item. Once that has occurred, norming tables for comparison purposes are created. 

These are the critical functions of the psychometrics team, who work together closely with our data science and nursing subject matter experts to bring all of these components together. The data analysis is only the first step in the process since these subject matter experts―nurse educators―also review each item's performance.

In the Spring of 2020, it became necessary for our partners to administer our Integrated Exams in a remote setting during the peak of the COVID-19 pandemic, which resulted in many of the exams and items becoming overexposed. Our teams have worked tirelessly to update the 9 most critical of these exams (consisting of over 900 new test items) to ensure that we provide updated, secure, stable, reliable, and valid assessments that our individual nursing students and institutional partners can trust. 

Predicting Future Success

One of the most critical functions of testing our nursing products is the ability to use performance measures on an assessment to predict future performance―and even the likelihood of passing the NCLEXⓇ. The key to having predictions that are stable and useful is to collect outcome data that is of the right quality and quantity. Our Secure Predictors are exams which do just that. In 2019, our psychometrics and learning analytics team worked together to collect outcome data from over a thousand nursing students in 21 states. After cleaning, data analysis was conducted that connected performance on these Kaplan-created exams and the likelihood of passing the NCLEX-RN (or NCLEX-PN). 

Additionally, we collected 900,000+ data points during the field testing of Kaplan Nursing’s Computer Adaptive Test (CAT), which allowed us to identify that CAT practice tests are a strong indicator of NCLEX success.  Based on data from 475 Kaplan Nursing NCLEX-RN CAT enrollments, from 27 academic programs/institutions in 14 different states, we identified that students who passed their first CAT were 5 times more likely to pass the NCLEX-RN, and students who passed any CAT attempt were 7 times more likely to pass the NCLEX-RN.*** 

At Kaplan, we remain vigilant in continuously analyzing, testing, and improving our products so that at the core they can be trusted. To learn more about Kaplan Nursing’’s learning science-backed pedagogy and our solutions for nursing institutions, request a demo today.

John Harnisher is Vice President of Psychometrics and Learning Analytics at Kaplan. In his role, he is charged with further developing and advancing Kaplan's capabilities in delivering industry-leading, world class learning programs for our students and partners and leading teams in learning analytics and psychometrics. An experienced practitioner and researcher in the fields of data modeling, analytics and assessment, John strives to ensure that the latest discoveries in learning are applied to Kaplan products. John holds a Ph.D. in Psychology from New York University and a B.A. in Psychology from Connecticut College.

See more posts by John V. Harnisher  Ph.D., Vice President, Psychometrics and Learning Analytics