Pre-Med Myths & Tips: Breaking Down the Realities of a Truly Transparent Score Improvement Claim
by Petros Minasi, Jr., Sr. Director, Pre-Health Programs | July 13, 2021
Pre-meds studying for the MCAT are understandably concerned with how much their score is going to improve after studying, and you can’t blame them—the MCAT is a challenging exam. And more so, while it is just one factor in the admissions process, the MCAT score is still weighed heavily by medical school admissions officers. Year after year, our medical school admissions officers survey shows that the MCAT is the most important admissions factor. So, while there is a lot of pressure to perform well on the MCAT, the problem students have in studying for the MCAT is this fixation on improvement. The reality is that there is no place on the medical school application for score improvement, only a spot for the final MCAT score. When students ask me how to get a higher MCAT score, my answer is quite simple: “Get more questions right. Don’t focus on the score; focus on understanding the content, learning its application, and practicing your execution, and the score will follow.”
But with that said, sadly there is still a lot of noise out there with grand, flimsily substantiated claims about “guaranteed score improvements” or “typical score increases.” But if you dig deeper into these claims, you will very quickly see how specifically engineered they are to make that prep company’s offerings sound like the magic pill that will boost a student to a competitive MCAT score. So, today we are going to discuss the myths and realities of what goes into a truly transparent score improvement claim.
Myth: Sample size and representativeness don’t matter.
Tip: Dig deeper into any claims that seem too good to be true to ensure that the sample demographic is truly representative of the population.
Basic statistics tells us that the larger and more representative the sample size, the more accurate and valid the conclusions we can draw from that sample. This concept has been in the news quite a bit this past year. In the early reports of vaccine trials, there were a lot of doubts raised when claims were being made about certain vaccines’ effectiveness because the participants weren’t necessarily representative of the population as a whole. Only as the sample size got larger and more representative did the results become more convincing.
The same is true with MCAT score improvement. One could easily find a small handful of students that showed great improvement, and claim that a given program raises scores by, say, 15 points. But a company making such a claim may be hiding more than they are sharing. When you’re looking at a small sample that shows big results, it’s quite likely that many of the students in that sample share certain attributes that may not be representative of students in general: for instance, many might have had a significant academic deficiency at the onset, or maybe had never learned certain content areas. It actually wouldn’t be a surprise then if these students made huge leaps in their scores, because effectively they were starting from scratch in those areas. But if you have a larger, more representative sample with students spread across starting ability levels, then your results become more compelling. Unfortunately, it’s very easy to cloud the reality of the data you are sharing.
Myth: Score improvement claims always compare scores across comparable exams.
Tip: The only way to truly measure improvement is to measure it against a comparable standard.
To be truly transparent, you can’t say, “For your initial assessment, we are going to look at how fast you run 10 miles, but for your final assessment we are going to look at how fast you run 1 mile, and then multiply by 10.” So much can happen in those 9 miles that the only way to see the true improvement is to have you run 10 miles, in similar conditions, for your final assessment. Same goes for the MCAT.
In psychometrics, which is the science of assessment creation, we say that the two tests you’re comparing need to be based on the same scale and equated to one another—this scaling and equating is what allows you to compare students who take different forms of the same test. But often in these faulty score claims, you’ll see companies use a diagnostic test they wrote themselves and that is shorter than the real thing, and then compare that to a self-reported score on the official MCAT. Always be skeptical if the initial and final assessments being compared are not based on the same standard, period.
Myth: How the population is qualified doesn’t matter.
Tip: Transparency about how the results were gathered, calculated, and presented is key to an accurate claim.
What often happens with misleading score improvement claims is that the population is sorted down according to a favorable set of circumstances to arrive at the boldest possible claim. For instance, we could start by surveying a population of 500 students, only hear back from 75 of them, get actual test results from about half of those, then exclude anyone who is a repeat test taker or didn’t have an initial score on record, and end up with only about 30 students that we then use for the study.
Since official MCAT scores are not available to test preparation programs, they must be self-reported by students, and this is an acknowledged source of respondent bias as students who did exceptionally well are clearly more likely to submit their scores than those who didn’t. With the added requirement of including the score report, the bias likely becomes even more skewed toward the high scorers who are likely less fearful of copies of their official score report floating around the web.
With all this said, rest assured that when Kaplan makes a score improvement claim, we will have a representative sample of students, truly comparable test scores that we are comparing, and a transparent process for conducting the analysis and presenting the results. We know that success on the MCAT is directly linked with the quality of work and time a student invests in their studies—it’s not about the number of hours, it’s about specifically diagnosing and targeting one’s areas of opportunity, learning from mistakes, mastering the application of the content, executing in the amount of time afforded by the exam, and building endurance. And of course, getting more questions correct on Test Day!
After more than two decades at Kaplan, I am often asked, “What keeps you here?” The answer is simultaneously simple and complex, but for now, I will keep it simple: it’s the people I interact with―advisors, students, and my colleagues, and the opportunities our programs and services open for students.
See more posts by Petros Minasi, Jr., Sr. Director, Pre-Health Programs