Formative assessments generally offer legal educators and law students a means of gathering pedagogically useful information. Our new Skills Diagnostic provides especially useful information as it is designed to better disentangle the intermingled effects of knowledge and skill upon performance. As such, it can better identify skill deficiencies and strengths. It is therefore a superior diagnostic instrument, which law schools and their students can use to their advantage.

Check out the full white paper with academic citations. 

The ABA’s Revised Standards and Summative Assessments

While law schools have long used summative assessments, Standard 314 posited the importance of jointly using summative assessments along with formative assessments. Admittedly, summative assessments provide meaningful feedback to students about the extent to which they have demonstrated a mastery of knowledge and skills. Nevertheless, these assessments are usually not well-positioned to improve learning. This is particularly--and most obviously-- the case when such assessments come at the end of a semester-long course.

The ABA’s Revised Standards and Formative Assessments

Formative assessments, in contrast, are intended to provide meaningful feedback to both professors and students about how to improve learning outcomes. In part, this is accomplished by using assessments during a course or at critical moments during a student’s legal education, when there is time for professor and student alike to adjust their teaching or learning practices, to change curricular or study emphases, or simply to revisit materials, concepts, and skills not yet sufficiently mastered. But, as importantly, this is accomplished by using assessments designed to inform students and their professors about underlying strengths and weaknesses across both substantive content areas and skills. This feedback is meaningful in a way that eludes summative assessments. It provides insights to students and professors whereby they can more knowingly and purposefully target study and practice efforts at specific areas and skills. In this way, formative assessments can empower learners and teachers, transforming learning challenges into learning opportunities and, in turn, learning deficits into learning accomplishments.

About Kaplan’s New Skills Diagnostic

Kaplan Bar’s new Skills Diagnostic is the result of a collaborative effort between Kaplan’s Academics and Learning Science Teams. The process began roughly two years ago, with an ideation phase in which the details and aims of a diagnostic redesign were contemplated and discussed. It will culminate with two new diagnostic instruments ready for use by our institutional partners in 2022. 

Our Skills Diagnostic actually consists of two aligned instruments. An institutional partner can opt to use one instrument, which it would administer annually, assessing a different cohort of students each year. Alternatively, an institutional partner can opt to use both instruments on an annual basis, with one administered to rising 2Ls and the other administered to rising 3Ls.

Each instrument consists of two components: a multiple-choice section and a writing section. Institutional partners will have the option of administering both components or administering only the multiple-choice section. In its MCQ-only version, the Skills Diagnostic will consist of 100 multiple-choice questions to be completed in three hours. In the version that includes both components, students will have an additional hour in which to complete a legal writing task. Both components are designed to assess core skills that students are expected to develop and master in law school, demonstrate on the bar exam, and competently implement as legal professionals.

The Multiple-Choice Skills Component

The multiple-choice component of the Skills Diagnostic is designed to assess the legal skills used to answer a bar-like multiple-choice question. Each of its multiple-choice questions ask students to answer a skills-focused question targeted at one of the following skills:

• reading comprehension,
• issue spotting,
• rule recollection,
• rule application, and
• evaluation.

These skills are essential to legal thinking generally and performance on the bar exam specifically. By targeting questions to a specific skill, we can generate a more robust data set concerning these skills both at the level of the individual student and at the level of the institutional cohort. In turn, students and law schools will be able to derive actionable steps more readily and confidently.

Conceived in this way, the multiple-choice questions on our skills-focused assessment are not intended to simulate a bar-like exam question in its totality. Rather, most of our Skills Diagnostic questions provide only excerpts from a bar-like multiple-choice question. As such, each question situates a student within a particular moment of answering a bar-like exam question, thereby simulating a moment when that student would rely upon a particular skill.

The Skills Diagnostic’s one hundred multiple-choice questions will be distributed equally across the five core skills, twenty questions per skill. But because legal skills cannot be readily demonstrated in a legal vacuum, each skills question will be situated within a context defined by one of five areas of substantive law: 

• Civil Procedure,
• Contracts,
• Criminal Law
• Real Property, and
• Torts.

Each of these subjects is appropriate for our purposes because each is heavily tested on the bar exam and is covered typically by a law school’s 1L curriculum. Along with the equal distribution across skills, the one hundred multiple-choice questions will be distributed equally across these five core legal subjects.  

The Writing Skills Component

The writing component of the Skills Diagnostic is designed to assess the legally pertinent skills used to perform a legal writing task. On this component, the specific skills to be assessed will include the following:

• language usage and organization,
• issue spotting,
• rule articulation,
• rule application, and
• concluding.

These skills largely correspond to those skills to be assessed on the Skills Diagnostic’s multiple-choice component.

Because this component is intended to be a skills-focused assessment of writing skills, it will present to students a writing task in a performance-test format. Like the performance tests offered by the National Conference of Bar Examiners and the California State Bar’s Committee of Bar Examiners, this testing format assesses legal writing skills in a setting where successful performance is largely independent of a student’s prior knowledge of substantive legal rules. This is accomplished by situating the student in a closed legal universe wherein the relevant legal rules of a fictional jurisdiction are provided to each student. With these rules accessible to them in a virtual file or library, students are directed to respond in writing to a legal supervisor’s task memorandum.

By relying on a performance-test format instead of a more traditional essay format, which requires the application of prior legal knowledge, our assessment of both a student’s and a cohort’s performance can more readily disentangle their legal knowledge from their legal know-how, their skills of memorization and recall from their analytical and writing skills. As a result, students and law schools will have more robust data from which action steps for improved performance can be more confidently derived and implemented.

Kaplan’s Skills Diagnostic is such a formative assessment, intended to be administered at key touch points during a law student’s educational journey so as to improve that student’s long-term educational outcomes. 

We believe that, with our new Skills Diagnostic, law schools will be able to more readily and confidently achieve better outcomes for their students.

Read the full white paper with academic citations.