How Real-Time Performance Data Helped Students at Different Levels of Academic Performance
Improving student learning outcomes using performance data is vital to the growth and development of post-secondary students. Students are increasingly seeking personalized information and feedback to guide their self-directed learning efforts and academic choices. The application of performance data can look different from student to student depending on their strengths, weaknesses, and goals, but the intention is the same – to improve student learning outcomes (See Table 1).
At The Ohio State College of Medicine (OSU), students are given detailed reports, made possible by the computer-based testing and reporting system provided by ExamSoft Worldwide, Inc., and a Tagged Electronic Database of Exam Questions (TEDEQ) developed by OSU. By utilizing a TEDEQ system, the lone measure of student performance is no longer their total score on the assessment, but rather a detailed breakdown of performance by question type, subject matter, question author, subtopic area, etc. This data is generated both on an exam-by-exam basis and longitudinal basis, and provided to the student in frequent intervals.
The usage of TEDEQ data and reports by students varies greatly. The academic standing of a student is a large determinant of how to get the most value from the information TEDEQ reports present. The strong student, average student, and struggling student all interpret performance differently. The real-time reporting data provided to the student means different things to each of them. Thus, to illustrate how these reports are used by students, we describe how three students in varying academic standings utilized student performance data and benefited from this information. These archetypes are based on actual OSU students; for both privacy and illustrative purposes, the specific data have been disguised.
Student A: Underperforming
Often, the struggling students have the most time and resources dedicated to improving their performance, because these students are the most at risk of not meeting the minimal competency requirements of their program. The need for improvement from the perspective of these students can be overwhelming. Student A is a student who has had difficulties reaching minimum required competency throughout the first year of medical school (See Exhibit 1 – orange box). Originally, the student ignored the performance feedback and focused on utilizing different study methods and increasing the amount of time spent studying. Only after those methods failed, did the student approach assessment and evaluation personnel for help in understanding the student’s performance and how to utilize the information.
“The ability to have information that outlines a student’s strengths and weaknesses enables our academic counseling and tutoring to be far more effective. Previously, we relied on students to tell us what was wrong so we could help them fix it. Part of the problem is they were frequently not accurate in their self assessment, which hindered our ability to intervene.”
– Dr. Dale Vandre, Director Integrated Pathway Program
Student A discovered that he was performing well in subject areas where the content lent itself easily to memorization, while struggling with subject areas that required information to be synthesized for understanding. Student A had been attempting to improve in these areas by memorizing more facts, rather than focusing on how to relate the information to other processes. This led Student A to pursue assistance from an academic counselor for help in developing study methods that would make content connections rather than memorization of fact.
Student A also began to better understand how to prioritize learning materials based on what was important enough to be assessed on exams versus what was secondary detail. Over time, Student A began to consistently meet required competency levels across the curriculum and was able to shift focus from meeting the minimum to finding ways to develop advanced areas of competency (See Exhibit 1 – green box).
Student B: Average
Student B scored consistently between the class average (~85%) and one standard deviation above the class average (See Exhibit 2). While at no risk of not meeting minimum passing requirements, Student B approached faculty to attempt to identify if there were specific content areas in which additional focus could improve performance on the 10-15% of material that was consistently being missed.
When they looked at the student’s longitudinal performance across all exams completed, it showed multiple explanations. The student was scoring well in recall questions and was scoring well on clinical science vignettes, so the ability to synthesize information was not an issue. However, the student was scoring poorly on basic science vignettes. Student B was not approaching basic science learning materials with a patient centered mindset. Thus, when the basic science material was presented in the context of a clinical scenario, the student was not recognizing the intention and focus of the question. (See Exhibit 2)
Additionally, Student B’s subject performance breakdown indicated poor performance in Biochemistry. In light of this discovery, the student utilized the breakdown of performance by question author to identify which faculty members to approach for additional understanding of the material. Combined with the feedback about performance on basic science vignettes the student was able to improve his scores in the trouble areas by applying a more clinical approach to the study of biochemistry with the help of the specific faculty.
Student C: High-Performing
The performance data generated not only is a tool to help students improve their learning outcomes, but it is also a valuable tool to help the highest performing students, by highlighting their strengths. Student C consistently performed at the top of the class. While occasionally a
topic or subject would come up that required some increased attention, for most part the student did not need to focus on performance improvement (See Exhibit 3). Instead, Student C utilized performance feedback to identify subject areas of greatest strength for further exploration. Analysis of the student’s longitudinal performance showed the most strength in the areas of Neurology and Pathology. These were subjects of focus the student had not considered before as potential areas of specialty or career pursuit.
As a result, the student pursued a position doing neurological research in a lab on campus during the summer between the med 1 and med 2 years. The experience was a very positive one for Student C, who became very interested in Neurology as a specialty during residency. As a result, Student C is approaching the clinical years of medical education with this goal in mind and choosing electives and extracurricular activities to best prepare for pursuing this specialty. (See Exhibit 3)
All students approach learning differently; they have varying needs and degrees of success. Regardless of these intricacies, real-time and specific student performance data is an invaluable tool for assisting them in reaching their academic goals. The personalized reports generated with the ExamSoft software, in conjunction with the TEDEQ system, provide a broad, clear picture of performance, and remove the guesswork from self-directed learning. Moreover, they empower faculty and administrators to target their interventions, and more effectively and efficiently (i.e. with less time) impact individual learning outcomes.
There are a multitude of ways students find to take advantage of performance feedback. These examples highlight just a few of the methods students use to interpret this data and apply it to their academic careers. As usage of this data by administrators and faculty becomes more tightly integrated with support systems, counseling, and intervention methods, impact from employing this information for those purposes will be measured.