Improving Retention and Remediation
Using a Data-Driven Feedback System to Fundamentally Alter Institutional Economics
Tuition rates have increased above the rate of inflation for three decades, and within the past ten years, they have skyrocketed. While it is clear this trend cannot continue indefinitely, educational institutions must rely on revenue flows in order to operate—but few know how these revenue streams can improve if tuition rates must plateau.
In this paper, Dr. David Lenihan of the Touro College of Osteopathic Medicine (TouroCOM) discusses how leveraging embedded assessments for remediation purposes has drastically altered the economic makeup of TouroCOM. In fact, in three years’ time of using ExamSoft’s solution in conjunction with a new dynamic curriculum, Dr. Lenihan and his team have brought board passage rates up from 70% to more than 90%, attrition rates down from 8% to 2%, and the school reclaims $2.2 million dollars per year in lost revenue.
Part I – Background
For nearly thirty years, and especially within the last decade, tuition rates at educational institutions have increased above the rate of inflation. This is an unsustainable process that cannot continue indefinitely; at some point, tuition rates must plateau if institutions are to continue to draw students. Still, like all entities that rely on cash flow in order to exist, institutions must continue to bring in revenue. But how does an educational institution improve revenue streams without increasing tuition?
Touro College of Osteopathic Medicine (TouroCOM) is a medical school situated in the heart of Harlem, NY. The two-fold mission of the school is to increase the number of primary care physicians in underserved communities, coupled with a desire to improve the diversity of physicians within the medical profession. When TouroCOM opened six years ago, a traditional, static curriculum was developed that was faculty-driven. The faculty wrote the syllabus, created and graded the exams, set the “curve” for these exams, and then posted the results to Blackboard.
Oversight of this process was done via the curriculum committee and administration at the end of the year. It was obvious early on in the development of TouroCOM that this process was not adequate. With its student board passage rate around 70%, its student dropout rate at approximately 8%, and a limited applicant pool for admissions, it was clear that a change had to occur. Furthermore, student feedback about the curriculum was described as disorganized and inefficient.
To meet this mission and to improve the existing curriculum, TouroCOM has developed a series of changes within Touro’s existing medical curriculum. Developing a new, dynamic curriculum included measuring student outcomes on objective learning on a daily basis, providing students with the objectives they are expected to learn each day at the beginning of the academic year, and being one of the first medical schools in the United States to deliver high-stakes exams with a computer-based testing platform.
Part II – Implementation
A new dynamic curriculum was developed using a triangle approach of lectures, exams and assessment (in healthcare—called the LEAH method) with outcomes from student performance in the center driving all aspects of the curriculum. In developing this new curriculum, TouroCOM administrators looked at many different educational models. Some of these models did very well in one part (lectures, exams, or assessment) of the triad, but were limited on the other two. Even fewer models looked at student outcomes as the main output of the system, and administrators were unable to find any model that looked at all three sides of the triangle as equals.
The first step in the implementation of the new approach was to establish objectives for each lecture given to the students throughout the entire medical school curriculum for the upcoming year. Before classes began, learning objectives for courses, as well as for each day the student would be in school for a course, were set and written in the syllabus. The syllabi for all courses were then compiled, bound in a book and given to the students the first day of class. This allowed students to know what objectives would be discussed in class five months later. The distribution of the syllabi to the students was important to the curriculum, as it kept faculty on track with their curriculum and ensured that core objectives in each course were taught in time with other courses. This longitudinal approach, which has been shown to improve student retention, works best if all classes keep to schedule.
Altering Touro with Data-Driven Feedback
Once all the objectives were set for the entire year, exams and classroom assessments were written. The questions for these exams and assessments were required to be directly linked back to the objectives detailed in the syllabus. The problems with this part of the implementation were 1) how to execute it and to track each question and its objective to allow for detailed assessments and 2) how to let the student know what objectives in the curriculum they were deficient in. To meet this challenge, TouroCOM administrators selected ExamSoft
ExamSoft allows the faculty members to enter their questions in an online database and to categorize each question based on an unlimited number of “tags,” including program objective, learning outcome, accreditation standard, Bloom’s taxonomy, etc. At TouroCOM, the faculty member tags each question back to the objective in the syllabus for that course, and the administration tags questions to national board areas, accreditation standards and other internal metrics, as defined.
By doing this, administrators and faculty are able to see how the students perform with respect to the objectives in each course, how they perform in national board areas, and assess how the students perform on the accreditation bodies’ core competencies. TouroCOM also uses ExamSoft’s biserial reports to give faculty feedback on course and program performance so that faculty might better adjust their lesson plans and syllabi toward students’ needs. Ultimately, this system of assessments allows TouroCOM to assess each student individually while giving scale to the institution to see how the curriculum performed globally.
Creating a Dynamic Lecture
The second step in building the new dynamic curriculum was to change the method of lecture delivery. Lectures are now delivered using iTunes. The professor records the lecture in the studio, and then it is uploaded to the iTunes server. The students download the lecture onto their computers and watch the lecture prior to coming to class. Classroom time is then devoted to assessment of the student’s knowledge from that lecture. The professor goes through a series of assessment questions in the classroom using the following structure:
- The professor prepares 6-10 questions from the iTunes lecture material and readings the students were to consume prior to coming to class. These questions are categorized similarly to the exam questions.
- Each question is given in sequence to the students.
- The professor presents the question to the class on a PowerPoint slide.
- The students answer the question using a student response system. The professor does not show the students the results. (1 minute)
- The students work in small groups to determine what the answer is to the question. (2 minutes)
- The students then answer the question again using the student response system. This is a graded question and counts toward their final grades. This gives the students the incentive to engage in the discussion in the second bullet point above. (30 seconds)
- The professor shows the results from the two questions.
- This is where the dynamic part of the lecture begins.
- If the students get the questions correct, the professor moves to the next question and begins the cycle again.
- If the students get the questions incorrect, the professor engages in a review of the material and works with the students for understanding as to why the answer is correct and the other answers are not. (5 minutes)
This method allows the professor to focus on the material the students did not understand from the lectures and bases this decision on the immediate outcomes from the student’s performance. This is called a “dynamic curriculum.”
One of the advantages to the dynamic curriculum is it gets students to go to class. The lectures are more fun, the students are engaged in the learning process, and the faculty is there to guide the students through the information they do not understand. And this pathway is based on the instant outcomes of the students in the class. One noticeable change is that often there are too many students attending class, which is a rarity in medical school. The reason that more students attend class is because the lecture now becomes more valuable to their learning process, and the students recognize this.
Additionally, the assessments provided in the classroom are written in ExamSoft and thus tagged in the same way as the exam questions. With this process, it is possible to perform a daily assessment of students in each class and in each categorized discipline—course specific, board area, and accreditation core competencies. The advantage to performing this daily assessment is that it is possible to identify students who are having difficulty early on in the curriculum, and to give them assistance.
Implementing an Early Warning System (EWS)
Educational institutions’ tuition rates have been increasing above the rate of inflation for almost three decades now, and the last 10 years has seen a dramatic increase in the tuition rate. It is clear that this cannot continue indefinitely. So, there remains a question: how does an educational institution improve revenue streams without increasing tuition?
The University of California system is considering a law that will require universities within the state to cap the tuition at the entering freshman rate for that student. Alternatively, this can be done quickly by reducing the dropout rate for the organization. At TouroCOM, administrators call measuring their students’ learning outcomes on a daily basis an “Early Warning System” (EWS), and the EWS does reduce TouroCOM’s dropout rate.
The EWS identifies students that move away from their expected norms (their average grades). This is often caused by an unexpected event in the student’s life—a personal, family, or other problem that is affecting his/her studies and may cause him/ her to drop out of school. When the student moves away from his/her expected norm, the administration can call the student in to review his/her academic progress.
Often by calling the student in and asking, “Is there anything wrong?,” while reviewing his/her academic performance, it is possible to determine what the issue is and fix it before it becomes a larger problem that could lead the student to drop out of the school. Thereby, TouroCOM’s administration reduces the number of students who drop out. In fact, by using this method, TouroCOM has been able to reduce the dropout rate from 8% to 2%, yielding an improvement of more than $2 million per year in cash flow.
By developing an EWS and measuring student outcome measures daily, it is possible to determine the area in which a student is struggling. By tracking subject areas and national board areas, it is possible to help direct students’ learning in areas where they are weak. This is not just for the students having difficulty. Top-performing students also benefit from this assessment because it allows them to see where they are weak in comparison with the students in their class—and, on a longitudinal scale, where they can be compared to students in pervious years. This system has allowed TouroCOM to improve its board passage rate from 70% to more than 90% in just four years. Furthermore, TouroCOM is able to matriculate students who have a ‘higher risk’ value because we monitor their performance daily. This allows the school to complete its mission.
A third point that the EWS provides the college is an upgraded risk tolerance in student admission. One of the main missions of the college is to get more underrepresented minorities (URM) into medical education. However, these students, on average, often do not perform well on standardized tests, such as the MCAT admission exam. This perceived risk often restricts the URM student from gaining access to medical school, even though the student may be an exceptional one who does not do well on standardized tests.
Because medical school is very competitive (and at TouroCOM there are more than 50 applications for each slot available), and given that if a student drops out they cannot be back filled, medical colleges are often reluctant to take the risk associated with a student with a low MCAT score. Since the EWS allows TouroCOM to have more comfort with that risk, TouroCOM has been able to accept more URM students compared with other medical schools, while maintaining the high level of education required to pass the national boards.
When developing a dynamic curriculum, it is important to recognize that lectures, exams and assessment are all equally important and should be used to assess the student outcome measures. These outcome measures should be reviewed on a daily basis to identify at-risk students on in the curriculum, which will yield improved revenues and academic performance.
In order to measure these student outcomes, the institution must have the technology necessary to administer online exams that will provide for assessment of objectives from the curriculum, competencies from the accreditation bodies, and metrics from national standardized exams. ExamSoft’s exam management and assessment analytics platforms supply these criteria. By performing these assessments, the institution will improve quality, reduce dropout rates, and drive more students to the organization—ultimately creating a positive and fundamental alteration in the workings of the institution’s economic structure.