Top 10 Ways to Engage Faculty When Implementing ExamSoft
The University of Texas Health Science Center at San Antonio (UTHSCSA) School of Dentistry (SoD) houses a four-year DDS program with just over 400 students, a dental hygiene program, and various dental residency and graduate programs. It was one of the first dental schools to implement an electronic curriculum with laptop support, and in the fall of 2013, the SoD began its transition from Scantron and paper-based assessments to computer-based assessments.
Reasons for the change included ailing Scantron equipment, the high cost of replacement parts, and the closure of the Scantron support facility. Because the UTHSCSA School of Nursing and School of Medicine had already transitioned to ExamSoft in 2011 and 2012, respectively, and because of security issues the SoD previously experienced with other electronic exams, the SoD chose ExamSoft as their electronic assessment system. The faculty was given one year to transition.
The administration and a faculty champion, Dr. Kyumin Whang, spearheaded the transition, which they completed by the fall of 2014. Faculty from the School of Nursing (already actively using ExamSoft), faculty from the Office of Blended Learning, and multiple webinars introduced the SoD to ExamSoft. Dr. Whang was the first to use ExamSoft in a course. Due to his early experience with ExamSoft, the administration asked him to join an ad hoc SofTest Implementation Committee and to help train faculty and staff at the 2013 Faculty Development Day. In early 2014, he further trained faculty and staff and started to troubleshoot at exams. In May 2014, the administration asked him to become the official ExamSoft go-to person.
Through his experiences with ExamSoft as a faculty member and champion, Dr. Whang gained many insights about implementing ExamSoft successfully program wide. Below are his ten most important pieces of advice. Some of these suggestions apply to anyone implementing ExamSoft, but most are specific to faculty champions of ExamSoft and course directors.
- Orient Faculty and Students and Troubleshoot Computers
To facilitate a positive first impression, Dr. Whang conducted faculty and staff training sessions, student orientations, and computer maintenance sessions. He also volunteered to troubleshoot at exams. The faculty and staff training workshops were broken up into three levels: question banking, assessment assembly and posting, and assessment performance reporting. Student orientation sessions focused on introducing students to the ExamSoft portal and SofTest and included a mock quiz to familiarize students with SofTest functionality. In addition, Dr. Whang administered multiple extra-credit quizzes in SofTest to acclimate his students to the software. Finally, the monthly computer maintenance sessions helped students troubleshoot any hardware problems they experienced during the academic year.
- Develop Best Practices and Exam Policy
While time-consuming, having one person troubleshoot most of the exams helped identify inefficiencies and procedural errors in exam administration using ExamSoft. Informed by reports of academic misconduct during exams from fellow students, the administration and Dr. Whang developed an updated and more relevant best practices and exam policy. If the troubleshooting had been conducted by several administrative assistants or staff members,identifying new methods of academic misconduct would have been more difficult and the level of communication with administration would not have been as effective.
Examples of academic misconduct using ExamSoft include but are not limited to these:
- Delaying the start of the exam to study a little more, knowing that ExamSoft will give them the full allotted time no matter when they begin. This includes students staying outside the exam room and deliberately coming in late to take the exam.
- Copying notes from their computers onto worksheets that were handed out as they walked into the exam room. Worksheets are now handed out one by one after verifying that students have started the test.
- On quizzes, texting answers to other students when secure review was enabled. The student receiving the texts would start the quiz late—again knowing that ExamSoft would give them the full allotted time—and the texting student would send the answers from his or her secure review upon completing the exam. Secure review is not enabled immediately after quizzes anymore.
- Use ExamSoft Resources Extensively
Faculty are interested in more than just the functions of ExamSoft. Fortunately, ExamSoft provides numerous webinars and white papers on how the software can help them improve their teaching and student learning. In addition, the ExamSoft community is active in asking for help, giving suggestions, and sharing new ideas on everything from enhancing student learning to question creation. Because these resources are not something resident ExamSoft implementation staff could offer on their own, it is crucial that faculty take advantage of them through ExamSoft, with the faculty onboarding champion leading this initiative.
- Collaborate with the Office of Blended Learning and Assessment Experts
The UTHSCSA is fortunate to have an Office of Blended Learning, which educates faculty on how to improve teaching and on new methods of teaching. Among other services, this office helps faculty make sure learning outcomes (LOs) are well stated and that exam questions directly reflect them. Doing so means questions are properly categorized (tagged) so student learning can be better tracked. By collaborating with the Office of Blended Learning, Dr. Whang was able to improve his LOs so they better matched the questions he was asking (and vise versa). He was then able to provide better feedback to students after an assessment. Dr. Whang has shared with other course directors (at a course directors’ forum) how this effort has helped his course. He will be giving permission (and support) to course directors to create new categories in order to implement LO alignment at a broader level.
5. Use Formative Quizzes to Enhance Learning
One of the first things Dr. Whang did when ExamSoft became available was begin giving formative quizzes in each class. Paper quizzes—though they could have been administered—were not being used because of the time it took to grade them (with 100+ students in the class) and because of an inability to give timely feedback. By the time the paper quizzes were graded and analyzed, it was too late to make quick adjustments to lecture material to cover topics students were struggling in. The addition of low-stakes formative assessments in ExamSoft added tremendous value by tracking student progress in real time.
6. Diversify Question Types to Better Assess Student Knowledge
Dr. Whang began by using only multiple-choice (MC) questions since, originally, ExamSoft was viewed only as a Scantron replacement. However, as he experimented with the different styles of questioning, he started using matching/fill-in-the-blank questions, multiple-answer MC questions, select-all-that-apply questions, and short-essay questions. He discovered how easy it was to grade and adjust the grading of matching/fill-in-the-blank questions, and he has promoted this function to other faculty to assess different types of learning. In addition, Dr. Whang realized that SofTest’s function of graying out other answer choices once the correct number of answers was selected in MC and multiple-answer MC questions helped students guess the answers. So he began using the select-all-that-apply question style and found that significantly fewer students got those questions right. Thus, diversifying question types is important in accurately assessing student knowledge.
7. Utilize Categories (Tags) to Help with Student Feedback
Dr. Whang found that several course directors were not letting students review their exams due to question security issues, and thus, the students were not receiving any feedback on which LOs they were struggling in. Dr Whang helped these directors understand how “tagging” questions to LOs would allow students to receive effective and constructive feedback without having to see the actual test questions. This explanation motivated faculty members to better categorize their questions, and thus, provide their students with valuable feedback on their performance.
8. Assess Student Performance in Preclinical Labs and in the Clinic
Dr. Whang wanted to know how his course (and other didactic courses in a student’s first two years) correlated to their clinical performance in the latter two years. He began tagging his questions with the different subjects (dental material) he was teaching. He hopes other faculty will begin tagging their assessments with these subjects so they can start to correlate the two types of assessment settings. Such correlations can be used to determine if a student who is weak in certain areas will continue to be weak in the clinical setting. This information can be used to intervene in a student’s education earlier. To help with this effort, Dr. Whang has also been developing rubrics (based on the current paper-based forms) for clinical course directors to use to assess student performance in preclinical labs and in the clinic.
9. Develop an Assessment Subcommittee
Dr. Whang became an advocate for establishing an assessment committee. He felt that if too many faculty members were given authority to create categories, there would be too many overlapping and irrelevant objectives listed. In addition, he felt that some exam questions might not be valid or ideally constructed. Thus, an assessment committee would streamline and control the categories—especially those available for multiple courses to use—give constructive feedback on high-stakes exam questions to improve question quality, and ensure that every question is tagged appropriately. While this committee will take time to construct, the SoD administration is supportive and is planning on implementing this over time.
10. Continue Faculty Development
The analytics provided by ExamSoft are invaluable tools for improving exam questions and lecture quality and for enhancing student learning. Specifically, these analytics can be used to revise exam items by eliminating poor distractors and identifying poorly written or ambiguous questions. In addition, items that the class did poorly on can be used to identify lecture material that may need to be revisited, ensuring students understand the material before proceeding to the next topic. These are all things that faculty can use to develop their teaching skills.
Finally, after Dr. Whang started using ExamSoft, he noticed a fivefold increase in the number of students coming to him to review their exams. This was mainly due to the immediate release of results, since they no longer had to wait one or two weeks to receive their grades. This increased interaction with students has enabled him to address their individual learning needs, further developing his teaching ability.
Faculty training in ExamSoft is now included as a module in a teaching-skills course called TExAS, which is given annually to new faculty in the SoD. Here, the development of LOs and exam questions will be directly tied to the use of ExamSoft. Faculty are trained in categorizing questions, using ExamSoft analytics and reports to improve question and lecture quality, and addressing individual student learning needs.
Following these ten best practices while implementing ExamSoft, the SoD administered over 450 assessments during the 2014–15 academic year. Over 60% of the courses used ExamSoft to assess students, a high percentage considering that many of the courses are clinical courses. In addition, student and faculty anxiety about using assessment software has significantly declined. Finally, analytics for overall student performance for each competency have been provided to the SoD administration periodically, and there is an increase in use of analytics for individual courses. While the ExamSoft implementation process is ongoing, the use of ExamSoft has already begun to positively impact teaching and student learning at the UTHSCSA SoD.