#AssessChat Recap: Writing Good Multiple-Choice Questions

August 12, 2015 Kristen Hicks

This week’s #AssessChat with Aaron Dewald (@Aaron_Dewald), the Associate Director of Instructional Design from the University of Utah College of Law was packed full of useful information for everyone that attended. We were pleased with the turnout, but we know not everyone who wanted to join us could make it, so we’ve collected some of the best insights from the chat for you to review here.

The theme of the chat was writing strong multiple-choice questions and Aaron came equipped with six questions to get the conversation going.

Question 1: What are your main challenges to writing good multiple choice questions?

#AssessChat The primary challenge of writing good multiple-choice questions for me is writing really good distractors

— G. Scott Weston (@gscottweston) August 11, 2015

a1 hard to write a good question that fits the criteria and doesn't give away the answer #AssessChat

— Anne Koerber (@DrAKoerber) August 11, 2015

@ExamSoft Q1: definitely construct validity! Also time, avoiding bias #AssessChat

— Jean Sumner Downs (@jeansumner) August 11, 2015

ensuring questions don't answer previous questions #AssessChat

— Walter Shorts (@skipatroler) August 11, 2015

Multiple choice questions are great for general knowledge objectives. but higher order cognitive skills are much harder #AssessChat

— Walter Shorts (@skipatroler) August 11, 2015

Question 2: What is your top tip for writing good multiple choice questions?

A2. Be intentional about student knowledge / learning you wish to assess #AssessChat

— Timothy Howell (@timsterhatc) August 11, 2015

@timsterhatc Love it. Intention. You need to have goals and ideas in mind before you write the questions. Hard to do. #AssessChat

— Aaron D (@aaron_dewald) August 11, 2015

#AssessChat Top Tip- contextualize the problem and have answer require a multi-step process to answer

— Charles G (@immune010) August 11, 2015

#AssessChat A2: Question is tied to one of the stated course/lecture objectives. Each distractor is incorrect for a different reason.

— G. Scott Weston (@gscottweston) August 11, 2015

A2. Think of the distractors as a scale from most correct to least correct. All must be viable options depending on case info #AssessChat

— Emma (@Emma1420) August 11, 2015

Question 3: What are some of your best practices for writing distractors?

common mistakes of present or past students #AssessChat

— Walter Shorts (@skipatroler) August 11, 2015

A3 class disscussions. #AssessChat

— Jennifer Brooks (@jbro1922) August 11, 2015

A3- keep all distractors of similar length and style…avoid technical item flaws #AssessChat

— Charles G (@immune010) August 11, 2015

Q3: Avoid wordiness and state distractors as clearly as possible; from http://t.co/tgWJs6nkW9 #AssessChat

— Jean Sumner Downs (@jeansumner) August 11, 2015

Question 4: If you’ve used item analysis, has it helped?

A4 Item Analysis has helped our faculty actually be mindful of the questions they write and be open to training #AssessChat

— Charles G (@immune010) August 11, 2015

A4 It shows which questions are missed most commonly and shows the distribution of Answers #AssessChat

— Walter Shorts (@skipatroler) August 11, 2015

#AssessChat A4: setting egos aside and just looking objectively at the performance of the question

— G. Scott Weston (@gscottweston) August 11, 2015

A4: helps to identify items that need further review #AssessChat

— Thomas J. Cook (@ProfTJCook) August 11, 2015

A4 if all distractors are used "evenly" by students – probably a good Tuff question with good distractors #AssessChat

— Walter Shorts (@skipatroler) August 11, 2015

Question 5: What is your primary barrier from using item analysis?

Q5: What is your primary barrier from using item analysis? #AssessChat

— Aaron D (@aaron_dewald) August 11, 2015

a5 if you are competency based then you may not want all questions to discriminate the best students? #AssessChat

— Anne Koerber (@DrAKoerber) August 11, 2015

Q5. Time and agreement between faculty about how to classify / code questions #AssessChat

— Timothy Howell (@timsterhatc) August 11, 2015

Q5. Time and agreement between faculty about how to classify / code questions #AssessChat

— Timothy Howell (@timsterhatc) August 11, 2015

Q5 Faculty like to review performance, but struggle to know what the data mean #AssessChat

— Sean Guinane (@SeanGuinane) August 11, 2015

Question 6: How might item analysis make you a better question writer?

A6 item analysis often challenges your assumptions about yourself…if you let it #AssessChat

— Charles G (@immune010) August 11, 2015

Q6 Item analysis lets faculty know what choices aren'c challenging students, and helps faculty encourage better discrimination #AssessChat

— Sean Guinane (@SeanGuinane) August 11, 2015

Q6: by looking at the difficulty index you can tell if your exams are too easy or adequately challenging #AssessChat

— Sheila Chelimo (@chelimosheila) August 11, 2015

#AssessChat A6: helps identify any obvious problems re: key errors, also helps (with student feedback) to pin down conceptual issues

— G. Scott Weston (@gscottweston) August 11, 2015

item analysis is a tool to look at the questions themselves not at the course…at least directly #AssessChat

— Charles G (@immune010) August 11, 2015

The chat also included an interesting debate on whether it’s best to write your tests before your classes or after. Opinions were mixed:

A2: Purposeful questions are easier to write in advance of, rather than after the instruction. #assesschat

— Leah Simpson (@TheLeahSimpson) August 11, 2015

@theleahsimpson Interesting. Knowing what you're going to test might help you know what to teach? #AssessChat

— Aaron D (@aaron_dewald) August 11, 2015

@theleahsimpson @aaron_dewald Begin with the end in mind; backwards design in all course and training designs #AssessChat

— Jean Sumner Downs (@jeansumner) August 11, 2015

@aaron_dewald @jbro1922 I have tried writing questions first, but then I focus too much on test prep and too little on content #assesschat

— Jeremy Crespo (@JeremyCrespo) August 11, 2015

Be careful writing the exam first – common mistake is to "Teach to the Test" and lots of content gets left out #AssessChat

— Walter Shorts (@skipatroler) August 11, 2015

@timsterhatc teaching to the test would be ok if it is well designed and covers all the learning objectives #AssessChat

— Jean Sumner Downs (@jeansumner) August 11, 2015

If you want more information on writing multiple choice questions, Aaron Dewold’s done a whole series on the subject for our blog that you can see here.

If you couldn’t make it this time, you can always add your input in the comments here. And to make sure you don’t miss any future chats, join our email list above to get up-to-date announcements on any future ones we hold. And, in case the extra knowledge isn’t incentive enough, all #AssessChat attendees are also eligible for prizes. We hope to see you at the next one.

Previous Article
How to Improve Programmatic Results by Focusing on High-Need Students
How to Improve Programmatic Results by Focusing on High-Need Students

Don't let students slip through the cracks - monitor their progress through exam analytics and intervene ea...

Next Article
Assessment Resources for Medical Education
Assessment Resources for Medical Education

Looking for assessment tips for medical education? Here are some great resources to get you started.