How to design multiple-choice questions

Taketheplunge

Our doubts are traitors,

And make us lose the good we oft might win


By fearing to attempt.

Shakespeare, Measure for Measure

 

Since taking the plunge and adding multiple-choice questions to my assessment repertoire, I’ve found they have refreshing and unexpected advantages.

They make assessment more reliable, marking less labour-intensive, pupil understanding and misconceptions more visible, and allow a wider breadth of knowledge to be assessed across a unit than just using essays or complex, holistic end-of-unit assessments. They save countless hours of marking downstream, and get pupils thinking deeply about subject content.

Both Alex Quigley and Cristina Milos have written perceptively about how tricky they are to create. How can we ensure that the advantages outweigh the limitations?

Research from Little, Bjork, Bjork and Angello (that Alex cites) suggests not only that they are as effective as short-answer tests for retention, but they also have an important advantage over them – that pupils have to think through incorrect alternatives. The key insight is that these alternatives must be plausible enough to enable pupils to retrieve why correct alternatives are correct and incorrect options are incorrect.

I’ve found these seven principles helpful in multiple-choice design:

Slide1

 

1. The proximity of options increases the rigour of the question

For instance, the question is, what year was the battle of Hastings? Options 1065, 1066, 1067, 1068 or 1069 are more rigorous than options 1066, 1166, 1266, 1366 or 1466. Of course, the question itself also determines the rigour: ‘80 is what percentage of 200?’ is much easier than ‘79 is what percentage of 316?’

 

2. The number of incorrect options increases rigour

Three options gives pupils a 33% chance of guessing the correct answer; five options reduces the chances of guessing to 20%; always create five rather than three or four options for multiple choice questions. A ‘don’t know’ option prevents pupils from blindly guessing, allowing them to flag up questions they’re unsure about rather than getting lucky with a correct guess.

 

3. Incorrect options should be plausible but unambiguously wrong

If options are too implausible, this reduces rigour as pupils can too quickly dismiss them. For instance, in the question: what do Charles Dickens and Oliver Twist have in common, an implausible option would be that they were both bank robbers. However, if answers are too ambiguously similar, this creates problems. For instance, in the question, ‘What happens in the plot of Oliver Twist?’, these options are too ambiguous:

a)     A young boy runs away to London

b)     An orphan falls in with a street gang of street urchins

c)     A poor orphan is adopted by a wealthy gentleman

d)     A criminal murders a young woman and is pursued by a mob

e)     A gang of pickpockets abduct a young boy

 

4. Incorrect options should be frequent misconceptions where possible

For example, if you know pupils often confuse how autobiographical ‘Oliver Twist’ is, create options as common confusions. These distractors flag up what pupils are thinking if they select an incorrect option:

a). Both were born in a workhouse

b). Both were separated from their parents and family

c). Both were put in prison for debt

d). Both had families who were put in prison for debt

e). Both were orphans

 

5. Multiple correct options make a question more rigorous.

Not stating how many correct options there are makes pupils think harder. For example:

Which characteristics of “Elegy Written in a Country Churchyard” can be seen as Romantic?

A. It celebrates the supernatural.

B. It is written in iambic pentameter.

C. It emphasises emotion over reason.

D. It deals with the lives of common people.

E. It aspires to nature and the sublime.

 

6. The occasional negative question encourages kids to read the questions more carefully.

Once they get a question like ‘Which of these is NOT a cause of World War 1?‘ wrong, and realise why, they’ll work out they need to read questions again to double-check on what it is they’re asking.

 

7. Stretch questions can be created with comparisons or connections between topics.

What was common to both the USA and Germany during the Great Depression?

a)     Jewish immigration increased

b)     Membership of Ku Klux Klan increased

c)     Public works projects were implemented

d)     Government social programs were reduced

 

Good, Better, Best

Here is an example of honing a question to take into account the key insight that multiple-choice options must be plausible, but unambiguously distinctive. Pupils aren’t able to work out the good, better, or best options from context alone. They’d need to carefully think through the nuances.

 

What does the word ‘resilient’ mean in this sentence?

Her resilient attitude toward life enabled her to overcome difficult situations.

Bad multiple-choice options are not plausible

A. depressing

B. dishonest

C. flexible

D. anxious

Good multiple-choice options are plausible and not too ambiguous

A. flexible

B. energetic

C. positive

D. enthusiastic

Better options are plausible and less ambiguous

Her resilient attitude toward life enabled her to overcome difficult situations.

A. flexible and durable

B. energetic and enthusiastic

C. positive and creative

D. logical and calm

Best multiple-choice options plausible yet distinctive

Her resilient attitude toward life enabled her to overcome difficult situations.

A. flexible and durable, with the ability to bounce back from setbacks

B. energetic and enthusiastic, with the ability to turn negatives into positives

C. positive and creative, with the ability to make something out of nothing

D. logical and calm, with the ability to solve complex problems

 

So, multiple-choice questions require us to design options carefully if they are to become a valuable part of our assessment mix.

 

Deep roots produce sweet fruits

rootsfruits

If Heads of Department invest the up-front time to create them for their subject, their teachers stand to benefit from a strong return on that investment in terms of time saved and pupil thinking – for years to come.

Advertisements

About Joe Kirby

English teacher, education blogger
This entry was posted in System. Bookmark the permalink.

23 Responses to How to design multiple-choice questions

  1. Laz77 says:

    Hi Joe, I always read your posts with eager interest and am intrigued with how to apply the use of multiple choice questions. How frequently would you usually set such multiple choice questions? Whenever frequent misconceptions occur, mid way through a unit? What frequency works best to help aid memory retention and mastery of the subject knowledge you wish them to gain?

  2. Ebefl says:

    Curious what evidence there is f

    • Ebefl says:

      Curious what evidence there is for these statements about test writing? Some seem at odds with things I’ve been told so if like to check. Ta.

  3. whatonomy says:

    Great post! Use Google forms to create and disseminate the tests and they mark themselves!

  4. Benjamin Wood says:

    Thanks for this Joe. Following your previous post on MCQs I have begun to work with them in Religious Studies. I ran one with a y12 class on the topic of Atheism and Postmodernism and it was successful in a number of respects. Firstly, it highlighted a number of common errors that I had always suspected were likely but this format gave me concrete evidence as to number of students making this error allowing me to to re-teach that point. Secondly, it was useful in helping them to learn from the questions they got wrong. Having done the 45 questions on one day, when asked to do it again a few days later, every single student improved their score, some by as much as 20%. Obviously, some of this improvement will be due to getting better at taking that particular test, but I think it clear it has helped students remember more than they might have otherwise.

    I am now going to to trying this type of assessment out with my y9 classes using Socrative on students iPads. I am planning to run regular tests, possibly weekly, as a starter activity to a lesson. Having written a bank of questions, I can construct a new test every week comprising a mix of new questions related to what has been taught that week and questions from previous work in the unit. If that helps students retain more of the knowledge they need then they ought to be able to write more informed essays at the end of units.

    I’ve really enjoyed following your blog and it has inspired to look carefully at how we assess in RS. Keep up the good work!

  5. Pingback: How to design multiple-choice questions | Pragmatic Education | Learning Curve

  6. dodiscimus says:

    Good examples of how to make MCQ effective – I use them a lot and am a big proponent. Did you see David’s post from last year, http://www.learningspy.co.uk/assessment/afl-have-we-been-doing-the-right-things-for-the-wrong-reasons/ which cites Roediger, Putnam and Smith (2011) available at http://psych.wustl.edu/memory/Roddy%20article%20PDF%27s/BC_Roediger%20et%20al%20(2011)_PLM.pdf. This paper just raises the issue of whether presenting incorrect answers can lead to pupils picking them up (p.30) and makes the point that feedback on correct and incorrect answers is important, although it only really cites one study demonstrating that this eliminates the problem. I can’t imagine a teacher (as opposed to a psychology researcher) not providing feedback but it would be nice to see some more robust evidence that doing so avoids the problem of familiarity with the wrong answer.

  7. Pingback: QuickKey, multiple choice and hinge questions. | mrbenney

  8. Pingback: Worksheet 2.0: learning concepts, deliberate practice and desirable difficulty | must do better...

  9. Pingback: A guide to this blog | Pragmatic Education

  10. PCS says:

    Interesting reading. I have twins and they have just sat their driving theory test, some of the multiple choice questions had completely implausible options. For example an old person is still crossing at a pelican crossing when the amber light flashes, should you a) sound your horn to hurry them up, b) Drive around them to proceed, c) wait patiently for them to cross, or d) Rev your engine getting ready to go. No studying required for some of these questions….
    Peter
    http://www.tutordoctorprivatetutoring.co.uk

  11. bt0558 says:

    Hi Joe, an interesting post. For me a good assessment item is one that assesses what it is supposed to assess, validly and reliably. I am struggling a little with some of the principles…e.g.

    If teaching a student that they should read questions carefully is the purpose of the assessment item then I can see a point in trying to trick them, otherwise I can’t.

    The issue of plausibility may have nothing whatsoever to do with the purpose of the question and what is plausible for one person will not be plausible for another depending upon their levels of knowledge and understanding.

    Is it not possible that the “principles” would not necessarily be identical, especially as the topic/issue/concept becomes more abstratct and/or the thinking required to solve the problem becomes more complex?

  12. Pingback: Increasing bandwidth – Planning a revision session. | @mrocallaghan_edu

  13. Pingback: Standin’ in line marking time…. | Distant Ramblings on the Horizon

  14. Pingback: Assessment | PGCEPhysicalEducation

  15. Pingback: The Signal & The Noise: The Blogosphere in 2014 | Pragmatic Education

  16. Pingback: How can I simply implement cognitive science findings to improve learning? | EviEd

  17. Pingback: Using Threshold Concepts to design a KS4 English curriculum | David Didau: The Learning Spy

  18. Pingback: 20 psychological principles for teachers #5 “Learning is dependent on practice” | David Didau: The Learning Spy

  19. Hi Joe,
    I’m developing a tool which uses a different approach to MCQ to introduce more rigour. One benefit is it makes writing the questions easier. I’d appreciate your thoughts.
    http://whatdoyoureallyknow.com/?q=node/7
    Regards,
    Nic

  20. Pingback: One Hundred Classics for Every Child | Tabula Rasa

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s