Improving "multiple answer" question type
We are moving more into automated tests where the system takes care of what the pass score or pass percent is, and then gives the user a certificate to download (more on our new Certify module coming soon).
In this process we have discovered a problem with the "multiple answer" question type, that can be best illustrated with this screen shot:
ATutor tells the user how many points are available, and in this case there are 10 questions with 10 point available, which for the not-very-observant student means that all alternatives are right. The problem is still there if there was 5 points available: the student could check all alternatives and reach maximum score for that question.
The underlying problem here is that you can only give point per question, not per question alternative, which means that you need a human beeing to correct all tests and this automatic correcting is not possible.
To solve this I believe we need to be able to set points per alternative, and also to set negative points (so you can't just check all and get away with max score).
let's take a look at one question:
This is how I believe this question type must work. Now, this will create a new issue, which is that creating these questions will be harder, so we need to take that into account as well.
As this is pretty simple calculating I believe the system should do it by taking the point value set on a multiple answer question, and then divide it as positive and negative scores on each alternative, which if needed can be changed by the creator at a later time:
We are willing to do this job, but need feedback on how to best solve it! I also wonder if there will be any consequences for backups and restoring of tests, and the IMS QTI Question and Test Interoperability Support?
![Syndicate content](/misc/feed.png "Comments for Improving "multiple answer" question type"){kind=small}
Comments
come to think of it..
May 6, 2009 - 19:47 — Vegard A. Johansenthere are some issues here to think more about.
one thing is that in the example above you would get 0 points by answering one right alternative and one false.
a different issue is that you could get negative points on a question. in the example above you could get -1 if you only answered the wrong alternative.
both these scenarios might be just fine, but should be discussed. the solution to the last one (if it needs one) is that we could say that if the student gets a negative score on a quiestion that would equal a score of 0, but I'm having a harder time to think of a solution for the first one (if it needs one).
you would be able to get decimals in results, which might be fine, but I don't know if the test results table in ATutor allows for floating values?
Re: Improving "multiple answer" question type
May 7, 2009 - 06:59 — Serhiy KostyshynNot sure that I get the point.
Currently, ATutor substracts number of mistakenly checked choices from the number of correctly checked ones. If that number matches the number of correct answers, the question is marked as correct. If not, it's wrong.
Does that cause any problems?
Or do you want the questions to be marked proportionally to the number of correct answers (like Matching questions)?
Re: Improving "multiple answer" question type
May 7, 2009 - 09:03 — Vegard A. JohansenThanks for clarifying Serhiy, I guess I was in a bit of a hurry when writing that post :)
Yes, we wish the questions to be marked proportionally to the number of correct answers. If you see my first screen shot all alternatives are correct, but for this course the instructor does not want to give 0 points if the user has 9 correct..
However, negative points will not work I can see. In this case the user would get 0 point if answering 5 alternatives (+5 and -5). Not giving negative points will cause the user to get a max score by filling out all alternatives.
Hm, it strikes me that this might not be easily solvable. Any ideas?
Also, if we find a way would it be better to create a new question type, or could we this "proportionally marking" available as an option on questions. On one hand we cannot destroy the current functionality of these questions, on the other hand we need to use the data we have on these questions for this case so we don't need to create the questions again, and require all the students to take them again..
Re: Improving "multiple answer" question type
May 7, 2009 - 14:16 — Serhiy KostyshynNow I see. In fact, in our local installation we've already changed the marking algorithm to be consistent with that of Matching type. I'll send you the changes if you need. I didn't look into the QTI compatibility though.
Still, in 1.6.2 the question type is marked as I describe above (I remember the code clearly since the customization, and also have tested once more in the demo). So there should be no possibility of cheating in that way.
I'd like that!
May 11, 2009 - 10:01 — Vegard A. Johansenthanks Serhiy, we haven't used the matching type questions much so I wasn't aware that they worked like this - I'd very much like to have a look at the changes you've done in the code, and I'll take it to a test run on a test install of this course here!
I guess I still struggle a bit to see how you are not able to cheat with checking all alternatives (how many point would you get if you checked all alternatives on a question where 8 of 10 are correct?), but I'll probbaly find out when I try it out :)
The export isn't a big issue unless this would be a part of core I guess..
Here you are
May 14, 2009 - 10:11 — Serhiy KostyshynSure, I'm sending you the changes by e-mail. Sorry for the delay. :)
Lessons learned from the above mentioned screenshots
May 7, 2009 - 16:30 — klaus melf (not verified)After having manually corrected the answers of five students to all the Multiple Choice and Multiple Answer questions of the MPW course (about 79 tests x 5 questions x 5 answer possibilities):
1. MAQs are easy for the author to design, while the production of good (!) MCQs is a quite difficult job.
2. (Probably because of 1!) Our MCQs seem to be easier for the students to answer (or guess) correctly.
3. To give negative points (and not only "0") for wrongly marked MCQs, increases the quality level of our course (it reduces the guessing): Please give negative points for wrong answer in MCQs!
4. To give only one point (positive or negative) per MAQ is frustrating for the student as he/she might have marked correctly several positive answers, but maybe not the last one (see Vegard's screenshot 2. However, there is a mistake in the red comment to choice 3; the student should get 0 for not-marking this correct answer, and in total +2 points for correctly marking choice 1 and 2.): Please give one point per correct answer in MAQs!
5. If we give several possible points for MAQs (because of several correct answers), but not give negative points for wrongly marked answers, this makes it quite easy for the student: Please give negative points also for MAQs! (We give therefore the following instruction to our students: "Multiple Choice means one answer is correct, Multiple Answers means one or several answer/s is/are correct. Please be aware that wrong answers give negative points. If you do not know, it is better not to mark an answer.")
6. If we in addition show the maximum possible point score in MAQs, in which all or most answers are correct, our questions become meaningless: Please don't show the maximum possible point score.
7. Summa summarum: Each MAQ choice should be evaluated individually with -1/0/+1 point, and then summed up per question. This can result, for example, in (up to) +10 points for one MAQ (see screenshot 1), or in -4 points for another MAQ.
8. I have ignored in our course evaluation the possibility to weigth one choice higher (more difficult = more points) than another one. This would make it even more complicated to calculate the correct point sum, and I think this option is not necessary either.
9. It takes now about one working day (!) to go through and evaluate all answers of one student. An automatic calculation of the correct score is highly requested.
thanks for this
May 11, 2009 - 10:46 — Vegard A. Johansenhi Klaus, thanks for your feedback here! I do agree that it is not possible to run a course like MPW in the long run without some form of automated test calculating and certificates.
After letting this sink for a bit my thoughts are that IF we can avoid negative points and at the same time make sure that it's not possible to cheat by just checking all alternatives that would be preferrable. I fear that giving +1 for correct answer, 0 for not answered and -1 for incorrectly answered will mean a whole lot of programming that could be solved easier, but I will run this by the programmers too, and also first try out the changes Serhiy has done with this.
You are still able to make the tests harder or easier to pass by adjusting the pass percent of them.
I agree that there is no particular good reason to show the max possible point score on questions. I know I had fixed this before with a quite simple theme change, but it must have been reintroduced in an upgrade. I'll have a look at it again.
First tests
May 20, 2009 - 12:33 — Vegard A. JohansenThanks Serhiy - it looks like this might be what we are looking for! Here's a screenshot of a test I made which shows the calculations:
So to make sure: the logic / flow is:
For me it looks like this solves the problem, however not entirely in the way Klaus describes above, as you will not be able to get a negative score for MCQs or MAQs. 0 is bottom. However it's still better to not mark if you're unsure as not maked equals 0 points while wrongly marked equals -1 point.
Now the tests are set to be passed when user has reached 80% of the maximum score. This can be adjusted up or down to make the tests easier or harder to pass..
One more issue, though: For this course we have a lot of tests that are taken, but where it seems that the scores are calculated from the original algorithm. I guess the score was calculated and stored when the user took the test, and for the new algorithm to calculate it the user would have to take the test again..?
We cannot ask for that here, so I believe we need some way to "flash" this, so the scores are calculated again, with the new algorithm.. Any ideas how this can be done? If we could manually run a SQL on the db to do that, that would work just fine!
And thanks again, Serhiy! :)
Re: First tests
May 20, 2009 - 13:38 — Serhiy KostyshynYou're welcome. Glad to hear it works for you.
Concerning existing submissions – yes, they remain unchanged. Indeed, you can re-mark them with a SQL query, taking "raw" data from AT_tests_answers (using AT_tests and AT_tests_questions along the way).
In an hour I'm to leave to a conference, so I won't be able to write anything this week; maybe after the weekend (if you don't come up with your own solution by that time).
Cheers!
SQL appreciated :)
May 26, 2009 - 09:12 — Vegard A. Johansenwell, we haven't been able to solve it yet so if you happen to have the SQL in the back of your head you are welcome to jot it down.. :)
(not that I believe you have these kind of SQLs in the back of your head all the time..)
we will probably use the SQL in a very small admin module created to do this and other update tasks we need for the coming certify module.
BTW: as you think this SQL, would it be on a test-by-test basis, or would you update all tests in the system at once?
..I kinda feel bad that every time I get your help I end up asking for more :)
Re: SQL appreciated :)
May 26, 2009 - 09:34 — Serhiy Kostyshyn> (not that I believe you have these kind of SQLs in the back of your head all the time..)
Not whole SQL queries of course, but the basic mental gymnastics with AT_tests* mix-and-matching happens to be always there – it's just used all too often. :)
> BTW: as you think this SQL, would it be on a test-by-test basis, or would you update all tests in the system at once?
And which flavor would you prefer? Limiting by test ID (or even by course ID) would be easy.
Re: SQL appreciated :)
May 26, 2009 - 09:40 — Vegard A. JohansenNot whole SQL queries of course, but the basic mental gymnastics with AT_tests* mix-and-matching happens to be always there – it's just used all too often. :)
Yes, I figured it would be easier for you than for our developer who is not that familiar with that part of ATutor :)
And which flavor would you prefer? Limiting by test ID (or even by course ID) would be easy.
Well, if I can decide, limiting by course ID would probably be just perfect!
Re: SQL appreciated :)
May 27, 2009 - 21:50 — Serhiy KostyshynFine, I'll write a query ASAP.
Not too bad...
June 19, 2009 - 14:37 — klaus melf (not verified)Hi all,
It seems to me that the main problems have been solved. Thank you for that!
I just want to question if it is not possible to give negative scores (e.g. "-1"), and why not? It would not mean very much in MAQs, however quite a lot in MCQ?
Right now we recommend to the student to leave the MCQ blank if the student is not sure about the right answer. And this would mean: 0 points (Leave blank).
But if the student marks a wrong answer (still not knowing the right answer, but using the chance of guessing 20-50% correctly), the student would still have 0 points. Guessing is therefore more favourable than being honest.
I would therefore appreciate if there could be negative scores, at least for the MCQ.
Cheers,
Klaus
See your point
June 24, 2009 - 08:11 — Vegard A. JohansenI see your point Klaus, and it makes sense, but I'm afraid changing this at this point will have too many implication for other courses so we will have to leave it like this atleast for the time being, sorry about that.
Post new comment