The Trouble With Exams

June 9, 2008

So the five month old DU Economics post had this reasonably long comments thread which instead of discussing DU or economics itself ended up discussing the stud-fighter framework, the best sort of examination system, what it is that an examination system should actually be examining, and the difference between CBSE Boards and the JEE. I promised a followup post by Sankranti, and then promptly forgot all about it and started blogging about other things like superhero underwear. However, I do get down to things… eventually, so here’s the post. Though by delaying it as much as this, I can now justifiably claim that it’s become topical.

Right. So. Question: what purpose do exams actually serve? I can think of:

  1. Establishing a minimum level of competence or knowledge or providing a pass-mark. Examples of this are getting 33% in the CBSE Boards, or 80% in my employer’s HIV-AIDS e-learning module, or 100% in my employer’s money laundering e-learning module (we take money laundering very seriously). Also, CA certification.
  2. Creating a filter so that you can select the top – the JEE works this way
  3. As a stage-gate between two-levels of education or employment, or between education and employment – in combination with #1 or #2 – the SAT/ GRE work a bit like this – being at the top isn’t essential because there are enough available seats to ensure that you can get in a long way down.
  4. To determine how much a student has learnt independent of formally certifying competence or university admission or things like that. For example, the practice tests a tutor or a coaching centre gives so that students can know how far ahead or far behind they are. Or the sample papers which a student practices ahead of the boards.
  5. To determine how successful the teacher or the examination system has been in passing on concepts. For example, the Azim Premji Foundation conducts its own tests at Class 5 and Class 8 (I think) for students in Chambal valley and North Karnataka rural schools to measure how well they are performing and which ones deserve aid.

Which is a fabulous framework if you stick to it. In practice, though, here’s the problem in real life. Where the CBSE is concerned, the board exam is being used for purpose #1: certifying minimum competence. Delhi University, on the other hand, uses it for purpose #2: as a filter to select top-performers, which is not what the boards are designed for. The boards are supposed to be high-scoring, easily crackable if you study the entire year, and use questions with standard answers that can be checked against a template to simplify the lives of the unfortunate examiner who has to check hundreds of answer sheets. Is it any surprise that college cutoffs start hitting 97% levels?

(An associated hoopla about the boards is that everyone taking them (and their parents, and their teachers) go up in arms every time a paper is difficult. All things being equal, it doesn’t matter, because a bad paper is just going to push the cutoff down – it’s still going to be the top five hundred people who convert the top five hundred seats. To be fair, all things aren’t equal, because students can have different subject combinations. So if the mathematics exam is particularly tough, someone who’s taken psychology or physical education instead gets an advantage.)

There’s another major problem with using the boards as an entrance / selection exam. To be fair, this is a problem with all entrance exams in India. It’s a one-shot exam which happens only once a year. It doesn’t measure how you perform in the classroom, and your long-term ability to learn, which is something that the teacher on the ground is much better placed to judge. And if you mess up a one-shot examination once, you don’t get a chance to make up for it until the next year. And by mess up I don’t only mean fail, I also mean get any score low enough to prevent you from getting the course/ college you want.

So basically, college admissions boil down to this: you get one opportunity once a year to display your excellence through an exam which is designed not to test expertise but minimum competence. As processes go, this is so thoroughly broken that re-engineering consultants would throw up their hands in despair and suggest restarting from scratch.

Not only is the process broken, it’s broken in a way that disproportionately hurts the poor. Why? Well, because:

  1. In the situation described above – you mess up your exams and need to wait another year, the opportunity cost of sitting out another year is much lower. If you’re rich, or even middle class, your family has enough savings (or enough of a standard of living to cut back a little) to not worry about starting your career a year later.
  2. If you have a one-year run up to the exams, that gives rich kids a year in which to hire tutors to help them prepare for the exams. This becomes even worse when the CBSE makes papers ‘easy’, because when papers are well-designed and difficult, topping them is truly a function of how smart you are. When they’re easy, students who have time to spare and money for tutors crack the scene. When they’re difficult because they’re badly designed, the top ranks become a lottery.

Which is why the new CBSE exam design scheme, called HOTS (Higher Order Thinking Skills) may turn out to be one of the more sensible things the CBSE has ever done:

HOTS was the new basis of the question papers in the Class 10 exams this year. It is an analytical problem solving process, geared to assess the students’ absorption of knowledge and its application.

“The average performance and pass percentage has increased across the board but the number of perfect scorers has gone down. The new question pattern to judge students’ knowledge base could be the reason,” said Ganguly.

There is a substantial decline in the number of candidates who have bagged the ‘perfect score’ in Mathematics. “As against 5,251 students last year, this year just 2,647 students have scored 100 out of 100 in Mathematics.”

Similarly, in Social Science, only 598 students have scored a full 100 as against 1,233 last year.

(Delhi Live)

In fact, HOTS may have resulted in better government school performance too, which sort of supports my point about conceptual examinations helping the poor:

That the results in government schools have improved is evident: it shows the largest increase in pass percentage, climbing from 77.12 per cent in 2007 to 83.68 per cent. And 100 of the 900-odd schools saw a pass rate of 100 per cent, as compared with 41 schools in 2007.

But Education Minister Arvinder Singh put the improved results largely down to “better teacher training” rather than HOTS.

(Indian Express)

So is it better training or HOTS? Probably both, but I suspect more HOTS than training – the effect of training shows up slowly over time, while HOTS was a sudden change. And I haven’t gone through enough data on the new HOTS papers to judge how conceptual they actually are.

But HOTS still doesn’t fix the core issue which is that university-entrance tests and school-leaving tests should be wholly different beasts. It also doesn’t fix the one-shot problem. Wonderful though the results of putting HOTS into place appear to be, HOTS is an incremental improvement on an examination system that needs radical redesign. Just to reiterate, the pieces which are still missing are:

  1. Measuring classroom performance
  2. Separating school-leaving competence and university entrance competence for everything, not just professional courses
  3. Making examinations better at measuring conceptual skills rather than mugging skills- HOTS is supposed to do this and may actually have accomplished it, but I would like that to be subjected to rigorous testing
  4. Redesigning exams (school leaving and university entrance ones) so that taking them over and over, or the amount of time you spend on preparation has less impact on how well you do- I think HOTS is meant to accomplish this as well, as was the JEE redesign a year or so ago.
  5. Making university entrance more dependent on multiple factors like extra-curricular skills, conceptual skills, ability to be a stud and a fighter, instead of just a since exam score. But this is probably not going to happen without a massive supply-side expansion of good universities (please see Abi and Ravikiran on this).

Tragically, I have been writing this post over a month and a half, after intending to write it six months ago, and it is now much less coherent than I hoped it would be. My apologies, and I aim to clarify confusion in the comments. Have away!

7 Comments | Education, Uncategorized | Tagged: , , , , , , , , , , , , , , , , , | Permalink
Posted by Aadisht

DU Economics

January 9, 2008

Skimpy has been writing about how people with an Economics background from Delhi University have abysmal conceptual clarity about the subject.

Ok I guess I’m likely to get flak for that comment about the “BA types”. I’ve had the opportunity to interact with a few Economics toppers (undergrad)  from Delhi University. And I’ve found them extremely weak on fundamentals. They know what the graphs look like. They know the definitions well. They know the formulae. But are frequently found wanting when it comes to absolute fundamentals.

Considering that there are hajaar people in my sample (technically, in IITese, anything greater than 2 is hajaar), and given that DU is considered to be the best univ in India for a UG in Economics, I think my generalization is justified.

Coming back, Aadisht says that this lack of fundamentals in these people might be attributed to inappropriate teaching. I won’t rule out that reason. I have the sneaking feeling that lecturers and professors in DU are also mostly from the same background – with an undergrad in Economics. And would have themselves learnt stuff the same way.

When I said inappropriate teaching, I was talking about the terrible textbooks and curriculum more than the teachers themselves. I haven’t seen it for myself, but I got the impression that the fundamentals of microeconomics and macroeconomics form only four papers out of about twenty or thirty in the Economics Honours course at Delhi University. Equal weightage is given to highly arbit courses like Economic History of India and whatnot.

After that GTalk chat, I realised that there are two other important factors at work: the DU admissions system and the DU examination system.

DU admission happens on the basis of your Class 12 board exam marks. Class 12 boards consist of one massive exam at the end of the academic year, with at least a month of study leave leading up to it. DU exams are the pretty much the same. Although some internal evaluation and mid-year exams have been introduced in recent years (I think), the major component of evaluation is still the end-of-year exam with lots of study leading up to it.

To use another framework invented by the Wimp, people who grasp concepts immediately are studs. People who can’t grasp concepts, but make up for it by devoting all their time to mugging and understanding the implications of the concept and how to use it without actually understanding the underlying logic are fighters.

The problem with a massive exam where you get a whole year to prepare for it is that unless you design it very well to test only for conceptual clarity, it ends up obliterating the differences between studs and fighters. The studs will always appreciate surprise quizzes where they can use their understanding of first principles to come up with answers while the more structured, studying-oriented fighters will be caught unawares. Similarly, studs will prefer open-book exams where you have to figure out which first principle to use, and then build theories from the ground up, while the fighters will prefer closed-book exams where they can peacefully obtain marks by regurgitating a given formula or derivation1.

So the DU admission procedure neutralises any advantage studs have over fighters, which leads to the intake consisting largely of fighters. And then three years of an examination process which once again neutralises the advantage any stud in DU might have, means that the toppers will usually end up being mugging-oriented fighters rather than concept-oriented studs.

By contrast, engineering colleges have semester-based continuous evaluation, where the advantage given to a fighter is heavily mitigated. For starters, you have only a four or five month semester to mug, instead of a whole year. Secondly, your concepts keep getting tested throughout the semester. This means that the lead time you have to absorb a concept goes down, and puts additional pressure on fighters.

In the IIMs, the situation is made even more brutal. You have trimesters instead of semesters, and the lead time for fighters to absorb concepts falls to almost Nil. Studs have a clear advantage in this environment (except in courses with lazy profs who set only a midterm and endterm).

Two related points:

  1. A couple of years ago, Annie Zaidi complained that she loved English literature, but used to keep getting outscored by people who didn’t understand it but just mugged it like robots; that this proved that merit in education was nonexistent, and so there was no merit-based argument against reservations.
    Actually, rather than proving that merit in education is nonexistent, it only proves that Annie’s university rewarded meritorious fighters rather than meritorious studs. The policy response therefore is to change the evaluation system, not to proceed with reservations.
  2. The question of what you should be evaluating in an educational system – actual conceptual clarity or the ability to be functional despite a lack of concepts of course remains open. Ideally, an evaluation system would reward both the ability to grasp concepts and derive from first principles, and the ability work with something even if you don’t understand it. But that only reinforces the case for continuous, multi-component evaluation systems.

1: Skimpy’s anguished outburst in the Financial Derivatives class on this very topic will remain forever etched in my memory. In the memory of everyone who attended that class for that matter.

17 Comments | Education | Tagged: , , , , , , , , , , , , , , , , | Permalink
Posted by Aadisht