Wokay!

So the five month old DU Economics post had this reasonably long comments thread which instead of discussing DU or economics itself ended up discussing the stud-fighter framework, the best sort of examination system, what it is that an examination system should actually be examining, and the difference between CBSE Boards and the JEE. I promised a followup post by Sankranti, and then promptly forgot all about it and started blogging about other things like superhero underwear. However, I do get down to things… eventually, so here’s the post. Though by delaying it as much as this, I can now justifiably claim that it’s become topical.

Right. So. Question: what purpose do exams actually serve? I can think of:

1. Establishing a minimum level of competence or knowledge or providing a pass-mark. Examples of this are getting 33% in the CBSE Boards, or 80% in my employer’s HIV-AIDS e-learning module, or 100% in my employer’s money laundering e-learning module (we take money laundering very seriously). Also, CA certification.
2. Creating a filter so that you can select the top – the JEE works this way
3. As a stage-gate between two-levels of education or employment, or between education and employment – in combination with #1 or #2 – the SAT/ GRE work a bit like this – being at the top isn’t essential because there are enough available seats to ensure that you can get in a long way down.
4. To determine how much a student has learnt independent of formally certifying competence or university admission or things like that. For example, the practice tests a tutor or a coaching centre gives so that students can know how far ahead or far behind they are. Or the sample papers which a student practices ahead of the boards.
5. To determine how successful the teacher or the examination system has been in passing on concepts. For example, the Azim Premji Foundation conducts its own tests at Class 5 and Class 8 (I think) for students in Chambal valley and North Karnataka rural schools to measure how well they are performing and which ones deserve aid.

Which is a fabulous framework if you stick to it. In practice, though, here’s the problem in real life. Where the CBSE is concerned, the board exam is being used for purpose #1: certifying minimum competence. Delhi University, on the other hand, uses it for purpose #2: as a filter to select top-performers, which is not what the boards are designed for. The boards are supposed to be high-scoring, easily crackable if you study the entire year, and use questions with standard answers that can be checked against a template to simplify the lives of the unfortunate examiner who has to check hundreds of answer sheets. Is it any surprise that college cutoffs start hitting 97% levels?

(An associated hoopla about the boards is that everyone taking them (and their parents, and their teachers) go up in arms every time a paper is difficult. All things being equal, it doesn’t matter, because a bad paper is just going to push the cutoff down – it’s still going to be the top five hundred people who convert the top five hundred seats. To be fair, all things aren’t equal, because students can have different subject combinations. So if the mathematics exam is particularly tough, someone who’s taken psychology or physical education instead gets an advantage.)

There’s another major problem with using the boards as an entrance / selection exam. To be fair, this is a problem with all entrance exams in India. It’s a one-shot exam which happens only once a year. It doesn’t measure how you perform in the classroom, and your long-term ability to learn, which is something that the teacher on the ground is much better placed to judge. And if you mess up a one-shot examination once, you don’t get a chance to make up for it until the next year. And by mess up I don’t only mean fail, I also mean get any score low enough to prevent you from getting the course/ college you want.

So basically, college admissions boil down to this: you get one opportunity once a year to display your excellence through an exam which is designed not to test expertise but minimum competence. As processes go, this is so thoroughly broken that re-engineering consultants would throw up their hands in despair and suggest restarting from scratch.

Not only is the process broken, it’s broken in a way that disproportionately hurts the poor. Why? Well, because:

1. In the situation described above – you mess up your exams and need to wait another year, the opportunity cost of sitting out another year is much lower. If you’re rich, or even middle class, your family has enough savings (or enough of a standard of living to cut back a little) to not worry about starting your career a year later.
2. If you have a one-year run up to the exams, that gives rich kids a year in which to hire tutors to help them prepare for the exams. This becomes even worse when the CBSE makes papers ‘easy’, because when papers are well-designed and difficult, topping them is truly a function of how smart you are. When they’re easy, students who have time to spare and money for tutors crack the scene. When they’re difficult because they’re badly designed, the top ranks become a lottery.

Which is why the new CBSE exam design scheme, called HOTS (Higher Order Thinking Skills) may turn out to be one of the more sensible things the CBSE has ever done:

HOTS was the new basis of the question papers in the Class 10 exams this year. It is an analytical problem solving process, geared to assess the students’ absorption of knowledge and its application.

“The average performance and pass percentage has increased across the board but the number of perfect scorers has gone down. The new question pattern to judge students’ knowledge base could be the reason,” said Ganguly.

There is a substantial decline in the number of candidates who have bagged the ‘perfect score’ in Mathematics. “As against 5,251 students last year, this year just 2,647 students have scored 100 out of 100 in Mathematics.”

Similarly, in Social Science, only 598 students have scored a full 100 as against 1,233 last year.

(Delhi Live)

In fact, HOTS may have resulted in better government school performance too, which sort of supports my point about conceptual examinations helping the poor:

That the results in government schools have improved is evident: it shows the largest increase in pass percentage, climbing from 77.12 per cent in 2007 to 83.68 per cent. And 100 of the 900-odd schools saw a pass rate of 100 per cent, as compared with 41 schools in 2007.

But Education Minister Arvinder Singh put the improved results largely down to “better teacher training” rather than HOTS.

(Indian Express)

So is it better training or HOTS? Probably both, but I suspect more HOTS than training – the effect of training shows up slowly over time, while HOTS was a sudden change. And I haven’t gone through enough data on the new HOTS papers to judge how conceptual they actually are.

But HOTS still doesn’t fix the core issue which is that university-entrance tests and school-leaving tests should be wholly different beasts. It also doesn’t fix the one-shot problem. Wonderful though the results of putting HOTS into place appear to be, HOTS is an incremental improvement on an examination system that needs radical redesign. Just to reiterate, the pieces which are still missing are:

1. Measuring classroom performance
2. Separating school-leaving competence and university entrance competence for everything, not just professional courses
3. Making examinations better at measuring conceptual skills rather than mugging skills- HOTS is supposed to do this and may actually have accomplished it, but I would like that to be subjected to rigorous testing
4. Redesigning exams (school leaving and university entrance ones) so that taking them over and over, or the amount of time you spend on preparation has less impact on how well you do- I think HOTS is meant to accomplish this as well, as was the JEE redesign a year or so ago.
5. Making university entrance more dependent on multiple factors like extra-curricular skills, conceptual skills, ability to be a stud and a fighter, instead of just a since exam score. But this is probably not going to happen without a massive supply-side expansion of good universities (please see Abi and Ravikiran on this).

Tragically, I have been writing this post over a month and a half, after intending to write it six months ago, and it is now much less coherent than I hoped it would be. My apologies, and I aim to clarify confusion in the comments. Have away!