Communications of the ACM
Tips for Successful Examinations in the Age of Remote Education
This is the second of two articles sharing experience with online teaching. In the first one, I described a "troika" of techniques that keep students interested and active. Here I will explain an approach to exams and other tests conducted remotely. The disclaimer is the same as before: there is no scientific study behind the advice given here, just a successful experience which I believe can help others.
How do we examine students in the age of Zoom? Many instructors are worried about students' access to the Web and the specter of cheating. The concerns are in fact separate.
Cheating can involve recourse to some third party, as in a traditional exam setting in which a student somehow communicates with an outsider. (Russian readers will remember the cult movie Operation Y.) With a remotely held exam, the helper can be in the exam-taker's immediate physical environment, unseen by the exam-giver; but the Internet opens up many more possibilities, even that of a juicy "remote professional helper" business. (That business already existed pre-Covid, with many sites providing remote homework help for a fee; the Zoom era can give it many new opportunities.)
Access to the Web, outside of cheating in the strict sense, limits the teacher's options for exams. It is important in a classical (non-remote) setting to have the freedom to define, for each exam, what materials are permissible, from nothing at all to some combination of dictionaries, the course's official textbook, the student's personal notes and other specified material. Phones and other communication devices are generally forbidden, a ban that must include computers if it is to be meaningful (unless you want to force students to disable networking, which is hardly realistic). In a remote-examination setting, defining such rules is pointless; by its very definition, such an examination requires good connectivity.
Heavy technical solutions are available from various companies, which put in place various measures to monitor the students regularly. I haven't used any of them, and would do so only in extreme circumstances; not that I take an angelic attitude, as cheating does occur, but I do not see intrusive surveillance and systematic mistrust as desirable components of a constructive and student-friendly pedagogical approach.
As part of cheating avoidance, we have devised at the Schaffhausen Institute of Technology an honor code that all students are required to sign. It would not eliminate the truly dishonest, but I believe it eliminates much of what would otherwise be casual, it-does-not-really matter cheating. (By the way, there is an absolute weapon against cheating, which removes the problem entirely. It's mentioned below -- stay tuned.)
In addition to the honor code, we require students to keep their Webcams turned on all the time and to ask explicitly, through a personal message in the chat area, if they want to leave their computer for a moment. This practice mirrors what happens in a traditional classroom exam with a proctor, whom the students must ask if they want to step out briefly. The possibilities or misuse exist, but so do they, albeit to a lesser extent, in the good old setup (you could have a textbook hidden in the toilet or an accomplice waiting for you in the corridor).
These techniques help make cheating harder but they do not rule out the remote-helper risk and, more importantly, they do not address the second issue identified above: the teacher's inability to define the precise amount of permitted material, based solely on pedagogical criteria. Let me now describe how we handle this problem.
The first step is to realize that it is pointless to hope for the pedagogical latitude afforded by traditional classroom settings. Once you let a cat out of your house, you cannot ensure that it goes to certain places and not to others. In the same way, once a student has a network connection, the whole big Internet is there at his or her disposal.
So what do we do? We recognize that there is no such thing as "a little bit on the Internet": Any hope of limiting or controlling Internet access is pointless. Instead, we make the exam such that if you need the Internet to find the answers, you'll never get far enough.
There are two kinds of exam, which to simplify we may call the French variant and the American variant. (Each variant exists in diverse countries, and there are American exams in France and French exams in the US, but the styles described here in an extreme form are, in my experience, dominant in their respective countries.) In the US an exam is designed for human beings. If you are a good student, you are actually supposed to complete the exam! In France, such an idea would make everyone laugh. The exam is so long and challenging that no normal person would be expected to go through it all; it is an endurance test as much as it is a knowledge test. (Legend has it that when the great Benoît Mandelbrot, future inventor of fractals, took the entrance exam at École Normale Supérieure, where he placed first although he decided to go instead to the equally prestigious Polytechnique, he caused a scandal: he answered all the questions, which none of the examiners had thought would be even possible. They could not believe the results and suspected foul play.) In fact out the maximum grade of 20 (why 20? I just found out from a Web search that the convention dates back to 1890 and that it was devised -- I am not making this up -- to "simplify the system") it would most of the time offend a teacher's honor to give more than an 18.
(An aside: a colleague mentioned to me some time ago that there is a sure-fire way to avoid cheating: make the exam competitive, with a fixed number of seats. Very simple. As anyone who has taken the entrance exams at Polytechnique or École Normale Supérieure -- both highly sought-after schools with many more candidates than positions -- will tell you, no student ever cheats there. No one would even think of cheating. Problem gone. In less competive exams, of course, this solution is not available and the temptation to cheat remains.)
Now I went through both systems, French and US, and actually enjoyed both very much; far from me then to criticize either or choose one as better overall in a normal situation. Most people would naturally point out that cruel and unusual punishment is anti-constitutional and that the US approach makes more sense. But for remote exams the French way has something going for it! Let's assume we stuff and stuff and stuff questions into the exam. To be precise, we make it possible for a good student actually to complete the test (after all, this is not the entrance exam to Normale Sup'), but leave absolutely no slack. So, sure, a student could look up many of the answers, given enough time -- but there just isn't enough time.
A few years ago I heard a talk from a renowned pedagogical expert whose core view was "if they can google it, I don't teach it". Very noble and fashionable meme, I thought (let's be smart and teach skills, not raw knowledge); still, if I am to be operated upon, I am not sure I would be too comfortable at the idea of the surgeon looking up, a few minutes ahead, how many kidneys we are supposed to have. Whether or not it conforms to the latest craze in educational-science fashions, I kind of like the idea that the surgeon internalized that insignificant tiny bit of knowledge a few years before. Of course we need to teach chunks of knowledge. And often, let's face it, a non-negligible part of what we test for in exams involves those chunks, rather than skills, which are harder to assess.
Now with a remote exam such pieces of knowledge can be found, in Wikipedia or wherever. The student who has learned the material will answer correctly. And the student who has not learned the material can also produce the answer: by searching for it! The only way to sort them out is to make sure that if they know the answers they will be able to answer many or most of the questions, and if they have to google them one by one they will run out of time. The result clearly separates the prepared from the lazy.
One final word about skills. If, as I do, you teach software concepts, you may give programming or design exercises. As mentioned in my previous article, we make students program on the Cloud through the Codeboard environment. Codeboard lets the students compile their code (and gives them feedback through compiler error messages, so if they understand the programming language they will be able to fix them, with again the good student -- who produces fewer errors in the first place -- at an advantage, but the less skilled student will get through as well, more slowly). Past compilation, the students can run their code and test it. Codeboard lets us make some of the tests public (typically, we do this with about 60% of the tests) and the rest secret. If some of the secret tests fail, students see how many did, but they can only use the public tests (plus any they devise themselves) to figure out what may be wrong in the program. Note that the overall time for the programming part is not very long (at most a couple of hours), so the programming exercises are of limited difficulty, but they still require most of the available time and are an excellent way to find out if students have mastered the topics.
It also helps, in our case, that we use a non-dominant language, Eiffel. While the whole world should be using Eiffel (and our planet would be a better place if it did), it doesn't yet, with the result that there are fewer programming templates available on the Web. Pick a programming problem or an algorithmic need and look for a solution in Java or Python: often you will find a ready-made code snippet, or many, that you can copy-paste. (In introductory programming education circles, this phenomenon is known as "Google programming".) With Eiffel, this is less often the case. (The growing availability of Eiffel code examples in places like Rosetta code and Stack Overflow may dispel this advantage at some point, but today it still helps make our exams meaningful.)
These are the solutions we use. They have worked well for us so far, enabling us to run exams with no evidence of actual cheating and, more importantly, results that (when correlated to our informal appraisal of students) appear accurately and fairly to reflect the students' actual progress in knowledge and skills. If you too have to examine students remotely, I hope these observations help you. And the best thing: you don't even have to learn French.
Acknowledgment: as with the first article, I am indebted to Marco Piccioni for essential help in devising this course and the associated techniques, as he has done for many others over the years.
Bertrand Meyer is professor and provost at the Schaffhausen Institute of Technology (Switzerland) and chief technology officer of Eiffel Software (Goleta, CA).
No entries found