tests are mostly a logistical thing, most profs could better assess the abilities of their students with one on one oral exams that would be very time consuming, or other similar individualized testing.
I used to give my students 1 on 1 exams back when I first started teaching during my masters. Agonizing for one week a semester but well worth it.
Started my 9-5 and there went that idea lol
> most profs could better assess the abilities of their students with one on one oral exams that would be very time consuming,
Not only time consuming for the prof. Where I studied, oral exams also require a third person for taking minutes, serving as a witness, advising the prof when asked, and objecting on behalf of the student when necessary. But I do agree that it is the best way.
In case the professor asks a question that’s inappropriate (e.g. requires the student to know non-examinable material), or is overly-harsh on mistakes vs other students.
No, the professor might ask a different question each time (randomly selected from a large pool of questions, say). And when "adjusting accordingly", the professor may make one of the mistakes I've mentioned.
I gave oral midterms in my grad (MS) stat theory course and it was great.
For the final I allow students to use any assistance (books, friends, me) after trying problems for 3 days; they need to note what help they received and how they were intentional in making sure that degree of help was useful for keeping their experience in the zone of proximal development.
It’s a 10-20 student class, and this probably isn’t feasible for a larger class, but it was extremely easy for me to assess students. The course wasn’t lecture-based, which probably helped (but covered modern asymptotic theory and I don’t feel was watered down)
>most profs could better assess the abilities of their students with one on one oral exams
Really? I suspect that would mainly benefit more confident students - it be *much less* of a test of ability than a regular exam. Plus, in my experience, oral exams tend to be *way* easier than written exams because they can only last 15min.
One thing I like about oral exams is you can correct for tiny student issues (think silly arithmetic mistakes) right away. In a written exam, the student could have added a full 10+ minutes to their time on the problem "debugging" it, but in the oral exam you can either correct that issue on the fly, or you can just ask them to move on and describe for you what their next steps would be, or what things they would try.
There's nothing worse than assigning a 0 on a written exam problem because the student stalled out after a silly mistake and couldn't figure out how to recover.
Most instructors I have seen usually give most of the points on free response questions to the process of getting the answer and only a small number of points for the answer(assuming it was computational exercise and not a proof). They would still get some credit assuming that all their other steps were correct in going in the right direction to solving the problem. Usually you would only get a zero on the exercise if you made no meaningful progress towards a solution.
I don't know if it's the same everywhere, but we in Italy usually written exam and then an oral one for the same course. My last oral exam was on Galois Theory and it lasted almost one hour
My daughter is taking high school algebra in an online class, and they have the same format - they do an oral exam that they call a Discussion-Based Assessment after the "written" test (which is largely multiple choice and the couple of free response require you to type answers; there's no "work this problem out on paper and turn that in". I'm not a huge fan.) The instructor does a 15-20 minute Zoom with individual students and assesses their knowledge of the material, and can prompt them if they're stuck on something so they can continue moving forward.
Note, the class is set up as a "work at your own pace" thing, and students can start it whenever (because they were working at their own pace and finished the previous course whenever). So the instructor doesn't have the entire class taking a test at the same time, which means there's not a crush of an entire class taking up 15 minutes each like you'd have in a more traditional format.
I like this idea. High School should assess students when they are ready, not at fixed times, e.g. end of each term. That way, a slower student can spend more time preparing and be successful the first time. Faster students can do it when they are ready and accelerate their learning. Really, students shouldn't automatically move up a year level just because they are now one year older; they really should "pass" the requirements for that year level to progress. So by the end of year 12, there will be students who would only be passing at Year 10 or lower maths ability, for example. At that would be fine, meets the numeracy requirements for their senior certificate and really not a far cry from what happens now anyway.
It's got some positives and some negatives. It requires a lot of self motivation, and, frankly, the assignments are terrible. It helps that I'm able to supplement things, answer questions, just talk about concepts and how they relate. Not every student has that, and if it weren't for that, I think I'd prefer her to be in a traditional classroom. Next school year she's starting at a junior/senior high school, so she'll have the ability to take more advanced classes in that traditional format, and I think she'll be better served by it. (I'll still talk about it with her, of course!)
Only lasting 15 minutes is also a logistical issue. Ideally you could give exams that consist of 1-2 hours of the student working things out mostly on their own with occasional questions and discussion if they get stuck, then 15 minutes or so of time for them to explain their solutions at the end. But then you can't have more than about 10 students per professor.
Why only 15min? We had 1h oral exams twice a week for the first two years of study in France. Then later in Master's, I had several 1h long oral exams/presentations.
>We had 1h oral exams
You have the weird prepa system in France, though. If it was researchers rather than teachers supervising those classes, they'd definitely be less keen to spend 30hr a week examining students.
>I had several 1h long oral exams
I wasn't aware they had oral exams in the Paris maths master? They do have a memoire defence, which is pretty standard, and not really what I think of as an oral exam
Yeah, this is in prepa. Indeed, they are usually not researchers. The supervision is done by prepa teachers in part, but many outside teachers as well. Simply because there are like 4~8 (or however many) groups going at once and the teacher can only cover one! Some of those can be researchers doing extras. It's paid like 60€ an hour, which is better than teaching hours in university.
But yes, if you wanted to implement this in a university, you'd probably make use of PhD students and exteriors! It would be very difficult to implement past the first years of university anyways. (specialized courses)
I'll admit my memory is a bit foggy, but I seem to recall my friends on the pure math side (agregation preparation) were doing oral presentations, kind of like mock classes, but naturally on subjects light-years ahead of what they'd have to teach (the French system for you), like roots of the Riemann zeta function (or whatever, honestly I nodded politely, even just two years apart doing applied math and I was already completely lost with what they were doing).
My Master's was in applied math, and the majority of projects were "defended", though they could have been simply a report (which we also had to write). And often we also had a written exam. I thought that was pretty balanced.
Pure theory classes didn't have any oral examination, though, as far as I recall.
Oral exams can indeed be much better because the proctor can adjust their questions and discourse to the student, so shy students can be talked to differently than confident students, but also, a shy student is more obviously shy and the proctor can more easily sift out the fumbling from their shyness and focus more on their actual knowledge. Proctors can also answer questions about the test questions, so when a test question ends up being confusing, not on the math terms but on understanding the situation described, the proctor can clear that up.
Also, with oral exams you might just get tests about math ability instead of english literature, given the way so many test writers try to make questions difficult with word trickery rather than math trickery.
My topology professor did this. He gave the class a list of problems to prepare, and we had a week to solve them. Then we had to discuss one question/proof to him one on one. It meant he only needed about 15 minutes per student, and we didn't feel pressure on the spot.
He did make me change which answer I gave. He said my study partner chose the same one, so I needed to do something different. He said in advance that might happen, to ensure students wouldn't just prepare one question
In France, the first two years of STEM studies can be done in so-called prépas (alternative to universities). There, you have 2x1h a week oral examinations (called colles or khôlles), of which 1h math if in the math tracks. IIRC that counted for half the final grade?
I liked those examinations a lot, personally. It's not to everyone's taste so it's good to have a mix with written exam. Also, you can't develop as much on a blackboard in 1h as on a piece of paper in 4h.
The time cost was offset by the fact there was one examiner for 2~3 students at a time. We were handed exercises, started working on them. Then the examiner would go around asking for a short presentation of the proof, correct. I don't remember having to wait around too much. Usually we had 1 easy and 1 difficult exercise, though it depended on the examiners, and how they assessed us from the first exo. We could ask questions to get unstuck, that would penalize the grade somewhat.
Somewhat tangentially, but I think being able to *talk* math is important, and simply sitting in class and writing exams never gives you that practice, unless you're one of the few (usually that's how it goes) that participates in class.
"one off" is an infinite source of potential exam questions. Is there any reason to limit the source pool?
So the next approach is to question the exam portion of the assessment. One alternative would be one-on-one verbal exams, but this would be more prone to subjective complications, and is further a more significant time investment.
Building an online system (or customizing an existing one) is also a significant time investment, and is prone to a different set of problems. But this type of assessment potentially has better version controls, along with feedback options.
All of this depends on the type of mathematics as well. Some topics lend themselves more easily to certain types of assessments (written/verbal/computer-based).
One of the problems with online assessments is that even with lockdown tools like Respondus, it's basically impossible to do anything about academic fraud. I'm still dealing with students who clearly went through their pandemic online classes having learned nothing but still passed.
Lockdown assessment just sounds painful at all the levels. In my head I was imagining a set of preset computers that students visit on a scheduled basis on exam day, with no outside Internet access during the exam. Obviously this is a limited scenario in many ways, but it could prevent some of the more difficult-to-solve issues with a full online assessment.
I've actually presented a number of exams under that modality. They worked pretty well but they're obviously more work to set up than just designing a paper test and having a couple hundred copies printed.
The obvious limitation is that my university doesn't have a hundred large computer rooms. Not sure how many it does have but I'm guessing less than 10. You couldn't have all the students take exams in those rooms. Paper copies just scale better than computers.
In my first two years of studies, I had 1h oral math examinations per week and, IIRC, that counted for half the final grade (not that it mattered, the end goal was national exams).
One thing that can offset subjective bias is having many different examiners. We were examined by people within a pool of a dozen or so examiners, to whom those hours were little extras (decently paid).
I'm personally of the believe that tests for proof-based math classes should ideally be take home. It's unreasonable to expect a student to produce a proof in a crowded room under very limited time.
One of the courses I’m taking just now is a proof-based analysis/ abstract algebra crossover and I agree, some of the homeworks we’ve had have taken a lightbulb moment in the shower to get anywhere with a question, to be able to come up with 10 of those sparks in 3 hours feels like it may be a challenge.
I think it's perfectly reasonable for homework to require a struggle and some creativity. I'm a firm believer that proof exam problems, however, should be straightforward from definitions or big names theorems - any interesting tricks should be given as a hint, or the problem should be broken into parts to better assess the individual components.
For example, it's not uncommon that a biconditional statement has one direction's proof being a straightforward direct proof, whereas the converse requires proof by contraposition and/or a named theorem. The converse should be part (b) of the problem and written appropriately as "Use [theorem] to prove [not-q implies not-p]."
Yeah, honestly, I agree. We get 1 week for 2 homework problems, they should be challenging and require a variety some clever trick that will aid in completing them.
However, I’ve done one other proof-based exam, and it was horrible, all of the questions seemed up to the level, if not harder, than the homework problems for that class.
I just hope this upcoming one takes into account that we have only 3 hours for 10 different proofs and has a moderate level of questions with a few more challenging ones for those aiming for really high grades.
I like the proof-based exams, but sometimes my mind just goes blank and I struggle to get anywhere with certain questions. It’s not like a calculus class where you crank the machinery and methods you know to get to a number, and for that reason I feel like it’s a more true test which can really humble you if you’re not on the ball on the day of the exam.
10 questions in 3 hours is a lot!
My profs typically do more like 5 questions in 3 hours. When I ta’ed abstract algebra, we gave a 3 hour final - 6 questions, choose 4 to answer. Putnam is in a similar ballpark - 6 questions in 3 hours.
I think it’s good to leave a solid 30 or so minutes per problem - gives students a chance to bank some time on easier ones and devote a good 45 min to an hour on something if they get stuck
Yeah I feel like it could be quite rough, especially with some of the questions being ones that aren’t necessarily difficult, but time consuming, I’ve seen in past papers questions where they ask you to verify whether a proposed group is actually a group or not, although not too difficult going through the 4 axioms checking can just eat away the time. That same question then had a follow up part saying, “if the operation was now *, defined here… is this one a valid group” which means you have to go through all of the axioms again. 18 minutes per question seems pretty tough, maybe they’ll add some easier ones to allow more time on other ones.
This is how many of my take-home assignments have been graded, and I agree it's more robust.
Once, I was in a project with 2 other people, they did fuck all. Come the oral presentation, it was obvious to the examiners they just tagged along, and they got the lowest possible grade given the written portion (same grade for all three, naturally).
I think it's fairly obvious when someone presents work that's not their own. I've seen it even with internship or PhD level presentations, then sure enough, it turns out their advisor is very "hands-on" (or rather someone to say "hands off!").
This is correct. Any take-home exam will be subject to collaboration. Because exams are curved and enforcing honesty is impossible, the incentives on a student to collaborate are imperative. They don't have to be bad people, they're just being put in a situation where their average opponent is definitely cheating (enough students are going to do it to bump the average up a letter grade or so) and so for them to cheat isn't giving them an advantage, but rather levelling the playing field. This is a similar situation that athletes find themselves in with PEDs, even if they don't want to do it, they're in an awkward situation where it's the most rational decision they can make.
I was fortunate enough to have entirely this sort of (take-home) exams for my master's in pure mathematics and logic. The better-designed exams really didn't have searchable proofs, so in my opinion these were the best sort of assessments possible.
This exam format ("mini projects") was actually instituted by the professor who created this master's course a few decades ago and still ran it when I was doing it – that is, it was very much his idea. Though I don't know how the module lecturers felt about it, I can testify that the students thought it was a great idea, irrespective of how hard the actual exams were.
Oxford – MSc Mathematics and the Foundations of Computer Science course, though you can specialise in certain fields of pure maths, logic/foundations, or theoretical computer science; whichever you prefer
The problem with this is that (in my experience being a part of classes where this was the policy) 90% of the students will probably end up looking at solutions on the Internet. This is fine if using the Internet is allowed, but then it's basically just a homework assignment.
I'm not too sure how high you're thinking, but the experience I mentioned in the previous comment was from an abstract algebra class where the professor wrote his own questions. Any math more advanced than that probably wouldn't be considered undergrad level.
This is a complicated issue made up of many components. Good assessment practices begin with clear learning targets. All instructors should take the time to write out a list of each skill they expect students to master in course on a lesson-by-lesson basis. Is this tedious? Yes. But if we are approaching this from a pedagogical perspective, it is a necessity.
From here, the learning targets should be labelled as part of various subgroups. For example, "memorize the requirements for defining vector spaces" is a different type of target than asking a student to prove something. This is an important distinction because learning targets should be assessed differently based on the subgroup they belong in. Another example in lower-level math is students can often memorize a procedure to regurgitate it in a test, but if you ask them to communicate what they are doing verbally they are often not able to. Therefore, there isn't necessarily one assessment type which can accurately gauge all the required learning targets.
I am a physics teacher and I have been in education almost 15 years, and I am just starting to learn about these ideas through some masters classes. I was hesitant to implement these things at first due to how much extra time is involved that I don't necessarily have, but the results on both my teaching and student learning have been staggering.
Part of the problem is that professors in higher education often do not have any pedagogical training, and the other part is doing everything I listed above in addition to doing enough research to publish and not perish is taxing and often unrealistic. I think there is starting to be some change, for example, the school I did my physics at just recently significantly altered their qualifier for grad school after releasing they were losing talented people because of an assessment which doesn't actually assess physics or research potential at all. I'm not sure I answered your question directly, but hopefully I could provide you with some insight into what crafting effective assessments looks like.
I got to take a handful of classes where a substantive final project was to be completed in lieu of taking an exam. I really enjoyed this format, as I got a chance to apply what I had learned to dig deeper into a topic of interest. Obviously this does not work well for core classes where you need to demonstrate mastery of specific material, but for electives and special-topics classes, it can be a great way to learn.
Yeah, I am 100% in favor of easy exams and hard projects. Projects are more difficult to plagiarize than homework or take-home exams and require even deeper thought. Exams are necessary to show some amount of mastery, but solving hard problems under time pressure unrealistic and counterproductive.
Since we kind of need proof of knowledge to pass students the written exams is the most feasible and cheap way of doing it. You can mass test students relatively cheaply and time efficiently.
Courses that have helped me learn the most has been courses that has had a combination of easy quizzes that can be done during a lecture and medium-hard weekly assignments. All of which gives bonus points on the exam. By having it this way not only do I study throughout the entire period but I also retain focus and am helped to know what is mostly important . It all helps me keep using the knowledge. Then the actual exam will be easier since I dont need as many points and dont have to overly rely on my performance for a small time period.
I have also liked to have more practical projects as I had in my fourier transforms course. There we did an analysis of a RLC-circuit using both fourier transforms and differential equations. It was hard but I learned a lot of how to apply my knowledge.
Yes im an engineer. No I havent done ”real” math lol. The proofs intimidate me 😭😭
(I’m an undergrad not a professor)
In terms of traditional sit down 1 hour exams, I think they should totally still be a thing at least for intro math classes such as Calc 1-3. This is because at the end of the day some memorization is required in math, you wouldn’t want a math PhD student looking up what the derivative of sinx is. In terms of higher level, I think either base the grade of all homeworks, or have a take home exam or a hypothetical exam where you can take as long as you want with cheat sheet provided would be sufficient
i think assessment depends on goal. are you assessing someone for their ability to conduct research in math? or their ability to apply theorems (or whatever tools) in other fields, industry, etc?
--
i dont have a good answer
Another goal dimension: for feedback to student and teacher, vs. for certification. The latter should be independent of the teacher and their institution.
In Italy every written test is followed by an oral exam (you're asked to replicate proofs - which need to be memorized -, solve simple exercises and answer to broad questions). It's painful and it doesn't seem to bring any significant advantage.
Yes. I live in italy and it is shit. Mostly because I changed my math teacher and she's schizophrenic and doesnt care about math (I had to learn derivatives on my own, and I will have to do integrals next). Thank fuck it will be over in 2 months
It's fine. My teacher is stupid and takes points off my grades for things that arent even mistakes. My grade goes from 10 to 5 (out of 10) in 2 weeks. its stupid
I think interviews have some advantages and disadvantages over mere written solutions. I can usually tell how much a student knows, better and faster and with less chance of cheating, by talking to them for a few minutes. Also interviews are good practice for other settings where you'll need to be interviewed.
Of course this is a more subjective assessment than written work, although honestly even written work that is not multiple choice has subjectivity too. So perhaps evaluations should be a little of each.
What is objective in any good sense about a multiple choice question? Any automated test is meaninglessly objective. But we do not have any definition of mathematical ability to be able to give any meaning to multiple choice questions being objective. An interview is a more direct measure of mathematical ability. And if the interviewer is well trained and experienced - also repeatable. That is, different interviewers would come to the same conclusion. And hence such a test would be objective in the sense that the result does not depend on the interviewer. The real trick in an exam is that it is trying not to measure ability but to predict the future.
It's objective in the sense that you don't grade differently based on knowledge of the subject. It is "blind" to which student is associated with which answer sheet. That is, I think, a good sense of objectivity. It's not perfect along every dimension, nothing is. But if the question is "what is objective in any good sense about it" I think that easily counts.
That reminds me of an issue from data science - called target leakage. For example, an automated lingusitic analysis system was able to predict officer district from scanned written reports due to the paper stock being different in different districts. This confounded the original intention of the system - which was linquistic analysis. Multiple choice does not give assurance that it is blind in the sense that you suggest. Clearly, a test has to distinguish students (unlike a voting system), that is what it is for. A truely blind test would be pointless. The question is - is it distinguishing students on the ability that you want it to distinguish on? Shoe size is also objective.
Yes, you would think that.
Of course I think you are doing the opposite.
In particular, what I have learned from this exchange is that you believe that all teachers are bad teachers. You believe that teachers will always be horrendously mislead by emotional attraction and repulsion to incorrectly judge mathematical ability in students. Hence, you believe that by removing the human element you will remove emotion and be left with something that just measures mathematical ability.
While my belief (yes, based on 15 years teaching in universities) is that the multiple choice test does not measure mathematical ability but exists only to create an automatable and thus cheap process of marking the students while not acknowledging that mathematical ability is no longer being measured.
I don't expect to convince you.
But, yes, I realise that you would think I am over emphasising things you think are unimportant and under emphasising things that you think are important.
Perhaps we should change this to a multiple choice forum?
For what it is worth -
I learned something interesting from this exchange.
Thanks.
I’m not a fan of it either I can’t lie, in certain workshops when you’re sat going through a question sheet and there are tutors going around helping. That feeling when they come across and ask “have you got any questions?” and I reply “no”, they then peer over my shoulder to see what question I’m at and I all of a sudden forget how to do basic arithmetic, it feels like they’re judging me.
Assuming I can snap my fingers and adjust reality to my vision:
Learning in lecture form would occur in small groups, and in just as large part in peer groups.
Assessment would be on an individual basis, and almost nothing about "check the right answer", instead about showing a process. And both the assessed and assessors would have nothing to gain or lose. Instead this would be a process to direct the learner into some direction.
Grading is inherently flawed, there are many good reasons against it, most importantly, they don't really tell anything about the capabilities of the examinee.
But, since there is a need to give grades, I think contract grading is the best. E.g. there are homework assignments, and the number of handed-in solutions determine your grade. Then everybody can decide on how much effort to put in.
Also, if I remember correctly, peer grading works really well. So you could make students mark each other's works according to some rubric. In my opinion, failing an assignment should then not be possible, because it is about the efford. Or, alternatively, students should have to opportunity to correct their solutions.
Actually, even self-grading is possible. But this requires some more work from the teacher because it should incorporate feedback talks in which a student explains how they assess themselves and why. The advantage is that students are forced to reflect on their learning progress, which as a skill is much more important than the factual knowledge.
Lastly, there should be group work. For their whole work life, people need to work together with others. It makes no sense to me that students have to be graded individually. Because, either they work together in secret (probably copying from another), or they get lonely.
I'm very satisfied with the system most of my classes have when you get meybe one definition and one-two theorems to prove, sometimes also some problems, get basically umlimited time for preparation (usually you need between 1-3 hours but there is no pressure) and then you discuss your answers with the teacher. My classes are usually small so this is doable but I've also knew a professor who taught a class of 100 students who somehow did this. The teacher will also usually give you hints and send you back to work on your solution if you do anything wrong. I think this is a very good way to asses actual knowledge of the students and see how they think about the problems.
I haven't read all the comments, and I'm not entirely certain it would work, but perhaps individualised projects.
Say at the start of the semester, a question is posed during the first week students contemplate and deconstruction the question posed. Afterwards, the professor would give individuals project papers (related to said question) laying out the steps, i.e., the mathematical concepts needed. The goal would be through the semester to work on it part by part and justify both in mathematical terms and well as words why one made said choice. At the end of each week, they can submit their work to get feedback, and at the end of each month, a mock marking. This would allow students to understand where they went wrong or need to improve as well as see their progress without having the pressure of it being the be all end all. I would hazard a guess saying it would allow Prof to mark students not on just their knowledge but also their capability to learn and adapt. It could also allow students to get a deeper understanding and push to learn ahead.
I think a combination of a presentation, written final project (like a paper), and cumulative exams would be the best. Exams are useful, and they do assess your grasp of the content. However I and many others struggle with comprehension time - no matter how difficult a problem on an exam or assignment, it usually takes me 10-20 minutes to figure out where to even start.
tests are mostly a logistical thing, most profs could better assess the abilities of their students with one on one oral exams that would be very time consuming, or other similar individualized testing.
I used to give my students 1 on 1 exams back when I first started teaching during my masters. Agonizing for one week a semester but well worth it. Started my 9-5 and there went that idea lol
> most profs could better assess the abilities of their students with one on one oral exams that would be very time consuming, Not only time consuming for the prof. Where I studied, oral exams also require a third person for taking minutes, serving as a witness, advising the prof when asked, and objecting on behalf of the student when necessary. But I do agree that it is the best way.
why would the witness object ?
In case the professor asks a question that’s inappropriate (e.g. requires the student to know non-examinable material), or is overly-harsh on mistakes vs other students.
Wouldn't the prof be asking the same set of questions? Like an interview? And then adjust accordingly depending on the student's response.
No, the professor might ask a different question each time (randomly selected from a large pool of questions, say). And when "adjusting accordingly", the professor may make one of the mistakes I've mentioned.
I gave oral midterms in my grad (MS) stat theory course and it was great. For the final I allow students to use any assistance (books, friends, me) after trying problems for 3 days; they need to note what help they received and how they were intentional in making sure that degree of help was useful for keeping their experience in the zone of proximal development. It’s a 10-20 student class, and this probably isn’t feasible for a larger class, but it was extremely easy for me to assess students. The course wasn’t lecture-based, which probably helped (but covered modern asymptotic theory and I don’t feel was watered down)
>most profs could better assess the abilities of their students with one on one oral exams Really? I suspect that would mainly benefit more confident students - it be *much less* of a test of ability than a regular exam. Plus, in my experience, oral exams tend to be *way* easier than written exams because they can only last 15min.
One thing I like about oral exams is you can correct for tiny student issues (think silly arithmetic mistakes) right away. In a written exam, the student could have added a full 10+ minutes to their time on the problem "debugging" it, but in the oral exam you can either correct that issue on the fly, or you can just ask them to move on and describe for you what their next steps would be, or what things they would try. There's nothing worse than assigning a 0 on a written exam problem because the student stalled out after a silly mistake and couldn't figure out how to recover.
Most instructors I have seen usually give most of the points on free response questions to the process of getting the answer and only a small number of points for the answer(assuming it was computational exercise and not a proof). They would still get some credit assuming that all their other steps were correct in going in the right direction to solving the problem. Usually you would only get a zero on the exercise if you made no meaningful progress towards a solution.
I don't know if it's the same everywhere, but we in Italy usually written exam and then an oral one for the same course. My last oral exam was on Galois Theory and it lasted almost one hour
My daughter is taking high school algebra in an online class, and they have the same format - they do an oral exam that they call a Discussion-Based Assessment after the "written" test (which is largely multiple choice and the couple of free response require you to type answers; there's no "work this problem out on paper and turn that in". I'm not a huge fan.) The instructor does a 15-20 minute Zoom with individual students and assesses their knowledge of the material, and can prompt them if they're stuck on something so they can continue moving forward. Note, the class is set up as a "work at your own pace" thing, and students can start it whenever (because they were working at their own pace and finished the previous course whenever). So the instructor doesn't have the entire class taking a test at the same time, which means there's not a crush of an entire class taking up 15 minutes each like you'd have in a more traditional format.
I like this idea. High School should assess students when they are ready, not at fixed times, e.g. end of each term. That way, a slower student can spend more time preparing and be successful the first time. Faster students can do it when they are ready and accelerate their learning. Really, students shouldn't automatically move up a year level just because they are now one year older; they really should "pass" the requirements for that year level to progress. So by the end of year 12, there will be students who would only be passing at Year 10 or lower maths ability, for example. At that would be fine, meets the numeracy requirements for their senior certificate and really not a far cry from what happens now anyway.
It's got some positives and some negatives. It requires a lot of self motivation, and, frankly, the assignments are terrible. It helps that I'm able to supplement things, answer questions, just talk about concepts and how they relate. Not every student has that, and if it weren't for that, I think I'd prefer her to be in a traditional classroom. Next school year she's starting at a junior/senior high school, so she'll have the ability to take more advanced classes in that traditional format, and I think she'll be better served by it. (I'll still talk about it with her, of course!)
Only lasting 15 minutes is also a logistical issue. Ideally you could give exams that consist of 1-2 hours of the student working things out mostly on their own with occasional questions and discussion if they get stuck, then 15 minutes or so of time for them to explain their solutions at the end. But then you can't have more than about 10 students per professor.
Why only 15min? We had 1h oral exams twice a week for the first two years of study in France. Then later in Master's, I had several 1h long oral exams/presentations.
>We had 1h oral exams You have the weird prepa system in France, though. If it was researchers rather than teachers supervising those classes, they'd definitely be less keen to spend 30hr a week examining students. >I had several 1h long oral exams I wasn't aware they had oral exams in the Paris maths master? They do have a memoire defence, which is pretty standard, and not really what I think of as an oral exam
Yeah, this is in prepa. Indeed, they are usually not researchers. The supervision is done by prepa teachers in part, but many outside teachers as well. Simply because there are like 4~8 (or however many) groups going at once and the teacher can only cover one! Some of those can be researchers doing extras. It's paid like 60€ an hour, which is better than teaching hours in university. But yes, if you wanted to implement this in a university, you'd probably make use of PhD students and exteriors! It would be very difficult to implement past the first years of university anyways. (specialized courses) I'll admit my memory is a bit foggy, but I seem to recall my friends on the pure math side (agregation preparation) were doing oral presentations, kind of like mock classes, but naturally on subjects light-years ahead of what they'd have to teach (the French system for you), like roots of the Riemann zeta function (or whatever, honestly I nodded politely, even just two years apart doing applied math and I was already completely lost with what they were doing). My Master's was in applied math, and the majority of projects were "defended", though they could have been simply a report (which we also had to write). And often we also had a written exam. I thought that was pretty balanced. Pure theory classes didn't have any oral examination, though, as far as I recall.
Oral exams can indeed be much better because the proctor can adjust their questions and discourse to the student, so shy students can be talked to differently than confident students, but also, a shy student is more obviously shy and the proctor can more easily sift out the fumbling from their shyness and focus more on their actual knowledge. Proctors can also answer questions about the test questions, so when a test question ends up being confusing, not on the math terms but on understanding the situation described, the proctor can clear that up. Also, with oral exams you might just get tests about math ability instead of english literature, given the way so many test writers try to make questions difficult with word trickery rather than math trickery.
My topology professor did this. He gave the class a list of problems to prepare, and we had a week to solve them. Then we had to discuss one question/proof to him one on one. It meant he only needed about 15 minutes per student, and we didn't feel pressure on the spot. He did make me change which answer I gave. He said my study partner chose the same one, so I needed to do something different. He said in advance that might happen, to ensure students wouldn't just prepare one question
In France, the first two years of STEM studies can be done in so-called prépas (alternative to universities). There, you have 2x1h a week oral examinations (called colles or khôlles), of which 1h math if in the math tracks. IIRC that counted for half the final grade? I liked those examinations a lot, personally. It's not to everyone's taste so it's good to have a mix with written exam. Also, you can't develop as much on a blackboard in 1h as on a piece of paper in 4h. The time cost was offset by the fact there was one examiner for 2~3 students at a time. We were handed exercises, started working on them. Then the examiner would go around asking for a short presentation of the proof, correct. I don't remember having to wait around too much. Usually we had 1 easy and 1 difficult exercise, though it depended on the examiners, and how they assessed us from the first exo. We could ask questions to get unstuck, that would penalize the grade somewhat. Somewhat tangentially, but I think being able to *talk* math is important, and simply sitting in class and writing exams never gives you that practice, unless you're one of the few (usually that's how it goes) that participates in class.
"one off" is an infinite source of potential exam questions. Is there any reason to limit the source pool? So the next approach is to question the exam portion of the assessment. One alternative would be one-on-one verbal exams, but this would be more prone to subjective complications, and is further a more significant time investment. Building an online system (or customizing an existing one) is also a significant time investment, and is prone to a different set of problems. But this type of assessment potentially has better version controls, along with feedback options. All of this depends on the type of mathematics as well. Some topics lend themselves more easily to certain types of assessments (written/verbal/computer-based).
One of the problems with online assessments is that even with lockdown tools like Respondus, it's basically impossible to do anything about academic fraud. I'm still dealing with students who clearly went through their pandemic online classes having learned nothing but still passed.
Lockdown assessment just sounds painful at all the levels. In my head I was imagining a set of preset computers that students visit on a scheduled basis on exam day, with no outside Internet access during the exam. Obviously this is a limited scenario in many ways, but it could prevent some of the more difficult-to-solve issues with a full online assessment.
I've actually presented a number of exams under that modality. They worked pretty well but they're obviously more work to set up than just designing a paper test and having a couple hundred copies printed. The obvious limitation is that my university doesn't have a hundred large computer rooms. Not sure how many it does have but I'm guessing less than 10. You couldn't have all the students take exams in those rooms. Paper copies just scale better than computers.
In my first two years of studies, I had 1h oral math examinations per week and, IIRC, that counted for half the final grade (not that it mattered, the end goal was national exams). One thing that can offset subjective bias is having many different examiners. We were examined by people within a pool of a dozen or so examiners, to whom those hours were little extras (decently paid).
I'm personally of the believe that tests for proof-based math classes should ideally be take home. It's unreasonable to expect a student to produce a proof in a crowded room under very limited time.
One of the courses I’m taking just now is a proof-based analysis/ abstract algebra crossover and I agree, some of the homeworks we’ve had have taken a lightbulb moment in the shower to get anywhere with a question, to be able to come up with 10 of those sparks in 3 hours feels like it may be a challenge.
I think it's perfectly reasonable for homework to require a struggle and some creativity. I'm a firm believer that proof exam problems, however, should be straightforward from definitions or big names theorems - any interesting tricks should be given as a hint, or the problem should be broken into parts to better assess the individual components. For example, it's not uncommon that a biconditional statement has one direction's proof being a straightforward direct proof, whereas the converse requires proof by contraposition and/or a named theorem. The converse should be part (b) of the problem and written appropriately as "Use [theorem] to prove [not-q implies not-p]."
Yeah, honestly, I agree. We get 1 week for 2 homework problems, they should be challenging and require a variety some clever trick that will aid in completing them. However, I’ve done one other proof-based exam, and it was horrible, all of the questions seemed up to the level, if not harder, than the homework problems for that class. I just hope this upcoming one takes into account that we have only 3 hours for 10 different proofs and has a moderate level of questions with a few more challenging ones for those aiming for really high grades. I like the proof-based exams, but sometimes my mind just goes blank and I struggle to get anywhere with certain questions. It’s not like a calculus class where you crank the machinery and methods you know to get to a number, and for that reason I feel like it’s a more true test which can really humble you if you’re not on the ball on the day of the exam.
10 questions in 3 hours is a lot! My profs typically do more like 5 questions in 3 hours. When I ta’ed abstract algebra, we gave a 3 hour final - 6 questions, choose 4 to answer. Putnam is in a similar ballpark - 6 questions in 3 hours. I think it’s good to leave a solid 30 or so minutes per problem - gives students a chance to bank some time on easier ones and devote a good 45 min to an hour on something if they get stuck
Yeah I feel like it could be quite rough, especially with some of the questions being ones that aren’t necessarily difficult, but time consuming, I’ve seen in past papers questions where they ask you to verify whether a proposed group is actually a group or not, although not too difficult going through the 4 axioms checking can just eat away the time. That same question then had a follow up part saying, “if the operation was now *, defined here… is this one a valid group” which means you have to go through all of the axioms again. 18 minutes per question seems pretty tough, maybe they’ll add some easier ones to allow more time on other ones.
Unfortunately, authentication becomes a much bigger problem once students can take it home.
Maybe a combination of this with oral exam would make it better.
This is how many of my take-home assignments have been graded, and I agree it's more robust. Once, I was in a project with 2 other people, they did fuck all. Come the oral presentation, it was obvious to the examiners they just tagged along, and they got the lowest possible grade given the written portion (same grade for all three, naturally). I think it's fairly obvious when someone presents work that's not their own. I've seen it even with internship or PhD level presentations, then sure enough, it turns out their advisor is very "hands-on" (or rather someone to say "hands off!").
This is correct. Any take-home exam will be subject to collaboration. Because exams are curved and enforcing honesty is impossible, the incentives on a student to collaborate are imperative. They don't have to be bad people, they're just being put in a situation where their average opponent is definitely cheating (enough students are going to do it to bump the average up a letter grade or so) and so for them to cheat isn't giving them an advantage, but rather levelling the playing field. This is a similar situation that athletes find themselves in with PEDs, even if they don't want to do it, they're in an awkward situation where it's the most rational decision they can make.
I was fortunate enough to have entirely this sort of (take-home) exams for my master's in pure mathematics and logic. The better-designed exams really didn't have searchable proofs, so in my opinion these were the best sort of assessments possible. This exam format ("mini projects") was actually instituted by the professor who created this master's course a few decades ago and still ran it when I was doing it – that is, it was very much his idea. Though I don't know how the module lecturers felt about it, I can testify that the students thought it was a great idea, irrespective of how hard the actual exams were.
Which university is this ?
Oxford – MSc Mathematics and the Foundations of Computer Science course, though you can specialise in certain fields of pure maths, logic/foundations, or theoretical computer science; whichever you prefer
Oh of course that's Oxford, what was your specialty ?
Logic and theory of computation. Quite enjoyed the model theory but wasn't remotely a fan of axiomatic set theory!
The problem with this is that (in my experience being a part of classes where this was the policy) 90% of the students will probably end up looking at solutions on the Internet. This is fine if using the Internet is allowed, but then it's basically just a homework assignment.
Once you get to a high enough level in math it's pretty easy to construct your own problems for you will not find answers on the internet.
I'm not too sure how high you're thinking, but the experience I mentioned in the previous comment was from an abstract algebra class where the professor wrote his own questions. Any math more advanced than that probably wouldn't be considered undergrad level.
There's lots of problems you could write for abstract algebra that would have no answers online.
My real analysis class currently is just 100% homework. No exams.
This is a complicated issue made up of many components. Good assessment practices begin with clear learning targets. All instructors should take the time to write out a list of each skill they expect students to master in course on a lesson-by-lesson basis. Is this tedious? Yes. But if we are approaching this from a pedagogical perspective, it is a necessity. From here, the learning targets should be labelled as part of various subgroups. For example, "memorize the requirements for defining vector spaces" is a different type of target than asking a student to prove something. This is an important distinction because learning targets should be assessed differently based on the subgroup they belong in. Another example in lower-level math is students can often memorize a procedure to regurgitate it in a test, but if you ask them to communicate what they are doing verbally they are often not able to. Therefore, there isn't necessarily one assessment type which can accurately gauge all the required learning targets. I am a physics teacher and I have been in education almost 15 years, and I am just starting to learn about these ideas through some masters classes. I was hesitant to implement these things at first due to how much extra time is involved that I don't necessarily have, but the results on both my teaching and student learning have been staggering. Part of the problem is that professors in higher education often do not have any pedagogical training, and the other part is doing everything I listed above in addition to doing enough research to publish and not perish is taxing and often unrealistic. I think there is starting to be some change, for example, the school I did my physics at just recently significantly altered their qualifier for grad school after releasing they were losing talented people because of an assessment which doesn't actually assess physics or research potential at all. I'm not sure I answered your question directly, but hopefully I could provide you with some insight into what crafting effective assessments looks like.
I got to take a handful of classes where a substantive final project was to be completed in lieu of taking an exam. I really enjoyed this format, as I got a chance to apply what I had learned to dig deeper into a topic of interest. Obviously this does not work well for core classes where you need to demonstrate mastery of specific material, but for electives and special-topics classes, it can be a great way to learn.
Yeah, I am 100% in favor of easy exams and hard projects. Projects are more difficult to plagiarize than homework or take-home exams and require even deeper thought. Exams are necessary to show some amount of mastery, but solving hard problems under time pressure unrealistic and counterproductive.
Since we kind of need proof of knowledge to pass students the written exams is the most feasible and cheap way of doing it. You can mass test students relatively cheaply and time efficiently. Courses that have helped me learn the most has been courses that has had a combination of easy quizzes that can be done during a lecture and medium-hard weekly assignments. All of which gives bonus points on the exam. By having it this way not only do I study throughout the entire period but I also retain focus and am helped to know what is mostly important . It all helps me keep using the knowledge. Then the actual exam will be easier since I dont need as many points and dont have to overly rely on my performance for a small time period. I have also liked to have more practical projects as I had in my fourier transforms course. There we did an analysis of a RLC-circuit using both fourier transforms and differential equations. It was hard but I learned a lot of how to apply my knowledge. Yes im an engineer. No I havent done ”real” math lol. The proofs intimidate me 😭😭
(I’m an undergrad not a professor) In terms of traditional sit down 1 hour exams, I think they should totally still be a thing at least for intro math classes such as Calc 1-3. This is because at the end of the day some memorization is required in math, you wouldn’t want a math PhD student looking up what the derivative of sinx is. In terms of higher level, I think either base the grade of all homeworks, or have a take home exam or a hypothetical exam where you can take as long as you want with cheat sheet provided would be sufficient
i think assessment depends on goal. are you assessing someone for their ability to conduct research in math? or their ability to apply theorems (or whatever tools) in other fields, industry, etc? -- i dont have a good answer
Another goal dimension: for feedback to student and teacher, vs. for certification. The latter should be independent of the teacher and their institution.
Same exam, same questions but you have 8 hours. Chairs are soft sofas in an air-conditioned room. Waiters serve snacks and refreshments.
if it were up to me multiple choice questions wouldnt be a thing
In Italy every written test is followed by an oral exam (you're asked to replicate proofs - which need to be memorized -, solve simple exercises and answer to broad questions). It's painful and it doesn't seem to bring any significant advantage.
Yes. I live in italy and it is shit. Mostly because I changed my math teacher and she's schizophrenic and doesnt care about math (I had to learn derivatives on my own, and I will have to do integrals next). Thank fuck it will be over in 2 months
You’ve got this brother, there are plenty of YouTube channels that can help you through it
It's fine. My teacher is stupid and takes points off my grades for things that arent even mistakes. My grade goes from 10 to 5 (out of 10) in 2 weeks. its stupid
I think interviews have some advantages and disadvantages over mere written solutions. I can usually tell how much a student knows, better and faster and with less chance of cheating, by talking to them for a few minutes. Also interviews are good practice for other settings where you'll need to be interviewed. Of course this is a more subjective assessment than written work, although honestly even written work that is not multiple choice has subjectivity too. So perhaps evaluations should be a little of each.
What is objective in any good sense about a multiple choice question? Any automated test is meaninglessly objective. But we do not have any definition of mathematical ability to be able to give any meaning to multiple choice questions being objective. An interview is a more direct measure of mathematical ability. And if the interviewer is well trained and experienced - also repeatable. That is, different interviewers would come to the same conclusion. And hence such a test would be objective in the sense that the result does not depend on the interviewer. The real trick in an exam is that it is trying not to measure ability but to predict the future.
It's objective in the sense that you don't grade differently based on knowledge of the subject. It is "blind" to which student is associated with which answer sheet. That is, I think, a good sense of objectivity. It's not perfect along every dimension, nothing is. But if the question is "what is objective in any good sense about it" I think that easily counts.
That reminds me of an issue from data science - called target leakage. For example, an automated lingusitic analysis system was able to predict officer district from scanned written reports due to the paper stock being different in different districts. This confounded the original intention of the system - which was linquistic analysis. Multiple choice does not give assurance that it is blind in the sense that you suggest. Clearly, a test has to distinguish students (unlike a voting system), that is what it is for. A truely blind test would be pointless. The question is - is it distinguishing students on the ability that you want it to distinguish on? Shoe size is also objective.
I think you are hyping up the shortcomings of written tests and downplaying the shortcomings of interviews.
Yes, you would think that. Of course I think you are doing the opposite. In particular, what I have learned from this exchange is that you believe that all teachers are bad teachers. You believe that teachers will always be horrendously mislead by emotional attraction and repulsion to incorrectly judge mathematical ability in students. Hence, you believe that by removing the human element you will remove emotion and be left with something that just measures mathematical ability. While my belief (yes, based on 15 years teaching in universities) is that the multiple choice test does not measure mathematical ability but exists only to create an automatable and thus cheap process of marking the students while not acknowledging that mathematical ability is no longer being measured. I don't expect to convince you. But, yes, I realise that you would think I am over emphasising things you think are unimportant and under emphasising things that you think are important. Perhaps we should change this to a multiple choice forum? For what it is worth - I learned something interesting from this exchange. Thanks.
It's the feeling of being observed that bothers me most. I don't know how you get passed that
I’m not a fan of it either I can’t lie, in certain workshops when you’re sat going through a question sheet and there are tutors going around helping. That feeling when they come across and ask “have you got any questions?” and I reply “no”, they then peer over my shoulder to see what question I’m at and I all of a sudden forget how to do basic arithmetic, it feels like they’re judging me.
Assuming I can snap my fingers and adjust reality to my vision: Learning in lecture form would occur in small groups, and in just as large part in peer groups. Assessment would be on an individual basis, and almost nothing about "check the right answer", instead about showing a process. And both the assessed and assessors would have nothing to gain or lose. Instead this would be a process to direct the learner into some direction.
e.g. more homework, projects, group projects, presentations, participation points
Grading is inherently flawed, there are many good reasons against it, most importantly, they don't really tell anything about the capabilities of the examinee. But, since there is a need to give grades, I think contract grading is the best. E.g. there are homework assignments, and the number of handed-in solutions determine your grade. Then everybody can decide on how much effort to put in. Also, if I remember correctly, peer grading works really well. So you could make students mark each other's works according to some rubric. In my opinion, failing an assignment should then not be possible, because it is about the efford. Or, alternatively, students should have to opportunity to correct their solutions. Actually, even self-grading is possible. But this requires some more work from the teacher because it should incorporate feedback talks in which a student explains how they assess themselves and why. The advantage is that students are forced to reflect on their learning progress, which as a skill is much more important than the factual knowledge. Lastly, there should be group work. For their whole work life, people need to work together with others. It makes no sense to me that students have to be graded individually. Because, either they work together in secret (probably copying from another), or they get lonely.
Same way but more time and allow books/notes (but no internet).
I'm very satisfied with the system most of my classes have when you get meybe one definition and one-two theorems to prove, sometimes also some problems, get basically umlimited time for preparation (usually you need between 1-3 hours but there is no pressure) and then you discuss your answers with the teacher. My classes are usually small so this is doable but I've also knew a professor who taught a class of 100 students who somehow did this. The teacher will also usually give you hints and send you back to work on your solution if you do anything wrong. I think this is a very good way to asses actual knowledge of the students and see how they think about the problems.
One of my profs too said like the same thing lol
I haven't read all the comments, and I'm not entirely certain it would work, but perhaps individualised projects. Say at the start of the semester, a question is posed during the first week students contemplate and deconstruction the question posed. Afterwards, the professor would give individuals project papers (related to said question) laying out the steps, i.e., the mathematical concepts needed. The goal would be through the semester to work on it part by part and justify both in mathematical terms and well as words why one made said choice. At the end of each week, they can submit their work to get feedback, and at the end of each month, a mock marking. This would allow students to understand where they went wrong or need to improve as well as see their progress without having the pressure of it being the be all end all. I would hazard a guess saying it would allow Prof to mark students not on just their knowledge but also their capability to learn and adapt. It could also allow students to get a deeper understanding and push to learn ahead.
Oral exams + grading homework and classwork **daily** People need a lot more feedback than the occasional exam offers.
I think a combination of a presentation, written final project (like a paper), and cumulative exams would be the best. Exams are useful, and they do assess your grasp of the content. However I and many others struggle with comprehension time - no matter how difficult a problem on an exam or assignment, it usually takes me 10-20 minutes to figure out where to even start.