Can student evaluations accurately measure the effectiveness of teaching?

[Andy Resnick]

Mathwonk's post struck a nerve with me. At the risk of random ranting:

1) most of my negative reviews can be summarized as "He is a bad teacher because I didn't understand anything". This is why student evaluations are generally ignored. Certainly, one of the functions of teaching is to *enable* understanding, but a teacher cannot understand things for the student.

Students arrive in my class with a wide range of preconceptions: Physics problems are basic plug-n-chug ("tell me what formula to use!"), Understanding something is equivalent to memorization, giving up means someone else will do the work ("work the problem out for us!") etc. etc. I encounter strong resistance and resentment when I directly confront their preconceptions. Far too many teachers fail to think about the purpose of the class they are teaching: what do I want the students to learn and remember long after the class? How can I evaluate the students against what I expect? What is the function of homework?

2) The classroom environment is highly asymmetric. Students have the opportunity to rant anonymously on a variety of platforms. Students can come into my office and yell, whine, cry, complain, cry some more, and even threaten. I, on the other hand, am highly restricted.

An example: one of the 'test like questions' we worked through in recitation was put on the (open book, open note) exam, *verbatim*. In spite of the students having the solved problem in front of them, a large majority still could not work out any substantive part of the problem. I'm sure you can imagine the string of profanities that issued forth while I was grading (in private). To the class, however, I am only permitted to express 'puzzlement': "I don't understand why so many people got this wrong, even when we worked this out before". Even then, many students interpret this comment to mean I am angry at them personally; somehow am impugning their intention to work hard and 'understand' the material. So their poor performance is really my fault, because they are working really hard and I don't explain anything.

</rant>. Sorry- it's been a long semester.

 

[Integral]

Just as a data point. I once saw, tacked on a prof's office bulletin board, 2 student review cards for the same class. One was a perfect 4.0, the other a perfect 0.0. What is to be learned from this?

 

[Sankaku]

Just as with any data set, you have to focus on the signal, not the noise.

I sympathize; If the data set is too noisy then, yes, discard it. However, don't discard the signal just because the noise is personally aggravating.

 

[Sankaku]

At the risk of random ranting:

...snip...

I forgot to add that I think ranting at such things is a perfectly natural response.

 

[mathwonk]

Andy, in the vein of "but they've done this before": In an abstract algebra class, intro level, I felt required to ask for proofs on tests. Since very few students could do them, I took the following approach. I would assign some easy proofs as take home tests, let them work out the proofs, hand them in, and I would grade them and hand them back. I also had some of them present them in class. Then I would ask the same proofs on the test. You guessed it, most students still could not do them. Even those who could come close, often just wrote down almost the same words as in a correct proof, but without the little modifier and connective words that made the logic correct and clear. I.e. they often seemed memorized and not understood.

Describing it now, I see this could only have been the result of a conscious lack of any attempt to understand the reasoning. Most students simply copied the proofs from the book, without adding any explanation or clarification of missing steps, and ones who did slightly better, just memorized the words as best they could also with no attempt to understand. This betrays a complete lack of understanding of what is wanted from the student, a lack of any idea of what college is about, learning to think, analyze, and verbalize.

Many of us assume teaching is about clear explanation. But in many cases, effective teaching seems to be about figuring out strategies to combat willful refusal to learn. This is depressing to those of us who do not see ourselves as personal trainers, or drill sergeants.

 

[Skrew]

If students feedback isn't considerd relevant then who's actually going to judge the instructors performance?

 

[mathwonk]

@ Skrew. When I was a freshman at harvard in 1960, we were told at orientation as follows: "For most of you, this is the first time in your life when your professors are not being judged by your performance, only you are."

The implication was that the professors were already known to be masters of their subjects and therefore capable of explaining it. The task for us as students was to get ourselves up to speed and be able to understand the material as explained by the professors. It was not their job to dumb it down until we could grasp it on our own level.

Now I admit there were some junior professors there, maybe grad students, who were not at all skilled at explaining the material, and =even some senior professors who were not as clear as others. So we communicated among ourselves, as to who we might prefer. Student evaluations were also extensively collected and published by the student newspaper and sold to the student body.

However whenever I chose my professor based on those popularity surveys, I was always sorry. I was not sorry when I chose based on my own observation and my esteemed friends' recommendations.

There was information available including student feedback in the form of evaluations, and the professors did know of their reputations on these evaluations and often took it to heart, but it was not deemed relevant to assessing of the professor by the university. This university had the best professors and the best lectures and classes in general of any I have ever been to since.

Since that time, class evaluations have become an official part of assessment of professors by the university, and in my opinion, some harvard classes since then have overemphasized appealing to the average student, and have declined somewhat in quality. I.e. the most popular professors are still not always the best, but with a student based evaluation system, they get more recognition.

Basically the best person to learn from is one who understands what you want to understand. it may be that at first it is easier to understand someone who just dumbs it down patiently, but eventually you are going to want to try again to get it from the real expert. The ideal situation of course is a patient expert with a gift for clear explanation, but in a pinch, I suggest you choose the difficult expert over the glib amateur.I would like to see a student evaluation system in which student opinions are assessed over years and years. It is my opinion that, like childrens' view of parents' wisdom, student opinions of professors change with maturity. the best professors will be appreciated much later. Those courses whose offerings can be completely absorbed in one semester are the most shallow.

Indeed this is the point of the first paragraph of the article Borek originally linked:

“I had a high school (or a college) experience like yours,” the poster typically said, “and I hated it and complained all the time about the homework, the demands and the discipline; but now I am so pleased that I stayed the course and acquired skills that have served me well throughout my entire life.”

My sincere apologies for popping off before reading that link. I now have done so, and I agree roughly 100% with it. Indeed it goes straight to the point very directly and eloquently.

 

[mathwonk]

Here are some questions about a professor: are his explanations easy to follow for even the most ill prepared student? does he give grades that please even the student who does not attend class? does he know enough about the subject that it is worth listening to him?think about it, which of these questions can be answered competently by an average student who got a C in the class? or a D? those students are giving input on evaluations. But if you are a good student, which question is more important for you in choosing the class? You might get more information from an evaluation if it included the grade received by the student writing the evaluation.[rant: Another question: is he/she well prepared for class?

This one used to get my goat at times. After spending decades learning the material thoroughly, reviewing iut the summer before the semester, and then maybe 3-10 hours preparing it thoughtfully the day before so as to try to make the connections clear even to a beginner, some of my students thought I was unprepared because I did not bring written notes to class in my hand!

It did not dawn on them that I had prepared so thoroughly that it was all in my brain, that I had written out 10 pages of notes the night before, then thought about it all night, and reviewed it mentally for one and a half hours on the long drive to school. (and taught it ten times before.)

The only time I need notes is when I do not understand the material completely, and need to refer to a note.

Indeed the one time I did that concerning a topic I found tricky, a weak student complimented me on being prepared because I had notes in hand! To me that was the only day that semester that I was not completely prepared. I.e. I could copy the material on the board correctly, but could not fully explain it to my own satisfaction, since I did not quite feel I understood it totally.]E.g. there is a set of notes, 15 pages long, on my webpage called primer of linear algebra. I sat down over christmas break on year before teaching the course in spring, and just for mental review I wrote out that set of notes off the top of my head, just to prepare mentally for the upcoming course. The point is that I can sit down and write a book on an elementary topic just off the top of my head, with time to think, revise and edit. So can any well trained professional.

That spring I did not teach of course anything as advanced or concise as those notes, but the exercise was for me, to make sure I had mentally reviewed the whole subject before trying to teach it. Of course then I also prepared every night for the next day, to make sure the examples were clear.

However I seldom consult notes even for an example, since I feel that if I myself cannot generate a correct computation just while doing it, then it will be harder for students listening to follow it. So I make sure every computation is fresh, and try to exhibit my thinking on the spot.

Nonetheless, there are certainly students who think this is poor teaching, especially those who are actually not trying to follow the reasoning as it is given.

 

[mathwonk]

sorry but I'm kind of on a roll here. in my opinion, a good teacher will not necessarily adhere to the ordering of the material in the book, but will think about the logic of the subject and try to present ideas in the order they most naturally fit into. this frustrates some students who cannot comprehend the fact that their book may not be doing things in the best way. A student reprimanded me once with the obviously sinful behavior of going from a topic in chapter 3 to a topic in chapter 8, but which was natural considering the logic of the material.

e.g. after decades of puzzlement i have realized that "non euclidean" geometry is the natural bridge from plane geometry to differential geometry. I.e. the big mystery about euclid's fifth (parallel) postulate concerns merely the fact that it says the euclidean plane is flat. non euclidean geometry is about geometry on surfaces that are not flat.but curvature is most easily understood if it is constant everywhere on the surface, such as on a sphere.

hence the most natural way to progress in learning geometry is to go from euclidean geometry on a flat euclidean plane (curvature zero), to spherical geometry (constant positive curvature).

Here one sees that the curvature causes the angles in a spherical triangle to sum up to more than 180 degrees, and indeed that excess is one definition of positive curvature. The question of what curves on a sphere should play the role of lines leads to the concept of shortest curves or "geodesics".

It is natural next to ask if there are surfaces where geodesics triangles have less than 180 degree sum. These would naturally be surfaces of negative curvature. Saddle surfaces and horns arise, and we can ask for an example with everywhere constant negative curvature, and one where geodesics are "infinitely long".

The classical example is the famous hyperbolic non euclidean plane, which in most geometry courses is led up to as some sort of freak mythical example, or modeled oddly, and without motivation, by a non standard metric on the upper half plane or the disc. the fact that all the weird behavior of these examples arises from their being curved is lost here, and often not mentioned, because the upper half plane and disc model look flat to the naive student. I.e. without sophisticated theories from calculus the fact that they are really curved in the goofy metric being used is not clear.

After studying the triangle sum theorems on these basic examples, one has a simple prototype of the gauss bonnet theorem, indeed the one found by gauss himself.

moreover if one considers these curved surfaces in families according to their "radii", as for an expanding family of spheres, one sees tht there do exist simialr triangkles in non euclidean geometry, but they occur on surfaces of different radii. I.e. there is a whole family of non euclidean planes and expanding a trianglke by a scalar multiple simply takes us onto a plane of differnt radius. why not?i am getting ahead of myself. anyway after this kind of preparation in the realm of non euclidean geometry, one is ready for differential geometry of surfaces, which are curved, but are allowed to have curvature which changes gradually from point to point. nonetheless, theorems like gauss bonnet, which involve limiting approximations using very small pieces of surfaces, are still true, by taking limits!This sort of natural progression from one simple idea, euclidean geometry, through slight variations, spherical geometry, saddle surface geometry, then to differential geometry of surfaces, makes the whole subject easier.

However very few books teach this way, partly because books used in elementary courses are not always written by real masters but by eager learners (e.g. my books).

What happens when an excited professor tries to expand his students' horizons and add in some of these in sights he has had after decades of studying the material and trying to make it more understandable? Well some students respond on their evaluation that he "presented material that was not in our book!" (Strangely, this was obviously meant as a negative comment.)Now I confess that I always read all my evaluations and try to take relevant input into account to improve my communication skills. But still I am not optimistic as to the value of much of the input received. (quote: "he needs to button the collar on his shirts." or: "professor thinks understanding is paramount, but the bottom line is the HOPE scholarship.") Moreover if a course contained 35 students and you read negative evaluations from only 3 of them from some unofficial source, they may not be entirely objective and useful.

one of my favorite quotes: "he is a good teacher...not real good, but good."

 

[mathwonk]

Let me give an example of deceptive but popular teaching. I once heard a public lecture by a famous professor at an elite school on the demise of the dinosaurs. He made a big deal out of debunking the popular myth, as he called it, that they died out because of the rise of the more intelligent species that led to mankind.

He explained that they died out because of an impact with a meteor that raised a big cloud of dust and suffocated them and denied them sunlight until they perished. Only then did the little creepy crawly things that preceded us come out from our caves to spread over the land. And he made a special point of emphasizing how stupid were the arrogant types who had taught us otherwise and exploited our desire to think that our being smarter than dinosaurs the was key to our rise. We were suckers he implied to be fooled by this flattering explanation.

The irresistible impression (to me) was that he was letting us in on the inside intellectual track, and saving us from the embarrassing plight of joining the ranks of the stupid and deceived. I bought it hook, line, and sinker.

Then a while later at Berkeley in a good newspaper I read an objective article explaining how this meteor theory was just one of several competing explanations for the dinosaurs disappearance. Al of sudden I realized I had been had, and suckered into accepting one explanation because I didn't want to feel stupid. There were (at the time) in fact several plausible points of view, all having some support, and all deserving consideration. None had been accepted by a consensus judgment of the community. The honest thing to do would have been to give us the competing arguments and let us decide.

The first professor it turns out from studying his popular books, had a standard technique of treating opposing theories to his own as myths, and "debunking" them for the reader, making the reader feel superior to the idiots who proposed theories different from his own. When you listened to his lectures the only way to feel smart, to feel like one of the "in crowd" was to agree with him. His evaluations were through the sky, and he was the most popular lecturer at his famous elite school for years, and he sold a lot of books to the public too.

But I think he was an intellectual fraud and snake oil salesman. He did not encourage the most important things in a science student, skepticism, questioning, doubt, and independent thought. In his class you had to agree with him. I felt really angry at being schmoozed by him, and felt like a sucker. But his class loved him, short term.

So whenever I too say something like how vastly superior my own view of math is, be very cautious. Ask me why I think that if you want, but then make up your own mind. E.g. I have implied in the previous post that authors who teach non euclidean geometry without mentioning curvature are misguided and are even misleading you. But maybe they have their reasons. You think about it and then you decide.

Of course from an evaluation standpoint it is always risky to put people on the spot and ask them to think for themselves. Have you read the interview between Jesus and the Grand Inquisitor in, what was it, Dostoesky? As I recall, the GI advised Jesus not to seek the decision of the masses on his teachings since those sheep hated being challenged to be free. He risked surely being condemned, indeed as he had of course been before, in comparison with a thief. As I recall Socrates got a bad evaluation too, for similar reasons. So don't feel bad if you are dissed by your class. You are in very fine company.

 

[Andy Resnick]

@Mathwonk- wow! I can't add anything to that. Well put!

@Skrew- You are asking a relevant question (who watches the watchers?)- I can only speak for myself, but I have peer evaluations every semester and a teaching center with Master teachers I can consult with- and a paper trail gets generated for my tenure review. But, the question "How can I (or the tenure review committee) determine if my teaching is effective?" is really difficult to answer. Honestly, it's an active discussion among us faculty, *especially* in the current financial environment (i.e. academic programs must demonstrate 'effectiveness' in order to get priority ranking for resource allocations).

So I'll put the question out there- 'Using your course goals, what criteria do you use to determine if the students have learned'?

 

[mathwonk]

Thank you Andy, that is indeed high praise!

Here is an extract from an article I once published, called "On teaching".
(my apologies, clearly this needs serious editing!)

If a concept is defined by how it is measured, our teaching evaluation forms suggest that good teaching can be appreciated by an average student in the class who has not yet even finished the course. I cannot fit my own great teachers into this paradigm. Take Raoul Bott, who was regarded as an outstanding teacher at Harvard. Once, in his class on algebraic topology, he remarked after proving the Brouwer fixed point theorem that all we had really needed was a "homotopy invariant functor that doesn't vanish on the sphere." I did not follow this remark. I did not even grasp what it meant much less why it would prove the theorem. The theorem had a lengthy proof, and I did not understand how he could pretend to summarize it in one phrase. Since he did not write his comment down, I even remembered it wrong as a "homotopy invariant functor that vanishes on the sphere."

We might say, that Bott indulged himself in making deep, succinct statements even though the statement was not comprehensible to the class at large. Is this a good quality or a bad one? You may feel the answer depends on how many in the class find the statement comprehensible: the more the better. But I suggest that this behavior of Bott's is valuable teaching even if not a single student understands the statement! In fact it is more valuable to the student who does not understand it, because that student is being helped the most.
That student has already been taught all he/she can take at the moment, and is being pointed to higher ground which he/she will eventually be able to tread. That student is receiving instruction not just for the moment, but also for the future; hence being given something to think about which will last a significant amount of time, and which will repay all the thought given to it.

In my case, several years passed before I understood Bott's statement. It occurred when I began to appreciate the difference between building a tool and using it, between definitions and existence theorems. The details that had obscured my vision were the nuts and bolts of constructing the tool, and Bott's lightning summary contained only the key features of the finished tool. As I finally understood his comment I savored the knowledge in it, his generosity in saying it, and the satisfaction of resolving a puzzle of many months standing.

How often does one encounter the grateful comment on teaching evaluations "He really gave us provocative questions to think about. I still have not settled them all!"? If this comment is missing, can the teacher really be excellent? When I took Freshman calculus from John Tate (at Harvard) his lectures were very dynamic, his course was very difficult, he knew instantly the answer to any question, and he could prove any statement in full detail apparently without a moment's reflection. But I did not know if he was a good teacher.
At Christmas I compared notes with a friend who had gone to a well known engineering school in the South. It was immediately clear that calculus was not the same everywhere, that my course was much more demanding, and that I was being given far more by Tate than my friend was getting from his professor. I began to realize that Tate was a good teacher.

The quality of a teacher was often measured by my undergraduate acquaintances, according to the quality and depth of the material being presented. The professor was praised for possessing a distinguished vision of the subject as much as, or more than, for a facility at making it easy. It was understood that deep material cannot be made easy. In Bott's course, for instance, his proof of the homotopy lifting property of covering spaces was sketchy and incomplete, and he seemed not to have any interest in writing out the details.
However I had no trouble finding it in every book on the subject, and eventually in working it out for myself. What I could not get for myself was the grasp of the big picture, the sense that it was possible to view all these things from a perspective from which they were quite trivial, and the inspiration to achieve that perspective. A teacher can be considered good in that sense only if she gives you something beyond what is in the books, and perhaps insists that you try to understand it. How often does one encounter comments like the following in a teaching evaluation:
"She really made this course hard by including points of view more sophisticated than those of the book, which appeared mundane in comparison. I have grown intellectually more in this quarter than ever before! Great teacher! The students last year were shortchanged by a professor who plodded through the syllabus, assigning only the easier problems."

One of the moments I remember best in a course by David Mumford was when he turned to the class and remarked, "the way to read Grothendieck is to find the topic you want, read that section (tracing back through the pages for all the references) and understand it, then go home and write it up yourself in two pages." This advice on how to extract information from tedious and lengthy source material is invaluable to the student who thinks he must slog through every book from the beginning.

How shall we progress beyond the minimal teaching skills associated with training people, displaying information, and instructing from a syllabus, to the deeper, more valuable ones of guidance, nurturing, illumination, and inspiration? I suggest we begin by emphasizing that these latter qualities are more important than simple information transferral. Steve Sigur, mathematics teacher at The Paideia School in Atlanta, has asserted that there is no point in teaching only for factual content, since after one year essentially no factual content is retained!
The truth of this brutal claim is evident to every teacher faced with verifying the prerecquisites in a new class. Indeed the lifetime of "learned" information often seems only weeks or days after the final, instead of one year. Therefore I suggest that advice to a new teacher include a reminder to volunteer to teach a variety of courses and to attend seminars and professional meetings, so that one's ability to inspire, enlighten, and draw connections between different topics continues to mature.

 

[member 392791]

With some people, I realized they are called ''lecturers'' and not ''teachers''. I think there is a difference between the two. One lectures at me and that is his or her sole responsibility, while the other tries to facilitate understanding of the material.

 

[Andy Resnick]

<snip>
How shall we progress beyond the minimal teaching skills associated with training people, displaying information, and instructing from a syllabus, to the deeper, more valuable ones of guidance, nurturing, illumination, and inspiration? I suggest we begin by emphasizing that these latter qualities are more important than simple information transferral. Steve Sigur, mathematics teacher at The Paideia School in Atlanta, has asserted that there is no point in teaching only for factual content, since after one year essentially no factual content is retained!
<snip>

Excellent post, I just wanted to focus on this paragraph, because it parallels Ken Bain's "What the Best College Teachers Do". It's full of really useful examples, is thought provoking, and is worth reading. However, I found the last chapter "How do they evaluate the students and themselves?" to be unsatisfying. To be sure, Ken is systematic:

"[...] the fundamental evaluation question, Does the teaching help and encourage students to learn in ways that make a sustained, substantial, and positive difference in the way they think, act, and feel- without doing them any major harm?"

To answer the question, Ken recommends a 'teaching portfolio'- this is a document that I generate, based on self-reflection, where I provide evidence that answers the fundamental question, hopefully in the affirmative. While I think the teaching portfolio is an excellent tool- the act of self-reflection is critically important- how do I know I am being objective? Similarly, what kinds of evidence can I gather that demonstrates I have "made a sustained, substantial, and positive difference in the way [my students] think, act, and feel"?

It's not clear. I suspect the teaching portfolio is like a democratic government- the worst possible system, except for all the other ones.

 

[mathwonk]

I reference to the question:

"[...] the fundamental evaluation question, Does the teaching help and encourage students to learn in ways that make a sustained, substantial, and positive difference in the way they think, act, and feel- without doing them any major harm?"inevitably the answer may be: yes for some, no for others.

Here are a few actual class evaluation comments for the same course of mine some years ago (this is all public information):

"His questions are too vague in class to illicit [sic] response and then he yells at us for not paying attention and not studying. He is not very organized in his presentations. NOT prompt on returning tests/homework."

"Has changed my concept of teaching. Will teach more like him in the future."

"The style is outstanding---I wish I had learned math in this fashion my entire career."

Here are a couple of comments from another course again by two different students in it:

"I did not like coming by on his office hours because he was sometimes busy and sometimes not very helpful at all."

"Prof _____ is truly a great teacher. He is always available to help students and his genuine love for what he does is evident."Sometimes it is hard to know what to learn from such evaluations, even if you want to. You tend to end up with something like: "I know I can improve, but I guess I can't please everyone."