Physics Challenge Exams: Placement & Evaluation Research

[Gamma]

Hello fellow physicists,
I teach first and second year physics at a community college. In particular, I teach two algebra based- and three calculus-based physics courses. I wanted to know if any of you have used any type of challenge exams to place out of the first year physics course. I am also researching to see if there are any standard exams that we can use to evaluate our students at the end of the semesters.

PS. I hope I am clear enough. My boss asked me to research, but I have not found any. I know chemistry has standard exams for intro classes.

- Thank you!

 

[Dr. Courtney]

I think the AP Physics tests are as good as it gets.

 

[Gamma]

Thanks for the response Dr. Courtney! Do you know if College Board would allow colleges to use their test as a placement test? I doubt it. I need to look into that.
I may end up making my own custom test that has the rigor of the AP exam.

 

[Andy Resnick]

Hello fellow physicists,
I teach first and second year physics at a community college. In particular, I teach two algebra based- and three calculus-based physics courses. I wanted to know if any of you have used any type of challenge exams to place out of the first year physics course. I am also researching to see if there are any standard exams that we can use to evaluate our students at the end of the semesters.
<snip>

We don't allow 'credit by exam' for our Physics courses; I'm not sure who approved AP credit policy here- it's a university policy, the registrar's office simply applies the policy.

As for 'standard exams' in Physics, there are a few, the Force Concept Inventory is perhaps the best-known:

https://www.ncsu.edu/per/TestInfo.html

But I'm not sure that's really what you mean.

 

[atyy]

MIT has placement exams for its first-year physics sequence: http://web.mit.edu/physics/prospective/undergrad/asc.html

 

[Dr. Courtney]

I'm sure that the rigor of the MIT placement exams is adequate. But having been intimately involved with placement exams at the Air Force Academy and in discussions about them at other institutions, I have a concern that at many institutions, there is often pressure to dumb down the placement and challenge exams so that credit gained by passing the exams represents much less knowledge, skills, and abilities than actually passing the course. In contrast, using a nationally normed and recognized exam (such as the AP) provides a recognized level of quality that the department, the downstream courses, and other institutions that may accept transfer credit can all have confidence in (or at least they can know what standard was used.)

It is very hard to write a 2-3 hour exam that properly represents the content of a 4-5 credit hour physics course, especially if the course includes a lab. If a student was awarded credit by earning a 4 or 5 on a given AP physics test, I (and many faculty who teach intro courses) have an immediate confidence level regarding what that means and what downstream courses the student is prepared for at our schools. In contrast, I have very little confidence in credit awarded by a challenge exam that is unique and proprietary to most state universities and community colleges.

 

[MidgetDwarf]

From my understanding, many schools in California, particularly the UC system, will not take CLEP test. Either AP credit or a letter grade are used to receive credit.

 

[Vanadium 50]

When I was a student, MIT simply did not offer credit for 8.02. (8.01, yes, but not 8.02) Their position was "you took it, but you didn't take it from us". It was felt that if you really knew the material, you could still register for the class, and it wouldn't take all that much of your time. If you didn't know the material, it would be a good thing to make you sit through the class again, no?

I agree that the AP test is as good as it gets. I don't think it's very good, though.

 

[bcrowell]

I teach first and second year physics at a community college. In particular, I teach two algebra based- and three calculus-based physics courses. I wanted to know if any of you have used any type of challenge exams to place out of the first year physics course. I am also researching to see if there are any standard exams that we can use to evaluate our students at the end of the semesters.

I teach physics at a community college in California, and in the past we have done this. It was very time-consuming for us to come up with an exam and agree on it as a department, then administer and grade the exam. We also had cases where we invested time and effort in the process, and then the student flaked out and said they didn't want to do the exam after all. In recent years we've just decided that there was no clear benefit to doing this, so we stopped doing it. A student who has a strong physics background could simply take the AP exam. Note that although it costs money to take AP exams, there is generous financial aid available for students who need it.

As for 'standard exams' in Physics, there are a few, the Force Concept Inventory is perhaps the best-known:

The FCI has been superseded by the IBCM. In any case, the authors of these tests have specifically be asked that they not be used in high-stakes situations, because that would provide an incentive for students to obtain copies of the tests, which is not particularly difficult to do. The purpose of these tests is pedagogical research and allowing instructors to get a reality check on the effectiveness of their own teaching. Using them for high-stakes purposes risks invalidating them as research tools.

 

[Gamma]

Thank you all! You have given a lot to mull over :) I don't get too many students who ask for a waiver. I needed a solution for those rare occasions and AP exam seems to be the route to go at this point. I looked for details about the physics CLEP exam in College Board website, and surprisingly physics was not listed!
(Link: https://clep.collegeboard.org/exams/offered). I will look into MIT exams, but I feel that they might be too rigorous for my need.

I had a second part to my question. I was looking for a common final exam for the intro physics courses. I am the physics lead faculty and as such I was given the responsibility of having the final evaluations same across the board. The reason is that many of our courses are taught by adjunct instructors and it becomes challenging to maintain a uniform evaluation, especially when adjunct faculties don't stick around for too long. From your expert input, I gather that making my own test would be the best. Thank you everyone!

-Gamma

 

[Dr. Courtney]

I had a second part to my question. I was looking for a common final exam for the intro physics courses. I am the physics lead faculty and as such I was given the responsibility of having the final evaluations same across the board. The reason is that many of our courses are taught by adjunct instructors and it becomes challenging to maintain a uniform evaluation, especially when adjunct faculties don't stick around for too long. From your expert input, I gather that making my own test would be the best. Thank you everyone!
-Gamma

At the Air Force Academy, Math, Physics, and Chemistry departments all used final exams common across sections in their intro courses. These exams were developed by senior faculty to have uniform evaluations, and it worked very well. I can see how a similar approach would be appealing when there is a high turnover rate among adjuncts.

Because there tend to be differences in learning objectives and topic coverage from one school to another, it probably is best for you to make a common final exam that accurately reflects the topics and learning objectives of the physics course at your school, but you want to make sure the adjuncts understand this from the beginning of the semester. They would be putting their students at a disadvantage if they emphasize conceptual understanding all semester and the final exam emphasizes quantitative problem solving.

 

[Gamma]

Because there tend to be differences in learning objectives and topic coverage from one school to another, it probably is best for you to make a common final exam that accurately reflects the topics and learning objectives of the physics course at your school, but you want to make sure the adjuncts understand this from the beginning of the semester. They would be putting their students at a disadvantage if they emphasize conceptual understanding all semester and the final exam emphasizes quantitative problem solving.

Agreed, Dr. Courtney! We do have an Adjunct conference prior to every semester, and this would be emphasized to them very well.

 

[alw34]

...At the Air Force Academy, Math, Physics, and Chemistry departments all used final exams common across sections in their intro courses.

When I took college physics as part of an engineering curriculum, the three professors of physics, who were also teaching physics majors as well as other engineering disciplines at the same time, also used jointly developed common exams. No matter your major, everybody took the same introductory exams. That was a rather long time ago.

Regarding other types of common or 'standardized' exams, might there be teachers at other schools, perhaps in your geographical area or accessible via professional groups, who might be willing to either exchange some past exams with you for yours or even get together to jointly develop some exams.? And you are not obligated to use a completely identical test. Always interesting to benchmark.

 

[Gamma]

Regarding other types of common or 'standardized' exams, might there be teachers at other schools, perhaps in your geographical area or accessible via professional groups, who might be willing to either exchange some past exams with you for yours or even get together to jointly develop some exams.? And you are not obligated to use a completely identical test. Always interesting to benchmark.

Great point alw34. We do have some fantastic 4-year institution surrounding my college, here in Maryland. Majority of our students end up in these colleges. So, it would be nice to look at that avenue as well. Thanks for your suggestion!

 

[bcrowell]

They would be putting their students at a disadvantage if they emphasize conceptual understanding all semester and the final exam emphasizes quantitative problem solving.

The empirical evidence is that devoting most of class time on conceptual understanding enhances conceptual understanding without any reduction in problem-solving ability. It's a win-win. There's some discussion of the evidence for this in Mazur, Peer Instruction: A User's Manual. Of course, this assumes that the students are being given homework and feedback so that they can practice the problem-solving skills, and also that they have a text that presents the techniques.

 

[Dr. Courtney]

The empirical evidence is that devoting most of class time on conceptual understanding enhances conceptual understanding without any reduction in problem-solving ability. It's a win-win. There's some discussion of the evidence for this in Mazur, Peer Instruction: A User's Manual. Of course, this assumes that the students are being given homework and feedback so that they can practice the problem-solving skills, and also that they have a text that presents the techniques.

I suppose this is possible in principle, and there may be some places where it is actually practiced.

In my experience, a lot of college credit in physics courses is being handed out under the pretense that students understand the concepts, when in reality they are incapable of solving almost any problem requiring the addition of vectors.

If it is not needed to pass the exams, many students will not learn it. It is a fantasy to pretend students can solve problems unless they are demonstrating it on exams. It is straightforward to write exam problems that require mastery of concepts. I haven't seen many exam problems that focus on concepts that also require mature problem solving skills.

 

[bcrowell]

If it is not needed to pass the exams, many students will not learn it. It is a fantasy to pretend students can solve problems unless they are demonstrating it on exams. It is straightforward to write exam problems that require mastery of concepts. I haven't seen many exam problems that focus on concepts that also require mature problem solving skills.

Yes, problem solving certainly needs to be tested on exams. The empirical evidence I referred to above (described in the Mazur book) was from exams that included problem solving.

In my experience, a lot of college credit in physics courses is being handed out under the pretense that students understand the concepts, when in reality they are incapable of solving almost any problem requiring the addition of vectors.

This may depend on whether their conceptual understanding is really being probed. The classic observation described in Mazur, which I have reproduced myself many times, is that if you make a scatter plot of students' scores on the FCI/IBCM versus their scores on plug-and-chug problem-solving, you get a triangle filling in one side of the main diagonal, with essentially all students doing better on problem-solving than on the conceptual stuff. But the IBCM does require deep conceptual understanding. It's quite possible that many instructors give points that they claim are for conceptual stuff when in fact the tasks don't really require much conceptual understanding.

Students who can ace the IBCM but can't do problem-solving are basically nonexistent. This is documented in the studies described in Mazur, and it's also something I've found very consistently with my own students for a couple of decades.

 

[Dr. Courtney]

One big challenge colleagues and I have faced focusing class time on concepts is that mission and curriculum creep keep the syllabus so packed with topics and the mathematical preparation (algebra and trig, mostly) of the average student is so poor, students really have no realistic chance at solving quantitative homework problems unless considerable time is spent modeling quantitative problems.

The best workaround we've found so far has been to shift some of the modeling of problem solving to instructional videos (1 per class). Students are motivated to watch the videos, because the problems that get solved are very close to assigned homework problems, and this frees up 10-15 minutes in each class that would otherwise need to be used to model problem solving.

See: http://arxiv.org/ftp/arxiv/papers/1204/1204.2462.pdf

 

[bcrowell]

The video project sounds interesting. Do you have a sample that's publicly available so I could see what one is like?

My general feeling about examples of problem solving is that that's why we have a textbook. If I do an example on the board, it's usually because I have some kind of a highly interactive shtick where I'm going to force the students to participate.

 

[Dr. Courtney]

The video project sounds interesting. Do you have a sample that's publicly available so I could see what one is like?

My general feeling about examples of problem solving is that that's why we have a textbook. If I do an example on the board, it's usually because I have some kind of a highly interactive shtick where I'm going to force the students to participate.

My experience is that students who are capable of learning from examples in the book are already pretty good at both math and problem solving. Book examples tend to leave out way too many steps for the average student (in my experience.)

The paper has links to the youtube channels of the co-authors. There is quite a bit of variability in style.

See:

https://www.youtube.com/user/BearTrueFaith/videos (Dr. Amy Courtney)

https://www.youtube.com/user/AgainstAllEnemies1/videos (Lt Col Ret Tom Slusher)

https://www.youtube.com/user/SupportAndDefend1/videos (me)

 

[Andy Resnick]

The FCI has been superseded by the IBCM. In any case, the authors of these tests have specifically be asked that they not be used in high-stakes situations, because that would provide an incentive for students to obtain copies of the tests, which is not particularly difficult to do. The purpose of these tests is pedagogical research and allowing instructors to get a reality check on the effectiveness of their own teaching. Using them for high-stakes purposes risks invalidating them as research tools.

I had a second part to my question. I was looking for a common final exam for the intro physics courses.
-Gamma

Right- as bcrowell points out (I was insufficiently clear), the battery of exams I linked to are more for evaluating the instructor than the students; these are not the exams you are looking for...

One thing to be mindful of- our Math department gives common final exams for the intro math sequence (pre-calc and calc); cheating is an ongoing horrendous problem as students in morning sessions will text/communicate the exam to students in afternoon sessions.

 

[Dr. Courtney]

Right- as bcrowell points out (I was insufficiently clear), the battery of exams I linked to are more for evaluating the instructor than the students; these are not the exams you are looking for...

One thing to be mindful of- our Math department gives common final exams for the intro math sequence (pre-calc and calc); cheating is an ongoing horrendous problem as students in morning sessions will text/communicate the exam to students in afternoon sessions.

At the Air Force Academy, the Math, Physics, and Chemistry departments all gave common exams in their intro courses. It may have been the culture of integrity (and assurance of severe consequences if caught cheating) that reduced cheating, but using 2-4 different exams probably played a role also.

The leadership team for the big intro courses would develop 2-4 exams that were very similar in terms of learning objectives tested and difficulty, but with different problems. Alternate seats in each final exam session had different exam versions, and there was no way a cadet could know before showing up which exam he or she would have. Given the high stakes of getting caught and the uncertain returns, there was very little attempt to communicate exam problems to other cadets. I expect that most students interested in cheating are unlikely to take the trouble to compile the exam questions from multiple exams and make sure they can work the problems on all of them. Students willing to work that hard may as well prepare for the final conventionally and honestly.

Of course, cadets at the military academies have also surrendered many privacy rights retained by most college students, and it was very easy to have the IT people pull communications records from the exam period/week if cheating was suspected. Leaving an easily discovered electronic trail of texting or emailing exam contents to another cadet is just about the easiest way to fail the course and most likely get kicked out.

 

[Andy Resnick]

At the Air Force Academy, the Math, Physics, and Chemistry departments all gave common exams in their intro courses. It may have been the culture of integrity (and assurance of severe consequences if caught cheating) that reduced cheating, but using 2-4 different exams probably played a role also.<snip>

Our situation is somewhat different. In Physics, we don't give common exams- call it 'academic freedom'- and each instructor has their own way to discourage cheating. Some use multiple versions and assigned seats, I just video the class with a GoPro. The math department has a single 'course coordinator' who writes the final exam in secret- does not tell the instructors what is on it- and while it's a rotating position, some of the coordinators just don't want to deal with multiple exams. That department is still trying to figure out solutions. The intro math sequences are not graded on a curve, so there's no sense of competition for grades.

And we are not allowed to request phone records, etc. etc. Even though we (Physics) see the same type of cheating in labs- students will text/email an image of their graphed data to a friend (typically a grad student at another institution) who then writes up their analysis section and texts it back for inclusion in the lab report. Students know that the burden of proof lies with the instructor/TA- it's not enough to simply witness academic misconduct, there must be actual evidence provided to the judicial affairs officer.

Honestly, the whole enterprise is not unlike infectious diseases and antibiotics: an arms race consisting of a constant series of mutations.

 

[Andy Resnick]

Yes, problem solving certainly needs to be tested on exams. The empirical evidence I referred to above (described in the Mazur book) was from exams that included problem solving.
<...>
This may depend on whether their conceptual understanding is really being probed. The classic observation described in Mazur, which I have reproduced myself many times, is that if you make a scatter plot of students' scores on the FCI/IBCM versus their scores on plug-and-chug problem-solving, you get a triangle filling in one side of the main diagonal, with essentially all students doing better on problem-solving than on the conceptual stuff. But the IBCM does require deep conceptual understanding. It's quite possible that many instructors give points that they claim are for conceptual stuff when in fact the tasks don't really require much conceptual understanding.

I think we are using very different contexts of the phrase 'problem-solving'.

Several of the physics faculty here use various incarnations of peer instruction- think-pair-share, A/B/C/D cards or clickers, etc. I call plug-n-chug problems exactly that- plug-n-chug, and plug-n-chug problems are used in peer instruction because they don't require much time to work out.

What I call problem-solving is more in line with George Pólya. [https://en.wikipedia.org/wiki/How_to_Solve_It] . This approach to learning is very different from plug-n-chug problem solving. 1/2 of my class time is spent working through a 'test-like question' to give students practice with this method.

And every time one of my students says "But I understand the concepts, it's just that I didn't use the right formula. I don't know how to start the problem." I point out (with a few quick questions) that they don't actually understand the concepts. Broadly speaking, my students are great at using Pólya's method in their everyday lives but horrible at using it for anything classroom related.

In fact, I believe the way students think of "understanding the concepts" is very different than instructors thinking "understanding the concepts". In my experience, students believe that if they can solve F = ma when given 2 of the 3 variables, then they 'understand' Newton's second law. They convince themselves they understand what 'Force' is (it's 'F'!). In my experience, many students believe that if they can learn the label of something, then they also understand what is being labeled.

Two examples demonstrate this fallacy: 1) I usually confuse a large fraction of my class simply by writing ma = F instead of F = ma. 2) If I use different symbols, for example I will say "Force equals mass times acceleration" while writing ϑ = ma, students ask me why I didn't use 'F' for Force.

I bet that what you are calling 'conceptual understanding' is close to what I call 'problem solving', or at least that the two concepts are related to each other. One demonstrates conceptual understanding of something by solving a problem involving that concept.

 

[bcrowell]

I bet that what you are calling 'conceptual understanding' is close to what I call 'problem solving', or at least that the two concepts are related to each other. One demonstrates conceptual understanding of something by solving a problem involving that concept.

What the studies described in Mazur used, and what I used, as a proxy for conceptual understanding was scores on the FCI/IBCM.

 

[Vanadium 50]

, students ask me why I didn't use 'F' for Force.

One of the most popular questions here is 'Why is m used for "slope"?'

This sounds a lot like magical thinking: there is power in knowing something's true name.

 

[Andy Resnick]

What the studies described in Mazur used, and what I used, as a proxy for conceptual understanding was scores on the FCI/IBCM.

Do you administer the test at the beginning of semester, the end of the semester, or both beginning and end? Are you basing student grades on FCI/IBC-M scores?

One interesting aspect of the FCI that does not appear to be carried over to the IBC-M is the insight gained by incorrect responses (Table 4-2 in http://www.physics.umd.edu/perg/dissertations/Saul/Chapter4.PDF)

Also, and this is true of any exam, there are fair doubts about what the test measures- specifically for the FCI: "from the students’ point of view, the FCI does not appear to test for a coherent, universal force concept, dimensions of a force concept, or any organized alternative beliefs (e.g., impetus)".

http://www.physics.emory.edu/~weeks/journal/hestenes-tpt95b.pdf

 

[Andy Resnick]

This sounds a lot like magical thinking: there is power in knowing something's true name.

The pessimistic cynic in me agrees wholeheartedly. After I've softened up with some holiday cheer, I think of the phenomenon more like "when drowning, grab anything that floats".

 

[bcrowell]

Do you administer the test at the beginning of semester, the end of the semester, or both beginning and end? Are you basing student grades on FCI/IBC-M scores?

I sometimes did it both pre and post, sometimes just post. It counts a little bit in their grade, but the amount is very small -- just an incentive to show up and try, basically.

Also, and this is true of any exam, there are fair doubts about what the test measures

If you show the FCI/IBCM to any physics teacher for the first time, their almost universal reaction is that the test is incredibly easy, and of course their students would get high scores on it. So in that sense, I think it has good "face validity," and the fact that most students in most courses get 50% or lower is something that we can't avoid taking very, very seriously.

 

[symbolipoint]

Right- as bcrowell points out (I was insufficiently clear), the battery of exams I linked to are more for evaluating the instructor than the students; these are not the exams you are looking for...

One thing to be mindful of- our Math department gives common final exams for the intro math sequence (pre-calc and calc); cheating is an ongoing horrendous problem as students in morning sessions will text/communicate the exam to students in afternoon sessions.

What a disappointment. What can you do to make that kind of cheating become unworkable for the student?

... that was before I read the posts which followed. Maybe making multiple versions of the test is how to do it.

 

[Vanadium 50]

. Maybe making multiple versions of the test is how to do it.

Even scramble tests (same questions, different order) work to a degree.

 

[Andy Resnick]

What a disappointment. What can you do to make that kind of cheating become unworkable for the student?

... that was before I read the posts which followed. Maybe making multiple versions of the test is how to do it.

Cheating, like any other kind of crime, cannot be completely and permanently eliminated. My approach is to disincentivize it as much as possible.

 

[Andy Resnick]

<snip>the fact that most students in most courses get 50% or lower is something that we can't avoid taking very, very seriously.

I agree, definitely. I'm struggling with how to best remediate the students.

 

[symbolipoint]

Hello fellow physicists,
I teach first and second year physics at a community college. In particular, I teach two algebra based- and three calculus-based physics courses. I wanted to know if any of you have used any type of challenge exams to place out of the first year physics course. I am also researching to see if there are any standard exams that we can use to evaluate our students at the end of the semesters.

PS. I hope I am clear enough. My boss asked me to research, but I have not found any. I know chemistry has standard exams for intro classes.

- Thank you!

Not really the type of thing you're looking/asking for, but
One of the university Physics departments had (maybe still do has) a mathematics skills assessment administered at the beginning of the class term in Phyics 1 - Fundamental Mechanics, part of the engineering Physics course sequence. If student performs too low, then student is REQUIRED to enroll in a short - few weeks remedial course to bring up students basic intermediate algebra and basic trigonometry skills; and if student does not enroll in this course and pass it, then the physics department will administratively drop the student from the Physics 1 - Fundamental Mechanics course.

 

[bcrowell]

Not really the type of thing you're looking/asking for, but
One of the university Physics departments had (maybe still do has) a mathematics skills assessment administered at the beginning of the class term in Phyics 1 - Fundamental Mechanics, part of the engineering Physics course sequence. If student performs too low, then student is REQUIRED to enroll in a short - few weeks remedial course to bring up students basic intermediate algebra and basic trigonometry skills; and if student does not enroll in this course and pass it, then the physics department will administratively drop the student from the Physics 1 - Fundamental Mechanics course.

We tried to do something like this. There are a lot of barriers.

1) We would want to convince ourselves that our math pretest was very strongly correlated with success in phyiscs. We constructed a test and didn't find such a strong correlation.

2) We're a public school (California community college), and there are court decisions that make it very difficult to do this sort of thing.

3) We would want to have some evidence that the required remediation was effective. Actually the best evidence I've seen is that math remediation in general simply doesn't work.

4) Many students with inadequate math skills have passed the relevant math courses at our school. Neither they nor the math department want to hear that a C in math equals total incompetence.