A few questions to assist in determining the source of your problem:
1) If you had a chance to redo the test now, still closed book but not for credit, just to solve the problems, could you get the answers right? In other words, do you just freeze up when you get nervous, or was there some problem in relating the concepts you learned in class in a new context?
2) Why does the exam become illegible? Are you scribbling quickly out of nervousness about being timed, or are you scribbling down anything you can remember out of order?
3) When you do your homework, do you write more neatly and keep equations/pictures more organized and legible than on the exam, or do you have a tendency to be a bit illegible all the time (perhaps in homework it's legible enough to you, but not to others)?
4) How do you approach taking the test: do you just start with question #1 and work through in order, or do you take a moment to read through the questions and start with the ones that look easiest, going back to the harder ones as you have time? Do you solve one question all the way to the end before going on to the next, or do you get to a point where you get stuck, and then jump to another question and another partial answer, and work back and forth on multiple questions at once?
So, there are a number of things you can work on depending on whether your difficulty is with test-taking or with the concepts on the test.
I'll start with the general test-taking tips.
First, stop studying/cramming about an hour before the exam. Get your mind off the exam and relax.
Eat properly. Good nutrition will help you maintain your focus, avoid the trap of too much caffeine, which will leave you more nervous and jittery and won't help with focus. You can use that hour that you're relaxing before the exam to go get a light snack; don't eat a heavy meal just before the exam, or you'll be sleepy trying to digest or will wind up with an upset stomach from too full of a stomach while nervous, but a light snack is good - a half sandwich, some fruit, some carrot sticks, a little peanut butter on crackers, any of those will be good.
Take your exam in pencil not pen and remember to take along a good soft eraser that won't tear up the paper when trying to erase.
As you get the exam, take a few deep breaths, and tell yourself to stay calm.
As soon as you're allowed to begin the exam, take a few minutes to read through ALL of the questions before you begin to answer them. Decide which ones you can answer most easily as this will help you calm down to have a few completed before tackling the time-consuming difficult ones. Also, sometimes something in one question will trigger your memory of something you need for another question. Take note of the point values of each question and try to allocate your time according to the point value (1 pt questions you should go through quickly, and skip over any that you can't get right away...you can always go back at the end if you have time to think through a few more; if a question is worth a half or third of the points on the exam, be sure to leave that much of the exam time to work on answering it, and remember those are the ones where showing your work will be most important).
Write neatly. If you need to jot down a bunch of unrelated equations or thoughts to remember them for later questions, do that on the back of the page, or off in the margins. Keep the space given for answering the question free of clutter. If you go back and realize you left something out, rather than trying to squeeze it in, number the steps in the order they should be followed and jot a note at the top or bottom to follow the numbered order.
If you make a mistake, either erase (if it's a small mistake that you can erase quickly, such as on a fill-in-the blank question), or draw a single line or X through it (better for long solutions to problems). There are two reasons for this; it keeps the work space uncluttered, so easier for the grader to read, and it also keeps your work visible in case you change your mind that something was right the first time. If that happens, all you need to do is make a note in a box next to the X'd out work to disregard the X, it is part of your final answer.
Okay, now for some tips if you're having difficulty with course concepts:
First, take your two exams to the professor. Also, bring along your notebook and homework solutions. I've found that really helps me evaluate where a student is running into trouble. Whether your notes are organized and complete, whether you're getting homework problems correct, which problems you're getting correct (are you getting the ones that are very similar to the examples given in class or in the textbook, but missing the ones that require synthesizing multiple concepts, or are you getting everything right for a given chapter, but not making connections between the material in separate chapters?). It could also come down to one key concept that you've misunderstood consistently that is leading to trouble in everything built upon that concept.
I've never taken any physics courses beyond "baby" physics (general physics geared toward the non-physics science majors...not quite as bad as the one geared toward non-science majors), but one thing I learned quickly is that diagrams are everything. If you need to draw pictures and diagrams to illustrate your point and help you solve the problems, take the time to draw them neatly and clearly label everything on it. Drawing a bunch of boxes and arrows is useless if nobody knows what they represent.
Out of all these general tips, from what you've written, I think there are two you should focus on first based on your description of the test and your self-assessment of what went wrong. 1) Keep organized and write legibly. If the grader can't follow your work, they can't even give you partial credit. And if the work is illegible, you also will have trouble going back to double check it. 2) Ensure you are understanding how concepts in each of the chapters covered on your exam relate to one another. It sounds like the questions you are getting require you to not just apply a single concept at a time, but to integrate your knowledge to demonstrate a complete understanding of the material, not just ability to solve problems in isolation from all other material in the course (the exam is testing understanding of concepts, not regurgitation of equations...this means you are entering the more advanced level of courses, so this is something you'll continue to need to do).
.
