I suspect you might not be in the US (China perhaps?). If that is true, have you visited both schools? They are very different types of schools: large state university versus small liberal-arts college/university. I don't think China (or indeed most other countries) has schools of the second type, so they might be a new thing for you. They're a very different environment than a large university like UW-Madison.
I did my undergraduate at a school similar to Lawrence but even smaller. Classes were small, and we had easy access to professors outside of class. We had a lot of freedom to pursue our own projects, subject of course to the limitations of the available facilities. All the upperclass physics majors had keys so we could get into the building and labs whenever we wanted. We socialized a lot with each other, usually in a study room in the department. The professors knew us well and could write letters of recommendation based on extensive experience with each of us.
On the other hand, we didn't have the large variety of courses, or the wide range of on-campus research, that a large university like UW-Madison offers.
My graduating class had five physics majors. All of us went to graduate school: three in physics (U of Michigan (me), Ohio State U, and Tufts U), one in engineering, and one in applied math (operations research). In the end, only one (me) became a physics professor, at a similar small school. One switched from physics to engineering and ended up at a large company. One became a technical writer. One became an engineering professor at a large university, doing research in robotics. I don't know what happened to the "math" person.
As hutchphd said, you can probably prepare well for graduate school at either place, but the living and learning experiences are very different. It comes down to which one is a better "fit" for you.