When I was an undergrad I met Bert Brockhouse (Nobel Prize in Physics 1994) and I asked him if he had any advice for a young person interested in physics. He told how important it was to develop "green thumbs." What he was talking about was the same thing Mary Boas introduced in her Mathematical Methods textbook when she talked about the difference between knowledge and skill. The cultivation of one's scientific skill set comes through practice. And I think that's at least one of the more important things one gains with research: skill development. And not only in terms of techniques, but project organization and design, systematic investigation, data analysis, working effectively as a member of a team, learning to write up results, learning to present and defend your work... even developing the ability to persevere on a problem on those days/weeks/months when everything you do doesn't seem to work. So to that end, when looking for an undergraduate project it's important to think about the specific skills you want to develop.
The other thing that comes to mind is that these kind of experiences are your best window into what life as a graduate student -> post-doc -> academic is going to be like. Even if you get nothing else out of them, they will give you insight into this life. So if you're interested in the "theory" side of things, look for research projects with people who work in this area, on problems that seem interesting to you. If it turns out that all the stuff you think is so cool in the lectures is actually quite mundane when you start working with it on a daily basis, it's a lot better if you figure that out over a summer as an undergrad than after you've committed to a four-to-six year project on the stuff.
.