I relate a lot to your idea of pushing fronteirs, but in a way that is unique to myself. I've always been smart, knew I was smart, and did well in school, but the reason for a lot of that has to do with my parents, and the idea of success that I was given. I always wanted to go further or learn something extra, but only if it was interesting to me.
On a very similar note, I have been and still am very competitive. I feel like I've always had this energy for winning or being better or more skilled (immature I know...I've been working on it haha). I feel like this was the process which I used to focus my energy. At points in my life I would almost describe winning as an obsession to me. I'm getting a little off topic, but allow me to bring it back together. Without the idea that I was smart, I wouldn't have had the confidence to pursue learning. I guess staying ahead of the curve was what I always used for motivation. I didn't want to downgrade to the average kid (I'm sounding like a terrible person
But my idea for you is to introduce some competition into the group of boys that don't seem to be motivated. Winning is fun, and losing is not. Both are methods of motivation. However, and this probably the same however that you are having, a lot of people shy away from competition, and it can alienate people, or put pressure on people who don't want it. If you can find a way to spice it up with a little competition, but make sure that the intensity threshold doesn't oppress anybody, I think you might spark some more interest.