Well if you set an object into spinning motion, Newton's first law of inertia states that the object will keep spinning at its same rate unless acted upon by an outside force. Rotational force is known as "torque" so by mentioning that the wheels are going to slow down and stop implies that there is a net torque acting on the wheels opposite to the direction of spin. I suppose this could be like a frictional force. But let's assume that there is such a force.
With this in mind, the wheel with the greater angular momentum will spin longer than the wheel with less angular momentum if a net torque is applied to each wheel equally. You said that both wheels are spun with equal energy. I'm not exactly comfortable with the word energy, because I'm not sure I want to get into energy considerations right now, so let's say, instead, that the wheels are spun with the same amount of torque over the same interval of time. This means that the angular momentum will be the same for each wheel, but the wheels will be spinning at a different rate. The wheel with the greater moment of inertia will spin slower and the wheel with the lower moment of inertia will spin faster. Now let's say you apply a net "frictional" torque of equal magnitude to each wheel in the opposite direction of their spin. Both wheels should come to a stop at the same time, since they both have the same angular momentum, and torque is merely the change in angular momentum over change in time, dL/dt.
But let's say you spin both wheels so that both are spinning at the same rate. This means that the wheel with the greater moment of inertia will have greater angular momentum than the wheel with the lower moment of inertia. Inertia is resistance to change in motion, so the wheel with the greater moment of inertia will be less wanting to slow down as the other wheel, and so the wheel with the greater moment of inertia will be spinning longer than the other wheel should an net torque act in the opposite direction of spin on each wheel equally.
Moment of inertia is an object's resistance to spinning, and it depends on how the mass of the object is distributed. So if you look at the list of moments of inertia, the moment of inertia of the wheel with most of its mass at its center can be approximated by the solid cylinder, and it has a much smaller radius. As for the wheel with most of its mass on the outside, its moment of inertia can be approximated by the thin cylindrical shell with open ends, and its radius would be significantly larger than that of the solid cylinder.
If you look at the formulas, the solid cylinder already has a smaller inertia (mr^2)/2 as opposed to that of the thin cylindrical shell mr^2 even with the same radius. So by reducing the radius of the solid cylinder, you're only making its moment of inertia even smaller.