This is exactly what I always figured.
Semesters of challenging coursework at best
give a 2.5% bump which is diminished after averaging. On the other hand the same coursework has the potential to hurt you more than that 2.5%.
I would say it is fairly insignificant since a 2.5% boost on a subsection is pretty insignificant.