too lazy to do a real schematic?
1. You need a 0.1 bypass across the shift register VDD and gnd. Don't use leads to connect it. Just put it over the IC.
2. When you have a 10K pulldown, the fall time when you release the button is slow. With 330 it is fast and either rings or is fast enough for the switch bounces to get through.
3. Switches bounce, causing multiple clock pulses. Usually one filters the heck out of them and uses a schmitt trigger to square up the edges. Look up switch bounce.
4. Adding a cap at the LED will do nothing. Add 100 ohms in series with the clock pin and a capacitor to ground at the clock pin. Or, just add a cap at the clock pin with no resistor. Experiment with the cap size. T=RC
Digital logic is very fast. Use the shortest possible ground. (you have a long inductor as a ground and power supply)