Is there a formula I could use to determine how far in a medium a seismic wave would move if I hit one end of it with a hammer? I know the density of the rock and the force of the hammer.
You wil need to know a lot more than that. And probably you don't actualy know even the force of the hammer. Rather the energy or momentum of the hammer before hitting.
You will need the atenuation coefficient of the medium for the frquency range produced by your hammer hitting the medium. And probably numrical simulation of both the excitation proces and propagation.