There are to elements of acceleration. Instant acceleration, and average acceleration.
High torque can give you a high instant acceleration. For example you could generate say, .6g's of acceleration in a specific gear, which is pretty hard acceleration in a car. However, if you only manage that for 2 seconds before you need to change up to the next gear, is it a fast car? If we round up 1G to 10m/s, then the difference in speed would only be 12m/s.
On the other hand, a petrol car might only manage .4g's of acceleration in a gear, BUT it will be able to maintain it for maybe 4 seconds. That would be 16m/s change in velocity for that gear. The rate of acceleration is lower, but the average acceleration is higher, and this is the effect of power. Being able to accelerate hard with lots of torque is meaningless if you cannot maintain it with power.
So yes, diesel cars are generally nowhere near as fast as they feel.
In the case of equally powered versions of either, then it ultimate still boils down to the thrust curve, which is dictated by power. You can plot a basic acceleration curve for a car, with velocity in m/s and power in KW, it is simply (P/V) / (Mass * 9.81). Acceleration will be on the Y axis and Velocity on the X. It will give you unrealistic values at low speeds as it will be limited by traction, but that curve is the acceleration capacity of any car and you see that torque does not factor into it.