This is fear, not restraint. It;s a reason as good as any other to not fight, but you must not misinterpret it as restraint.
Only that it doesn't work that way. Humans prefer to beat targets which are easy to beat or prey upon. Targets of physical violence , bullying, rape (or conn actions for that matter) are usually selected for from categories of persons which are least likely to represent a real threat to the attacker. That's it, easy targets. Some humans are very good at selecting least dangerous opponents. Uncannily so. It works the same way like when you see a women and in several seconds you decide she is hot or not. You just sit and look at the flow of pedestrians, and you will just know who is more likely to be a victim. It doesn't take more than seconds of observation. We are very good at this. In this case, reading a book by its cover works. The process is surprisingly accurate.
And agression is not about "jolly". Its mainly about emotive responses to stressors, domination and instrumental goals.
But despite the accuracy of this process, you may be in for a surprise some times.
And the statement "your not afraid to die" is a bit melodramatic. Youll **** your pants if it comes to that. And to be honest, it takes much less than a life threat to make an average person be very afraid.