What you are feeling in a roller coaster and provoke an effect on you is the acceleration. The acceleration when going downhill, but even more the lateral acceleration when you are entering a curve or going upside down.
Acceleration means that your body is changing its speed and/or its direction. Normally your brain is supposed to control that and the fact that something else is controlling your body's position (especially so violently) can be interpreted as a sign of danger by your brain, thus creating fear.
If it is scarier in the first car, it is mostly because in the other cars, your brain can expect what will happen by seeing what happened to the cars in front of you. In the first car, your brain have more difficulties determining the acceleration you will be subjected to, making the experience more frightening.