statdad is right.
A normal distribution has only two parameters, its mean and its standard deviation. The average of your data is an unbiased estimator of the mean, and the mean square error is an unbiased estimator of the standard deviation.
this is why these two statistics are used. they are unbiased estimators for the defining parameters of the unknown normal distribution.
In reality, a lot of data is close to normal. For instance stock price returns are nearly normal. In these cases one can estimate the distributions directly with the sample mean and standard deviation. If data is not normal then one can take averages before estimating parameters. Before computers, this is what statisticians did because they needed to know something about the mathematical form of their sample distribution. if you average your data to create a new sampling distribution then this new distribution of averages is approximately normal. this reduces the problem of data analysis to estimating which normal distribution your sample data comes form.