Dot product is a linear map. It's really a whole lot more complicated once you look at how dual vector spaces work, but let's forget that for a moment. Focus on linearity. You want property that a.b+a.c = a.(b+c) That tells you right away that a.(2b) = 2 a.b In other words, the result must be proportional to magnitude of b. Similarly, you can show that it's proportional to a. So you know the format must be ||a|| ||b|| * something. That something can only depend on directions of the vectors.
Next, if you dig a bit deeper into linear algebra, you'll see that (Ua).(Ub) = a.b, where U is any unitary transform. That is, if for example, you rotate both a and b vectors, their dot product doesn't change. That tells you that the dot product can only depend on the angle between the two vectors, not individual orientations. So you know that a.b = ||a|| ||b|| f(θ).
Showing that f(θ)=cos(θ) requires one more constraint. We want to make sure that length of our vectors is defined by the dot product itself. That is ||a||²=a.a That requires a function that satisfies f(0)=1, and then with some more hairy math you can get it to cos(θ). Unfortunately, I don't know a simple way to prove it that doesn't involve complex numbers.