It seems to me that -3/2 is indeed the minimum. Denote three unit vectors in 3-space by a,b,c with components a(i), b(i), c(i) where i=1,2,3. Then we wish to minimize a.b + b.c + c.a subject to a^2=b^2=c^2=1. Using Lagrange multipliers L,M,N we wish to minimize:
a(i)b(i) + b(i)c(i) + c(i)a(i) - L(a^2-1) - M(b^2-1) - N(c^2-1) where I have used summation convention for repeated indices.
Differentiating by a(i), b(i), c(i) respectively we get
b(i) + c(i) =2La(i)
a(i) + c(i) =2Mb(i)
b(i) + a(i) =2Nc(i)
it is immediately apparent from this that all the vectors are coplanar. furthermore, subtracting the first two of these gives
b(i) (1+2M) = a(i) (1+2L)
But by inspection the minimum is not achieved by having a(i) and b(i) be the same vector, since the cosine is then maximized, so we must have L=M=N= -1/2
Therefore b(i) + c(i) = -a(i), or a + b + c = 0. Hence at the extremum, a,b,c are at the vertices of an equilateral triangle, as conjectured.