Let the vectors from the origin to the 3 points you know be [itex]\vec{p}_1,\vec{p}_2,\vec{p}_3[/itex], and call the vector from the origin to the center point [itex]\vec{c}[/itex]. Our goal is to find [itex]\vec{c}[/itex].
The vector from point 1 to the center, [itex]\vec{p}_1-\vec{c}[/itex], must be the same length as the vector from point 2 to the center, [itex]\vec{p}_2-\vec{c}[/itex], which must be the same length as the vector from point 3 to the center, [itex]\vec{p}_3-\vec{c}[/itex]. In other words, we have the following 3 equations:
[tex]\left(\vec{c}-\vec{p}_1\right)\cdot\left(\vec{c}-\vec{p}_1\right)=\left(\vec{c}-\vec{p}_2\right)\cdot\left(\vec{c}-\vec{p}_2\right)[/tex]
[tex]\left(\vec{c}-\vec{p}_1\right)\cdot\left(\vec{c}-\vec{p}_1\right)=\left(\vec{c}-\vec{p}_3\right)\cdot\left(\vec{c}-\vec{p}_3\right)[/tex]
[tex]\left(\vec{c}-\vec{p}_2\right)\cdot\left(\vec{c}-\vec{p}_2\right)=\left(\vec{c}-\vec{p}_3\right)\cdot\left(\vec{c}-\vec{p}_3\right)[/tex]
Focus on the first one. We can multiply this out and simplify (note the cancellation of [itex]\vec{c}\cdot\vec{c}[/itex]):
[tex]\left(\vec{c}-\vec{p}_1\right)\cdot\left(\vec{c}-\vec{p}_1\right)=\left(\vec{c}-\vec{p}_2\right)\cdot\left(\vec{c}-\vec{p}_2\right)[/tex]
[tex]-2\vec{c}\cdot\vec{p}_1+\vec{p}_1\cdot\vec{p}_1=-2\vec{c}\cdot\vec{p}_2+\vec{p}_2\cdot\vec{p}_2[/tex]
[tex]\vec{c}\cdot\left(\vec{p}_2-\vec{p}_1\right)=\left(\vec{p}_1\cdot\vec{p}_1-\vec{p}_2\cdot\vec{p}_2\right)/2[/tex]
The same thing could be done for the other 2 starting equations, in which case we get the following 3 results:
[tex]\vec{c}\cdot\left(\vec{p}_2-\vec{p}_1\right)=\left(\vec{p}_1\cdot\vec{p}_1-\vec{p}_2\cdot\vec{p}_2\right)/2[/tex]
[tex]\vec{c}\cdot\left(\vec{p}_3-\vec{p}_1\right)=\left(\vec{p}_1\cdot\vec{p}_1-\vec{p}_3\cdot\vec{p}_3\right)/2[/tex]
[tex]\vec{c}\cdot\left(\vec{p}_3-\vec{p}_2\right)=\left(\vec{p}_2\cdot\vec{p}_2-\vec{p}_3\cdot\vec{p}_3\right)/2[/tex]
If we recast these 3 equations in matrix form, we get,
[tex]\left(\begin{matrix}\vec{p}_2-\vec{p}_1 \\ \vec{p}_3-\vec{p}_1 \\ \vec{p}_3-\vec{p}_2\end{matrix}\right)\cdot\vec{c} = \frac{1}{2}\left(\begin{matrix}\vec{p}_1\cdot\vec{p}_1-\vec{p}_2\cdot\vec{p}_2 \\ \vec{p}_1\cdot\vec{p}_1-\vec{p}_3\cdot\vec{p}_3 \\ \vec{p}_2\cdot\vec{p}_2-\vec{p}_3\cdot\vec{p}_3 \end{matrix}\right)[/tex]
Then multiplying both sides by the matrix inverse, we get the center, [itex]\vec{c}[/itex].
[tex]\vec{c} = \frac{1}{2}\left(\begin{matrix}\vec{p}_2-\vec{p}_1 \\ \vec{p}_3-\vec{p}_1 \\ \vec{p}_3-\vec{p}_2\end{matrix}\right)^{-1}\left(\begin{matrix}\vec{p}_1\cdot\vec{p}_1-\vec{p}_2\cdot\vec{p}_2 \\ \vec{p}_1\cdot\vec{p}_1-\vec{p}_3\cdot\vec{p}_3 \\ \vec{p}_2\cdot\vec{p}_2-\vec{p}_3\cdot\vec{p}_3 \end{matrix}\right)[/tex]
You can find the radius by taking the length of the vector from the center to one of the points on the circle,
[tex]R =\left|\vec{c}-\vec{p}_1\right|[/tex]