I am trying to create an interactive whiteboard with an IR camera (wiimote actually), IR pen, and an arbitrary surface. For simplicity I'll assume the surface is a screen. To know how an IR dot from the camera's point of view relates to a point on the screen's surface, I must calibrate it with a series of points. I have a matrix [A] that represents the location of a series of points on the screen's surface S and a matrix that represents the location of those points as projected on to the camera's viewing plane C. From those points I am trying to figure out the transformation that must be applied to all subsequent points. If S is thought of as a plane in an ideal position relative to the camera (i.e. z=1, Sx,y=Cx,y), I know the situation can almost be modeled by: [P][T]=[A] where [T] is a 3D transformation matrix, [P] is a 2D-3D projection matrix, [A] is a set of three 3D vectors, and is a set of three 2D vectors. If that was correct, I could simply perform -1[A] to obtain the transformation/projection matrix for subsequent points. Unfortunately, I don't know the Z of the points in and don't know how to compensate. Also, I want to use more than three calibration points, but that would make non-square. I have found many different materials on the topic, but none of them where exactly what I am looking for. Any help would be greatly appreciated. This problem is driving me nuts.