To calculate the rotation of a gear in a rack and pinion system, the effective motion geometry is based on the pitch circle diameter of the gear. The linear displacement of the rack corresponds to the tangential displacement of the pinion at this pitch diameter. The formula for linear displacement can be derived from the angle turned by the pinion, using either the pitch circle diameter or radius. The pitch of the linear gear rack and the number of teeth on the gear wheel are essential for establishing the relationship between linear position and rotation angle. Understanding these parameters allows for accurate calculations of gear rotation and displacement.