When I was first learning torque and moments of inertia, I had some of the same problems with it that the OP did, so I'm going to take a stab at answering this from a different approach than those above. Conceptually, this approach worked for me, so I hope it will help the OP as well.
Imagine a non-rigid rod, but rather one made of a lattice of point particles of some arbitrary mass connected by springs. Prior to the application of a force, suppose that all of the particles are at rest with respect to each other, and all of the springs are in their equilibrium position.
Now, apply a force to a particle at one end of the rod in a direction perpendicular to the axis of the rod. Initially, you will get a deformation in the shape of the rod as that particle begins to move in the direction of the force. Hooke's Law will propagate the force through the rod along the direction parallel to it, but what happens with respect to the perpendicular directions? Those springs will become elongated as the rod deforms, creating an internal restoring force which acts (roughly) along the axis of the rod. However, because our external force is being applied only at one end of the rod, this restoring force is unbalanced, leading to a net acceleration along the axis of the rod in addition to the net acceleration in the direction of the external force. Thus, rotation.
Now, add in a second force, anti-parallel to the first and of equal magnitude, some distance between the end of the rod and the center of mass. The particles at the top of the still feel the same two forces: that generated by external force one and the restoring force. At the point where external force two acts, the same thing happens; the external force causes a deformation, and the deformation generates a restoring force. However, in the case, the restoring force is less unbalanced, because portions of the rod are on both sides of the point where the external force acts. Thus, while this force also wants to generate rotation (in a direction opposite to the first force), it does so less strongly because of the cancellation of some of the restoring force. If the force were acting at the center of the rod, the restoring would cancel completely and no rotation would be generated at all. It's easy to see that the magnitude of the component along the axis of the rod in this picture is roughly proportional to the distance from the center of mass, so at a qualitative level this gives the justification for torque having the same property.
Now, granted, this picture changes as soon as you allow the system to evolve in time, due to the deformation of the rod. However, if you assume the spring constants to be arbitrarily large, then the deformation stays small and this picture holds some validity. Taking the spring constants to be infinite gives you the rigid body limit.