Hi Kyouran,
that is a good question. As you wrote, the work is always done by a force. This force always has its origin in some other body:Imagine man pushing a box on ground from right to left.
BOX MAN
--> <--
-F .. F
<--BOX...
The force acting on the box is [itex]\mathbf{F}[/itex], the force on the man is [itex]-\mathbf{F}[/itex].
Because these are two distinct forces, the have two distinct works.
1) The work of the force acting on the box
When the box has moved to the left by displacement [itex]\mathbf{s}[/itex], the force [itex]\mathbf{F}[/itex] has performed work
$$
W_1 = \mathbf F\cdot \mathbf s
$$
Since the force originates in the man, it is also often said that the man has done the work. In our case it is positive; this means the box has acquired equivalent energy and man has lost it.
2) The work of the force [itex]-\mathbf F[/itex] acting on the man is
$$
W_2 = (-\mathbf F)\cdot \mathbf s
$$
This is negative of the above work. Since the force originates in the box, we can equally say this work has been done by the box.We can choose anyone of the two above alternatives to describe the energy transfer; but only one at a time. Usually we choose that which has positive work, here it is 1).
So, when we say body A (a man) is doing work on body B (a box), we actually mean that there is a force due to body A that has some positive component along the velocity of the body B.