Just looking at the horizontal component of the displacement, force, velocity etc.:
Initially you apply a small force to get the object moving in the horizontal direction. This does work, and in the absence of friction, imparts some velocity to the object. The work, therefore, has gone into increasing the kinetic energy of the object.
When the object eventually comes to a rest, it has no horizontal velocity. Its kinetic energy has been removed, therefore some negative work has been done on it. This means you applied a force in the reverse direction. Again, in the absence of friction, the negative work done to stop the object is equal in magnitude to the work done to get the object moving horizontally. Therefore, if you move a brick a km horizontally, you don't do any net work. (In your equation W = F.d, F is not constant, it changes sign). The only work done goes towards increasing the potential energy of the object.
This may be hard to imagine because there is always friction in the real world, so one is always doing some work to overcome energy dissipation. But in an "ideal" world, there is no work done when something is moved horizontally.