Having thought about this, these are my thoughts:
The definition of the Newton is the force it takes to accelerate a 1Kg mass (in space, outside of a gravitational field) at 1 m/s/s.
So, really, whatever work done that it takes to move that 1 Kg, can be expressed in terms of Kg per metre of movement - of a body in space. If the mass is 2 Kg, then work done doubles. Move a 1 Kg mass at an acceleration of 1 m/s/s, 1 metre and work done is 1 J.
You can also express the work done in terms of Kg per second of the movement of a body in space. Double the time, and work done doubles. Move a 1 Kg mass at an acceleration of 1 m/s/s for 1 second and work done is 1 joule.
Of course, here were talking about a mass hanging in space. And referring to mass, time, distance, acceleration.
If we get away from mass in space, and units associated with that situation, we may talk about Newton metres.
A force of 1 Newton moving anything, whatever it's mass, no matter how long it takes, 1 metre, equals 1 J of work done. There will be no acceleration if the force remains constant. And it is this Newton metre measure, that is really an equivalent to the other measures.
My 2 cents. I think I solved my problem.