PeterDonis said:
how is force defined when the system is not of constant mass?
To wrap up the discussion in this thread, I'm going to answer this question and then give some brief follow-up. The total force on the spacecraft in the scenario given in the OP of this thread is
$$
\vec{F} = \vec{F}_\text{external} + \left( \vec{u} - \vec{v} \right) \frac{dm}{dt} = m \frac{d \vec{v}}{dt} = m \vec{a}
$$
The total force here is frame invariant and has the obvious physical interpretation of the force exerted by the dust on the spacecraft , plus the "magical" external force exerted to oppose the dust force and keep the spacecraft moving uniformly (i.e., so that ##\vec{a} = 0##).
This also shows that Newton's Second Law, in the form ##\vec{F} = m \vec{a}##, with the total force ##\vec{F}## properly defined, works just fine for systems with time-varying mass.
What about the equation ##\vec{F} = d \vec{p} / dt##? We can obtain that from the above equation by rearranging it as follows:
$$
\vec{F}_\text{external} + \vec{u} \frac{dm}{dt} = m \frac{d \vec{v}}{dt} + \vec{v} \frac{dm}{dt} = \frac{d \vec{p}}{dt}
$$
The LHS is the sum of a frame invariant term and a frame-dependent term, so the LHS as a whole is frame-dependent; thus, the RHS must also be frame-dependent. So this equation, unlike the equation above, does
not express a relationship between frame-invariant quantities.
Furthermore, the "force" appearing on the LHS of this equation is
not the total force, since the term ##\vec{v} \, d \vec{m} / dt##, which is part of the total force, at least if "total force" has the obvious physical meaning described above, is included in the RHS, not the LHS. Defining the LHS of this equation as "force" is giving a
different, frame-dependent meaning to the term "force".
It is true that, mathematically, both equations are equivalent in terms of what they will say about actual experimental predictions, and what one chooses to mean by the term "force" is a question of choice of words, not physics. The first equation seems to me to be a clearer representation of the actual physics, since it relates invariant quantities. But if someone wants to use the second equation, as long as they are able to get the right answers with it, it's up to them.