One way to think about this is that PdV is only dW for quasi-static situations (i.e., gases evolving from one equilibrium state to another). If both the gas inside and outside are at the same P and evolve quasi-statically, then the distinction between the work done by the interior gas and the work done by the exterior gas gets blurred because it is not an important distinction to make when they are the same (except for sign). But the distinction is always there-- we should never say that the work is done on the exterior gas rather than by the interior gas-- instead, we should simply note that both the interior and exterior gases are doing work, and we will need to analyze that work separately if we do not have a reversible quasi-static process.
I'm really not sure where the idea comes from that the work done by an irreversible process is the PdV on the external gas, that just isn't consistent use of language (I'm not saying anyone on here made up that language, you see it in a lot of places). Even for an irreversible process, there is a work done by the internal gas, and a work done by the external gas, and they can be different. Whether or not either one will be PdV just depends on whether or not that particular gas is maintaining an isotropic pressure, or if there is significant dynamical energy appearing that cannot be treated as quasistatic.
Apparently the case people often talk about in this context is so-called "free expansion", where you have an interior gas at much higher pressure than the exterior gas, and there it is clear that the interior gas is going to carry a lot of dynamical bulk-flow energy that cannot be treated as PdV work on its surroundings because P will not completely characterize the momentum flux density. But the surrounding gas should also generate significant bulk flows in this case, so PdV isn't going to be right for the work done by the external gas either unless the external gas has a much higher sound speed than the internal gas (so a shock won't be driven into the external gas). I think a lot of what is said about free expansion just doesn't make a whole lot of sense.