You need to be aware that charge particles strike a lot of things, including conductors, all the time. Cosmic rays consist of such particles. So this is not uncommon.
So now, use this experience that you live with everyday to come up with a plausible answer. After all, if charge particle strikes are common, why haven't you been electrocuted yet due to this buildup in voltage? Most conductors are attached to other conductors and other things. Most are grounded. Any net charge that a conductor gets from such strikes will flow out of that conductor. Even one that is isolated is not truly "isolated", because it is surrounded by air, water vapor, etc. where at some point, access charge will be able to escape.
Zz.