You are right that they don't actually orbit the nucleus (and therefore that the Rutherford atom that's the default avatar here isn't a correct picture :) ).
However, popping into and out of existence is also a misconception - it's widely promulgated in the pop-sci press, but that doesn't make it right.
There are a bunch of threads here on this subject, but probably the best way of describing it is to say that the electron has no position (and that is literally "no" position, not "it's somewhere but we don't know exactly where") unless and until it interacts with something else in a way that locates it (an "observation" in the lingo). The mathematics of quantum mechanics give the probability that we will find the electron at various locations if we measure it, but says nothing about where it is when otherwise.