There is something called a
pointed set. It is simply a set with a 'distinguished' element. Such an object has no structure whatsoever, aside from the fact that one of its elements is 'distinguished'. The simplest way to define such a thing is
A pointed set is a pair (S, x), where S is a set and x is an element of S.
If you wanted a formal system instead, then the theory of a pointed set is presented simply by specifying that the language has a constant symbol. (usually, '*' is used) In particular, no axioms are given -- this theory consists only of tautologies.
Pointed topological spaces are a useful object for some purposes. As the name suggests, it's simply a topological space with a distinguished point. A familiar example would be a Euclidean line or plane with a specified origin.
But these things don't really have a notion of "primeness".
Incidentally, Peano's axioms don't yield a notion of primeness either -- there are lots of ways to put an ordering or an algebraic structure on an object satisfying Peano's axioms. For example, one can define addition recursively by:
a + v = v
a + S(b) = S(a + b)
(often, one would use the symbol '0' instead of 'v' if one intends to use this definition of addition)
or, one might define addition by
a + v = S(v)
a + S(b) = S(a + b)
(and one would typically use the symbol '1' instead of 'v')
One could even define addition so that v+v is undefined, and
S(a) + v = a
v + S(b) = b
a + S(b) = S(a + b)
(one might use the symbol '-1' instead of 'v' if one were to adopt this definition)
For the above examples I was using S as if it were an "add one" operation -- but one could define an addition operation in an entirely different way, if one so desires!
The point is that primeness doesn't automatically make sense -- it is only meaningful relative to an algebraic structure.
The (usual) ordering on the ordinal numbers is indeed given by containment: \alpha < \beta if and only if \alpha \subseteq \beta. And for the usual set-theoretic model of the natural numbers, this ordering does agree with the usual ordering on the natural numbers.
Incidentally, the main practical reason for studying ordinal numbers is that they are very useful for analyzing and proving things -- the practical content of the fact the finite ordinals model the natural numbers is that it allows us to transfer our expertise with natural numbers into a set-theoretic context.
Probably the most pervasive notion of "primeness" in mathematics is that of a prime element in a
lattice. For example, the notion of primeness you're familiar with -- primeness of integers -- is a special case of this. You can organize the positive integers into a lattice by defining a \leq b if and only if a divides b. Or equivalently, by defining a \vee b = lcm(a, b) and a \wedge b = gcd(a, b).