I am not sure if the OP's specific question is resolved, but I think I can answer it.
The OP is asking given a theory T, with a collection of nonisomorphic models, how to specify a subset of the models. I am unsure how much model theory does the OP know, and whether they have in mind are additional assumptions I missed. What I know below is from undergraduate model theory, so there may be more to it.
The first thing to check is whether T is complete. A complete theory is one which derives every sentence or it's negation. If T is not a complete theory, then there is some sentence σ of first order logic that is independent of T, which is to say that both T∪{σ} and T∪{~σ} are consistent, and by completeness have models. Then simply adding σ to you set of axioms will remove all models of T∪{~σ}, and you have specified a subset of the models of T.
An example of this is when T is the group axioms and σ is the sentence asserting commutative. This parses down from all groups to abelian groups
It is however, possible for T to be complete, yet have nonisomorphic models. In fact, it is a theorem of model theory that if T has an infnite model, it has models of arbitrarily large infinite cardinalities. An easy way to specify a subset of models would be to specify a cardinality, since models of different sizes can never be isomorphic. This specification will not be first order in general unless you specify a finite cardinality. In the previous example this may mean requiring your groups to have 3 elements, which would give you the cyclic group of order 3, or countable etc.
An example of a complete theory with nonisomorphic models is the theory of dense linear orders without endpoints. It has only one model in countable infinity, i.e. ℚ, and it is a theorem that if a theory has only one isomorphism class in some uncountable cardnality, then it must be complete. Several nonisomorphic dense linear orders without endpoints in cardinality continuum include ℝ and (a copy of ℝ followed by a copy of ℚ)
As an aside, In model theory, theories with only one model in cardinality λ are called λ-categorical. Categoricity is an important concept in model theory, as with completeness. There are other important concepts the OP may be interested in, including "stability" which roughly means that there are not too many elements that could exist but don't. An example of an unstability is in dense linear orders, because in the model ℚ there are only countable many elements, but you can describe uncountably many real numbers by dedekind cuts using only the language of order and parameters from ℚ.
Going back to the OP's question again, in general whatever mathematical property you want to add to a theory T to restrict the models, it will likely be first order, but in the language of sets rather than your original language. For example the property of being noetherian in a ring is not first order in the language of algebra, but you can formula it in terms of sets.
EDIT: Thinking of the OP's example of SO(3), you can never find first order axioms such that the only model is SO(3), because SO(3) is an infinite model, and you can build models of arbitrarily large cardinalities that satisfy the same first order sentences (and your axioms) as SO(3) by ultrapowers, or the compactness theorem etc.
stevendaryl said:
I spent a fair amount of time pretending to do mathematical logic, and I think I can answer this question.
A model is definitely not a collection of axioms. It's a mathematical object.
BTW this is true, but the completeness theorem constructs models by taking equivalence classes of strings of symbols. When I think about the completeness theorem, models in some sense literally become quotients of infinite lists of sentences, and this gives me weird philosophical thoughts about the nature of language, reality, etc.