ndung200790,
everything you said so far can be understood as a symmetry operation represented by a (finite or infinite or even continuous) group.
In some cases it makes sense to discuss generators (in case of Lie groups and algebras), in some other cases (like C, P, T or crystallographic groups) not.
There are rather general cases of symmetry structures like the diffeomorphism or mapping class group for GR, infinite dimensional Kac-Moody algebras (as generalizations of finite dimensional Lie algebras) with central extension in string theory, supersymmetry / supergravity / graded algebras, quantum deformations of U(1), SU(2), dynamical symmetries (ordinay symmetry groups like SU(n) for the n-dim. harmonic oscillator) and perhaps many more which I am not aware of. I haven't seen anything else that does not belong to such a (generalized) symmetry structure.
The reason is rather simple.
Consider you have a Lagrangian L[x]; now you do something with it and get L[x'] where x has been transformed using something called 'g', but L is invariant (b/c it's a symmetry ;-). No you do something else with it called 'h' and get L[x'']. You can now write formally
x'= g*x
x''= h*x' = hg*x
which automatically results in a group structure!
I have no idea how to talk about a symmetries or transformations which do not form to a group.
Please have a look at
http://en.wikipedia.org/wiki/Group_(mathematics)#Definition
Which property can be relaxed?
Closure: you have a symmetry transformation, a second symmetry transformation, but both transformation together are not a symmetry ?
Associativity: OK, perhaps one could play around with Octonions, non-associative algebras ...
Identity element: having no identity element would mean that it's not possible to do nothing!
Inverse element: having no inverse means that certain transformations cannot be undone; I have no idea how this would look like
Please give me a hint what you have in mind