You'll have to be more specific about the subject that you want to study. "Optimization theory" could be the study of how to maximize and minimize continuous functions, or the sutdy of how to maximize and minimize functions defined only on the integers ("Integer Programming"), or the study of how to minimize risk ( as defined mathematically) in situations involving probability ("Optimal Statistical Decisions"), or the study of how to plan experiments for fitting multinomial models to data ("The Design Of Experiments").
One of my professors said "Every problem can be phrased as an optimization problem."
I suppose the sub-fields I'd be interested in involve maximization and minimization of continuous functions using an associated cost function of all of the parameters. Optimal statistical decisions also sounds interesting. Any references to good books on these topics would be much appreciated.