If you can reference a copy of ANSI B92.1 you will get a ton of information on spline/gear measurement techniques.
Are you referring for inspection purposes or just general measurement? If accuracy is not a major issue, an optical comparitor is the easiest way that many places will go. If accuracy is required for the acceptance of a part, then measurement over pins is used in most cases. A drawing for the gear will give data on the gear tooth geometry that will be used in inspection. I attached a portion from a drawing as a reference.
As Fred points out, there are inspection methods for gear teeth but there are also simple identification methods. I've seen gear gauges used before, they are simply a small piece of metal with gear teeth cut into them. They generally come in sets. You take out each coupon and try to match it to a gear. The one that matches gives you the pressure angle and gear tooth size. Its not meant for inspection, but they're good for identifying a specific gear. If you want to purchase one, they cost about $40 to $120 from McMaster Carr. Type in "gear gauge" and search their catalog.