These brackets refer to the norm and the norm of a vector in R^n is simply is the sum of the components squared. [This norm is simply the ||x-0|| for some vector x]
Also you can't use this to prove what AXB is, but you can prove that for some directional normal vector n^ then you can prove that AXB = nhat*length where length is |AXB|.