cristo said:
Yes, the answer should be a number. I think your command '* may be incorrect. A quick google search tells me that the scalar triple product [itex]A\cdot (B\times C)[/itex] should be entered like this dot(A,cross(B,C)).
Matlab will always "do the right thing" with dot(A,cross(B,C)) if A,B,C are three vectors.
The function cross(B,C) produces the same type of vector (column vector or row vector) as the inputs. If B and C are column vectors, cross(B,C) will be a 3x1 column vector. If A is also a column vector, then the notation A.'*cross(B,C) does produce the scalar triple product.
OTOH, if A, B, and C are row vectors (1x3), cross(B,C) is 1x3 and A.' is 3x1. Following the rules of matrix multiplication, the product of a 3x1 matrix with a 1x3 matrix is a 3x3 matrix. If you are using row vectors, the appropriate expression is A*cross(B,C).'
If A, B, C are of mixed row/column type (not a recommended practice), use dot(A,cross(B,C)).