Unless you are using infinite precision arithmetic, you cannot tell whether a matrix truly is or is an orthogonal matrix. You can't use infinite precision arithmetic, so your answer will necessarily be approximate.
What you can do is roll your own function, using the fact that an orthogonal NxN matrix has rank N, that each row (or column) is a unit vector, and that each row is orthogonal to all other rows (or columns). Matlab has the rank function to help with the first item. This one is fairly cut and dry. An NxN matrix that has rank less than N cannot be an orthogonal matrix.
The other two items are going to be problematic. You'll need some tolerances such that a row whose magnitude is not quite 1 but is within some ε of 1 is still deemed to be a unit vector, and such that a pair of rows whose inner product is not quite zero but is within some other ε of 0 are still deemed to be orthogonal to one another.