I think that direction of angular momentum is only a convention. Someone thought of that rule, it seemed practical since intensity of cross product of r and p really defines its amplitude. I am not sure, but think you could define another rule and get same physical results.
Such thing is with phasor diagrams, you choose phase of one quantity to be zero, and according to that do everything else.
This is only my opinion.