And what does that mean three vectors to have the same extent against fourth vector?

In the relations you have shown, what are the operations? Are yu multiplying the magnitudes of the vectors, taking the dotproduct, what?
Well, r is vector from the origin the point on the boundary where wave is reflected and transmitted.

It is an arbitrary vector then? (Since the choice of origin is arbitrary.) Therefore there is no reason that the rays should be in a plane defined by the incedent ray and some arbitrary vector unless the origin is in some special place.
This sort of setup is normally used when you want to invoke the principle of least time.
Well, I think Pedrotti takes only the equations I have written in his argument (Introduction to Optics, third edition, page 493).

I don't have that one. You should doublecheck the context that he's doing this in.