I don't understand. Do you want R to be a binary relation? Let's go back a minute. Or let's start over.
S is a set. We don't care what its members are.
R is a 3-ary relation on S, meaning that R is a subset of S3, i.e., the set of all 3-tuples whose members are all in S.
P is a binary relation on S.
Q is a binary operation on S, meaning that Q is a subset of S2xS.
Do you want to define R in terms of P and Q? You can do this, yes. However, saying 6R5S4 only makes sense if R and S are both binary relations, and even then it is not clear whether you mean (6R5)S4 or (6R5)(5S4) or something else. Infix notation isn't as suited to relations or operations involving more than two inputs because there isn't an obvious way to separate things. Prefix or postfix works better: R(a, b, c), (a, b, c)R, (or if you only use atomic symbols) Rabc, abcR.
And an n-ary relation contains only n-tuples. It doesn't contain any (n-1)-tuples. If I want to define a set of relations sum-of-n that contain tuples whose last member is the sum of the preceding members, I need a different relation for each number of addends that I want. That is, I need a 3-ary sum-of-2 = {(a, b, c) | a + b = c}, a 4-ary relation sum-of-3 = {(a, b, c, d) | a + b + c = d}, etc. But sum-of-2 is not a subset of sum-of-3. They might contain the same information under certain circumstances (so you might define sum-of-2 = {(a, b, c) | (0, a, b, c) is in sum-of-3} where 0 is your additive identity), but an n-ary and m-ary relation do not have the same structure if n != m.
It is possible for an n-ary relation to include another n-ary realtion, as < includes <.
Note also that an n-ary operation does not have the same structure as its related (n+1)-ary relation. For example, the binary operation + on N is itself a binary relation. That is what the equal signs signifies. ((a, b), c) is in + iff a + b = c. They are two ways of saying the same thing. Now you can use + to define a 3-ary relation sum-of-2 such that (a, b, c) is in sum-of-2 iff ((a, b), c) is in +. But clearly (a, b, c) and ((a, b), c) do not have the same structure.
Note also that you can't necessarily go the other way and use an n-ary relation to define an (n-1)-ary operation by just gathering up the first n-1 terms into a tuple because you need to ensure that your operation meets the uniqueness condition. (1, 1, 2) and (1, 1, 3) can both be in the same relation but ((1, 1), 2) and (1, 1), 3) cannot both be in the same operation.
Does this address what you are trying to do?