Your plant has the transfer function: G = (s^2-1) / (s^4-s^2-1)
In #2 I have suggested a transfer function like: H = K*(s+a)(s+b)(s+c) / ( ( s+d)(s+e)(s+f) )
The two transfer functions are connected in series, so the overall transfer function as for the open loop will be: GH(s) = G(s) * H(s)
It is tempting to place a zero/pole pair in H(s) matching the righthand pole/zero pair in G(s), to cancel/get rid of them, but that doesn't work in practice because you cannot hit them exactly. Thus you must do as stated in #2:
Hesch said:
Now, calculate the algebraic characteristic equation as for the closed loop transfer function, H1(s) = 0
Express the desired characteristic equation, H2(s) = 0.
By inspection of H1(s) = H2(s) you can setup 7 linear equations.
Solve them and you have found the values of K, a . . f.
So having found the 3 zeroes and 3 poles in H(s), you must place the overall 5 zeros and 7 poles in the same root locus.
Now, calculate the characteristic equation (Mason's rule) as for the
closed loop transfer function, plot the root locus by varying K from 0 to ?. You should get something like this ( just an example with only 3 curves):
At the calculated K-value, you should see all the 7 curves passing the desired locations, left to the imaginary axis ( stable area ).
Normally you can see by intuition, where about zeroes and poles are to be placed, but that's impossible ( for me ) in a 7. order system.
Likewise I cannot (any longer) solve 7 linear equations, with complex number results, by mental calculations.

. . .
