You seem to be applying both maximum forces at the same time. It won't be the case that for a given load the maximum force occurs in both elements simultaneously; one will reach its maximum before other (but you can't tell which yet).
So, start by assuming that the maximum in the jib element is reached first. Set up static equilibrium equations and solve for the tension in the tie. Does it exceed its maximum? If not, find the load and this will be your maximum load. If it does, then clearly the tie maxes out first so redo the calculations beginning with the tie tension set to maximum.