This thread seems to have gone on some irrelevant tangents. It makes no difference what the sails do when the motor either speeds up or slows the boat down. It doesn't make any difference what the hull speed of the boat is (except in that clearly it is higher than 10 knots because somehow the boat got there). The original question said nothing about increasing the speed to 15 knots. It doesn't matter how the boat got to ten knots. Etc etc.
The question asked was: given a motor and prop that can propel a boat to 5 knots by itself, if the boat is made to travel at 10 knots will the motor add or subtract? I.e will said motor be a thrust or a drag when it is moving through the water at ten knots. All we are interested in is the sign and all that matters is the motor and prop except perhaps to argue that the 5 knot limit was somehow indicative of something about the design of the motor.
At five knots the thrust from the motor equaled the drag from the boat. That is clearly not zero thrust. The drag from the boat is not zero, so the thrust from the motor is not zero. There is no reason to think the thrust will suddenly drop to zero if the speed through the water is increased. The drag of the boat would increase and the motor doesn't have enough thrust to overcome the increase in drag, thus the top speed, but the thrust cannot and will not instantly drop to zero. if the boat is forced through the water at say 5.1 knots by some other agency whether it be sails, a separate stronger motor, or the boat is being towed behind a battleship, there is absolutely no question that the motor will continue to provide thrust at some speed which is at least slightly higher than the speed the motor alone could achieve.
There is certainly a speed higher than 5 knots at which the answer to the question which was asked is positively yes, the motor will provide extra thrust.
There is a speed through the water where the rotational speed of the prop and the pitch of the prop give a negative angle of attack. The prop cannot provide thrust at a negative angle of attack. There is certainly a speed high enough that the answer to the question which was asked is positively no, the motor will not provide extra thrust. It will be a drag.
Therefore there is a speed higher than 5 knots at which the answer changes from yes to no. The only question then is will that speed be above or below 10 knots.
The only way the boat enters into this question is what it's drag is at five knots. That gives us a single point on the motor's thrust to speed curve, although I'm not sure even that is much help. Everything else about the boat is irrelevant.
Instead of talking about sailboats and hull speeds and points of sail, this question should focus on propeller and motor design.
I feel very strongly that it is possible to create a plausible situation where the thrust of the propeller equals the drag of the boat at 5 knots and yet the thrust of the propeller is greater than zero at 10 knots. So I don't believe that there is a fundamental reason that the propeller can't have positive thrust at ten knots.
I am equally sure that it is possible to create a plausible situation where the propeller turns to drag below 10 knots.
The argument then is not about what is required by first principles, but rather what is likely by engineering choices of the designer of the propeller and motor, not to mention the choices of the boat owner who doesn't always buy the motor and propeller best optimized for how he intends to use it.
There is a perfect example curve here:
http://atljsoft.com/html_help/Example, 34' Sailboat.htm
For the weakest prop the thrust crosses the drag a 7 or 8 knots. The thrust curve is rather shallow. They didn't draw much of it and it surely has shape making extrapolation iffy at best, but you can see it is heading off to hit zero at MUCH higher than the 8 knot top speed. Maybe 14 or 15 knots? Could it be double the thrust equals drag point? Sure. Could it be less? Sure.
So despite all the noise the answer is "physics doesn't tell you from the information provided". It can easily be engineered to happen either way, and the thrust vs speed curves I've been looking at suggest 10 knots would be on the edge of where the answer changes from yes to no.