No. If ##ab < 0## you cannot take its square root (at least, not in the real numbers), and of course, if ##ab < 0## you cannot have ##a = b = 0## (so you should never, ever write that). However, you can get a contradiction another way:
(1) You have just shown that ##ab = -a^2 - b^2##, which is ## \leq 0##.
(2) You can write ##a^2 + b^2 + ab## as ##(a+b)^2 - ab##, so you also have ##ab = (a+b)^2##, which is ## \geq 0##.
(1) and (2) are compatible only when ##a = b = 0##.

I should rephrase some stuff in my post, when I say must be positive or must be negative, I mean, ## ≥ 0 or ≤ 0##
So, ##ab ≤ 0##
If ##a+b=(ab)^{1/2}##
The only real solution is if ##ab = 0##, exactly because there is no real root of a negative number, which means that ##a=0## or ##b=0## or ##a=b=0##
if ##ab = 0##, then ##a+b=0##, so ##a=-b##
Since one of them is 0, so must be the other.

Well late to the party but I've been thinking about extending the domain of this question to including all numbers including complex numbers. This is my proof for it and I'd like to know if it is valid.

Given ##a^2+ab+b^2=0## ∀ ##a,b ∈ ℂ##
##a^2+b^2=-ab## ⇒(Realise that for this to be true only either ##a,b∈ℝ## or both ##a,b∉ℝ##)[tex] ab =
\begin{cases}
≤0 & \text{if } a,b ∈ ℝ \\
≥0 & \text{if } a,b ∉ ℝ
\end{cases}
[/tex]
##(a+b)^2-2ab=-ab##
##(a+b)^2=ab##
Suppose ##a,b≠0##
##⇒(a+b)^2/(ab)=1##
Since ##(a+b)^2## and ##ab## have to be of the same sign, it is a contradiction with the case statement above and hence a,b cannot not be 0.

^complex numbers can have sign other than +1 and -1.
suppose a=r b
then
0=a^2+ab+b^2=r^2 b^2+r b^2+b^2=(1+r+r^2)b^2
1+r+r^2=0 or b^2=0
if r is real 1+r+r^2 cannot be zero
if r is complex it can be zero

@lurflurf aha you're right. I was thinking only about imaginary numbers when I wrote it out and forgot that the variables doesn't have to only be real or imaginary numbers! Thanks for the correction!

##a^2## is positive OR ZERO.
##b^2## is positive OR ZERO.
therefore ##ab## MUST be negative OR ZERO, in order for the sum to be 0.
I'd point out too that ##(a+b)^2## is positive (or zero), which implies by ##(a+b)^2-ab=0## that ##ab## MUST be positive (or zero) in order for the sum to be zero. This implies that if ##ab## is nonzero, it must be positive and negative at the same time, which is impossible.

I'm going through the same book again. You already pointed me in the right direction before, but think I got it now. However, I could be wrong.

[tex]\text{Prove if}\ a^{2}+ab+b^{2}={0} \
\text{, then}\ a=0 \ \text{and } b=0 \\
a^{2}+ab+b^{2}=0 \\
a^{2}+b^{2}=-ab[/tex]
a², b² are always positive, so their sum must be positive. It follows, if a and b have the same sign, the right hand side is negative,
[tex](-a)^{2}+(-b)^{2}=-(-a)(-b) \\ a^{2}+b^{2}=-ab[/tex] so the equation only holds when a=b=0.

If a and b have opposite signs, then a>b or b>a, and ab becomes positive,
[tex]a^{2}+(-b)^{2}=-a(-b) \\ a^{2}+b^{2}=|a||b|[/tex]
so the equation can only hold when a=b=0 because a² + b² (and either a² or b² seperately as well) will otherwise always be larger than |ab|.

Try a proof by contradiction. The original statement is this: If ##a^2 + ab + b^2 = 0##, then a = 0 and b = 0.
Suppose the hypothesis is true, but the conclusion is false, i.e., that If ##a^2 + ab + b^2 = 0##, then it's not true that a = 0 and b = 0.
The negation of the conclusion is equivalent to ##a \ne 0 \text{ or } b \ne 0##, by De Morgan's law.
The assumptions that ##a \ne 0## or ##b \ne 0## can be handled in four cases.
1. a < 0, b ≥ 0
##a^2 + ab + b^2 = 0 \Leftrightarrow a^2 + 2ab + b^2 = ab \Leftrightarrow (a + b)^2 = ab##
Show that this assumption leads to a contradiction.
2. a > 0, b ≥ 0
##a^2 + ab + b^2 = 0 \Leftrightarrow a^2 + b^2 = -ab##
Show that this assumption also leads to a contradiction
Cases 3 and 4 are similar, but in these cases, we make assumptions about b being negative or nonnegative.

By the way, in your work you seem to think that a and -a must necessarily be positive and negative, respectively. That's not true. If a = -3, then a < 0 while -a > 0.

You can use a proof by contrapositive. Your statement is of the form if ##P## then ##Q##. This is a conditional statement. The contrapositive is: if ## not~ Q## then ##not~ P##. Conditional statements are equivalent to their contrapositives.
##Q## is of the form (##A## and ##B##) where ##A## is ##a=0## and ##B## is ##b=0##. ##not~ Q## is ##not~(A## and ##B)## which is also (##not~ A## or ##not~ B##) according to De Morgan's laws.
So ##not~ Q## is ##a\neq0## or ##b\neq0##
NB ##not~ A## is ##a\neq0## since the negation of ##a## equals ##0## is ##a## is not equal to ##0##. You do the same for ##not~ B##, and ##not~ P##

Proof:
You can rewrite the statement as: if ##a\neq0## or ##b\neq0## then ##a^2 +ab +b^2 \neq0##
Now ##a\neq0## or ##b\neq0## means at least one of them is non-zero. So if you pick one to be ##0## and the other to be a non-zero number ##c## you would get:
##c^2 + 0c + 0^2##
##=c^2## which is a non-zero number.

I suppose it's a matter of taste, but I prefer to bound things over most 'contra' proofs.

Via ##GM \leq AM## or Cauchy's Inequality or playing around with ##(a-b)^2 \geq 0##, while dealing in ##\mathbb R##, we know real numbers squared are always non-negative. We put this together and get the below bound

subtract ##\frac{1}{2}\big(a^2 + b^2\big)## from each side and get

##\frac{1}{2}\big(a^2 + b^2\big) \leq 0##

and if the sum of two real non-negative numbers is zero, then both must be zero. (A fancier way of saying this would refer to positive definiteness but it's not really needed.)

A year ago I thought I'd never become good at skateboarding because I kept failing at one simple trick. After lots of practice, something clicked this month, and things start to feel natural. I really hope this happens to proofs [math] as well, because I'm sure as hell feeling quite pathetic at the moment.

I've posted this on Reddit and apparently my proof is sloppy, but not wrong:

"I'm explaining it very broadly because it's been too long that I've been struggling with this. This problem can be found in the book Algebra - I.M. Gelfand, A. Shen. Problem 124. I'm currently self-studying math, and never had a formal training or experience with proofs. Also, we can use determinants, or the factorization of a³-b³ to prove the problem, but those I understand. I do not understand why the one below is wrong. Thanks in advance, any help is appreciated and probably much needed.

Prove that if a²+ab+b² = 0, then a = 0 and b = 0.

a²+ab+b² = 0
a²+b² = -ab

Let's assume a and b are not 0. On the left hand side a² and b² are always positive, so their sum is also always positive. On the right hand side there are four possible situations:

1. a and b are both positive (they have the same sign)

2. a and b are both negative (they have the same sign)

3. a is negative, b is positive (opposite signs)

4. a is positive, b is negative (opposite signs)

Situation 1 and 2: if a and b have the same sign, the right hand side is negative:
1. a and b are positive: a²+b² = -ab
2. a and b are negative: (-a)²+(-b)² = -(-a)(-b)
-> a²+b² = -ab

Since the left hand side must be positive, the right hand side must be positive as well for the equation to hold. However, the right hand side must be negative in situation 1 and 2 so the equation does not hold. It only holds when a = b = 0.
Because if only a = 0, then 0²+b² = -0*b -> b² = 0 -> b = 0
and if only b = 0, then a²+0² = -a*0 -> a² = 0 -> a = 0

Situation 3 and 4: if a and b have the opposite sign, the right hand side is positive:
3. a is negative, b is positive: (-a)²+b² = -(-a)b
-> a²+b² = ab
4. a is positive, b is negative: (a)²+(-b)² = -a(-b)
-> a²+b² = ab

If the absolute values of a and b are not equal, then a>b or b>a, therefore the absolute value of the right hand side must be lower than the left hand side because respectively a²>ab and b²>ab.
If the absolute values of a and b are equal, then a=b, and a²+b²=ab becomes 2a² = a², in which evidently 2a²>a² when a is not 0.

So, in all four situations we have shown that the equation only holds when a = b = 0."

The result is false for complex numbers. Let ##\omega \neq 1## be a (complex) cube root of unity; that is, ##\omega^3 = 1##. If ##b = \omega a## we have $$a^2 + a b + b^2 = a^2 ( 1 + \omega + \omega^2) = 0$$
for any ##a##, because ##1 + \omega + \omega^2 = 0##.

You can verify all this explicitly by taking
$$ \omega = -\frac{1}{2} + i \frac{\sqrt{3}}{2} .$$

It's easy to simplify this to ##(a+b)=\sqrt{ab}##. But extending AM-GM to include negative numbers (which is fine since this setup won't invalidate the underlying stipulations) tells us that ##(a+b) \geq 2\sqrt{ab}##. Substitution gives ##\sqrt{ab} \geq 2\sqrt{ab}##. The only time this is true is the equality case, i.e. ##\sqrt{ab}=0##. Thus at least one of ##a## and ##b## is zero. Solving for the other gives ##a^2+a(0)+(0)^2=0## so the other is zero as well.

Notwithstanding what someone on Reddit has to say, this is not a good proof. For one thing, it looks like you are concluding that ##a^2 + b^2 = -ab##. You should not conclude this -- you are assuming it is true. Since you are assuming that ##a^2 + ab + b^2 = 0##, it follows trivially that ##a^2 + b^2 = -ab##, regardless of whether a and b are positive, negative, of mixed signs, whatever. The two equations are equivalent. You don't need any machinery to get from one to the other, apart from adding equal quantities to both sides of either equation.

If a and b are both positive or both negative, then the equation ##a^2 + b^2 = -ab## can't possibly be true, and is therefore a contradiction. The left side is positive and the right side is negative. I mentioned this in my earlier post.

The worst part of your proof is that you seem to think putting a minus sign in front of a variable makes it negative. That is NOT true in general. For example, if x = -2, then -x is positive.