I like Serena said:
Well, you had just found this limit:
Let's invert the fraction.
Then we have:
$$\lim_{n \to \infty} \frac{4n^5 \log^2 n}{n^6} = 0$$
This means that:
$$\forall\, \varepsilon > 0 \ \exists\, n_0 \text{ such that }\forall\, n>n_0: \left| \frac{4n^5 \log^2 n}{n^6} \right| < \varepsilon$$
If we pick $\varepsilon=\frac{4}{10}$, we get that there is an $n_0$ such that for each $n > n_0$:
$$4n^5 \log^2 n < \frac{4}{10}n^6$$
So we don't have a specific value for this $n_0$, but we do know that it exists.
That is enough for what you want to proof. (Mmm)How could you apply this reasoning to the whole expression? (Wondering)
So, could we do it like that? (Thinking)
$$\lim_{n \to \infty} \frac{4n^5 \log^2 n}{n^6} = 0$$
This means that:
$$\forall\, \varepsilon > 0 \ \exists\, n_0 \text{ such that }\forall\, n>n_0: \left| \frac{4n^5 \log^2 n}{n^6} \right| < \varepsilon$$
If we pick $\varepsilon=\frac{4}{10}$, we get that there is an $n_0$ such that for each $n > n_0$:
$$4n^5 \log^2 n < \frac{4}{10}n^6$$
So, we have:
$$g(n)=n^6-4n^5 \log^2 n -10-5n^3 \geq n^6-\frac{4}{10}n^6-\frac{10}{64}n^6-\frac{5}{27}n^6=\left ( 1-\frac{4}{10}-\frac{10}{64}-\frac{5}{27} \right ) n^6=\frac{1117}{4320}n^6, \forall n \geq \max \{ \lfloor n_0 \rfloor+1,3 \}$$
So, we pick $n_1= \max \{ \lfloor n_0 \rfloor+1,3 \}, c_1=\frac{1117}{4320}$.
$$g(n)=n^6-4n^5 \log^2 n-10-5n^3 \leq n^6, \forall n \geq 1$$
So, we pick $n_2=1, c_2=1$
Now, we pick $n_0=\max \{ \max \{ \lfloor n_0 \rfloor+1,3 \} \}, c_1=\frac{1117}{4320}$ and $c_2=1$ and we have that:
$$g(n)=\Theta(n^6)$$
Or is it better like that? (Wasntme)
$$\lim_{n \to \infty} \frac{4n^5 \log^2 n}{n^6} = 0$$
This means that:
$$\forall\, \varepsilon > 0 \ \exists\, n_1 \text{ such that }\forall\, n>n_1: \left| \frac{4n^5 \log^2 n}{n^6} \right| < \varepsilon$$
If we pick $\varepsilon=\frac{4}{10}$, we get that there is an $n_1$ such that for each $n > n_1$:
$$4n^5 \log^2 n < \frac{4}{10}n^6$$$$\lim_{n \to +\infty} \frac{10}{n^6}=0$$
This means that:
$$\forall\, \varepsilon > 0 \ \exists\, n_0 \text{ such that }\forall\, n>n_2: \left| \frac{10}{n^6} \right| < \varepsilon$$
If we pick $\varepsilon=\frac{10}{64}$, we get that there is an $n_2$ such that for each $n > n_2$:
$$10 < \frac{10}{64}n^6$$
$$\lim_{n \to +\infty} \frac{5n^3}{n^6}=0$$
This means that:
$$\forall\, \varepsilon > 0 \ \exists\, n_3 \text{ such that }\forall\, n>n_3: \left| \frac{5n^3}{n^6} \right| < \varepsilon$$
If we pick $\varepsilon=\frac{5}{27}$, we get that there is an $n_3$ such that for each $n > n_3$:
$$5n^3 < \frac{5}{27}n^6$$
So, we have:
$$g(n)=n^6-4n^5 \log^2 n -10-5n^3 \geq n^6-\frac{4}{10}n^6-\frac{10}{64}n^6-\frac{5}{27}n^6=\left ( 1-\frac{4}{10}-\frac{10}{64}-\frac{5}{27} \right ) n^6=\frac{1117}{4320}n^6, \forall n \geq \max \{ n_1,n_2,n_3\}$$
So, we pick $n_4=\max \{ n_1,n_2,n_3\}, c_1=\frac{1117}{4320}$.
$$g(n)=n^6-4n^5 \log^2 n-10-5n^3 \leq n^6, \forall n \geq 1$$
So, we pick $n_5=1, c_2=1$
Now, we pick $n_0=\max \{ 1, \max \{ n_1,n_2,n_3\} \}, c_1=\frac{1117}{4320}$ and $c_2=1$ and we have that:
$$g(n)=\Theta(n^6)$$
(Thinking)