Hi jacks,
You've no idea how much I like this problem! It is like one of the most delicious problems that I've seen this year! Thank you for bringing this problem up here so that we can have a stab at it and at the same time, enjoy it to the fullest!
One way of approaching this problem is to use the substitution technique. But the hardest part is to decide which part of the integrand should be replaced by the substitution.
We also know we can't simply let $u=\sqrt[4]{x^{10}+x^8+1}$ because it will give us $\dfrac{du}{dx}=\dfrac{x^7(5x^2+4)}{2(x^{10}+x^8+1)^{3/4}}$, and it gives us more a headache integrand, that complicated the problem even more.
So, we need some u substitution that after differentiation is done, we would get something like $\dfrac{du}{dx}=\dfrac{3x^{10}+2x^8-2}{?}$, and our next trial is to let
$u=\dfrac{\sqrt[4]{x^{10}+x^8+1}}{x}$, and notice that
1. after differentiation, we get $\dfrac{du}{dx}=\dfrac{3x^{10}+2x^8-2}{2x^2(x^{10}+x^8+1)^{\dfrac{3}{4}}}$, or $dx=\dfrac{2x^2((x^{10}+x^8+1)^{\dfrac{3}{4}})}{3x^{10}+2x^{8}-2}$
2. $(x^{10}+x^8+1)^{\dfrac{3}{4}}=u^3x^3$$\begin{align*}\therefore \int \frac{\sqrt[4]{x^{10}+x^8+1}}{x^6}\cdot \left(3x^{10}+2x^{8}-2\right)dx&=\int \frac{\sqrt[4]{x^{10}+x^8+1}}{x}\cdot \left(\dfrac{3x^{10}+2x^{8}-2}{x^5}\right)dx\\&=\int u\cdot \left(\dfrac{3x^{10}+2x^{8}-2}{x^5}\right) \dfrac{2x^2((x^{10}+x^8+1)^{\dfrac{3}{4}})}{3x^{10}+2x^{8}-2}du\\&=\int \dfrac{2u(u^3x^3)}{x^3} du\\&=\int 2u^4 du\\&=\dfrac{2u^5}{5}+C\\&=\dfrac{2\left(\dfrac{\sqrt[4]{x^{10}+x^8+1}}{x}\right)^5}{5}\\&=\dfrac{2(x^{10}+x^8+1)^{\dfrac{5}{4}}}{5x^5}\end{align*}$
Honestly speaking, the choice of the substitution of $u=\dfrac{\sqrt[4]{x^{10}+x^8+1}}{x}$ is my 5th or 6th trial.