For ##y \ge 0##, we can rewrite the fractional part as ##\{y\} = y - \lfloor y \rfloor##. Thus
$$
\begin{align*}
\int_0^1 x \left\{ \frac{1}{x} \right\} dx &= \int_0^1 x \left( \frac{1}{x} - \left\lfloor \frac{1}{x} \right\rfloor \right) dx \\
&= \int_0^1 dx - \int_0^1 x \left\lfloor \frac{1}{x} \right\rfloor dx \\
&= 1 - \int_0^1 x \left\lfloor \frac{1}{x} \right\rfloor dx
\end{align*}
$$
Setting ##y \equiv 1/x##, ##dy = -1/x^2 dx##, we have
$$
\begin{align*}
\int_0^1 x \left\lfloor \frac{1}{x} \right\rfloor dx &= \int_\infty^1 \frac{1}{y} \left\lfloor y \right\rfloor \left(-\frac{1}{y^2}\right) dy\\
&= \int_1^\infty \frac{1}{y^3} \left\lfloor y \right\rfloor dy
\end{align*}
$$
Since ##\left\lfloor y \right\rfloor## is piecewise constant, the integral can be written as the sum of integrals
$$
\begin{align*}
\int_1^\infty \frac{1}{y^3} \left\lfloor y \right\rfloor dy &= \int_1^2 \frac{1}{y^3} 1 dy +
\int_2^3 \frac{1}{y^3} 2 dy + \int_3^4 \frac{1}{y^3} 3 dy + \cdots \\
&= \sum_{n=1}^{\infty} \int_n^{n+1} \frac{1}{y^3} n dy \\
&= \sum_{n=1}^{\infty} n \int_n^{n+1} \frac{1}{y^3} dy \\
&= \sum_{n=1}^{\infty} n \left[ - \frac{1}{2y^2} \right]_n^{n+1} \\
&= \sum_{n=1}^{\infty} n \left[ \frac{1}{2n^2} - \frac{1}{2(n+1)^2}\right]
\end{align*}
$$
Here I have to admit that I cheated because I couldn't find a better way to write the sum, I thus couldn't find a simple expression for it. Mathematica told me it is ##\pi^2/12##, which is nice, but I still don't see how to rewrite the sum. Anyway, using this result, we finally find
$$
\begin{align*}
\int_0^1 x \left\{ \frac{1}{x} \right\} dx &= 1 - \frac{\pi^2}{12}
\end{align*}
$$