We have:
\begin{align*}
\int_0^{\infty}\frac{e^{-t}\cos(t\sqrt{3})}{\sqrt{t}} \, dt&=\text{Re}\int_0^{\infty}\frac{e^{-t}e^{it\sqrt{3}}}{\sqrt{t}} \, dt \\
&=\text{Re}\int_0^{\infty}\frac{e^{t(-1+i\sqrt{3})}}{\sqrt{t}} \, dt \qquad \left[u=\sqrt{t},\quad 2 \, du=\frac{dt}{\sqrt{t}}\right] \\
&=2\,\text{Re}\int_0^{\infty}e^{u^2(-1+i\sqrt{3})} \, du.
\end{align*}
Let $\displaystyle I=\int_0^{\infty}e^{u^2(-1+i\sqrt{3})} \, du$. Then
\begin{align*}
I^2&=\int_0^{\infty}e^{u^2(-1+i\sqrt{3})} \, du \cdot \int_0^{\infty}e^{v^2(-1+i\sqrt{3})} \, dv \\
&=\int_0^{\infty} \int_0^{\infty}e^{(-1+i\sqrt{3})(u^2+v^2)} \, du \, dv \\
&=\int_0^{\pi/2}\int_0^{\infty}e^{(-1+i\sqrt{3})r^2} \, r \, dr \, d\theta \qquad (\text{switching to polar})\\
&=\frac{\pi}{2}\int_0^{\infty}e^{(-1+i\sqrt{3})r^2} \, r \, dr \qquad \left[ s=r^2, \quad \frac{ds}{2}=r \, dr \right] \\
&=\frac{\pi}{4}\int_0^{\infty}e^{(-1+i\sqrt{3})s} \, ds \\
&=\frac{\pi}{4}\left(\frac{e^{(-1+i\sqrt{3})s}}{-1+i\sqrt{3}}\right)\Bigg|_{0}^{\infty \, \to \, 0} \\
&=\frac{\pi}{4} \, \frac{1+i\sqrt{3}}{4} \\
I^2&=\frac{\pi(1+i\sqrt{3})}{16} \\
I&=\pm\frac{\sqrt{\pi}}{2\sqrt{2}} \, e^{i\pi/6}.
\end{align*}
Now then,
\begin{align*}
\int_0^{\infty}\frac{e^{-t}\cos(t\sqrt{3})}{\sqrt{t}} \, dt&=2\,\text{Re}\int_0^{\infty}e^{u^2(-1+i\sqrt{3})} \, du \\
&=2 \, \text{Re} \left[ \pm\frac{\sqrt{\pi}}{2\sqrt{2}} \, e^{i\pi/6} \right] \\
&=\pm\frac{\sqrt{\pi}}{\sqrt{2}} \cdot \frac{\sqrt{3}}{2} \\
&=\pm\frac{\sqrt{6\pi}}{4}.
\end{align*}
Now here is the justification for ruling out the minus sign. The function $1/(e^t \sqrt{t})$ is monotone decreasing on the interval of integration. $\cos(t\sqrt{3})$ is oscillating. Let us examine the first period of this function carefully - this would be the interval $\displaystyle (0,2\pi/\sqrt{3})$. On $(0,\pi/(2\sqrt{3}))$, the cosine function is positive, and on $(\pi/(2\sqrt{3}),\pi/\sqrt{3})$, the cosine is negative. Because the multiplying function is decreasing, the positive portion will outweigh the negative. Next, we compare the interval $(\pi/\sqrt{3},3\pi/(2\sqrt{3}))$ to $(3\pi/(2\sqrt{3}),2\pi/\sqrt{3})$. The cosine function will be negative for the former, and positive for the latter. Hence, the integral, considered on these two intervals, will be negative overall. So, on the first half-period, the integral is positive, and on the second half-period, it is negative. Because the multiplying function is decreasing, however, the positive portion will dominate the negative, making the entire integral positive. We can repeat this analysis for any period, hence we can rule out the negative sign.
