I would say a "gauge connection" is simply a connection on a principal fiber bundle, i.e. a Lie algebra-valued 1-form ##\omega## that maps fundamental vector fields to their generators, and transforms as ##R^\star_g\omega = ad(g^{-1})\omega## under the right-actions of the structure group (the "gauge group" in physicist's terminology). (But I'm not sure I heard the term "gauge connection" before, so maybe something else is meant?) A gauge potential is what you get if you pull ##\omega## back to the base space via a local section. In other words a gauge potential is defined on spacetime with respect to a local "choice of gauge". So, both are closely related, but different concepts.
Unfortunately it is difficult to explain all this without introducing a lot of jargon first. A book I like, that explains all this in quite simple terms is David Bleecker, "Gauge Theory and Variational Principles".
I think it's the same thing. It's just discussed in different language by theoretical and mathematical physicists. For a physicist the gauge potential is introduced to extend a partial derivative to a derivative covariant under gauge transformations, and this leads him to introduce a gauge potential in the derivative,
$$\partial_{\mu} \rightarrow \partial_{\mu} + \mathrm{i} g \mathcal{A}_{\mu},$$
where ##\mathcal{A}_{\mu}## is a Lie-algebra valued vector field.
From the mathematical point of view this introduces an affine connection on the fiber bundle with the ##\mathcal{A}_{\mu}## the connection coefficients (like the Christoffel symbols in affine differentiable manifolds in differential geometry).
I believe what mathematicians call a "connection on a principal bundle" is the object ##\omega## I defined above, which lives on the total space of the bundle. Physicists usually only discuss the pull-back ##\sigma^\star\omega ## to the base space (##\sigma## a local section), which in your notation would be ##\sigma^\star\omega = \mathcal{A}_\mu \mathrm{d}x^\mu##, and which posesses the characteristic transformation law of a gauge potential