I may be contributing to your confusion by being a bit unclear and sloppy myself.
Every group has an adjoint representation as a function mapping the group to itself. That, even for Lie groups,
has no direct "linear structure". It is a point mapping. For Lie groups it has differential geometric structure. I think that is what is usually denoted by \operatorname{Ad}.
Now there is the adjoint representation of a Lie algebra: \operatorname{ad}_\mathfrak{g}. When physicists says it is "linear" they mean several equivalent things.
Firstly every Lie algebra can be embedded in its universal covering algebra which is an associative algebra within which the Lie product is a commutator of the associative product. The adjoint representation is linear in the sense that it extends to a representation of the universal covering algebra within the operator algebra over some linear space (= vector space). Most every time a physicist says "linear" the imply an embedding into a vector space or if multiplication is occurring embedding into the operator algebra on some space. In this adjoint representation it is the operator algebra over the Lie algebra as a linear space.
(I like to say linear space rather than vector space since in many applications there is an implied metric structure on the vector spaces and "linear space" implies less.)
In the operator algebra \operatorname{Op}(\mathbf{V})= \mathbf{V}\otimes\mathbf{V}^* for some vector space \mathbf{V} is the set of invertible operators which defines the Lie group \operatorname{GL}(\mathbf{V}). There is also in the same operator algebra, via the commutator product, the corresponding general linear Lie algebra. Each is n^2 dimensional (n = \mathop{dim}(\mathbf{V}).
Thus for the Lie algebra \mathfrak{g} = (\mathbf{V},\triangle) ( = vector space with a Lie product):
\operatorname{ad}: \mathfrak{g} \to \mathfrak{gl}(\mathbf{V}) = (\operatorname{Op}(\mathbf{V}) , [\cdot,\cdot] )
(This is the 2ndly part, mapping into the general linear group.)
The "thirdly" part refers to the induced group representation which I will here denote as \operatorname{ad}_G. It is the extension of this representation of the Lie algebra (within the associative operator algebra) via the exponential mapping. Note that all of these representations are acting on the vector space \mathbf{V}. Group products are the operator product, Lie products are the commutator product.
Here is the commutative diagram:
This picture is valid for any linear representation if you relabel the rep mappings and generalize \mathbf{V}. [See Footnote]
Now on top of all this, we can treat the operator algebra as a vector space and thus as a manifold. The general linear group is a sub-manifold and the group adjoint mapping is a continuous point map. The first adjoint mapping I mentioned in this post, where the group acts on itself as a point mapping is here extended to the action of the group on the whole algebra as a linear space via the
adjoint action:
g: X\mapsto \operatorname{ad}_G(g) X \operatorname{ad}_G(g^{-1})
for X some element in the operator algebra.
When restricting the this adjoint action to act only on representatives of the group it expresses the adjoint rep \operatorname{Ad}_G embedded in a larger linear representation \operatorname{GL}(\mathbf{V}\otimes\mathbf{V}^*). In most physics applications the speaker/author will specify this representation by using the term
adjoint action. The point here is that, though we can construct a linear "re-representation" of the [imath]\operatorname{Ad}[/itex]joint representation it is not, in-and-of-itself, a linear representation.A final comment about the "Ad" adjoint representation of the Lie group. Since the adjoint action of anyone element on the whole of the Lie group is a Lie group automorphism, this big A adjoint representation is a homomorphic mapping from the group into its automorphism group \operatorname{Ad}: G \to \mathbf{\mathop{Aut}}(G). These actions are thus called
inner automorphisms and that's the best way to think about this big A adjoint rep. There may also be additional (outer) automorphisms (though not for simple groups) and that's a fun topic for another thread.
I hope my exposition has been clear enough to help resolve your issues. Let me know if/where I've fallen short on that.
[Footnote] There is a slight caveat when generalizing though as the induced adjoint map on the group may be a projective representation having multiple representatives of the identity elements. The obvious example being spin representations of the orthogonal groups which are not technically speaking not quite representations of the Lie groups.