But do you get the same strong existence results as one gets for infinite-dimensional Lie groups, say?
The statement about the full embedding that I quoted means precisely that all standard theory embeds.
What I meant is: To understand the concept of a smooth set you apparently need the whole category.
Not in any non-trivial sense, no. A smooth sets is, as I mentioned, a declaration for each Cartesian space (abstract coordinate system) of what counts as a smooth function out of that space into the smooth set, subject to the evident condition that this is compatible with smooth functions between Cartesian space (abstract coordinate changes).
Or can you define a smooth set without reference to a category, just as you can define a group without reference to a category?
Sure. But this is a triviality unless you read some superficial scariness into the innocent word "category". When you define what a group is, you also want to know what counts as a homomorphism between two groups. The high-brow term for this is "the category of groups" but it means nothing else but "groups and maps between them".