Abstract algebra is an incredibly fun thing, and I wouldn't consider it difficult. There are way more difficult things out there than groups.
What abstract algebra tries to do, is to generalize the arithmetic rules to a broader concept. For example, you (should) know how to solve 2x+1=4. Well, the techniques that you use for this are not limited to R. Instead, abstract algebra generalizes these techniques to groups and rings. So in a way, a group (or a ring) is the most general concept in which you can solve equations. Then a course in abstract algebra will try to find links between several groups and they will prove very beautiful theorems about them.
Now, why is this all useful? Because, in the end, we can prove things in this new framework which we couldn't prove normally (or whose proof would be very difficult). One example is the unsolvability of the quintic. You've undoubtely seen how to solve quadratic polynomials. Well, the same is possible for cubic and quartic polynomials, but not for quintics.
You need almost nothing for abstract algebra (but of course: the more you've seen, the better you'll be at it. Not because of the background information, but more because of "mathematical maturity"). You'll only want to be able to handle "abstraction". I'll give an analogy: when you were 6, you learned how to add two numbers, and that was it. But after a few years, you added letters in your equations. You "abstracted" the concept. This is what abstract algebra does... a lot. So you will have to accept abstraction.
You'll also do a lot of proofs in algebra. If you don't like proofs, then abstract algebra is not for you. (or, generally, mathematics is not for you)
I wouldn't say that you need linear algebra for abstract algebra. I actually argue that it's the other way around. But colleges give linear algebra first because it's applications are very important to all kinds of fields.
In short: linear algebra studies systems of equations. It considers the most general place where you can solve equations: the vector space. It also deals with matrices (and it are these that you need in abstract algebra).
Now, a very, very, very good book in abstract algebra is "a book of abstract algebra" by Pinter. (
https://www.amazon.com/dp/0070501386/?tag=pfamazon01-20). They claim in the comments that even a ninth-grader can understand it. This is very true, the book is THAT easy. You'll only need to work a little for it (i.e. make all the exercises).
There will be some matrices in the book, but they're not fundamental. They can easily be skipped as they only arise in the exercises.