I'd say that while you don't need linear algebra for ODEs, I'd still take it first. That way, when you do take ODEs, you'll be ahead of the class.
In terms of difficulty, linear algebra is pretty easy. DEs vary from person to person. I found them incredibly easy, but some people think they are...