It took me maybe 2 months to understand the bigger picture, I know exactly what you mean. I realized that linear algebra is basically just the abstract study of vectors. Vectors can be represented as matrices, which have many different properties, etc.. The most important part (for physics) that I learned was actually part of my calculus II sequence, which included plane geometry.
I don't know if you've already gotten into vector analysis, but learning this subject from a linear algebra perspective is very rewarding. Learning the geometry and mathematical manipulation of objects such as lines and planes is crucial for physics. Applications and worth make themselves relevant here.Linear algebra, however, is still taught as pure mathematics subject. The pure math comes out in studying operations, particularly linear operations, on mathematical objects such as vectors and matrices. Since the fulfillment of all these operations creates a vector space, we can define an abstract set of functions or vectors on this "space". For example, we know that every vector in R3 can be represented as a linear combination of the 3 perpendicular unit vectors. This leads to the study of the cross product and how perpendicular vectors can be made or represented in the first place. Understanding concepts such as these, from a mathematically rigorous linear algebra perspective, helps give creativeness to the concept and can also be useful in physics (since we also use coordinate systems). It's very abstract, but it's pure math!