If you understand matrix products, I can work with that.
First of all, it's much easier to work with QM like that using Dirac notation.
<a| - This is a row matrix 1xN, also known as a bra.
|a> - This is a column matrix Nx1, also known as a ket.
Bra-ket, get it? Anyways. The dimension N corresponds to total number of states you are working with. For the most basic pair of entangled particles, you need at least 4 states, so we'll be dealing with 1xN, Nx1, and NxN matrices.
Naturally <a|b> is just a sum of corresponding elements of the two matrices. That means <a|b> behaves like a dot product of vectors a and b. It also means that <a|a> is the square of the norm. Since our state vectors will be normalized, you can assume that <a|a> = 1, unless otherwise specified.
Finally, |a><b| is the outer product. The result is an NxN matrix. So if you have vectors <a| = (1, 0, 0, 0), <b| = (0, 1, 0, 0), <c| = (0, 0, 1, 0), and <d| = (0, 0, 0, 1), then |a><a| + |b><b| + |c><c| + |d><d| = I, the identity matrix. That tells you that vectors a, b, c, and d form a complete orthonormal basis.
Now, I'm going to keep using marble analogy for the rest of this, but all the math is exactly the same with whatever four states you end up having. The four states of the system in the conventional basis are going to be |rr>, |rb>, |br>, and |bb>. The labels on each one denote the colors of the first and second marbles in that order. The states themselves are orthonormal, so that <rr|rr> = 1, but <rr|rb> = 0. In other words, dot product with itself is 1, dot product with any of the other three is zero. Then the general state can be written as some |ψ> = x|rr> + y|rb> + z|br> + w|bb>. Requirement that <ψ|ψ> = 1 gives us x² + y² + z² + w² = 1. Otherwise, these can be any number. (In general, a complex number, but we don't need that for this example.)
To find probability that both marbles are red, we must project the state onto the basis. This is trivial here, because the basis corresponds to our state, but hopefully it's still clear.
P(rr) = <ψ| (|rr><rr|) |ψ> = <ψ|rr><rr|ψ> = <rr|ψ>² = x²
The |rr><rr| is the NxN projection matrix. It is placed between row and column versions of our state vector, and gives us the probability of finding system in that state, which is x². Probabilities for rb, br, and bb are y², z², and w² respectively. Since all these add up to 1, total probability is conserved, which is good.
Now, suppose I want to know what the probability is of finding just the first marble being red? The corresponding projection matrix is M = |rr><rr| + |rb><rb|. Probability is computed as before.
P(r_) = <ψ|M|ψ> = <rr|ψ>² + <rb|ψ>² = x² + y²
Again, this makes perfect sense. But the more interesting question is, what is the state of the system after the measurement if we indeed found the first marble to be red? The system must collapse to a state that corresponds to the result. In other words, the state of the system after measurement |ψ'> = M|ψ>/(<ψ|M|ψ>)1/2. The division ensures normalization. If applied to our state ψ, we get the following.
|ψ'> = M|ψ>/(<ψ|M|ψ>)1/2 = (|rr><rr|ψ> + |rb><rb|ψ>)/(<rr|ψ>² + <rb|ψ>²)1/2 = x/(x²+y²)1/2 |rr> + y/(x²+y²)1/2 |rb>
In other words, we get a superposition of the two states corresponding to the measurement result weighted in same proportion as originally, but re-normalized so that total probability is one. Naturally, if measurement of the first marble came up blue, instead, the resulting state vector would change accordingly.
|ψ'> = z/(z²+w²)1/2 |br> + w/(z²+w²)1/2 |bb>
So that's just measurement in QM. Let's look at entanglement. Specifically, suppose that the system was prepared in the following state.
|ψ> = \frac{1}{\sqrt{2}}(|rb> + |br>)
Note that <ψ|ψ> = 1 as required. Furthermore, probabilities of state rb and br are 0.5 each, and probabilities of rr and bb are 0.0 each. For simplicity, you can also just look up all of the previous results with x=w=0 and y=z=\small 1/\sqrt{2}
What is probability of measuring the first marble to be red? As before, it works out to be x²+y² = 0.5. Same for probability of first marble being blue, z²+w² = 0.5.
Suppose, first marble was, indeed, found to be red. What is the new state?
|ψ'> = x/(x²+y²)1/2 |rr> + y/(x²+y²)1/2 |rb> = |rb>
This is where things get interesting. What is now the probability of finding second marble to be blue?
<ψ'|(|rb><rb| + |bb><bb|)|ψ'> = <rb|rb><rb|rb> + <rb|bb><bb|rb> = 1 + 0 = 1.
So P(_b) is now 1, and P(_r) is now 0. What if the first marble was measured to be blue? Then the new state |ψ'> = |br> and P(_b) = 0 and P(_r) = 1.
You can go through this and check what happens if you made measurements on the second marble first. You'll find exactly the same thing. It will have probability or red or blue coming up at 1/2 and 1/2, but following that, the other marble will have 100% of being the opposite.
This is the mathematics of entanglement. You can replace labels of red and blue marble with "spin up" and "spin down", or any other 2-valued quantum measurement, and you'll have it work out exactly the same way.