Frustration occurs when a system is unable to simultaneously achieve a minimum energy for each entity involved. For example, if you take the antiferromagnetic Heisenberg model on a triangular lattice, the ground state cannot be the Néel state, since there will always be one unhappy bond. The system has to come to a compromise, and the ground state is some canted antiferromagnetic arrangement.
It occurs for other things than non-bipartite lattices as well. If you take the square lattice antiferromagnetic Heisenberg lattice and include exchanges other than nearest neighbour, you can frustrate the system.
So, frustration can either occur for geometric reasons, as in the first example, or due to competition between interactions, as in the second.
In your example, suppose we have a nice Néel ordered antiferromagnet in the parent compound. When you start doping, you will remove one of the spins, and thus break two bonds. The hole cannot move around without breaking more bonds, or putting two of the same spin next to each other. This leads to something that is named in the literature 'stripes'.
The Coulomb interaction prevents the material from phase separating into a bunch of electrons in one part of the material, and a bunch of holes on the other. So, the material splits into a pattern of hole rich and hole poor regions. This is 'Coulomb frustrated phase separation' in the literature.
