This is going to be a slightly technical post; apologies if this isn't the sort of thing you're after.
You've struck a point which actually requires a little mathematical sophistication to deal with. It's not very easy to define a singularity.
A first attempt when you see the singularity in the black hole solution, for example, is that you might be tempted to say a singularity is a point of infinite curvature or similar. But from a strict mathematical point of view, such a point cannot be part of the manifold as it doesn't have an honest metric defined there. So you leave a 'hole' in the manifold.
We can't just call any hole like that a singularity though, since if we think about the 2D plane in polar coordinates, for example, we can't include the origin because the coordinates are degenerate there. But we know perfectly well that that's just a problem with our coordinates, and we can change to cartesians, say, and extend the manifold to include the origin with no problems.
Another attempt might be to look at what happens near our 'hole': at the centre of a black hole, the curvature tends to infinity as r tends to 0. So we could try to say that a singularity is defined by nearby 'bad behaviour'. But this is no good; think of a cone. At the vertex it isn't smooth so it can't be part of the manifold, but it's flat everywhere else. So this is no good either.
The other thing we want to exclude from the definition is the edge of spacetime itself, at infinite distance for example (loosely speaking).
The proper way to do things is in fact to use the slightly technical property of geodesic completeness. Roughly speaking, geodesic completeness means you can extend a straight line to an arbitrary length. A singularity exists if there is no way to extend the spacetime manifold to a geodesically complete one. (In fact this doesn't quite work, a little more is needed. But this captures the essence of the idea).
Here's a much more useful glib summary: A singularity means that someone could conceivably fall off the edge of or into a hole in spacetime, and there's no way to extend beyond the edge or patch over the hole to save them.