About the only thing it's too late for at 22 is a Happy Meal.
From a practical point of view you'll have about a decade worth of schooling ahead of you. You'll be a few years behind kids who started out at 18 or so, but that itself is not going to have any influence on things like graduate school acceptance, scholarships, post-docs, etc. There are far bigger factors at play than your age.
What tends to be the rate limiting factor for most people is life itself. By the time you're finishing your PhD you'll likely be your early 30s. That's a time when a lot of people get married, start families, etc. and those things can complicate an academic lifestyle. But a lot of that comes down to personal choice and your ability to manage your circumstances.