There are many components of physical fitness, including metabolic, morphological, bone integrity/density, body composition, cardiovascular fitness, muscular endurance/strength, agility, balance, etc. Read more on those,
http://en.wikipedia.org/wiki/Physical_fitness#Components_of_physical_fitness".
Some of these, such as muscle strength and endurence, can decline quite rapidly with lack of use. But they're easily rebuilt, as well, in just a few weeks or months time, provided one doesn't try to push it too hard.
Others, such as agility/balance/coordination can be left unused for years, decades even, and regained in a very short period of time (days, if not even hours). Ever heard the expression, "just like riding a bike?" Case in point: I used to roller skate a lot as a kid, but hadn't done it since I was 13. Last fall, for the first time in 33 years, I tied on a pair of roller skates and hit the rink with friends and our kids. For the first five minutes I was waving my arms all over the place, as if I'd never skated in my life! By the end of the three hours, however, while I wasn't back to my former level of youthful expertise, I was skating backwards, and doing various turns and manuevers with relative ease, about 70% of what I used to be able to do.
Many people believe physical fitness involves just cardio and strength training, but that's not at all it by a long shot. If you really want to be all-around fit, you need to engage in a variety of indoor and outdoor activities, not to the point where you're in the top 10% in any given
http://en.wikipedia.org/wiki/Physical_fitness#Components_of_physical_fitness", but so that you're exercising all components.
The three most important aspects of physical fitness are
consistancy (3-4 times a week),
variety (exercising all the components), and
moderation (to prevent injury). The last is important enough I should have put it first, as injury is the number one reason people cite for not engaging in a physical activity!
Even if you're really out of shape, getting back into shape is easy, provided you don't push it. Exercising lightly for half an hour a day using a simple on-off-on-off-off cycle of days, doing strength training on the first "on" and cardio on the second "on," while varying your activities for achieving each, will pull you out of the couch potato slump in a week or two, provided you stick with it. After a couple of weeks of consistent, well-rounded activity, you can turn it up a notch to the moderate level.
Unless you're in competition, no one needs to go beyond the moderate level. When doing cardio, the best gauge is your heart rate. 220-age will give you your max. You should exercise between 70% and 90% of your max, with a target heart rate of 80% of your max. It's actually good to vary your heart rate within this range, but never exceed 90% unless you're under the direct and immediate supervision of a cardiologist! I personally don't exceed 85% of my max heart rate, yet I have the cardio capacity to climb 14ers and the physical strength to haul 100lbs of gear around all day (though my knees can't take it more than a few dozen steps).