But there's also the centrifugal force, so a plumb bob does not point directly at Earth's center and an object wouldn't fall to the center, just because of the spin of the Earth, unless the hole was cut through the rotation axis. The role of the orbit around the Sun is even trickier-- there the centrifugal effect is of order the Sun's tidal effect, so I think that would be small (the scale of tides is a meter or less, on average), but the coriolis effect could be large.
I said above that the coriolis deflection scales like 1/v, where v is the falling speed in the rotating frame, which comes from the idea that the acceleration is proportional to v, and the deflection scales like the acceleration times the time squared, while the time scales like 1/v. For some inexplicable reason, I concluded that the coriolis effect wouldn't matter if air resistance slowed the fall down, because then the air would carry the object along with it, but if air resistance makes v small, then 1/v gets large. So air resistance could make the coriolis deflection even worse, it depends on some additional details.
In truth it had been bothering me to discount the coriolis effect if the falling was slow, because usually the coriolis effect is most prominent for motions that occur on a timescale that is comparable to the rotation period of the frame. Motions that happen much more quickly (like draining a bathtub) cannot be affected by the coriolis effect, and motions that happen much more slowly (like the settling of metals to the Earth core) must average out the coriolis influences. So I think we should be expect maximum coriolis deflection for processes that require about a year for the man to fall, and the deflection would be of order the Earth radus. If we have no air resistance, the free fall time is of order the radius over the escape speed, so that's roughly 500 seconds, way shorter than a year. Since the deflection scales like acceleration times the time squared, and the coriolis acceleration scales like the inverse of the falling time, the deflection scales like the falling time. The ratio of the falling time, with no air resistance, to a year is about 10-5, so to order of magnitude, the coriolis deflection would be about 10-5 times the Earth radius, or say 10-100 meters.
However, if the air resistance was cranked up until the fall time was about a year, the deflection would be much more. But, if the air resistance was so great that the fall time was way more than a year, the effects of the Earth's orbit would cancel out, and we'd be back to a small effect. If the fall speed was the terminal speed in air now, say 100 m/s, then the fall time is on order a million seconds, which is like a few weeks or so, so that would produce a pretty large coriolis deflection indeed, maybe several to ten percent of the Earth radius, though a full calculation is needed to be more accurate. If the air gets even denser, it would eventually become too thick to allow any coriolis deflection, and the effect magnitude would drop again.
So, as usual, when you dig deeper into a seemingly simple problem, and start layering on all the complexities that a real-world calculation would need, you get surprised! Of course, the problem is never going to be a "real world" problem in the first place, but you were certainly right to bring up the possibility of not falling straight through the hole, even if it is drilled through the rotation axis.