Well, it was a fruitful discussion... but I guess you people deviated. A different approach could have yielded the result in less time.
Let the line joining origin, O, and the given arbitrary location of the particle, P, be denoted by OP. Draw a line, AB, perpendicular to OP in the xy-plane. This line AB divides the xy-plane in two parts, one containing O and the other not containing O.
Now, distance of the particle from the origin, O, would decrease if velocity vector of the particle lies in the part containing origin. (Satisfy yourself with the help of a diagram.) Thereby, we can say that angle between velocity vector and position vector (call it 'theta') must be greater than 90 degrees.
Thus, Cos(theta) < 0 (zero)
=> |velocity vector|.|position vector|.Cos(theta) < 0
=> dot product of velocity vector and position vector < 0
=> X.Vx + Y.Vy < 0.
Hence, the (C) option.