I am a bit confused. Your title is "Differentiation- Why open set?" but you don't mention differentiation in the body.
If you are asking about mappings in general, no U does not have to be open. But if you are asking about points at which you can define the derivative of f, that's a different story.
Think about what happens in the "Calculus I", R1 case. The derivative of f(x), at x= a, is defined by
\lim_{h\rightarrow 0} \frac{f(a+h)- f(a)}{h}
In order for that to exist, f(a+ h) has to be defined for all h close to 0- positive or negative. And that means that a has to be in the interior of some interval on which f is defined. If a is on the boundary, we can only define the "one sided derivative", using a one sided limit.
Now, in the more general, n dimensional, case, we have to be able to define f(x) for all x close to a. In other words, a must be an interior point of the domain of f. The set of all interior points of a set is, of course, an open set.
