As has been pointed out, it is possible to make the metric g_{\mu \nu} be equal to the Minkowski metric \eta_{\mu \nu} at any given (nonsingular) point, and to make all of its first derivatives vanish there, by using Riemann normal coordinates. However, the tensor R \indices{^{\mu}_{\nu}_{\sigma}_{\rho}} (the Riemann curvature tensor) is an isometric invariant of the manifold, which is to say that if you keep the metric g_{\mu \nu}, then the curvature stays, too.
However, if we are only dealing with weak gravitational fields, then we may regard general relativity as the theory of a symmetric tensor h_{\mu \nu} propagating against a flat, Minkowskian background; this is called linearized gravity, and is used to study gravitational waves. Specifically, we can write g_{\mu \nu} = \eta_{\mu \nu} + h_{\mu \nu}, where the perturbation h_{\mu \nu} is assumed to contribute significantly to measurable quantities only to first order (this is the "weak-field" assumption). We then have g^{\mu \nu} = \eta^{\mu \nu} - h^{\mu \nu} (again, to first order), and we can raise and lower indices using \eta (in fact, h^{\mu \nu} is defined here as \eta^{\mu \sigma} \eta^{\nu \rho} h_{\sigma \rho}). We can then go on to derive the Riemann, Ricci, and Einstein tensors, and the Ricci scalar, to get the field equations. Alternatively, and in keeping with the viewpoint of a field theory on a flat background, we can define the Lagrangian
\displaystyle \mathcal{L} = \frac{1}{4} [2h \indices{^{\mu}^{\nu}_{,\mu}} h_{,\nu} - 2h \indices{^{\rho}^{\sigma}_{,\mu}} h \indices{^{\mu}_{\sigma}} + \eta^{\mu \nu} h \indices{^{\rho}^{\sigma}_{,\mu}} h_{\rho \sigma, \nu} - \eta^{\mu \nu} h_{,\mu} h_{,\nu} ] \textrm{,}
which, when varied with respect to h_{\mu \nu}, gives the linearized Einstein equations.