The idea of the Green's function is based on Green's integral theorem, which is an application of Gauss's theorem,
$$\int_{V} \mathrm{d}^3 \vec{x}' \vec{\nabla}' \cdot \vec{A}(\vec{x}')=\int_{\partial V} \mathrm{d}^2 \vec{f} \cdot \vec{A}(\vec{x}'),$$
where ##V## is some volume and ##\partial V## its boundary surface. The orientation of the surface-normal vectors is out of the volume.
Now set
$$\vec{A}(\vec{x}')=G(\vec{x},\vec{x}') \vec{\nabla}' \Phi(\vec{x}')-\Phi(\vec{x}')\vec{\nabla}' G(\vec{x},\vec{x}')$$
Then
$$\vec{\nabla}' \cdot \vec{A}(\vec{x}')=G(\vec{x},\vec{x}') \Delta' \Phi(\vec{x}') - \Phi(\vec{x}') \Delta' G(\vec{x},\vec{x}').$$
If the Green's function now satisfies
$$\Delta' G(\vec{x},\vec{x}')=-4 \pi \delta^{(3)}(\vec{x}-\vec{x}')$$
and if ##\Phi## fulfills (Gauss units)
$$\Delta' \Phi(\vec{x}')=-4 \pi \rho(\vec{x}')$$
you have
$$\vec{\nabla}' \cdot \vec{A}(\vec{x}')=-4 \pi G(\vec{x},\vec{x}') \rho(\vec{x}') + 4 \pi \Phi(\vec{x}') \delta^{(3)}(\vec{x},\vec{x}').$$
Now integrate this over the volume ##V##, you get for ##\vec{x} \in V##
$$\int_V \mathrm{d}^3 \vec{x}' \vec{\nabla}' \cdot \vec{A}(\vec{x}') = -4 \pi \int_V \mathrm{d}^3 \vec{x}' G(\vec{x},\vec{x}') \rho(\vec{x}')+4 \pi \Phi(\vec{x}).$$
According to Gauss's Law, this equals the surface integral
$$-4 \pi \int_V \mathrm{d}^3 \vec{x}' G(\vec{x},\vec{x}') \rho(\vec{x}')+4 \pi \Phi(\vec{x})=\int_{\partial V} \mathrm{d}^2 \vec{f}' \cdot \vec{A}(\vec{x}') = \int_{\partial V} \mathrm{d}^2 \vec{f}' \cdot [G(\vec{x},\vec{x}') \vec{\nabla}' \Phi(\vec{x}')-\Phi(\vec{x}')\vec{\nabla}' G(\vec{x},\vec{x}')].$$
By definition the Dirichlet problem means that you know ##\Phi(\vec{x}')## for ##\vec{x}' \in \partial V## but nothing about derivatives of ##\Phi## along the surface. Thus you define the Green's function such that it has to fulfill the boundary condition
$$G(\vec{x},\vec{x}')|_{\vec{x}' \in \partial V}=0,$$
then you get
$$4 \pi \Phi(\vec{x})=4 \pi \int_V \mathrm{d}^3 \vec{x}' G(\vec{x},\vec{x}') \rho(\vec{x}')-\int_{\partial V} \mathrm{d}^2 \vec{f}' \cdot \Phi(\vec{x}') \vec{\nabla}' G(\vec{x},\vec{x}').$$
Since further ##\mathrm{d}^2 \vec{f}' =\vec{n} \mathrm{d}^2 a##, dividing by ##4 \pi## gives your formula.