First begin with the electromagnetic fields since these are the really observable quantities. In your case, we have to deal with the electric field in the static case. For static fields the Maxwell equations decouple into equations for the electric and the magnetic field. So you can treat them independently from each other.
The electrostatic equations read
\vec{\nabla} \times \vec{E}=0, \quad \vec{\nabla} \cdot \vec{E}=\frac{1}{\epsilon} \rho,
where \epsilon is the dielectric constant of the material (which I assume to be (piecewise) homogeneous and isotropic for simplicity) and \rho is the charge density.
These equations are local equations since they only contain differentials of the fields, and you have to be careful at boundaries since there the field can have singularities. However, you have to remember the definitions of the vector operations, here the curl and the divergence of a vector field, through integrals to give these operations clear meanings when you consider the neighborhood of a boundary (where, e.g., two dielectrics meet; or you have conductors around etc.).
The curl tells you to take a line integral along some closed path and the enclosed surface element. To investigate the neighborhood of a boundary surface, you chose a rectangular path with two sides tangent to the surface and two very small sides perpendicular to the surface. Let's call the enclosed rectanglular surface S and its boundary \partial S. Then Stoke's Law tells you
0=\int_S \mathrm{d} \vec{F} \cdot \vec{\nabla} \times \vec{E}=\int_{\partial S} \mathrm{d} \vec{x} \cdot \vec{E}.
Now you can neglect the contributions from the two perpendicular pieces of the rectangle since we make them infinitesimally small. Then the above integral tells you that the components of \vec{E} tangent to the boundary must be continuous.
A similar argument can be made with Gauß's Theorem and a little prism parallel to the boundary. That tells you that the normal component of \epsilon \vec{E} has a jump equal to the surface charge along the boundary.
Now that we know the behavior of the electric field across boundaries, where the matter can change discontinuously (i.e., either the dielectric constant changes or you have a conductor, and their may be some surface charges sitting on the boundary etc.), we can see what this means for the electric potential. Since electrostatic fields are curl free according to the first equation of electrostatics this means that (at least locally) there exists a scalar potential, \Phi, such that
\vec{E}=-\vec{\nabla} \Phi.
Plugging this into the second equation of electrostatics gives
\Delta \Phi:=\vec{\nabla} \cdot \vec{\nabla} \Phi=-\frac{\rho}{\epsilon}.
The latter is known as Poisson's Equation (for \rho=0 the Laplace equation). Now since the components of \vec{E} tangent to the boundary are continuous and the normal components have at most finite jumps across the boundary, the potential itself must be continuous across the boundary. That is so because, suppose you know the electric field, the potential is given (up to an uninteresting constant) by the line integrals connecting a fixed point, \vec{x}_0, with the point under consideration, \vec{x} (let's call this lines C(\vec{x}_0 \rightarrow \vec{x})):
V(\vec{x})=-\int_{ C(\vec{x}_0 \rightarrow \vec{x})} \mathrm{d} \vec{x} \cdot \vec{E}.
The first equation of electrostatics guaranties that the value of the potential is independent of the particular line chosen (as long as the considered region in space is simply connected). Since \vec{E} has at most finite jumps in the normal component across the boundary, V thus must be continuous.
