I learned about Chern numbers from Topology, Geometry, and Physics by M. Nakahara. It's very well written, and aimed at a physics audience. I'm an undergraduate physics major and I found it very accessible.
The reason that the Chern number linked to Berry's phase must be an integer becomes clear when you consider the phase of a wave function. Berry's phase, of course, arises from curvature in the parameter space of possible Hamiltonians. Consider adiabatic transport of an eigenstate around a small loop in the parameter space. Assuming some reasonable 'niceness' conditions on the parameter space, when you complete the loop, the particle will be in the same eigenstate it started as, with a change of phase e^{i\phi_1}. By Stoke's theorem, this \phi_1 should be some function of the area inside of the loop taken. Then, by adiabatically transporting the eigenstate in the opposite direction around the loop, the change in phase should be e^{-i\phi_2}, where \phi_2 is some function of the area of the parameter space outside of the loop. But, going around the loop clockwise should result in the same change of phase as going around the loop counterclockwise. So we set
e^{i\phi_1} = e^{-i\phi_2}
Setting \phi = \phi_1 + \phi_2, we get the condition e^{i\phi} = 1, meaning that \phi MUST be an integer multiple of 2\pi. This integer is the Chern number.
What is really going on here is that you have a fiber bundle with a base space consisting of the parameter space, and the fiber at each point is the associated eigenspace with the Hamiltonian at that point (perhaps with a cutoff at some energy to avoid the complication of an infinite dimensional fiber). Then, imagine the evolution of the eigenvector as it takes a trip around the parameter space. Assuming some 'niceness' conditions on the parameter space (namely that there is no degeneracy, energy bands never touch, and the dimension of the eigenspace never changes), then you can imagine the eigenvector traveling along with the frame of the eigenspace as it is transported around the manifold. In the presence of say, a magnetic field, the fiber twists around the parameter space. But, in order for it to be able to be 'glued' back to itself appropriately, it has to twist around the parameter space an integer number of times. Otherwise, when you imagine the eigenvector traveling along the frame of the eigenspace, it will end up in the wrong spot when you travel along a loop. I.e., you'd start with some state \psi and instead of transporting \psi around a loop and ending up with e^{i\phi} \psi you might get some other eigenstate \xi. This makes no physical sense, so it must be that the fiber 'twists' around the parameter space an integer number of times.