There is nothing saying that the L2 cache needs to be on another chip, or if it is on the same chip, that it needs to be distinctly separate from the core processor. Putting cache of any type on a chip other than the CPUs would drastically reduce the advantage of having a cache. The primary reason cache exists is because main memory access is so very slow compared to the rate at which the CPU operates. If memory access was just as fast as the CPU itself there would be little need for cache at all. (If main memory was just as fast as the CPU, it might still be handy to have separate data and memory caches so that the machine could be Harvard architecture at its core (faster) but Von Neumann architecture to the outside world (simpler).)
Specialized memory can be made that has very high clock rates. This specialized memory is necessarily small, and increasing the size decreases the operating speed. Hence multiple levels of cache. Primary cache, or L1 cache, is very small but very fast. Because L1 cache is so small, it is helpful to have an intermediate cache(s) to bridge the gap between main memory and the CPU. L2 cache is significantly larger than but is also slower than the primary L1 cache. Because the gap between main memory and CPU operating speeds has widened, some newer machines even have a tertiary cache on the same chip as the CPUs.