If your Hamiltonian is not spherically symmetric, then l is not a good quantum number in the first place: If H does not commute with rotations, you cannot find a common eigensystem of H and the L^2 operator[1].
I guess what you mean is "why is the energy dependent on n and l for for spherical potentials which are not 1/r" (i.e., not hydrogen). If this is the case: Because there generally is no reason for all energy to be independent of l. Rather, this degeneracy in the hydrogen atom is a special case for 1/r potentials, ultimately related to a hidden additional conserved symmetry (the so called Runge-Lenz-Vector already known from classical mechanics). So it is the 1/r potential which is the special case, not the other ones.[1] of course you could still prepare a wave function in a specific l state with respect to some point, but such states would not be eigenstates.