"such as an ellipse"
Because of precession, none of the non circular orbits of a black hole are exactly elliptical.
"an unbound orbit" is an orbit which, as time goes to infinity, can exceed any number in radius. For Newtonian gravity and a point mass, this would be a hyperbola, or the limiting form (between ellipses and hyperbolas), a parabola. It's possible that some would also define an orbit that ends at the singularity as an unbound orbit.
"an unstable orbit" is one where small changes in the initial conditions create large changes in the behavior of the orbit. This is kind of hard to describe.
Suppose we have an orbit which runs between some minimum and maximum radius. Let the initial condition be a point on the orbit where the radius is maximum. We can consider taking the particle, keeping its velocity unchanged, but changing the initial radius by an infinitesimal amount (or we could leave the radius unchanged but change the initial velocity by an infinitesimal amount). If the original orbit was stable, the new orbit will have a minimum and maximum radius that is only infinitesimally different.
A black hole has circular orbits all the way down to the event horizon at r = 2M. The circular orbits larger than r=6M are stable, while the ones smaller than r=6M are unstable.
Of a black hole's unstable circular orbits, the ones smaller than r=4M are particularly unstable in that infinitesimal change to their initial conditions will either send the particle into the black hole, or escape to an unbounded orbit (unless the change in initial condition moves the particle to a new unstable circular orbit, but that would require a certain amount of "just right".)
The instability in the orbits between r=4M and r=6M is manifest in that infinitesimal changes to the radius create new orbits with changes to the radius that are unbounded only in the ratio. That is, if r is the initial condition, and R is the maximum (or minimum or possibly both) radius, then dR/dr = infinity. This is what "unstable" means, as far as I know.
By the way, you might enjoy my simulation of black hole orbits. It includes cases where several test particles are introduced with almost the same initial conditions. You can see that for r<4M some of these particles get vacuumed up by the black hole and some get sent off to infinity:
http://www.gravitysimulation.com/
P.S. The calculations that are behind the above simulation got an "honorable mention" at the annual gravity essay contest. This is apparently the first amateur paper to get an award for the last 30 years or more:
http://www.brannenworks.com/gravity2009.pdf
The above paper and simulation do the orbits in two different coordinate systems, Schwarzschild and Gullstrand-Painleve(GP). These coordinate systems are identical except for time. That is, to convert an event (x,y,z,t) in Schwarzschild coordinates to an event (x',y',z',t') in GP coordinates, you take x=x', y=y', z=z', but t' = f(x,y,z,t). As a consequence of this reorganization of the time coordinate, in GP coordinates particles do not get stuck on the event horizon, they get sucked into the singularity in finite coordinate time.