Hi, first:
the term "M-theory" is reserved for a vacuum of string theory (in the post-1995 sense) that admits an 11-dimensional description. In 11 dimensions, there are no strings at all. In 11-dimensional M-theory, one only finds M2-branes (membranes) and M5-branes (fivebranes) as the analogous extended objects. They have 2 or 5 spatial dimensions, respectively.
(For a while, some people wanted to use the term "M-theory" for all of versions of string theory. However, this meaning of "M-theory" wasn't adopted. Instead, this unified network of all string theories (in plural, in the old sense) is called string theory (in singular, in the new sense) and the term M-theory is reserved for the newly found 11-dimensional limit only.)
If you return to string theory - whose supersymmetric version has 9+1 dimensions of spacetime - there must always exist closed strings. For example, the graviton is always a mode of a closed string. Open strings can also sometimes exist but their endpoints have to be attached to a D-brane. The dimensionalities and types of D-branes are limited. But in general, e.g. in type IIB string theory, there can even exist spacetime-filling D-branes allowing the open strings to end anywhere.
At any rate, two endpoints of an open string terminating on the same D-brane may meet and the open string may be allowed to become a closed string; that's really why you can't ever ban the closed strings in any theory of strings. This process of joining two endpoints is always allowed to occur - i.e. its probability is nonzero - unless a conservation of charges, energy, or spin is violated.
Spectrum of open strings in the five 10D string theories:
1) heterotic strings have no open strings. Open heterotic strings are impossible because the boundary conditions have to identify the left-moving and right-moving degrees of freedom - a standing wave on an open string is a wave going back and forth, if you wish. But the heterotic strings have different - bosonic vs supersymmetric - types of left-moving and right-moving excitations. So the two types can't be identified. Consequently, there are no open strings and no D-branes in E8 x E8 or SO(32) heterotic strings. All their spacetime fields, including gauge fields and fermions, are coming from the closed strings.
2) type IIA has D0, D2, D4, D6, and D8-branes (besides uncharged branes of odd dimensions). The D8-branes have codimension one fill too much space and their number is constrained because the dilaton and other fields start to run in their presence. 8 D8-branes on an orientifold O8-plane may actually become the Hořava-Witten domain wall in a strong coupling limit of type IIA which is 11-dimensional M-theory. The D-branes are forced to sit on the orientifold plane in this limit and they become the Hořava-Witten boundaries of the M-theory world. Other D-branes are unrestricted. D0-branes are a new kind of particles whose dynamics is governed by the attached open strings.
3) type IIB has D(-1), D1, D3, D5, D7, and D9 branes (aside from unstable even-dimensional D-branes). The total number of D9-branes and anti-D9-branes has to vanish by an anomaly cancellation and 0+0 is the only stable arrangement. D7-branes still have a big dimension and they cause a "conical deficit angle" in space. F-theory is the appropriate description of possible types of D7-branes. The lower-dimensional branes are unrestricted. The D(-1)-brane is the D-instanton, an instanton localized in all dimensions of the Euclidean spacetime, that contributes terms to many processes (much like instantons in gauge theories) and whose calculations are fully analogous to other D-branes (unlike gauge theory).
4) type I theory is type IIB string theory with a spacetime-filling orthogonal orientifold O9-plane that requires one to add 16 D9-branes and their mirror images to cancel a world sheet anomaly, leading to SO(32) gauge group - S-dual theory to the SO(32) heterotic string. Type I string theory has stable D5-branes and D1-branes; the D-instantons and D3-branes disappear.
5) bosonic strings would formally have Dp-branes with any p between -1 and 25. All of them are unstable. They don't carry any charge because there's no Ramond-Ramond sector of the bosonic string.
All these things can be understood from the standard literature but one must actually study it. These are qualitative, somewhat superficial questions. Of course that the textbooks try to answer not only qualitative questions such as whether open strings exist or may become closed strings; they teach the reader to calculate the actual probabilities of any such a process.
Cheers
LM
