I'm back!
here is how i went about proving it, trying to use CompuChip's hint.
the octic group is:
[tex]D_4=\{ (1),(1234),(13)(24),(1432),(24),(14)(23),(13),(12)(34)\}[/tex]
for convinience let [tex]\rho =(1234), \theta=(24)[/tex]
So, what i did is, i found all cyclic subgroups of the octic group, namely i found all subgroups generated by the 8 el of [tex]D_4[/tex], that is i found
[tex][(1)],[\rho],[\rho^2],[\rho^3],[\rho \theta],[\rho^2 \theta],[\rho^3 \theta][/tex]
I also found two non-cyclic groups, that is
[tex]K=\{(1),(13),(24),(13)(24)\}, M=\{(1),(12)(34),(13)(24),(14)(23)\}[/tex]
So, as we can see, all the above subgroups are of orders, 1, 2 or 4, so in order to find subgroups of order 3,5,6, or 7 i claimed that we need to take the union of any of these subgropus. In more details here it is how i proceeded:
I said, let A and B, be any of [tex]\{[(1)],[\rho],[\rho^2],[\rho^3],[\rho \theta],[\rho^2 \theta],[\rho^3 \theta],K,M\}[/tex] or the union of any of the elements in this set, such that, neither [tex]A\subset B, nor, B\subset A[/tex]
Now, my claim was that [tex]A\cup B[/tex] does not form a subgroup in [tex]D_4[/tex] thus we cannot have subgropubs of order 3,5,6 or 7 in it.
Proof:
I used proof by contradiction, i said, let's suppose that [tex]A\cup B[/tex] is a subgroup in [tex]D_4[/tex]. Thus, it means that it is also closed under the opertaion of [tex]D_4[/tex].
Now, let [tex]a\in A \setminus B[/tex] and [tex]b\in B \setminus A[/tex] I took these two elements this way, because i wanted to increase the nr. of elements when we would take the union of A and B. So, since AUB is a subgroup it means that
[tex]ab=h\in A \cup B[/tex] but since [tex]b=a^{-1}h[/tex] this cannot be possible, since it would mean that
[tex]b=a^{-1}h\in A or b=a^{-1}h\in B[/tex] but this is not possible, since we have supposed that [tex]b\in B \setminus A[/tex].
So, this means that [tex]A \cup B[/tex] is not a subgroup in [tex]D_4[/tex] and thus we cannot have subgroups of order 3, 5, 6 or 7.
So, is this even close to being the right way of proving it?
P.S. Now i know how to prove this one using Lagrange's theorem, i learned it.