Let me draw threads together and point out some issues.
First treating this as slightly more-than-average hard exercise routinely posed to students. What I do is I believe what I'm told, suppose accuracy of measurements good. Atomic masses are near whole numbers and it is faster, you will see, if I work with those at first. The analysis gives me a C/H ratio close to 2/5 - an empirical formula you could say C
2H
5. Then C
8H
20 fits the actual molar mass of the compound. But chemically no such compound is possible. We then looked for a smaller molecule with the same C/H ratio, but with O and/or N to make up the molar mass to 116. Initial candidates must be based on C
2H
5, C
4H
10 and C
6H
15. Using the whole-number atomic mass approximation I rapidly eliminate the first and third of these. These part-molecules having an odd number molar mass, combined with any number of O or N which have even number mass (16 and 14) cannot make the molar mass of 116 which is even. That narrows it down to C
4H
10 with approximate mass 58. From 116 we need to make up 116 - 58 = 58 with O and N. 58 is not a multiple of either 16 or 14, so only with both could we find an answer. From 58 subtract 16 once, twice, three times and see if any result is a multiple of 14. Turns out the only one corresponds to N
3O. So we would conclude the molecule is C
4H
10N
3O. When I calculate the exact molar mass of that I get 116.14. That would normally be considered excellent and conclusive agreement with the #1 question figure of 116.28 but more anon.
The didactic point is that this is just an example calculation of the kind students are required to perform, usual principles, though slightly more complex than average. I cannot see anything not straightforward and even obvious about it (at most you might miss some shortcut) and do not understand why it took so long. Maybe the student now realizes what point he was missing and needs to do some more exercises to make sure he is at ease with them.
There are however some dubious points. My calculated molar mass of 116.14 is extremely close to 116.28 but we are not really comparing theory with experiment which is approximate - we are comparing theory with theory. You would only work out a figure like that from a composition and the precisely known atomic masses, so I don't know what explains the discrepancy. It is possible that this was an invented school exercise, and the authors thought of C
8H
20 not noticing it was impossible. For that I calculate a MM of 116.24 which is nearer though still not quite the one of the problem. At this point I should say the atomic masses that are used in such calculations are updated from time to time. I am using
H 1.00794, C 12.010107, N 14.00674, O 15.9994 Perhaps someone will check, if there are other figures around.
Molecular masses as far as I know are these days mostly determined by mass spectroscopy. But then from the fragments you can determine the composition too, and for that matter the molecular structure from mass spectroscopy. But maybe one machine cannot do everything and I am not very familiar with the state of the art.
I said ‘believe the data’. My faith wavered a bit when the OP pointed out four formulae which were within about 1% of right. In fact my shortcut rather depends on these being ruled out beforehand. But actually you can weigh within 0.1% easily so I guess this is right (though accuracy downstream does nothing if there is a problem with the sample). But you see how critical the accuracy question is to the scope of the method. The didactic exercises like
http://www.chm.davidson.edu/vce/stoichiometry/ch.html just give nice numbers and pass over this. The Addison-Wesley book cited surely devotes consideration to this question and maybe the student could re-read that bit with new appreciation. Accuracy, how to check this and what you think the experiment is telling you, problems encountered, precautions etc., essentially scientific and methodological questions enriching the bare academic exercise.
Finally I found it quite difficult to find a structure that corresponds to the formula C
4H
10N
3O. But if we can find only one we can certainly find a number of others. Is there anything wrong with
NH
2NHCCHOHCH
2CH
2NH OR NH
2NHCCH
2OCH
2CH
2NH?
The question looks like a mistake but it has given us some useful issues to examine.
is 2:5 from 8:20 ratio but I don't undestand what you said there about something having only 7 compounds with the above mentioned ratio.. to be quite honest, I don't really know if this question was meant to be this complicated. I guess I do need some more hints.
One of those is more convincing and is very close to the molar mass given in the problem, though not quite exact. 
