This set should be open in Q since for any a in Set S, we can find b>a such that b is also in S
This doesn't prove that it's open, but it's on the right track. You have to show something similar for being able to find a b < a, which you can easily do. But I would also try to be more precise. The above would be enough to show that it is open, but that's because it implies something else, which in turn, by definition, implies that the set is open. To be rigorous, you should show these implications (and it should be very easy).
This set is not open in the Reals since when a= 3^1/2, we cannot find b>a such that b is in S
But 3^1/2 is not in S. Note that S only contains rationals, and the root of 3 is irrational. This set is not open in the reals, but for a different reason. If S is open in
R, then for each x in S, there is an open interval I such that x is in I, and I is a subset of S. Note that an open interval in
Q is different from an open interval in
R. An open interval of
Q around each x in S exists such that Q is a subset of S. An open interval of
R around some x in S would contain rationals and irrationals, but since S contains only rationals, this open interval cannot be a subset of S.