If sin(pi/3) is approximately .8660254038, then it is obviously (and again, approximately) equal to the fraction 8660254038/10000000000. Now you can try to reduce this fraction, for example by looking for the unique prime factors of 8,660,254,038 and 10,000,000,000 and cancelling the common ones; or, which is the same thing, by dividing 8,660,254,038 and 10,000,000,000 by its g.c.d. (in the ti-83, this is math-num-9:gcd). Since gcd (8660254038 ,10000000000) = 2, and 8660254038 / 2 = 4330127019, and 10000000000 / 2 = 5000000000, the number .8660254038 equals the fraction 4330127019/5000000000.
But notice three things:
1) .8660254038 equals the fraction 4330127019/5000000000, but none of them equals sin(pi/3). Both of them only approximate sin(pi/3), up to a certain numbers of decimals. With less decimals, the approximation would have been 8/10, or 86/100, or 866/1000, or ...
2) Everything they have been telling you is true; these numbers are irrationals, and you can only approximate them with a fraction. On the other hand, if what you wanted was to convert a given decimal like .8660254038 to a fraction, that can be done as above, but has nothing to do with sin(pi/3), except as an approximation.
3) Of course this is just a guess of what you want, because you are not explaining yourself very well, and by repeating "it can be done" the question doesn't get any clearer.