Argh, I have to go; I was hoping to keep giving you hints so you could figure out all the important steps yourself. I'll write out what I had been planning to say with answers... but please try to answer the questions yourself before reading my answer, and moving onto the next question! (I'll put the answers in white -- highlight for easier reading)
(From here on, I'm going to set g = f^-1, to make it easier to write)
There is a loose end I missed before -- we have to assume the domain of our function does not contain zero. Do you see why?
--------------
f(0) = a
g(a) = 0
g(a) = 1/f(a)
1/f(a) = 0 <==== there is no possible value of f(a) that works here
[/color]
--------------
So, for the rest of this post, let's assume that we do not require 0 to be in the domain of f.
We know f(a) = b and g(b) = a. I've already said you can tell me the values of f(b) and g(1/a). What are they?
----------
f(b) = 1/a
g(1/a) = b
[/color]
----------
Are there any other values of f or g that you can compute?
----------
yes: we can compute both f and g at the points a, b, 1/a, 1/b.
[/color]----------
What are those values?
----------
f(a) = b
f(b) = 1/a
f(1/a) = 1/b
f(1/b) = a
g(a) = 1/b
g(b) = a
g(1/a) = b
g(1/b) = 1/a
[/color]----------
Do you think your equation g(x) = 1/f(x) can be used to relate the values we just computed to anything else?
----------
No. I don't have a rigorous proof of this, but it seems clear to be just from playing with the equation.
----------
So, we know that if we pick the value b for f(a), then this forces upon us a particular choice of f(b), f(1/a), and f(1/b), and nothing else.
Play with some possible orderings of a and b, such as:
1/b < 1/a < a < b
and plot the values of f at those points. If f is continuous, can it possibly be invertible?
Can you tell me the possible values of f(1) and f(-1)? (Hint: we've already done the work: set a = 1, and see what that tells us)
----------
f(a) = b, so f(1) = b
f(1/a) = 1/b, so f(1) = 1/b
So, b = 1/b, and thus b = 1, or b = -1.
The same is true if we set a = -1.
So, we can either choose f(1) = 1 and f(-1) = -1, or we can choose f(1) = -1 and f(-1) = 1
[/color]
----------
Now, what about the rest of the values of f? All we have to do is to organize the rest of the numbers into groups of four, of the form: {a, b, 1/a, 1/b}, then define f(a) = b, et cetera. It seems clear to me that there is no problem in doing this organization, and thus we've constructed an f satisfying your equation.