That is very nice because your problem is now discussed deeper and deeper!
Though my research is very different from yours,I still decide to spend some time to discuss it.
(1) For your first problem,i.e."Now I have a wavefunction defined in reciprocal space (in DFT-VASP language, CPTWFP(G,K=0)) and I need to determine what its irreducible representation is under the C3v symmetry. One thing to do is to transform this wavefunction back to real space and then determine its property, or I would like to work on it directly in reciprocal space."
I think M.Quack is right,i.e.,"To determine the IR of the wave function, you do not need to transform back to real space. You can do the symmetry analysis directly in reciprocal space, using the exact same symmetry operators."
Here,I may supply a suitable formula for the symmetrized planewaves as follows
\Phi_{s}=\frac{1}{N_{op}}\sum_{R}exp[iRG(r-t_{R})]=\frac{1}{m_{s}}\sum_{m}\varphi_{m}exp[iR_{m}Gr]
From its mathematics,we know that it is fit for pseudo potential (or your software mentioned above,DFT-VASP) or interstitial repersentations in so-called APW method.
(2)For your second problem,i.e.,"Then, the issue is how to define the symmetry operations in the reciprocal space."
Of course,you can use the exact same symmetry operators in reciprocal space as in real space.
But, "how to relate these symmetry operations in reciprocal space to real space operations.",obviously,you can use Fourier transform (or more efficiently,FFT) to make it.However,(I think) those symmetry operations in reciprocal space should not be transformed into the real space,or you can't find the precise mathematical relation about their oprations between these two space (as mentioned above by me).