Well, we should really be clear on this since "confidence" has a very technical definition in statistics and it is not a synonym for "probability".
If you are looking for statement that you can make in a scientific publication then the best thing to do is to look at work that got published and see exactly what statistics those papers used that passed the peer review process in your field. (Statistics is highly subjective and what procedures are favored in various academic arenas is largely a matter of tradition.)
If you are seeking an answer that satisifes you personally then you must be clear about what you are asking. If you are seeking what it takes to make statements like "There is a 0.97 probability that the true value of theta is 2 , given the data I observed" then you must use Bayesian statistics. This is also the situation if you want to make a statement about a "credible interval", such as "There is a 0.97 probability that the true value of theta is between 1.98 and 2.02".
The situation in non-Bayesian ("frequentist") statistics is that you make statements such as "There is a 0.97 probability that the value of theta observed in the experiment is within plus or minus 0.2 of the true value", but you can't take a particular numerical value, such as 2, and substitute it in place of "the true value of theta" in that statement. You also can't take a particular numerical value such as 2.2 and subsitute it for "the value theta observed in the experiment" . This is because a statement of a "confidence interval" says something about the general quality of the sampling process you use. It doesn't make any claims about one particular outcome of that process.