Here is an example.
Let's try find a polynomial function of degree 2 (a parabola) that has integer values exactly at x=1, 2 and 3. Assume there is such a function f(x).
We can find a linear function g(x) such that g(1)=f(1) and g(2)=f(2), namely g(x)=(f(2)-f(1))x + 2f(1)-f(2). Note that both coefficients are integers [note*], and g(x) is an integer for every integer x.
If f(x) has the properties we want, then h(x)=f(x)-g(x) has it as well, because we are only subtracting integers. We also know h(1)=h(2)=0. Every parabola with the last property can be expressed as h(x)=a(x-1)(x-2) for some a.
Let's plug in x=3: h(3)=2a. We want this to be an integer. That is possible, of course, but what happens if we look at other values? h(4)=6a. No matter which a we choose, if 2a is an integer, then 6a=3*2a will be an integer as well. This contradicts our requirement that our function has integer values only for 1,2,3.
I used three specific points here (1,2,3), but you can generalize this to any three points, e.g. 1,4,6 or 2,3,5. Similarly, you can ask for a polynomial function of degree 3 that has integer values only at 1,4,6,8 or 2,3,5,7 (see above: It doesn't matter if you take the set of primes or the set of non-primes).
I picked the degree first here, but that doesn't matter. If there is a polynomial function that has integer values only at prime numbers, then it has some degree n, and we can look at the first n non-prime numbers (or prime numbers) to determine the polynomial up to a constant (the "a" from above). While you have some choice in the constant, I can't see how any choice would lead to the correct result for every number. This is not a strict mathematical proof, but I'm quite confident that there is no such polynomial function.
*note: this doesn't generalize that nicely, but the coefficients stay rational with small denominators, and g(x) keeps getting integer values elsewhere