I don't know of any matrix that has all the primes as its only eigenvalues. But there appears to be a matrix such that its most negative eigenvalue (one eigenvalue per matrix) is a prime plus minus a small number.
T(n,1)=1, T(1,k)=1, n>=k: -\sum\limits_{i=1}^{k-1} T(n-i,k), n<k: -\sum\limits_{i=1}^{n-1} T(k-i,n)
\displaystyle T(n,k) = \begin{bmatrix} +1&+1&+1&+1&+1&+1&+1 \\ +1&-1&+1&-1&+1&-1&+1 \\ +1&+1&-2&+1&+1&-2&+1 \\ +1&-1&+1&-1&+1&-1&+1 \\ +1&+1&+1&+1&-4&+1&+1 \\ +1&-1&-2&-1&+1&+2&+1 \\ +1&+1&+1&+1&+1&+1&-6 \end{bmatrix}
For which the list of the most negative eigenvalue for the first 100 matrices is:
{-1., 1.41421, 2.65544, 3.43931, 4.77106, 5.24392, 6.84437, 7.15539, \
7.47476, 7.57341, 10.9223, 11.096, 12.9021, 13.0453, 13.259, 13.4055, \
16.9724, 17.0824, 18.9443, 19.0552, 19.2282, 19.307, 22.9972, \
23.0759, 23.1576, 23.2173, 23.2976, 23.3972, 29.0103, 29.0407, \
30.963, 31.0104, 31.1008, 31.1505, 31.268, 31.34, 37.0284, 37.0658, \
37.1289, 37.174, 41.029, 41.0503, 42.9921, 43.0326, 43.0807, 43.1149, \
46.996, 47.0293, 47.0619, 47.1025, 47.1582, 47.2011, 53.0192, \
53.0497, 53.1076, 53.1419, 53.1893, 53.2117, 59.0477, 59.0681, \
61.0248, 61.0474, 61.0812, 61.1071, 61.1644, 61.1812, 67.0341, \
67.059, 67.0929, 67.1062, 71.027, 71.0496, 73.014, 73.0331, 73.0575, \
73.0829, 73.1282, 73.1427, 79.024, 79.0442, 79.0633, 79.0799, \
83.0154, 83.0287, 83.0648, 83.0806, 83.1091, 83.1312, 89.032, \
89.0463, 89.0784, 89.0973, 89.1237, 89.1374, 89.1731, 89.1921, \
97.0597, 97.0753, 97.0963, 97.1128}
which when rounded is:
{-1, 1, 3, 3, 5, 5, 7, 7, 7, 8, 11, 11, 13, 13, 13, 13, 17, 17, 19, \
19, 19, 19, 23, 23, 23, 23, 23, 23, 29, 29, 31, 31, 31, 31, 31, 31, \
37, 37, 37, 37, 41, 41, 43, 43, 43, 43, 47, 47, 47, 47, 47, 47, 53, \
53, 53, 53, 53, 53, 59, 59, 61, 61, 61, 61, 61, 61, 67, 67, 67, 67, \
71, 71, 73, 73, 73, 73, 73, 73, 79, 79, 79, 79, 83, 83, 83, 83, 83, \
83, 89, 89, 89, 89, 89, 89, 89, 89, 97, 97, 97, 97}
Compare this to the previous prime with the Mathematica command:
http://www.wolframalpha.com/input/?i=Table[NextPrime[i,+-1],+{i,+1,+101}]"
https://oeis.org/A191898"