How Effective Is a Neural Network Model in Predicting Poker Hands?

[tomeram]

Hey

I have a neural network model that produces as an output a vector of 169 variables which represents the probabilty of having a certain hand in poker (2 random cards dealt from a regular deck - 169 possibilities if considering only if the cards are from same suit or not).
The model predict for each spesific situation and action made by a player in the game the distribution of having all possible hands.
I kept a random sample from the data for testing the model, and now I want to test it. Each row in the testing set contains the data of the situation, the action the player made and the hand he had, however the model produce a vector of 169 values (which represent the probabilty of having each of the possible hand).
I am looking for a statistical method to estimate the accuracy of the model - some kind of method that can say what is probabilty that the observation came from the distribution produced by the model.
Thanks
Tomer

 

[Cexy]

Cross-validation to test the accuracy of the model?

To test how well the observations fit the probability distribution produced by the model you could construct a test statistic (eg Pearson's chi square statistic) and repeatedly sample from your model's prob distribution to produce a distribution for the test statistic - this allows you to give a p-value (e.g. test statistic is greater than Z with y% probability) and you can then compare the value of the test statistic from your observations to see how well they fit the model.

Edit: I should say that in order for this to be reliable, you should be training the model on a different data set to the one you're later using to test its accuracy.

 

[tomeram]

Hi
Thanks
The problem id that the output of the model is already a distribution - the distribution of getting a certain value. The problem is that each observation comes from a differnt observation and I need to know if the model predicts the distribution correctly. It is rare to get two point from the same observation, so I have to build a test based on single point from each distribution. I don't think cross validation will help this time.

 

[tomeram]

Hi
Thanks
The problem id that the output of the model is already a distribution - the distribution of getting a certain value. The problem is that each observation comes from a differnt observation and I need to know if the model predicts the distribution correctly. It is rare to get two point from the same observation, so I have to build a test based on single point from each distribution. I don't think cross validation will help this time.