Testing a prediction model

[tomeram]

Hey

I have a neural network model that produces as an output a vector of 169 variables which represents the probabilty of having a certain hand in poker (2 random cards dealt from a regular deck - 169 possibilities if considering only if the cards are from same suit or not).
The model predict for each spesific situation and action made by a player in the game the distribution of having all possible hands.
I kept a random sample from the data for testing the model, and now I want to test it. Each row in the testing set contains the data of the situation, the action the player made and the hand he had, however the model produce a vector of 169 values (which represent the probabilty of having each of the possible hand).
I am looking for a statistical method to estimate the accuracy of the model - some kind of method that can say what is probabilty that the observation came from the distribution produced by the model.
Thanks
Tomer

 

[Cexy]

Cross-validation to test the accuracy of the model?

To test how well the observations fit the probability distribution produced by the model you could construct a test statistic (eg Pearson's chi square statistic) and repeatedly sample from your model's prob distribution to produce a distribution for the test statistic - this allows you to give a p-value (e.g. test statistic is greater than Z with y% probability) and you can then compare the value of the test statistic from your observations to see how well they fit the model.

Edit: I should say that in order for this to be reliable, you should be training the model on a different data set to the one you're later using to test its accuracy.

 

[tomeram]

Hi
Thanks
The problem id that the output of the model is already a distribution - the distribution of getting a certain value. The problem is that each observation comes from a differnt observation and I need to know if the model predicts the distribution correctly. It is rare to get two point from the same observation, so I have to build a test based on single point from each distribution. I don't think cross validation will help this time.

 

[tomeram]

Hi
Thanks
The problem id that the output of the model is already a distribution - the distribution of getting a certain value. The problem is that each observation comes from a differnt observation and I need to know if the model predicts the distribution correctly. It is rare to get two point from the same observation, so I have to build a test based on single point from each distribution. I don't think cross validation will help this time.

 

[blue_raver22]

I am excited to see your work on developing a neural network model to predict the probability of hands in poker. Your model seems to have a comprehensive approach, taking into account various factors such as the player's action and specific situation.

To test your model, you have mentioned keeping a random sample from the data and now wanting to test it. I suggest using a statistical method such as cross-validation to evaluate the accuracy of your model. This method involves splitting your data into training and testing sets, where the training set is used to train the model and the testing set is used to evaluate its performance. This will give you an estimate of how well your model can generalize to new, unseen data.

Additionally, you can also use metrics such as precision, recall, and F1 score to evaluate the performance of your model. These metrics will provide a more detailed understanding of how well your model is predicting each hand.

I also suggest considering using other testing methods such as k-fold cross-validation or leave-one-out cross-validation, which can provide a more robust evaluation of your model.

Overall, I am impressed with your work and look forward to seeing the results of your testing and the potential applications of your model in the field of poker. Keep up the good work!