Over-searching error (statistics)

  • Context: Graduate 
  • Thread starter Thread starter 0rthodontist
  • Start date Start date
  • Tags Tags
    Error Statistics
Click For Summary
SUMMARY

The discussion focuses on the concept of "over-searching" error in the context of machine learning statistics. It explains that when evaluating models from a search space S, selecting a subset K that equals S can lead to overfitting, resulting in poor performance on unseen data. The key takeaway is that while a larger K may yield a better fit for the training data D, it can also introduce biases that negatively affect the model's generalization ability. Understanding the balance between model complexity and generalization is crucial for effective model selection.

PREREQUISITES
  • Understanding of machine learning model evaluation
  • Familiarity with concepts of overfitting and generalization
  • Knowledge of search space and model selection techniques
  • Basic statistics related to bias and variance
NEXT STEPS
  • Research "Bias-Variance Tradeoff in Machine Learning"
  • Learn about "Cross-Validation Techniques for Model Evaluation"
  • Explore "Regularization Methods to Prevent Overfitting"
  • Study "Model Selection Criteria: AIC and BIC"
USEFUL FOR

Data scientists, machine learning practitioners, and statisticians interested in improving model selection processes and understanding the implications of overfitting in predictive modeling.

0rthodontist
Science Advisor
Messages
1,229
Reaction score
0
I guess this should properly go in the Programming forum, but I think I might get a better response here.

My question is with respect to statistics (context of machine learning) about "over-searching" error. You have a search space S of all possible models, from which you choose a subset K. Then you have some test data D, and you evaluate how well each model in K fits D. You pick the best model in K and use that as your model.

Over-searching says that it is bad to do exhaustive sampling, where K = S. Though the model you end up with fits D better than the model you end up with when K is much smaller than S, for some reason the model when K = S does not work as well when it's tested against new data that's not in D.

I didn't quite catch the reason for this and I still do not understand. I wrote down, "two or more search spaces contain different numbers of models. The maximum scores in each space are biased to different degrees." I understand this but I don't see its relevance to over-searching.
 
Physics news on Phys.org
Difficult to answer without measures for quantifications. The general idea could be: the bigger K the better the fit, but also the tolerances are smaller. This means by additional data to check against, sub optimal solutions can still be solutions, whereas an optimum is likely to be destroyed.
 

Similar threads

High School Statistical Errors, Type I and Type II
  • · Replies 43 ·
2
Replies
43
Views
6K
Undergrad Does inference help forecasting?
  • · Replies 2 ·
Replies
2
Views
2K
Undergrad Questions about error range from Bayesian statistics
  • · Replies 26 ·
Replies
26
Views
5K
Graduate Computing the Fisher Matrix numerically
  • · Replies 1 ·
Replies
1
Views
2K
Undergrad What Are the Key Differences Between OLS and GLM Models in Statistical Analysis?
  • · Replies 3 ·
Replies
3
Views
3K
Undergrad Nonlinear Least Squares or OLS for Nonlinear Models?
  • · Replies 7 ·
Replies
7
Views
3K
Undergrad Data Analaysis -- How to choose the best statistical model to use?
  • · Replies 5 ·
Replies
5
Views
2K
Undergrad Error bars in Excel (standard deviation, standard error, percentage)
  • · Replies 4 ·
Replies
4
Views
3K
Undergrad Linear regression and random variables
  • · Replies 30 ·
2
Replies
30
Views
5K
Undergrad Right way to account for non-statistical uncertainties
  • · Replies 16 ·
Replies
16
Views
2K