Register to reply

How to quantify error in model

by bradyj7
Tags: error, model, quantify
Share this thread:
bradyj7
#1
Dec27-12, 03:02 PM
P: 122
Hello,

I was hoping that somebody could advise me how to do evaluate the performance of a model.

The model simulates the power demand from a fleet of electric cars over 24 hours.

For example take the blue line as the actual demand for a given day and the black lines (100) are simulated demands.



Could anybody advise me on how I could quantify the performance of a the model? For example it varies by 5%, 10% etc? Would you use RMSE?

I'm not sure if error is the correct word because the blue line varies from day to day depending which day you pick.

Thank you
Phys.Org News Partner Science news on Phys.org
Scientists develop 'electronic nose' for rapid detection of C. diff infection
Why plants in the office make us more productive
Tesla Motors dealing as states play factory poker
mfb
#2
Dec28-12, 01:55 PM
Mentor
P: 11,917
What do you want to do with your performance value?
How do you simulate those 100 curves?
I think the blue curve is an average over many cars?

To start, you could consider the mean squared deviation between blue line and simulations as time-dependent function.
bradyj7
#3
Dec29-12, 04:28 PM
P: 122
Hi,

Thanks for your reply.

The black curve were generated by a model that I have made to predict the power demands of the cars over the day. The model uses data collect by cars.

The blue is the actual average power demand used by the cars. I create it my averaging the power demand of all the days in the dataset.

I want a performance value to quote how accurate or not the model is in general. Is it capable of generating reliable power demand profiles? How reliable?

Is mean squared deviation the same as mean square error?

http://en.wikipedia.org/wiki/Mean_squared_error

Is it essentially computing the difference between the blue line and the black line?

Would it be a good idea to compute the mean squared deviation between the blue line and the 100 black lines and then take the average mean squared deviation?

I computed the mse between the blue and 1 blak line ans an example in this workbook. I got an answer of 309. But what does this tell me about it accuracy?

https://dl.dropbox.com/u/54057365/All/MSE.xlsx

Thanks

mfb
#4
Dec29-12, 05:05 PM
Mentor
P: 11,917
How to quantify error in model

Is mean squared deviation the same as mean square error?
If that deviation is the error of your simulation, right.
Is it essentially computing the difference between the blue line and the black line?
The average difference.
Would it be a good idea to compute the mean squared deviation between the blue line and the 100 black lines and then take the average mean squared deviation?
I would average over the 100 black lines first, and check the time-dependence. If that is flat and you don't care about the deviation at specific times, you can average over all.

I computed the mse between the blue and 1 blak line ans an example in this workbook. I got an answer of 309. But what does this tell me about it accuracy?
Lower=better. sqrt(309)≈17 - on average, your black line is wrong by 17 kW.
bradyj7
#5
Dec30-12, 04:05 PM
P: 122
I would average over the 100 black lines first, and check the time-dependence. If that is flat and you don't care about the deviation at specific times, you can average over all.
Thank you for your reply. I don't quite follow the point above.

Are you implying above that I compute the MSE between the blue line and the 100 black lines? Getting 100 MSE values?

How would you check for time-dependence?

Thank you
mfb
#6
Dec30-12, 04:29 PM
Mentor
P: 11,917
Quote Quote by bradyj7 View Post
Are you implying above that I compute the MSE between the blue line and the 100 black lines? Getting 100 MSE values?
No, just a single one (for each point in time): The meanSE.

How would you check for time-dependence?
Can be seen in the resulting graph.


Register to reply

Related Discussions
Why Can't We Quantify Wffs Set Theory, Logic, Probability, Statistics 3
Is it possible to quantify human language? General Math 3
How to quantify seebeck effect Classical Physics 0
How to quantify knife-edge sharpness Mechanical Engineering 4
How do we quantify the viability of a scientific model? General Discussion 4