How to quantify error in model

[bradyj7]

How to quantify performance of a model

Hello,

I was hoping that somebody could advise me how to do evaluate the performance of a model.

The model simulates the power demand from a fleet of electric cars over 24 hours.

For example take the blue line as the actual demand for a given day and the black lines (100) are simulated demands.

https://dl.dropbox.com/u/54057365/All/demand.JPG

Could anybody advise me on how I could quantify the performance of a the model? For example it varies by 5%, 10% etc? Would you use RMSE?

I'm not sure if error is the correct word because the blue line varies from day to day depending which day you pick.

Thank you

 

[mfb]

What do you want to do with your performance value?
How do you simulate those 100 curves?
I think the blue curve is an average over many cars?

To start, you could consider the mean squared deviation between blue line and simulations as time-dependent function.

 

[bradyj7]

Hi,

Thanks for your reply.

The black curve were generated by a model that I have made to predict the power demands of the cars over the day. The model uses data collect by cars.

The blue is the actual average power demand used by the cars. I create it my averaging the power demand of all the days in the dataset.

I want a performance value to quote how accurate or not the model is in general. Is it capable of generating reliable power demand profiles? How reliable?

Is mean squared deviation the same as mean square error?

http://en.wikipedia.org/wiki/Mean_squared_error

Is it essentially computing the difference between the blue line and the black line?

Would it be a good idea to compute the mean squared deviation between the blue line and the 100 black lines and then take the average mean squared deviation?

I computed the mse between the blue and 1 blak line ans an example in this workbook. I got an answer of 309. But what does this tell me about it accuracy?

https://dl.dropbox.com/u/54057365/All/MSE.xlsx

Thanks

 

[mfb]

Is mean squared deviation the same as mean square error?

If that deviation is the error of your simulation, right.

Is it essentially computing the difference between the blue line and the black line?

The average difference.

Would it be a good idea to compute the mean squared deviation between the blue line and the 100 black lines and then take the average mean squared deviation?

I would average over the 100 black lines first, and check the time-dependence. If that is flat and you don't care about the deviation at specific times, you can average over all.

I computed the mse between the blue and 1 blak line ans an example in this workbook. I got an answer of 309. But what does this tell me about it accuracy?

Lower=better. sqrt(309)≈17 - on average, your black line is wrong by 17 kW.

 

[bradyj7]

I would average over the 100 black lines first, and check the time-dependence. If that is flat and you don't care about the deviation at specific times, you can average over all.

Thank you for your reply. I don't quite follow the point above.

Are you implying above that I compute the MSE between the blue line and the 100 black lines? Getting 100 MSE values?

How would you check for time-dependence?

Thank you

 

[mfb]

Are you implying above that I compute the MSE between the blue line and the 100 black lines? Getting 100 MSE values?

No, just a single one (for each point in time): The meanSE.

How would you check for time-dependence?

Can be seen in the resulting graph.