Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

How to quantify error in model

  1. Dec 27, 2012 #1
    How to quantify performance of a model


    I was hoping that somebody could advise me how to do evaluate the performance of a model.

    The model simulates the power demand from a fleet of electric cars over 24 hours.

    For example take the blue line as the actual demand for a given day and the black lines (100) are simulated demands.

    https://dl.dropbox.com/u/54057365/All/demand.JPG [Broken]

    Could anybody advise me on how I could quantify the performance of a the model? For example it varies by 5%, 10% etc? Would you use RMSE?

    I'm not sure if error is the correct word because the blue line varies from day to day depending which day you pick.

    Thank you
    Last edited by a moderator: May 6, 2017
  2. jcsd
  3. Dec 28, 2012 #2


    User Avatar
    2017 Award

    Staff: Mentor

    What do you want to do with your performance value?
    How do you simulate those 100 curves?
    I think the blue curve is an average over many cars?

    To start, you could consider the mean squared deviation between blue line and simulations as time-dependent function.
  4. Dec 29, 2012 #3

    Thanks for your reply.

    The black curve were generated by a model that I have made to predict the power demands of the cars over the day. The model uses data collect by cars.

    The blue is the actual average power demand used by the cars. I create it my averaging the power demand of all the days in the dataset.

    I want a performance value to quote how accurate or not the model is in general. Is it capable of generating reliable power demand profiles? How reliable?

    Is mean squared deviation the same as mean square error?


    Is it essentially computing the difference between the blue line and the black line?

    Would it be a good idea to compute the mean squared deviation between the blue line and the 100 black lines and then take the average mean squared deviation?

    I computed the mse between the blue and 1 blak line ans an example in this workbook. I got an answer of 309. But what does this tell me about it accuracy?

    https://dl.dropbox.com/u/54057365/All/MSE.xlsx [Broken]

    Last edited by a moderator: May 6, 2017
  5. Dec 29, 2012 #4


    User Avatar
    2017 Award

    Staff: Mentor

    If that deviation is the error of your simulation, right.
    The average difference.
    I would average over the 100 black lines first, and check the time-dependence. If that is flat and you don't care about the deviation at specific times, you can average over all.

    Lower=better. sqrt(309)≈17 - on average, your black line is wrong by 17 kW.
  6. Dec 30, 2012 #5
    Thank you for your reply. I don't quite follow the point above.

    Are you implying above that I compute the MSE between the blue line and the 100 black lines? Getting 100 MSE values?

    How would you check for time-dependence?

    Thank you
  7. Dec 30, 2012 #6


    User Avatar
    2017 Award

    Staff: Mentor

    No, just a single one (for each point in time): The meanSE.

    Can be seen in the resulting graph.
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook