Systems and complexitySystems thinking and modelling

Optimization and Complexity: Assessing Model Accuracy [Systems thinking & modelling series]

This is part 58 of a series of articles featuring the book Beyond Connecting the Dots, Modeling for Meaningful Results.

We first import our historical data into a converter primitive. We then assess the qualitative and quantitative accuracy of the model. To assess how well our model fits the historical data qualitatively, we plot the simulated and historical data series next to each other. Ideally, they will match closely but if they do not we should pay close attention to how they differ.

If they have the same general shape (except for a vertical or horizontal displacement) that is good news, as it indicates that most likely the general dynamics of your model are correct and you may just need to fine-tune the relationships and parameter values. If the results look considerably different you may have more work to do in improving the model.

You can also assess the accuracy of models quantitatively. A standard tool used to assess the accuracy of a model is the R2 metric1. R2 is the fraction of the squared error explained by the model compared to the “null” model. It ranges from 0 (the model basically provides no predictive power) to 1 (the model predicts perfectly). Mathematically, R2 is calculated like so:

Naively used, R2 has a number of issues that we will discuss later in this section. However, it is still a useful tool that many people use and with which they are familiar. It is also relatively straightforward to calculate. The following code calculates an R2 for a model fit. This is code written in JavaScript and can be placed as the Action for a button primitive in Insight Maker2. The code assumes two primitives: a converter Historical Hamsters containing historical population sizes and a stock Hamsters containing simulated population sizes. You can edit the code to reference the actual names of the primitives in your model.

var simulated = findName(Hamsters); // Replace with your primitive name

var historical = findName(Historical Hamsters); // Replace with your primitive name

var results = runModel({silent: true});

var sum = 0;
for(var t = 0; t < results.periods; t++){
sum += results.value(historical)[t];
}

var average = sum/results.periods;

var nullError = 0;
var simulatedError = 0;
for(var t = 0; t < results.periods; t++){
nullError += Math.pow(results.value(historical)[t] – average, 2);
simulatedError += Math.pow(results.value(historical)[t] – results.value(simulated)[t], 2);
}

showMessage(Pseudo R^2: +((nullError-simulatedError)/nullError));

Next edition: Optimization and Complexity: Calibrating the Model.

Article sources: Beyond Connecting the Dots, Insight Maker. Reproduced by permission.

Notes:

  1. Though this metric is not often used in systems dynamics or agent-based models, it is widely used for statistical models such as linear regressions.
  2. Button primitives let you add interactivity to your model. You can place custom JavaScript code in them to be executed when a user clicks the button.

Scott Fortmann-Roe and Gene Bellinger

Scott Fortmann-Roe, creator of Insight Maker, and Gene Bellinger, creator of SystemsWiki, have written the innovative interactive book "Beyond Connecting the Dots" to demystify systems thinking and modelling.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top button