Motivation: Why validate simulations?
- when not rigorously calibrated and validated, simulations are neither a reliable research method nor a reliable tool for practical decision support
Definition: Verification, Validation, Calibration
(Steps, that subsume validation)
Verification
Does the code do what the specification asks?
- code could include faults
Definition: Verification, Validation, Calibration
(Steps, that subsume validation)
Structural Validation
Does the model correctly represent the problem space?
- are relevant relationships included in the model etc?
Definition: Verification, Validation, Calibration
(Steps, that subsume validation)
Input/Output validation
Does the simulation correctly represent the problem space?
- some kind of description of the status quo based on data
- parametrise the shape of the distribution
Calibration: adjusting input values to get valid output values
-> danger: overfitting the model when calibrating it
-> agent-based models are subject to over parametrization -> customer learning -> we can make assumptions but we don’t know the parameter for sure (no input parameters)
Simulation Data
Different layers of data
Input data
Process data
Output (result) data
Simulation Data
Different layers of data
Input data
- collected of inferred from empirical status quo
- e.g. customer arrival rate, range of products a company offers
Simulation Data
Different layers of data
Process data
- generated during the simulation process, provides insights into the simulation model that are not empirically available or relevant for the simulation purpose
- process or event logs from the real world can be compared
Simulation Data
Different layers of data
Output (Results) data
- indicators calculated for validation or what-if analysis, matching empirical indicators
- in the status quo these are the indicators we are interested in -> what happens when we change them? Outcome?
Simulation data
Input data
Assumption:
- Structure of relevant simulation components and parameters has been determined
Direct observation:
- for transparent systems
- example: machine run times
- or: what is the actual outline of the shop
Indirect inference:
- based on empirical process and result indicators
- example: customer choice
- > we know what happened but not why, so we apply data analysis to find a model how things work in the real world
Simulation data
Input data: scenarios
Stochastic scenarios
Worst- and best case scenarios
Qualitatively discrete scenarios
Simulation data
Input data: scenarios
Stochastic scenarios
- follow empirical distributions
- stochastic input scenarios lead to stochastic process and result data
- > when we are uncertain about data -> consider stochastic scenarios
Simulation data
Input data: scenarios
Worst- and best-case scenarios
- model extreme cases
- results indicate the modeled systems’ robustness
- > robustness = does it behave the same in each of these cases?
Simulation data
Input data: scenarios
Qualitatively discrete scenarios
- model discrete alternative cases
- test robustness and the necessity of individualized strategies
Simulation data
Example: railway ticket as simulation input
Directly observable:
- supply: products, capacity, price categories, availabilities
- demand: historical sales
Indirect inference (also: estimation):
- customer loyalty (e.g. we have the name on the ticket -> we can analyse how often they bought in the past)
- reference prices
- willingness to pay
- > can use this to calibrate customers in a model
Simulation data
Input data on the future
- Insights:
- analyzing empirical data to parametrize the simulation input data is based on insights, the pre-condition for predictive analytics - Forecast:
- to simulate future scenarios, the future values of input data have to be forecasted
Simulation data
Process data
Simulation systems are fully transparent - all data that is generated in the process of the simulation can be observed
We may look at: empirically-available process data
- forecasts
- plans
- documented events
And we can compare it to: simulation-exclusive process data (= generate process data that wouldn’t have been availble in the real world):
- decisions
- learned experiences
- communication
Simulation data
Result data
Usually empirically-available
- system reports
- transaction data
Simulation-exclusive
- long-term developments
- what-if developments
- internal agent- and system-states: e.g. customer satisfaction, emergent strategies
Simulation dara
Result data
Usage
- Output validation: compares result data to empirical data
- sensitivity and meta-modeling: analyse relationship in the “black box”
- data farming: simulations generate artificial transaction data -> data of the real world is not sufficient so we need more data
Simulation data
Data farming
= simulation models run thousands of times can provide insights into the different consequences of different options
- generates as much reproducible data as desired (amount is only limited to time and storage space)
- success depends on validity - which is difficult to determine for human, social, cultural and behavioral modeling
Simulation data
Data farming
Which types of data farming are there?
Simple: Monte Carlo
- Create data from known distributions
Mechanic: Discrete event-based white box
- create data given varying scenarios
- look at everything that happens in the real world -> input data
Emergent: Agent-based black box
- create data based on assumptions and theories
- we don’t know what’s actually going on in the empirical agents
Simulation data
Data farming with Monte-Carlo Simulations
Problem: Idea: Chance: Risk: Example:
Problem:
- empirical data set is not large enough to allow for significant statements about variable relationships
Idea:
- create additional data based on distributions fitted to the empirical sample (fill in potential gaps)
Chance:
- draw more meaningful conclusions from the enriched data set
Risk:
- enriched data set is “tainted” by a priori assumptions about distributions (gaps are filled based on assumptions)
Example:
- survey responses on the influence of consultative committees
Verification
aims to ensure that the code does what the specification asks - identify and eliminate “bugs”
Problem: when to stop testing?
- testing takes a creative and destructive mind
- avoid testing your own code
- test cases should be formulated by subject matter experts (don’t use the same people to develop and test -> won’t notice mistakes)
- test cases only prove the absence of those errors that they were designed to test
Structural validation
Does the model correctly represent the problem space?
- systematically explicate model components and relationships
- expert walkthrough with mode stakeholders
Questions:
- Are all concepts and structures relevant to the problem included?
- is the model structure consistent with relevant knowledge of the system?
Structural validation
ODD protocol
- used to describe individual-based models, agent based models, simulation models
Overview:
- Purpose
- Entities, state variables, and scales
- Process overview and scheduling
Design:
4. Design Concepts (Basic principles, emergence, adaptation, objectives, learning, prediction …)
Details:
- Intitialization
- Input data
- Submodels
-
1 - Introduction16
-
2 - Forecasting and Estimation37
-
3 - Data Warehouses and OLAP44
-
4 - Unsupervised learning: clustering and association rules37
-
5 - Supervised Learning, Classification26
-
6 - Supervised learning, classification13
-
7 - Validating and Evaluating Data Models19
-
8 - Data Visualization26
-
9 - Simulation Modelling45
-
10 - Stochastic Simulation Modelling19
-
11 - Validating simulation models28
