Walk-Forward Simulations in Synergy

The Walk-Forward Simulator is used to test the stability and robustness of a given market timing model that has been retained during a Synergy data mining run. It is probably the most important test to apply when considering a model for use.
The Lookback Period consists of n rows of input data leading up to the row being processed and never beyond. The default value for the Lookback Period is 1,500 rows or approximately 6 years of daily data. The particle swarm optimizer is applied to the input data over the Lookback Period to determine the model parameter values to use walking-forward until model parameter re-optimization is due to be done again.
The Walk-Forward Period is the number of rows that the optimized model parameter values will be applied for before using the particle swarm optimizer to re-optimize the model parameter values. The default value for the Walk-Forward Period is 50 rows. At the expiration of each Walk-Forward period the model parameter values are likely to change as a result of the re-optimization.

Synergy Walk-Forward Procedure

Figure1. Synergy Walk-Forward Procedure

A single walk-forward run is referred to as a path. Synergy runs 25 paths during a walk-forward simulation by default and averages the resulting equity curves. A run of 25 paths with all paths using the same lookback and walk-forward periods is referred to as a scenario. The particle swarm optimizer does not do an exhaustive search. The PSO computes near optimal model parameter values and ranges and is likely to arrive at slightly different values each time it is run.

The model parameter value search space is defined by the Function Block Min and Max values listed on the Function Blocks tab. The search space is often significantly large. Synergy is doing a true walk-forward simulation by ‘starting from scratch’ each time the PSO is run.

The number of paths to run, lookback period and walk-forward period are user defined parameters of a walk-forward simulation and define a single walk-forward scenario.

Single-Scenario Walk-Forward Simulation

The following steps describe how to run a Single-Scenario walk-forward simulation for a given model.

Step 1. Go to the Models tab and select the model that you would like to run walking-forward.

Figure 2. Model Selection

Figure 2. Model Selection

Step 2. Either select Models and then WF Simulator on the main menu or click the WF Simulator button that is under the Model Details information. There must be at least 500 rows of data from the start of the Build period to the final row of data loaded or the Walk-Forward Simulator cannot be run.

Figure 3. Opening the Walk-Forward Simulator

Figure 3. Opening the Walk-Forward Simulator

Step 3. The Single-Scenario Simulator is located on the second tab.

Figure 4. Single-Scenario Walk-Forward Simulator

Figure 4. Single-Scenario Walk-Forward Simulator

Step 4. Change the number of paths to run, lookback period, walk-forward period or the simulation start and end dates if the default values are not suitable.

Figure 5. Single-Scenario Simulation Parameters

Figure 5. Single-Scenario Simulation Parameters

Increasing the Lookback Period will move the minimum Start Date forward because more data is required for the initial optimization of the model parameter values.

If a simulation was previously run, then changing any of the Simulation Parameters will cause the results of the previous run to be cleared.

Step 5. Click the Start button to start the walk-forward simulation. The Status will change to Running… and the number of completed paths will update until the simulation completes. The simulation will be run using all available CPU cores.

Figure 6. Single-Scenario Simulation Progress

Figure 6. Single-Scenario Simulation Progress

You will notice equity curves being added to the chart as walk-forward paths complete. Each of the gray equity curves results from a single walk-forward path.

When the simulation has completed the Status will change to Completed.

Step 6. Analyze the chart of the equity curves. The green line is the original model’s equity curve generated using fixed model parameter values that were determined using data over the Build Period during the model building process.

Figure 7. Single-Scenario Equity Curves Chart

Figure 7. Single-Scenario Equity Curves Chart

The orange line is the walk-forward equity curve average. If the walk-forward equity curve average is very poor then try re-running using lookback periods of 1500, 2000 and 2500. Some models require longer lookback periods. If the walk-forward equity curve average is still poor, then the model may not be robust.

The End Build Period and End Validation Period markers are displayed on the chart for reference only. The original Build and Validation Periods do not apply when walking-forward.

Step 7. Analyze the Statistics list below the Equity Curves Chart to evaluate the walk-forward performance. The average values for all paths are computed and listed.

Figure 8. Single-Scenario Summary Statistics

Figure 8. Single-Scenario Summary Statistics

Multi-Scenario Walk-Forward Simulation

The equivalent of the Multi-Scenario walk-forward simulation can be run using the Single-Scenario walk-forward simulator by manually changing the lookback period and re-running the simulator. The advantage of the Multi-Scenario walk-forward simulator is that it enables the average of the walk-forward equity curves for each scenario to be easily compared.

Step 1. To run a series of walk-forward simulations over a range of lookback periods click the Multi-Scenario tab.

Figure 9. Multi-Scenario Walk-Forward Simulator

Figure 9. Multi-Scenario Walk-Forward Simulator

The length of the lookback period has, by far, the greatest effect on the walk-forward results. By comparison, the length of the walk-forward period is insignificant.

Step 2. Modify the Paths per Scenario, Walk-Forward Period, Initial Lookback Period, Number of Scenarios and the Scenario Interval if the default values are not suitable.

Figure 10. Multi-Scenario Simulation Parameters

Figure 10. Multi-Scenario Simulation Parameters

Different values for the Lookback Period are what distinguishes one scenario from another. The Initial Lookback Period is the shortest lookback period that will be evaluated. The Scenario Interval is the amount to increase the lookback period by from one scenario to the next. In the above image a total of 5 scenarios will be run. The lookback periods for the scenarios will be 500, 1000, 1500, 2000 and 2500.

The Final Lookback Period is read-only and is displayed to help the user determine if they have spanned a large enough range in lookback periods. The number of rows available for the walk-forward simulation is determined by the Build Start Date and the Final Lookback Period.

The Autovalidation Tests include an automated walk-forward test that is optionally applied by Synergy when building models. Models that do no walk-forward well should not be considered robust.

Step 3. Click the Start button to run the walk-forward simulation. The number of completed scenarios and completed paths for the current scenario will update until the simulation completes.

Figure 11. Multi-Scenario Simulation Progress

Figure 11. Multi-Scenario Simulation Progress

Step 4. Analyze the results. Summary statistics for the original model and each of the walk-forward scenarios are displayed in the Statistics report. Selecting one of the Scenarios in the Statistics report will highlight the corresponding equity curve in the chart.

Figure 12. Multi-Scenario Statistics

Figure 12. Multi-Scenario Statistics

Note that the statistics and equity curve for a given scenario are computed by averaging the scenario walk-forward paths.

Figure 13. Multi-Scenario Equity Curves Chart

Figure 13. Multi-Scenario Equity Curves Chart

When optimizing model parameter values there is no ‘fudging’ of the parameter ranges. The optimization is always done using the parameter ranges that are defined on Function Blocks tab within Synergy. The parameter range for the period of a given oscillator would typically be set to range anywhere from 2 to 50. Moving average periods range from 2 to 50 or 2 to 100 by default.

The walk-forward simulators in Synergy not only test how well a given model performs walking-forward. There is variation in the performance of the walk-forward paths because the particle swarm optimizer will not arrive at exactly the same model parameter values each time it is run. As a result, the walk-forward simulations also provide an indication of how sensitive a given model is to variations in the model parameter values.

Kind Regards,

James

Leave a Reply

Your email address will not be published. Required fields are marked *