facebooktwitteryoutube
Home Managed Accounts About Glossary
in Modeling Software - 03 Oct, 2016
by James - no comments
Synergy Ensemble Report

The Ensemble Report is a relatively simple, but very useful report. When a modeling run completes, the first task is to analyze the results displayed in the Ensemble Report. No models should be deleted from the modeling run before viewing this report. If models are deleted then the statistics displayed on the report are no longer meaningful.

The Ensemble Report lists the percent of perfect over the in-sample and out-of-sample periods from percentile zero through to 100 in increments of 10 based on the ranked performance of all models belonging to the modeling run.

Figure 1. Ensemble Report for Gold Futures Modeling Run

Figure 1. Ensemble Report for Gold Futures Modeling Run

The above image is of an Ensemble Report for an ensemble of models that trade gold futures (GC). This is an example of a successful modeling run. The percent of models that produced a hypothetical profit out-of-sample is 92.6%. Ideally, greater than 90% of models should be profitable out-of-sample. The next indication that the modeling run was successful is the high bias toward profitability over the out-of-sample period. The median percent of perfect is 9.4% out-of-sample. This is roughly equivalent to the in-sample median percent of perfect which is 8.5%. These results indicate that the input data is useful for modeling gold futures and that it is OK to proceed to the next step which is the analysis of individual models.

Note that the out-of-sample period is usually significantly shorter than the in-sample period. This is one of the reasons why there is a greater range in percentile values over the out-of-sample period. The percent of perfect is a measure of model efficiency. That is, what percent of the total possible profit did the model obtain. You can imagine that over a 5 day period some models will have a percent of perfect of 100% and others will have a very poor percent of perfect. A ‘normalized’ version of the in-sample results would be a nice additional feature in a future version of Synergy.

The following Ensemble Report is an example of a poor result. A modeling run was done using very low fitness thresholds and no autovalidation testing to produce the this report.

Figure 2. Ensemble Report Showing a Poor Result

Figure 2. Ensemble Report Showing a Poor Result

In the above Ensemble Report we see that only 60% of models produced a hypothetical profit over the out-of-sample period and that there was no bias toward profitability. Having obtained the above result, the recommendation is to not proceed any further with the modeling run. All models are suspect because it is unlikely that the input data is useful for modeling the traded series. This may be an over-simplification, but it is better to have a cautious approach.

Kind Regards,

James

 

Leave a Reply