Author: rgearyiii

Subject: How not to backtest Sum-of-Ranks

Date: 12/23/2016

Recommendations: 32

While investigating a backtester crash last week produced by running some variant of http://gtr1.net/2013/?~TrendingValue_20120222_tetranomad:s19... (unfortunately I don`t know what parameter values were entered in the parameter box because at the time of the crash, the CGI script that calls the backtester was set to clean up after itself and delete the parameter file, which now prevents me from re-producing the crash), it occurred to me that tetranomad (who attached his username to the URL), and probably others, could save themselves a lot of time by following some advice that I often give for constructing GTR1 backtests. That advice doesn`t appear to be followed very often, so I thought I would back it up with an example that illustrates its performance-related benefits. While the above URL is a poorly constructed backtest in several respects, if the date in the screen`s name is accurate, then it preceded the introduction of some features needed for better construction (thus, no reason for tetranomad to be embarrassed). The advice I refer to is this: Always encapsulate complex calculations inside their own backtests and import the results into other backtests using either importf (for regular field values), imports (for signal fields) or picktally. In some cases, it isn`t even possible for a backtest to work as intended without following this approach. For example, tetranomad`s screen calculates a bunch of different field ranks, but there is no de-blanking or crap filtering for any of them, and the result is that a lot of junk finds its way to the top or bottom of the sorts by the sum of ranks. Since each ranked field has its own unique set of steps that need to be taken to ensure meaningful values, combining all of that into a single screen would be a complete mess, if it can be done correctly at all. It is far safer and simpler to create a separate backtest for each field of interest and import its ranks into the backtest for summing and final screening. The field translations that I provided to Larry for GTR1 Helper were deliberately designed to exemplify the recommended approach. However, the focus of this post is on the performance advantages to what I recommend. Fixing tetranomad`s screen would be too big of a project, so I`ve put together a simple example (with a URL that I could construct with Excel formulas). This first version uses the ill-advised method of calculating everything within one backtest, where "everything" includes the percentile ranks for RRS with twelve different lookbacks: http://gtr1.net/2013/?dspo%280%29al252:adv%281,63%29tp50:RRS... If the backtest has already been run for the day, you will get results almost instantly. If not, then you will have to wait about seven or eight minutes. You can force the backtest to run without cache files by including "-nocache" as a command line-only option. I just did so and the backtester reported Clock: 45175 which indicates that it took 451.75 seconds to complete. Furthermore, if you make a minor change to the screen, e.g., by changing "adv(1,63) top 50%" to "adv(1,63) top 55%", you will have to wait another seven or eight minutes (a little longer than the first backtest, since more stocks are ranked) for results. These wait times are well past the level that starts getting on my nerves. This next version uses the method I recommend, where each ranked field is ranked within its own backtest and the ranks are imported with importf: http://gtr1.net/2013/?dspo%280%29al63:adv%281,63%29tp50:min%... When I run that with the "-nocache" option, I get Clock: 4623 which indicates that the backtest took 46.23 seconds to complete. That`s still a long backtest (though twelve different RRS variants calculated for the entire stock market back to 19251231 is a non-trivial job), but it`s almost ten times faster than the first. Furthermore, you can tweak the second screen in just about any way you want and still get results in under a minute. That could easily make the difference between spending all weekend backtesting variants of the screen or getting the same information in half an hour of work. The reason the second version performs so much better is the way the importf function works: It causes the referenced screen to save the requested field`s values to a hidden intermediate field file, and the backtest that makes the importf call then reads that field file like any other. For a calculation as intensive as RRS, reading the calculations from a file is faster than re-calculating them from stock prices in RAM, especially since Jamie installed the 1TB SSD. Though in the case of RRS, where values and ranks change for just about every single stock every day, the intermediate field files are massive, which is why the backtest still takes as long as it does. If the twelve different imported fields were, say, your typical valuation ratios based only on weekly fundamental data and limited to the SI Pro era, the backtest would probably take just a few seconds due to the much smaller intermediate field files. Note that the recommended version will still take quite a bit of time to run the first time for the day. However, subsequent variations on the screen (i.e., those that import the same ranks but vary in what they do with them) will be as fast as I have indicated. I can`t demonstrate how long the initial backtest would take because "-nocache" does not hide intermediate import field files, but even the initial field file creation is faster than the unadvised version. This is because the twelve intermediate field files, each calculated within its own backtest, are calculated in parallel. The unadvised version of the backtest affords no (easy) multi-threading. A few other tips on constructing fields for import: 1. Speed up the backtest that makes the calculation to be imported by adding a final step that guarantees no stocks pass, thus eliminating portfolio management from the workload. 2. Set the holding period to one market day, which also minimizes portfolio management (since there is a separate portfolio for each trading cycle) in case you forget (1). 3. For your own sanity, move all lag control to the "top" backtest making the import calls, so that it can all be controlled in one visible place. Do this by setting field file lag adjustment and price lag adjustment to -1 in the "lower" backtests with default lags (or one-day lags) specified in all formulas. This results in zero net lags in all calculations, which can then by lagged as desired in the top backtest`s settings, namely, by either individually lagging imported fields using the lag argument for importf, or leaving those settings to defaults and controlling lag adjustments globally. As always, though, complete details on applied lags can always be found in the command translation, even with the nested backtester calls that result from using importf. So please, folks, take this advise and enjoy the many days, weeks, months and perhaps even years added to your extra-GTR1al life. It also makes debugging (which is what prompted this post) a lot less time-consuming for me as well. Robbie Geary