Author: Staka

Subject: Benchmark Investing SIPro backtests

Date: 7/23/2004

Recommendations: 35

I have promised to create a backtest of Benchmark Investing (BI) with the SIPro backtester from Keelix. It took some time but now preliminary backtests are available. Here are the results: Executive Summary The backtests of BI with the SIPro database are short (1997-today) and only snapshots (e.g. no all months test). Not a single optimization has been done, I have straight implemented the algorithm as I understood it. It is not the real BI because I (have to) use fixed holding periods and fixed numbers of stocks. It should give a "feeling" on how BI stocks behave. On average they beat the indices but by less than usually advertised. Yearly holds seam to work better as do midcap stocks (instead of large caps). Test script Define {BI} Uses [SI Price] [SI Volume--Average Monthly 3m] [SI Book value/share Y7] [SI Book value/share Y6] [SI Book value/share Y5] [SI Book value/share Y4] [SI Book value/share Y3] [SI Book value/share Y2] [SI Book value/share Y1] [SI Return on equity Y7] [SI Return on equity Y6] [SI Return on equity Y5] [SI Return on equity Y4] [SI Return on equity Y3] [SI Return on equity Y2] [SI Return on equity Y1] [SI Price--Low Y7] [SI Price--Low Y6] [SI Price--Low Y5] [SI Price--Low Y4] [SI Price--Low Y3] [SI Price--Low Y2] [SI Price--Low Y1] [SI Book value/share Q1] [SI Return on equity 12m] [SI EPS Est Q0] [SI EPS Est Q1] [SI EPS-Continuing Q4] [SI EPS-Continuing Q3] Deblank [SI Price] [SI Volume--Average Monthly 3m] [SI Book value/share Y7] [SI Book value/share Y6] [SI Book value/share Y5] [SI Book value/share Y4] [SI Book value/share Y3] [SI Book value/share Y2] [SI Book value/share Y1] [SI Return on equity Y7] [SI Return on equity Y6] [SI Return on equity Y5] [SI Return on equity Y4] [SI Return on equity Y3] [SI Return on equity Y2] [SI Return on equity Y1] [SI Price--Low Y7] [SI Price--Low Y6] [SI Price--Low Y5] [SI Price--Low Y4] [SI Price--Low Y3] [SI Price--Low Y2] [SI Price--Low Y1] [SI Book value/share Q1] [SI Return on equity 12m] [SI EPS Est Q0] [SI EPS Est Q1] [SI EPS-Continuing Q4] [SI EPS-Continuing Q3] Keep :[SI Price]>5 Keep :[SI Price]*[SI Volume--Average Monthly 3m]/21>500 Keep :[SI Book value/share Y7]>0 Keep :[SI Book value/share Y6]>0 Keep :[SI Book value/share Y5]>0 Keep :[SI Book value/share Y4]>0 Keep :[SI Book value/share Y3]>0 Keep :[SI Book value/share Y2]>0 Keep :[SI Book value/share Y1]>0 Keep :[SI Return on equity Y7]>0 Keep :[SI Return on equity Y6]>0 Keep :[SI Return on equity Y5]>0 Keep :[SI Return on equity Y4]>0 Keep :[SI Return on equity Y3]>0 Keep :[SI Return on equity Y2]>0 Keep :[SI Return on equity Y1]>0 Create [BookValuesAve] :([SI Book value/share Y7]+[SI Book value/share Y6]+[SI Book value/share Y5]+[SI Book value/share Y4]+[SI Book value/share Y3]+[SI Book value/share Y2]+[SI Book value/share Y1])/7 Create [RoesAve] :([SI Return on equity Y7]+[SI Return on equity Y6]+[SI Return on equity Y5]+[SI Return on equity Y4]+[SI Return on equity Y3]+[SI Return on equity Y2]+[SI Return on equity Y1])/7 Create [LowPricesAve] :([SI Price--Low Y7]+[SI Price--Low Y6]+[SI Price--Low Y5]+[SI Price--Low Y4]+[SI Price--Low Y3]+[SI Price--Low Y2]+[SI Price--Low Y1])/7 Keep :[SI Book value/share Q1]>0 Keep :[SI Return on equity 12m]>0 Create [Price7yDownside] :[LowPricesAve]/[BookValuesAve]*[SI Book value/share Q1]*[SI Return on equity 12m]/[RoesAve] Create [PriceToDownside] :[SI Price]/[Price7yDownside] Create [EarEst] :[SI EPS Est Q0]+[SI EPS Est Q1] Keep :[EarEst]>0 Create [EarY0] :[SI EPS-Continuing Q4]+[SI EPS-Continuing Q3] Keep :[EarY0]<>0 Keep :OR([EarY0]<=0,([EarEst]/[EarY0]-1)*100>=10) Sort Ascending [PriceToDownside] ; Top :5 End I have two slightly modified scripts for the test of BI in the S&P500 and in the S&P400 universe (see backtester protocols referenced below). Although SIPro has an indicator for membership in these S&P universes this indicator is not available before 2000. Therefore I have used approximations based on market cap, e.g. top 500 in market cap and the following 400. The script has been developed based on the BI rules found for the calculation of the downside price at http://boards.fool.com/Message.asp?mid=20916205. BI requires a stock to be below its downside price target, the more the better. An additional rule requires that the stock has an estimated earnings growth of >= 10%. I have implemented this rule by using the earnings estimates for the current and next fiscal quarter and compare them (the sum) to the earnings in the corresponding year ago quarters. If the year ago earnings were in sum negative it was impossible to calculate a growth rate. In this case if the estimates give a positive number I let the stock pass the filter. If the previous years earnings were exactly zero I must skip the stock to avoid a zero division. This is a bit inconsistent but I see no other way to circumvent this with the backtesters script language. It should not have a big influence anyway. For the all universe backtest I have used a minimum dollar volume filter to ensure an average daily dollar volume > 500,000. I have not used this for the market cap restricted S&P 500/400 backtests. For the downside price calculation I have required that the roe and bookvalue data is available for each of the 7 years covered in SIPro. The original BI algorithm requires the data to be available in at least 7 of the last 10 years. As far as I have checked a stock which has data 7 years back usually has this data for all 7 years. I have required the bookvalue and the roe to be positive each year. This excludes a number of high leveraged stocks with low or negative bookvalue but these aren`t usually the targets of the BI method (e.g. the originaly used DOW30 index has no such stocks and even among the S&P500 it is difficult to find stocks with a negative bookvalue). Therefore this should not be a serious restriction. Test scenarios For each universe I have tested 3 holding periods. Each test holds the top 5 stocks. The results can be seen as output from Keelix` backtester. You can reference them via a link similar to http://kjeldahl.net/cgi-bin/siproscreener.cgi?lookupjob=XXXX where XXXX stands for the job number. Here are the job numbers: Monthly Quarterly Yearly All stocks 4357 4358 4359 S&P500 4370 4371 4372 S&P400 4373 4374 4375 I have done the quarterly and the yearly runs only once starting on first possible date 1997-08-31. Keelix provides no all month backtester. If somebody is interested to get more data he can easily try all the other start months. Results All results are given in the format CAGR/GSD. The values are taken from the backtester output. Monthly Quarterly Yearly All stocks 5/37 7/39 16/14 S&P500 8/34 3/25 0/14 S&P400 29/37 23/31 18/18 S&P500 index 3/20 4/18 2/25 Nasdaq index 4/39 4/40 2/69 Conclusions - 7 of the 9 backtests are better than the indices. - The S&P400 backtests are most impressive. Here you have to keep in mind that the midcaps have performed better than the S&P500 over this time period. Unfortunately I have no comparision with the S&P400 index but I made a rough calculation which shows that this index had a CAGR of about 10 over this time frame. - The yearly backtests are especially convincing because they have such a low GSD. - If you look at the details of the backtester output you will see that the strategy in all cases has worked much better then the indices from 2000-2004 and in most cases were worse from 1997-1999. - BI seems to be a reasonable method for bear markets but might have some weaknesses in bull markets. - Be aware that the tests do NOT represent the exact original BI idea. They should rather be taken as supportive evidence on whether BI might work or not. Backtester issues The backtester output contains some comments for a number of periods which might hint to errors but I don`t know because I`m not always sure what they mean, e.g.: - stopped (GET) -> happens quite often, no idea how it influences results - missing data, stopped (RUS, TDW, OEA, HP, LEN) -> leads to zero period result - [] -> no selections although this seems wrong From my experience with the backtester here some hints what could be improved (if Keelix finds the time): - Whenever I go to http://kjeldahl.net/cgi-bin/siproscreener.cgi I first see another web page (Investor forum home page) and must enter the URL again to go to the backtester site. Am I doing something wrong here? - A big source for errors is the Uses statement. Keelix said that he is parsing the code twice anyway. Would this not allow to automatically generate and add such a statement based on all fields found? - It would be a nice idea to (optionaly) get an email when my backtester job has been finished! - It would be nice to be able to store an URL which can reproduce the whole backtest including time periods, trading frequency, stocks hold, job description and comparision indices. - If the previous is to difficult it would be a good idea to have on the job output page a button which would allow to take over the whole backtest into the backtester start page. This would support an easy way to make incremental changes. - I would like to see a way to search for previous backtests which I did, otherwise it is extremly difficult to find them if I have missed to note the number. Another issue are substitutions. A number of variables do not exist before 2000. To get a backtest from 1997 I was forced to make some substitutions. In most cases it would be a good idea to be able to use the newer data fields (e.g. S&P500 membership) and make an automatic substitution when the field does not exist. I propose to have the following automatic substitutions: - "Standard and Poor stock" -> Market cap based filter - "Volume--Dollar Daily Avg 3m" -> "Price" * "Volume--Average Monthly 3m" / 21 - "EPS-Diluted Continuing Q#" -> "EPS-Continuing Q#" Staka