Author: rgearyiii

Subject: Portfolio123 Auditing & Validation

Date: 3/5/2016

Recommendations: 26

I`m splitting this topic off from http://boards.fool.com/Message.aspx?mid=32145718 as a separate thread because it obviously has nothing to do with the main topic there. People are obviously free to discuss whatever they want here, but I`ll be focusing on a few anomalies in the Portfolio123 database that I saw when a couple of its users posted some test results on the GTR1 Helper mailing list a year ago. Since I`m not a customer, I don`t anticipate having much to contribute after that, other than describe how I would go about validating Portfolio123 (or any backtester) against benchmarks, as described in the post I just linked to. Universe Composition The first thing I would do is determine exactly what the P123 universe consists of from the non-OTC stock market. We did this almost exactly one year ago on the GTR1_Helper mailing list and found the following: 1. The P123 non-OTC universe as of 3/6/2015 included only nine stocks that were not part of the GTR1 universe on the same day. Five of them were in fact OTC, three of them were ADRs representing preferred stock (which P123 appeared to generally exclude), and one was a unit bundling common stock with warrants (which they also appeared to generally exclude), where the common stock would qualify for the GTR1 universe only once they were unbundled. So the non-OTC P123 universe was essentially a subset of the GTR1 universe, with the tiny handful of exceptions appearing to be the result of security mistyping or exchange listing information that was out-of-date. 2. The P123 universe excluded absolutely all ETFs and CEFs, i.e., stocks with security type codes 14, 15, 44, 45, 73, 74 and 75 in the GTR1 field file styp.a. 3. 53 stocks excluded from the P123 universe were issues of companies with another share class that was included in the P123 universe. It would be helpful to know how consistent P123 was in including only a single share class for each company, but I don`t appear to have determined that. But it appears that this was their intent, and if they are simply defining their universe according to Capital IQ Snapshot coverage, then that would be the result. 4. Coincidentally, there were also 53 stocks excluded from P123 for unapparent reasons. -- Exactly one of these was a REIT, and thus the only exception to P123 generally including all REITs. -- Five were ADRs, and thus exceptions to P123 generally including all ADRs representing common stock. -- Twelve were unit trusts, a.k.a. "K-1 stocks" of various types (Limited Partnerships, LLCs, Royalty Trusts, etc). Since they included the vast majority of unit trusts, these once again appeared to be the exceptions to generally including them all. -- The remaining 35 missing stocks were ordinary common shares. Many of them were shell corporations aiming to take private companies public through acquisition, but this did not disqualify other stocks from inclusion in the P123 universe. The bottom line: One could expect the non-OTC P123 universe to consist of all non-ETF/CEF stocks in the GTR1 universe, excluding redundant share classes, and perhaps excluding a few dozen stocks for unknown reasons. This conclusion more than likely still holds today, but here is a URL that customers can use to test it against the current P123 universe: http://gtr1.backtest.org/2013/?lf-1lp-1h1::styp.a:ne14!15!44... Just click "Run Screener" and download the spreadsheet report. -- The field labeled "MultiClass" indicates whether a stock is one among more than one share class for the same company (1 = yes, 0 = no). -- The field labeled class.a gives the share class number that the GTR1 backtester assigns to each stock. A stock with a share class number of 1 is the primary class identified by the backtester, which is likely, but not always, the share class included by P123. -- The field styp.a gives security type as a code explained by borisnand here http://boards.fool.com/share-type-styp-counts-31909642.aspx . Historical Universe Size After determining the rules for current universe composition, the next thing I would check for is that they were consistently applied in the past. Since I don`t think P123 allows historical screening, all we can do to determine this is to count non-OTC stocks over the course of a P123 backtest from universe inception and compare the counts to what the GTR1 backtester predicts. To get the GTR1 stock counts, go to http://gtr1.backtest.org/2013/?lf-1lp-1::styp.a:ne14!15!44!4... and click "Count Stocks". This screen selects the primary active share class for each company among all non-CEF/ETF stocks. While the primary share class may not match what P123 includes, that shouldn`t matter for the sake of counting. What we found a year ago was that at the beginning of the backtest on 1/4/1999, P123 was missing 1,031 non-ETF/CEF stocks, but this steadily diminished to 441 missing stocks on 6/9/2003. Then, on 6/16/2003, the number of missing stocks abruptly dropped to just 25. The discontinuity was due almost entirely to a jump from 5,583 to 5,995 in the number of stocks in the P123 non-OTC universe, whereas the GTR1 count was essentially unchanged from 6,024 to 6,020. Such a pattern where stocks are missing from the beginning of a backtest, but where the number of missing stocks gradually decreases over time could signify some survivorship bias. What I think is quite possible is that the P123 database is based on some source other than Compustat/S&P Capital IQ before 6/16/2003, because the abrupt change in the number of stocks does not appear in my 2010 vintage Compustat Point-In-Time database. The count of CPiT stocks in the GTR1 database can be obtained from http://gtr1.backtest.org/2013/?lf-1lp-1::cprc:gt0 . A year ago, P123 was missing 682 CPiT stocks on 1/4/1999, which gradually diminished to 393 missing CPiT stocks on 6/9/2003, and then abruptly fell to -24 (meaning P123 somehow contained more 24 more stocks than actually active in CPiT) on 6/16/2003. But there is no need to speculate--customers should be able to get an explanation for this anomaly from Marco Salerno. Missing Returns The particular P123 backtest that was done for me was simply the entire non-OTC universe with the minimum holding period of one week. That backtest, oddly, was missing forward 1-week returns for the following dates: 20000703 20010910 20011224 20011231 20060703 20071224 20071231 20121224 20121231 The stock market appears to have been closed on all of these days, and perhaps the prior week`s forward returns are actually two-week forward returns. I would imagine that any P123 customer would be able to confirm this. Implausible Returns It appears that a separate backtest of the entire non-OTC universe held in equal weight with a holding period of one market day was also done for me. In this backtest, I noticed two daily returns that were totally implausible: A 1-day forward return of 118.7816% from the close of market day 1/30/2015, and a 1-day forward return of 10.1288% from the close of 10/22/2002. One of the P123 customers can let us know whether either of these errors have been corrected. This is enough for today (and there isn`t much more I can do without further assistance), but it will be interesting to see how far this discussion goes. Robbie Geary