Reviewing My Own Lack of Skepticism
Good tools don't do much good if you use them wrong
My post three years ago checking for signal in SqueezeMetrics GEX and DIX indicators got more engagement than anything else I’ve written. That’s probably because it contains “positive” results, which people are absolutely ravenous for.
That’s evidence of a strong incentive for people publishing trading content: people want to be told things that work, skepticism isn’t sexy. Even better if they can stay vague and say it’s your fault if something doesn’t work!
Today, I find my lack of skepticism in that post deeply embarassing. Let's review what’s wrong here.
High Density Intervals (HDI)
I’ve used HDIs to illustrate the range of probability mass in my posterior distributions. These regions illustrate where the hypothetical true population difference of means most probably lies. For my particular charts, I plot the 97% HDI interval, which is where 97% of the probability mass lies.
A layman’s understanding of this interval would be something like “I’m not sure exactly what the true long term average edge is, but I’m pretty sure its within this range”.
It’s important to keep in mind that this is the probable range of an aggregate statistic, in this case the long term average edge. It’s not the probable range of a random draw. When you’re trading, you’re subject to individual random draws which will lie well outside of that range, but they should be scattered around this mean value if the analysis is correct.
Region of Practical Equivalence (ROPE)
The ROPE is another interval used to contextualize the significance of a result in comparison with an HDI. It’s set manually. You can see it in my results as a pink line.
In the above result we’re looking at 1-month returns. I decided that I would consider anything within the +-0.5% range insignificant. The ROPE illustrates the position and size of this range. If we’re being honest with ourselves, that’s definitely too narrow.
If I were to do this analysis again I’d use the monthly standard deviation of the security (S&P 500 in this case) for this monthly analysis and use that as the ROPE. I’d want to see a result larger than this before I’d be interested.
The monthly standard deviation of the S&P 500 is on the order of 2%. The width of the entire HDI above is about 3%. The ROPE using the standard deviation would be 4% wide. This makes a huge difference in the analysis and immediately invalidate every conclusion I made.
The ROPE is the area in which the null hypothesis lives. If the ROPE is plausible, the null hypothesis is plausible.
This is where I failed. Even with my overly narrow ROPE I still drew the wrong conclusions about significance.
Almost Implausible Is Not The Same As Plausible
Every single one of the results on that post have the ROPE lying within the range of the HDI. Statement 6 is almost an exception, but it still has a significant amount of overlap.
If the HDI and the ROPE overlap, then the ROPE is consistent with the HDI and the null hypothesis is a plausible explaination for your data and your model. You need more data. to be more certain.
The HDI is our best guess given the amount of data we have. You make this narrower by getting more data, or improving your causal understanding of the underlying process that you’re modelling. If your best guess includes the null hypothesis, you can’t be confident that it’s false.
It’s important to carefully consider your null hypothesis for each analysis. I was reusing my hard-coded ROPE from daily return analysis. Half a percent for a one-month forecast is damn tiny. We don’t expect that kind of statistical precision in financial markets. We do expect the S&P 500 index to wiggle by at least 2% over that kind of period, so our ROPE should be at least larger than that.
Take-aways
Everyone involved in trading is ravenous for a positive result, and I was no exception here. I was also fueled by the embarassment of having a clear error in my earlier analysis of the same indexes.
I’ve always believed that any positive result you see anywhere should be considered wrong until you verify it yourself; but can you really trust yourself to be objective and careful when you can almost taste the fortune you’ll make?
You can’t trust anyone with skin in the game, especially yourself.