My Analysis Approach and What Motivated It
How and why I attempt to separate signal from noise
Traders hope to do better than Jack Bogle’s “buy everything and hold it forever” approach to investing by seeking an edge over the market. Certainly most of them fail, as few would disagree, but that does not stop anyone. I’m addicted like the rest of the gamblers, but I have pretty good control of my margin account these days. This is a short narrative of how I blew up an account with futures, and what I learned from it.
You will be embarrassed for how badly I traded, and why. However, it does lead to what I think is a rigorous model for filtering the tools we allow in our toolbox.
TL;DR: don’t put a garbage tool in your toolbox. Be very selective of what inputs you allow in your decision making process.
The “Before Time”
I tried my bit-shifting abilities at markets early on, as most programmers do. The MetaTrader platform made data acquisition, programming, and optimization very easy and with a tight development loop that allowed me to try out anything I could think of. I kept optimizing for the best PnL and was sure the money machine will soon be ready for takeoff! 🚀
Unfortunately every positive result vanished when I tried with real money. Despite hundreds of hours spent hacking away, I had nothing to show for it but a greater humility towards markets. I left the attempt assured that this sort of alpha was impossible.
I accepted my mediocre returns from low-cost ETFs and did other things with my free time.
Working with Professionals
Years later, I got a job at a family office that traded livestock futures. I built them a data acquisition, databasing, and visualization system that saved them from manually entering numbers in Excel. Working directly with the traders, I collected and manipulated the data they needed to make decisions with.
I saw that they seemed to know what they were doing. There were severe drawdowns, but the long-term success was impressive. I was feeding them input, and they used it to extract edge out of their markets.
My faith in edge was restored.
Renewed Hunt for Edge
I collected and read books, consumed Twitter, and subscribed to Real Vision. I wrote down nearly every claim I could find in my notebook, and set out to test them or otherwise combine this into some kind of wisdom. Most of what I wrote down were unsupported claims, and many of them were very difficult to test - requiring obscure data, or subjective judgement that eluded long-term testing.
I briefly tried a some technical analysis, trading wedge patterns exclusively as a way of testing the approach as they subjectively seemed to work. Sometimes they did.
All of this was extremely haphazard. If I made money from any of it, it would have been impossible to distinguish this from pure dumb luck. This is a critical point.
At the time though, I thought I had some tools that worked very well. What does a gambler do when they find edge?
LEVER UP 🤘
I wrote a lot of code. I automated the statistical testing and backtesting that would guide my asset selection. I had them spit out handy spreadsheets summarizing the hours of testing that I could use for reference. It’s almost like this was similar to a job I used to do or something!
I started trading via the futures market. I became accustomed to having my PnL swing by thousands of dollars in mere hours. Having previously worked for a futures trading firm, I knew at least a few things about risk management and I did keep my positions small. Sometimes I would hedge with options, but mostly I used something like a value at risk approach. I now know that this approach is stupid.
Eventually hubris came to bite me. I was massively overconfident and over-leveraged. I shorted gold, for a variety of stupid reasons. Lets look at what this past-version-of-me wrote in his notebook:
2019-07-10
Short due to bearish positioning on multiple timeframes: 1H, 4H, 30 min. MACD weak.
Spiked against me when Powell said something bearish. Saw lower volume, bear wedge. Shorted more.
Stopped out based on old Kumo
In % terms, it only moved ~1.4% against me. The power of leverage amplified my loss to 20%. I held it and let it bleed me. I gave back all of my gains from trading, but no more, thanks to my hedges. I went from a Y/Y return of ~100% to 0%.
Honestly I didn’t remember how stupidly I traded this until I sat down to write this. The truth is much harder for my ego to stomach. I was stupid beyond belief. Every tool I was using to trade I have since rigorously tested and found to be totally worthless.
Eventually I capitulated and went back to the drawing board.
Cleaning Out the Toolbox
My overleveraged and emotional trading did not prove that my tools did not work, necessarily. All it shows is that I was a pretty terrible trader. However, it did motivate me to dig deeper and critically evaluate my tools from a different angle.
A core aspect of my previous approach was ordinary conditional probability. There is a trap in conditional probability: if your sample sizes are small, your results are probably uninterpretable. For a sample size of 300, a conditional probability “edge” over marginal probability of 20% is entirely ordinary when comparing two completely random series. This was the first nail in the coffin that I buried my approach in.
I also had backtests supporting my tool use. However, as I learned in my earlier days, backtests don’t necessarily mean a damn thing. They are enough to stop you from proceeding with a strategy, but they add no positive proof.
Having lost my primary “edge” detector, I needed a replacement. For me, that was Bayesian inference.
The Bayesian Razor
I dug into applying using Bayesian inferencing, leveraged by computing via pymc3. There are many of libraries out there for this, but Python is my go-to language and this is the dominant library in Python. This article essentially lays bare the model I use for testing, though mine is tweaked slightly for market data.
I approach signal evaluation like we would approach medical testing. You say some signal (medicine) improves returns (treats a condition). Okay. Let’s compare what returns (patients) look like with and without your signal (medicine). This is broadly called hypothesis testing in statistics.
Here’s a tweet thread where I explain this in more detail:
Most of this kind of testing is done with the sorts of methods we learn in first year statistics courses. My reading has pushed me towards Bayesian methods, and I don’t honestly know when or specifically how this happened. They are far more intuitive to interpret, IMO. I think they are also best used for smaller sample sizes. Here’s a good discussion on the difference.
With this approach I can evaluate any “signal” and allow the data to separate it from noise, if it can do so. The downside is admitting our uncertainty. Low sample sizes curse us with a wide credible interval - but that does not mean there are no indications of edge available with this approach for small sample sizes.
I also look at the same results a minimum of three different ways, to avoid fooling myself with the Bayesian approach1.
The below chart is the old conditional probability approach that burned me before - except now it’s overlayed with a distribution of results one would expect from random chance. If the signal advantage is significantly within or below the random edge distribution, that’s a strong count against its value.
The second approach is a scatter. If there’s something “there”, it should be apparent on a scatter diagram. It could also indicate if there is a time dependence: did this used to work, and no longer? The scatter also warns you of the fat tails of the distribution these returns come from: the below shows something that seems to work quite well, but there’s one -15% loss and several -5% losses as well. This does not include the max drawdown, which may well have been worse.
Finally, if all else is positive, I’ll check a backtest. A positive edge is worthless if it isn’t tradable.
I haven’t given up my search for edge, I’ve just raised my standards for accepting it. I am very protective of my toolbox, and I will only allow good tools in it. This new level of skepticism enables me to swing big when opportunity comes.
This leaves me back where I’ve begun. I’m fully invested in passive index-tracking ETFs, waiting and watching for an opportunity in my sweet spot.
I also check my traces and posterior distributions to ensure my Bayesian model is well described.