Confirm Signal Newsletter

Share this post

My Analysis Approach and What Motivated It

confirmsignal.substack.com

My Analysis Approach and What Motivated It

How and why I attempt to separate signal from noise

Aug 14, 2021
1
Share this post

My Analysis Approach and What Motivated It

confirmsignal.substack.com

Traders hope to do better than Jack Bogle’s “buy everything and hold it forever” approach to investing by seeking an edge over the market. Certainly most of them fail, as few would disagree, but that does not stop anyone. I’m addicted like the rest of the gamblers, but I have pretty good control of my margin account these days. This is a short narrative of how I blew up an account with futures, and what I learned from it.

You will be embarrassed for how badly I traded, and why. However, it does lead to what I think is a rigorous model for filtering the tools we allow in our toolbox.

TL;DR: don’t put a garbage tool in your toolbox. Be very selective of what inputs you allow in your decision making process.

The “Before Time”

I tried my bit-shifting abilities at markets early on, as most programmers do. The MetaTrader platform made data acquisition, programming, and optimization very easy and with a tight development loop that allowed me to try out anything I could think of. I kept optimizing for the best PnL and was sure the money machine will soon be ready for takeoff! 🚀

strategy_tester_optimization_result
I spent a lot of time looking at screens like this thinking I was sifting for gold

Unfortunately every positive result vanished when I tried with real money. Despite hundreds of hours spent hacking away, I had nothing to show for it but a greater humility towards markets. I left the attempt assured that this sort of alpha was impossible.

I accepted my mediocre returns from low-cost ETFs and did other things with my free time.

Working with Professionals

Years later, I got a job at a family office that traded livestock futures. I built them a data acquisition, databasing, and visualization system that saved them from manually entering numbers in Excel. Working directly with the traders, I collected and manipulated the data they needed to make decisions with.

I sure know more about these kinds of reports than I ever thought I would

I saw that they seemed to know what they were doing. There were severe drawdowns, but the long-term success was impressive. I was feeding them input, and they used it to extract edge out of their markets.

My faith in edge was restored.

Renewed Hunt for Edge

I collected and read books, consumed Twitter, and subscribed to Real Vision. I wrote down nearly every claim I could find in my notebook, and set out to test them or otherwise combine this into some kind of wisdom. Most of what I wrote down were unsupported claims, and many of them were very difficult to test - requiring obscure data, or subjective judgement that eluded long-term testing.

I briefly tried a some technical analysis, trading wedge patterns exclusively as a way of testing the approach as they subjectively seemed to work. Sometimes they did.

Sometimes, wedge patterns seem to work!

All of this was extremely haphazard. If I made money from any of it, it would have been impossible to distinguish this from pure dumb luck. This is a critical point.

At the time though, I thought I had some tools that worked very well. What does a gambler do when they find edge?

LEVER UP 🤘

I wrote a lot of code. I automated the statistical testing and backtesting that would guide my asset selection. I had them spit out handy spreadsheets summarizing the hours of testing that I could use for reference. It’s almost like this was similar to a job I used to do or something!

A screenshot of my statistical summary reference spreadsheet

I started trading via the futures market. I became accustomed to having my PnL swing by thousands of dollars in mere hours. Having previously worked for a futures trading firm, I knew at least a few things about risk management and I did keep my positions small. Sometimes I would hedge with options, but mostly I used something like a value at risk approach. I now know that this approach is stupid.

Twitter avatar for @ryxcommar
Senior Data Masseuse @ryxcommar
Image
11:40 PM ∙ Aug 11, 2021
3,797Likes370Retweets

Eventually hubris came to bite me. I was massively overconfident and over-leveraged. I shorted gold, for a variety of stupid reasons. Lets look at what this past-version-of-me wrote in his notebook:

2019-07-10

Short due to bearish positioning on multiple timeframes: 1H, 4H, 30 min. MACD weak.

Spiked against me when Powell said something bearish. Saw lower volume, bear wedge. Shorted more.

Stopped out based on old Kumo

In % terms, it only moved ~1.4% against me. The power of leverage amplified my loss to 20%. I held it and let it bleed me. I gave back all of my gains from trading, but no more, thanks to my hedges. I went from a Y/Y return of ~100% to 0%.

This unassuming candle was enough to blow up an idiot

Honestly I didn’t remember how stupidly I traded this until I sat down to write this. The truth is much harder for my ego to stomach. I was stupid beyond belief. Every tool I was using to trade I have since rigorously tested and found to be totally worthless.

Eventually I capitulated and went back to the drawing board.

Cleaning Out the Toolbox

My overleveraged and emotional trading did not prove that my tools did not work, necessarily. All it shows is that I was a pretty terrible trader. However, it did motivate me to dig deeper and critically evaluate my tools from a different angle.

Twitter avatar for @confirmsignal
Confirm Signal @confirmsignal
A basket full of garbage signals does not make a single good signal, unfortunately; and if you add two garbage signals to one good signal, you underweight the only thing you have that's worth a damn. Be careful with what input you feed your brain when making decisions.
12:23 AM ∙ Aug 14, 2021

A core aspect of my previous approach was ordinary conditional probability. There is a trap in conditional probability: if your sample sizes are small, your results are probably uninterpretable. For a sample size of 300, a conditional probability “edge” over marginal probability of 20% is entirely ordinary when comparing two completely random series. This was the first nail in the coffin that I buried my approach in.

I also had backtests supporting my tool use. However, as I learned in my earlier days, backtests don’t necessarily mean a damn thing. They are enough to stop you from proceeding with a strategy, but they add no positive proof.

Apparent 'Edge' via Conditional Probability When Comparing Two Random (Normal) Series 
80 
9-—60 
40 
20 
50 
100 
150 
200 
Total Size of Series 
250 
300

Having lost my primary “edge” detector, I needed a replacement. For me, that was Bayesian inference.

The Bayesian Razor

I dug into applying using Bayesian inferencing, leveraged by computing via pymc3. There are many of libraries out there for this, but Python is my go-to language and this is the dominant library in Python. This article essentially lays bare the model I use for testing, though mine is tweaked slightly for market data.

A test of a modified version of Ray Dalio’s All Season portfolio on gold, showing no benefit

I approach signal evaluation like we would approach medical testing. You say some signal (medicine) improves returns (treats a condition). Okay. Let’s compare what returns (patients) look like with and without your signal (medicine). This is broadly called hypothesis testing in statistics.

Here’s a tweet thread where I explain this in more detail:

Twitter avatar for @confirmsignal
Confirm Signal @confirmsignal
@adambaybutt @lopezdeprado Pardon my weird phrasing! Lets say you have a magic box. When it says buy, you buy. It does this maybe twice a year. It's written on the box that you must close your position after three days to profit handsomely. We want to test it, and see if it means anything.
8:15 PM ∙ Aug 13, 2021

Most of this kind of testing is done with the sorts of methods we learn in first year statistics courses. My reading has pushed me towards Bayesian methods, and I don’t honestly know when or specifically how this happened. They are far more intuitive to interpret, IMO. I think they are also best used for smaller sample sizes. Here’s a good discussion on the difference.

With this approach I can evaluate any “signal” and allow the data to separate it from noise, if it can do so. The downside is admitting our uncertainty. Low sample sizes curse us with a wide credible interval - but that does not mean there are no indications of edge available with this approach for small sample sizes.

A test of a modified version of Ray Dalio’s All Season portfolio on a utility index showing probable benefit

I also look at the same results a minimum of three different ways, to avoid fooling myself with the Bayesian approach

1
.

The below chart is the old conditional probability approach that burned me before - except now it’s overlayed with a distribution of results one would expect from random chance. If the signal advantage is significantly within or below the random edge distribution, that’s a strong count against its value.

The second approach is a scatter. If there’s something “there”, it should be apparent on a scatter diagram. It could also indicate if there is a time dependence: did this used to work, and no longer? The scatter also warns you of the fat tails of the distribution these returns come from: the below shows something that seems to work quite well, but there’s one -15% loss and several -5% losses as well. This does not include the max drawdown, which may well have been worse.

Finally, if all else is positive, I’ll check a backtest. A positive edge is worthless if it isn’t tradable.

I haven’t given up my search for edge, I’ve just raised my standards for accepting it. I am very protective of my toolbox, and I will only allow good tools in it. This new level of skepticism enables me to swing big when opportunity comes.

This leaves me back where I’ve begun. I’m fully invested in passive index-tracking ETFs, waiting and watching for an opportunity in my sweet spot.

1

I also check my traces and posterior distributions to ensure my Bayesian model is well described.

Share this post

My Analysis Approach and What Motivated It

confirmsignal.substack.com
Comments
TopNewCommunity

No posts

Ready for more?

© 2023 Confirm Signal
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing