Mean Absolute Deviation and MAD About Shaming Science

A spotlight on a replacement for standard deviation and the existential value of science

I have two things I’d like to write about this week: the first is quantitative, the second is a bit of an unrelated rant.

Mean Absolute Deviation v. Standard Deviation

In the early chapters of Nassim Taleb’s Statistical Consequences of Fat Tails the use of mean absolute deviation rather than standard deviation is promoted as a yardstick for how much variation from the mean occurs in a sample. The reason is pretty simple: standard deviation squares everything, so larger deviations have massively larger weights in the aggregate calculation. When your distribution is fat tailed this measure will be pretty misleading. Standard deviation only maps to the intuitive sense of “ordinary deviation” for the normal distribution.

Here’s a 10 minute video by Nassim Taleb discussing the same:

Mean absolute deviation is exactly what it sounds like. It’s an arithmetic average of how much the samples vary from the aggregate average. It’s probably what you’ve always intuitively thought standard deviation means.

Correlation analysis uses standard deviation at its core, and so has the same flaw regarding large deviations. If you do correlation analysis on sets that are fat tailed, you’re fooling yourself. I’m still interested in correlation though, and to that end I found a paper discussing the use of a mean absolute deviation measure for calculating something similar.

Confirm Signal @confirmsignal

A straight-forward paper by @SGorard about assessing correlation via mean absolute deviation rather than standard deviations: dro.dur.ac.uk/13873/ This is an important problem for me, since I rarely have the privilege of working with normal distributions; STDEV is misleading.

dro.dur.ac.ukAn absolute deviation approach to assessing correlation. - Durham Research Online

The results are promising! I implemented the RA1 measure mentioned in the paper against all securities in my database and found a surprising result: this measure tells you when correlated measures move in the same direction, but one moves in that same direction more.

RA1 Measure of Correlation, Full Version Available on Google Sheets

These correlation coefficients are not bounded between -100% and +100% like a Pearson R2 correlation constant. A value of 1.00 means perfect correlation. Zero means no correlation. But those values greater than 1.00 and less than -1.00 tell you that the correlated security moves with more “leverage”, essentially. That’s interesting, and not something ordinary correlation will tell you.

This is still a new measure for me, but it’s exciting to see it working well so quickly.

A Rant About The Value of Science

Warning: I unironically quote Gandalf below

First, some motivating tweets:

Jackie Champagne @WhataMergerJr

my opinion is that astronomy is a luxury field -- not a societal necessity, but good for the advancement of human knowledge. but in an era characterized by unnameable suffering around the world, it often feels futile, depressing, downright silly to worry about photometric errors.

nick saunders @nksaunders

the discourse about astronomy having no tangible benefits is back and it always stops me in my tracks. do astronomers have a moral obligation to apply their skills to stop climate/health/social catastrophes and injustice? I’ve asked myself often and don’t have a good answer.

This discussion breaks my heart. People that passionately dedicate their careers to uncovering the mysteries of the cosmos should not be ashamed. There is an undercurrent today that if you’re not actively part of the solution with your entire being, you’re part of the problem. Perhaps some of this comes from that youthful ego that says we’re going to be the ones that save the world, that our impact is going to be amazing. With great power comes great responsibility, right?

This narrative that we live in an era characterized by unnamable suffering around the world is not a fact-based statement; it’s a headline-based statement. Check out Factfulness for a thorough discussion on why things are better than we think. By nearly every measure the world is better than it was.

People are passionate about what they are passionate about. They will do far more following that passion then they will by doing things out of a sense of obligation. Individuals can still donate to charity, make ethical hiring decisions, and stand up for people in the workplace when the situation presents itself. You can do a lot of good without dedicating your life to feeding the poor.

Saruman believes it is only great power that can hold evil in check, but that is not what I have found. It is the small everyday deeds of ordinary folk that keep the darkness at bay. Small acts of kindness and love.

— Gandalf, in The Hobbit: An Unexpected Journey

Learning and teaching our place in the universe is value enough. It’s easy to get caught up in our local problems and lives, but on the cosmic scale: why are we here? Where is here? What else is out there? These questions matter.

“Pale blue dot”. Image credit: NASA.

My primary motivation for this newsletter and my Twitter is the pursuit of truth, though in the niche area of finance. Not because I think this is the most important place for truth, but because that is where I am driven. My approaches here are applicable anywhere - and if I can motivate critical thinking in this mad speculative world, then I’ve done some good. Maybe some rationality will spill out into other areas, too. That’d be fantastic.

The careful application of science, whether directed towards practical or abstract pursuits, is how we know what’s true. Science does not belong to universities or academics, it belongs to all of us who would choose rationality over madness. The pursuit of truth in any area is valuable and important, and what separates us our base animal instincts.

Don’t let shame drive you. It’s a bad passenger and a worse navigator. If you have passion for something that doesn’t harm people, feed it.