This is my collection of those. For each one I give the plain version, what is underneath it, and a source you can check. I’ve left out anything that rests only on a psychology experiment, however famous. Those can be overturned by the next replication. These can’t.

Probability and statistics

“Extraordinary claims require extraordinary evidence.”

This is Bayes’ theorem in one sentence. Your belief after seeing evidence is your belief before it, multiplied by how much more likely the evidence is if the claim is true than if it’s false. When the starting belief is tiny, that ratio has to be enormous before the result moves anywhere near “probably true”. The idea goes back at least to Laplace. Carl Sagan made this wording famous.

“Never be certain of anything you could be wrong about.”

This is Cromwell’s rule, named by the statistician Dennis Lindley. A probability of exactly 0 or 1 can’t be changed by Bayes’ theorem, whatever the evidence, because anything multiplied by zero is still zero. Certainty, in this precise sense, is the refusal to learn. Lindley borrowed the name from Oliver Cromwell’s letter to the Church of Scotland: “I beseech you, in the bowels of Christ, think it possible you may be mistaken.”

“Absence of evidence is (weak) evidence of absence.”

The popular version says the opposite, and it’s wrong. If you looked for something properly and found nothing, the probability that it exists has gone down. How far it drops depends on how likely you were to find it if it were there. A careful search that comes up empty tells you a lot. A glance tells you almost nothing. Either way, the probability can’t go up.

“Before asking how good the test is, ask how common the thing is.”

This is the base rate. A test that is 99% accurate for a condition that affects one person in ten thousand will produce about a hundred false positives for every true one. Most positive results for rare conditions are wrong, even with good tests. Science has the same problem when most tested hypotheses are false to begin with, which is the argument in Ioannidis (2005).

“Extreme results are usually followed by ordinary ones.”

This is regression to the mean, and it needs no psychology at all. Any measurement is part skill and part luck, so an extreme result is likely to include extreme luck, and luck doesn’t repeat on schedule. Praise someone after their best performance and the next one will probably be worse anyway, which teaches managers the wrong lesson about praise. Galton described it in 1886, measuring the heights of parents and children: Galton (1886).

“Look at the planes that didn’t come back.”

This is survivorship bias. During the Second World War, Abraham Wald was asked where to add armour to bombers, using damage patterns on the planes that returned. His answer was to armour the places with no holes, because planes hit there were the ones that never made it back. The full story, from Wald’s declassified memos, is in Mangel and Samaniego (1984).

“With enough chances, the miraculous is guaranteed.”

This is what Diaconis and Mosteller called the law of truly large numbers. If something has a one-in-a-million chance of happening to a given person on a given day, it happens to about ten people in Portugal every day. Somebody, somewhere, will dream of an old friend the night before that friend calls. The surprising thing would be if nobody did. Diaconis and Mosteller (1989).

“Your friends have more friends than you do.”

This is the friendship paradox, and in any network where people have different numbers of friends, it holds on average. Popular people appear in many friend lists, so when you average over friends, they get counted many times. It applies to followers, partners and colleagues too, which makes it a useful thing to remember when scrolling makes everyone else’s life look fuller. Feld (1991).

“The whole can tell the opposite story of every part.”

This is Simpson’s paradox. In 1973, Berkeley’s graduate admissions looked biased against women overall, yet most individual departments admitted women at similar or higher rates. Women had applied more often to the departments that rejected most applicants of either sex. The aggregate and the parts were both correct, and they pointed in opposite directions. Bickel, Hammel and O’Connell (1975).

“If you only see a filtered group, you’ll find links that don’t exist.”

This is Berkson’s paradox. Berkson showed that two diseases unrelated in the population can look related among hospital patients, because having either one makes admission more likely. The everyday version: if you only date people who are attractive or kind (or both), you will notice that the attractive ones seem less kind. The filter created the pattern. Berkson (1946).

Decisions, risk and time

“Avoid ruin above all. The average doesn’t happen to you.”

This is non-ergodicity. A bet can have a positive expected value across many people and still ruin nearly every individual who keeps taking it, because the average across people and the average over one person’s lifetime are different numbers. A repeated bet that loses half your money on a bad round and gains 60% on a good one looks profitable on average and shrinks your wealth over time. Kelly worked out how much to stake so that this doesn’t happen. Peters (2019); Kelly (1956).

“The longer something has lasted, the longer it will probably last.”

This is the Lindy effect, and Richard Gott gave a version of it a formal argument. If you are observing something at a random moment of its life, there’s a 50% chance you are in the middle half, so its remaining life is probably between a third and three times what has already passed. He used it in 1969 on the Berlin Wall, which fell twenty years later, inside his range. Gott (1993).

“Sunk costs aren’t costs. They’re history.”

Only what happens from here on should enter a decision: the costs you will still pay and the benefits you can still get. Money and time already spent are the same whichever option you choose, so they can’t help you choose. This one is pure decision theory. The psychology is only needed to explain why people ignore it.

“Speeding up what isn’t the bottleneck changes nothing.”

This is Amdahl’s law. If a fraction of a job can’t be sped up, that fraction sets a hard limit on the speedup of the whole, however many processors you add to the rest. If 10% is stuck, the best you can ever do is ten times faster. The management version is the Theory of Constraints. Amdahl (1967).

“Work in progress equals throughput times lead time.”

This is Little’s law, and it holds for any stable queue: a coffee shop, a support desk, a team’s backlog. If a team finishes the same number of tasks per week, starting more tasks at once can only make each one take longer. It’s a theorem, not a productivity tip. Little (1961).

Models and measurement

“All models are wrong, but some are useful.”

George Box’s line. A model is useful because it leaves things out, so being wrong in detail is the price of being usable at all. The question is never whether a model is true, only whether it is wrong in ways that matter for the decision in front of you. Box (1976). The older philosophical version is Korzybski’s “the map is not the territory”.

“When a measure becomes a target, it stops being a good measure.”

This is Goodhart’s law, in the wording Marilyn Strathern gave it. Its rigorous cousin in economics is the Lucas critique: a relationship you observed in past data will change once you start making policy based on it, because people respond to the policy. Every metric dashboard eventually teaches this lesson. Lucas (1976).

“There’s no best method for every problem.”

This is the No Free Lunch theorem. Averaged over every possible problem, every optimisation algorithm performs the same. Any method that does better on some problems does worse on others. All real performance comes from assumptions that happen to match the problem you have. Wolpert and Macready (1997).

“Of two equally good explanations, prefer the simpler one.”

This is Occam’s razor, and it has a real justification in Bayesian model comparison. A complex model can explain many different outcomes, so it has to spread its probability thinly across all of them. A simple model that predicts what you actually saw puts more probability on that outcome, and wins. David MacKay explains this in chapter 28 of Information Theory, Inference, and Learning Algorithms.

Logic and philosophy

“You can’t get an ought from an is.”

This is Hume’s guillotine. No amount of data about how things are will tell you how they should be, on its own. Values have to come in from somewhere else. It’s worth remembering whenever a chart is presented as if it settled a decision.

“No voting system is fair to everyone.”

This is Arrow’s impossibility theorem. With three or more options, no method of combining people’s rankings into a group ranking can satisfy a short list of conditions that all seem reasonable. Every voting rule gives up at least one of them. Arrow (1950).

“No hypothesis is ever tested alone.”

This is the Duhem-Quine thesis. When a prediction fails, the theory might be wrong, but so might the instrument, the measurement, or one of the background assumptions you didn’t think of as assumptions. Experiments test bundles, never single claims.

“Before removing a fence, find out why it was put there.”

This is Chesterton’s fence. Read it in probability terms: anything that has survived for a while probably solved a problem, even if you can’t see the problem yet. That doesn’t make it right. It means the burden is on you to find out what it was for before removing it.

“Invert, always invert.”

This is the mathematician Carl Jacobi’s advice, made popular by Charlie Munger. Many problems become easier if you solve the opposite one first. To avoid failure, work out in detail how you would cause it. In a project, this is the pre-mortem: assume it has already failed, and explain why.

Coda: the sides of a story

People say there are three sides to every story: yours, mine and the truth. The rigorous version combines the map and the territory with Duhem-Quine. Every account, including the one that claims to be objective, depends on an instrument: who was watching, what was written down, what was left out. So there’s a fourth side, and it’s the one to ask for first: what was actually measured.

Whatever happens, happens. It just happens with a probability.

We all love the myth of the “10x developer”, the engineer who outpaces ten others. Chasing that directly mostly produces burnout. The less glamorous route is to get 1% better each working day. A work year is about 250 working days, and 1.01^250 ≈ 12.03, so you end the year roughly 12× where you started. That’s a gain of about +1,100%, not +1,200% (a multiple of 12 means you added 11 of yourself). If you insist on improving on weekends and Christmas too, 1.01^365 ≈ 37.8×, which is the number you’ll see in Atomic Habits [1].

Before anyone emails me: no, productivity doesn’t actually compound like a savings account. Skills plateau, some gains don’t stack, and the same arithmetic works in reverse (0.99^250 ≈ 0.08). The point of the model isn’t the exact multiple. It’s that small, boring habits kept up for a long time beat heroic bursts, and small bad habits quietly do the opposite.

After years of trial and error, from nootropics to meditation to a $15 kitchen timer, my conclusion is that developer productivity has surprisingly little to do with coding technique and a lot to do with sleep, attention, and protecting your time. Here’s what worked for me.

1. Count Your Pomodori

How many hours of genuinely productive work do you get done per day?

Not hours at your desk. Not hours with your IDE open. Focused work that produces something.

For most people the honest answer is two or three. A UK survey of nearly 2,000 office workers put it at about 2 hours and 53 minutes out of an 8-hour day [2]. It’s a self-reported survey, not a lab study, but it matches what most of us would admit after a drink.

The Pomodoro Technique is a way to make those few hours count. Whether you like 25-minute sprints or 90-minute deep work blocks, the principle is the same: work in focused bursts, take real breaks, repeat.

The part people skip is taking breaks before you’re fried. When the timer ends, stop, even if you’re in flow. Stand up, get water, look at something more than a metre away. Before the next round, ask: “What’s the highest-impact thing I can do in the next block?” Then set the timer and go.

Use a physical timer. Twisting a dial and hearing it tick keeps you honest, and there’s no app to open, which means no “while I’m here, let me just check…”. A cheap kitchen timer is the single best productivity tool I’ve bought.

Over time, count focus sessions, not hours. Five properly focused 30-minute sessions beat eight hours of half-distracted “work”. Interruptions are expensive: in Gloria Mark’s observational studies, people took on average about 23 minutes to get back to an interrupted task [3]. Not all of that is lost time, but enough of it is.

2. The Biology You Can’t Ignore

Sleep

When you’re stuck on a tough problem, the worst thing you can do is brute-force it at 2 AM. The problem usually isn’t effort. It’s that you’re tired.

Sleep consolidates what you learned during the day, and it seems to help with insight too. In a well-known experiment, participants who slept between sessions were more than twice as likely to discover a hidden shortcut in a task (59% vs 23%) as those who stayed awake [4]. Everyone has gone to bed annoyed at a bug and woken up knowing exactly where it is. There’s at least some science behind that.

Something I learned late: when you feel drained and unmotivated and just can’t push further, a nap or an early night often fixes it better than another coffee. Sleep deprivation measurably disrupts the brain’s dopamine system [5], so “I have no willpower today” is sometimes just “I slept five hours”. Give yourself permission to recharge.

Don’t skimp on sleep. Everything else in this article works better on top of it.

Caffeine + L-Theanine

I’m not anti-caffeine. If there’s one “performance enhancer” I’d recommend, it’s caffeine paired with L-theanine, an amino acid found in tea.

Caffeine blocks adenosine receptors, so you feel less tired, but on its own it can bring jitters and anxiety. L-theanine takes the edge off, and the combination has been shown to improve attention-switching and accuracy more than caffeine alone in small trials [6]. A common ratio is 1:2, for example 100 mg caffeine with 200 mg L-theanine.

Timing matters more than dose. Caffeine’s half-life is around 5 hours (it varies a lot between people), so a 4pm coffee is still about a quarter-strength at midnight. In one study, caffeine taken even 6 hours before bed cut total sleep by more than an hour, and participants didn’t notice [7]. Huberman suggests stopping 8–10 hours before bed [8], which for most people means nothing after early afternoon. He also suggests delaying your first coffee by 90 minutes after waking. The evidence for that one is thin, so try it and see if it does anything for you.

Also drink water and eat lunch. Even mild dehydration has been linked to worse mood and concentration [9], and the “forgot to eat, shipped the feature” session feels heroic until 4pm, when it doesn’t.

Exercise and Morning Light

Not novel, but it works: regular exercise, even brisk walking, and morning daylight will do more for your energy and focus than any nootropic or gadget. Aerobic exercise improves mood and cognition, and in one year-long trial walking three times a week increased hippocampal volume by about 2% in older adults, roughly reversing a year or two of age-related shrinkage [10].

Getting outside for 10 minutes in the morning helps anchor your circadian rhythm, which helps you sleep, which, as we covered, helps everything.

On supplements: if you live far from the equator or work indoors all day, there’s a decent chance you’re low in vitamin D. Get it tested, and if you’re deficient, supplementing is cheap and low-risk [11]. Just don’t expect miracles if your levels are already fine. A large trial of over 25,000 adults found supplementing didn’t reduce cancer or cardiovascular events in a general population [12]. Omega-3s are similar: worth it if you never eat fish, unconvincing as a brain booster otherwise. There’s no magic pill for motivation or IQ.

3. Dopamine: The Molecule of More

Dopamine is the neurotransmitter of motivation and wanting. Some call it the “molecule of more” because it drives the pursuit of the next reward rather than enjoyment of the current one [13].

Social media notifications, a new episode, a donut, a whisky: they all deliver a hit that is far more immediate than writing code or reading docs. Feed yourself enough of those and ordinary work starts to feel dull by comparison.

Avoid Dopamine Stacking

“Dopamine stacking” is piling multiple stimuli on at once: coding while checking Twitter, blasting music, and working through coffee and sweets. It might get you through a grind, but in my experience it raises the bar for what feels stimulating, and plain work gets harder.

Music is the sneaky one. If you always work to intense music with lyrics, you may find you can’t work without it. I still listen to lo-fi or instrumental stuff, but I try not to make every task depend on it. Save the big playlist for the genuinely tedious jobs.

Doom-scrolling is the same, only worse: spikes of outrage and novelty followed by a hangover that makes focused work less appealing. The more the work itself becomes the reward (“I want to see this test go green”), the less you need to borrow motivation from elsewhere.

Rituals and Cues

Your brain is good at learning associations. If you always use the same playlist, the same tea, or the same tidy desk for deep work, those cues start to signal “it’s go time” and you settle in faster. This is plain habit formation, and it’s cheap to set up.

It works in reverse too. If you watch Netflix at your desk, your desk starts to mean Netflix. Keep your work spot for work where you can.

4. The Mental Game

Meditation

I used to think meditation was new-age fluff. Then I tried it and found it’s mostly attention training: sit, focus on your breath, notice when your mind wanders, bring it back. Ten minutes a day is enough to notice a difference.

The bigger benefit for me is noticing a feeling without obeying it. Developers deal with a lot of impostor syndrome, anxiety, and frustration. Meditation is practice for feeling anxious and still writing the code, or feeling frustrated and still debugging methodically.

You may have read that meditation grows grey matter. The early studies were small, and a larger randomised trial found no measurable structural brain changes after an 8-week course [14]. I’d do it for the attention training and leave the brain-scan claims alone.

Emotions Come in Waves

However stoic you think you are, your mood will go up and down. Productivity isn’t about never feeling down. It’s about not letting the down days sink the week.

When I’m in a funk or panicking about a deadline, I sometimes literally tell myself: “Fine, I’m anxious. I’m going to chip away at this for 15 minutes anyway.” Giving yourself permission to feel bad while working often makes the feeling pass faster.

Also expect the crash after a big push. If you’ve been sprinting on adrenaline to finish a feature before a demo, you’ll probably feel flat afterwards. Plan a lighter day rather than pretending peak output is sustainable.

Trust the Process

The more you panic about being stuck, the longer you tend to stay stuck. Acute stress impairs the prefrontal cortex, the part you need for working memory and reasoning [15], so panic is the worst state to debug in.

It helps to keep a short mental list of hard bugs you’ve solved before. Not as self-help, but as evidence: “This is tricky, but I’ve done tricky before. Break it down.”

When you’re truly stuck, step away. Walk, shower, cook, or sleep on it. I’ve lost count of how many times the answer turned up while I was making dinner.

Rubber duck debugging (explaining the code line by line to an actual rubber duck) and a second pair of eyes, human or AI, work for the same reason: explaining the problem clearly often reveals the answer.

5. Learn the Craft, Not Just the Hot Tools

This is career advice more than daily productivity, but it pays off massively over time. The best developers I know keep learning, and not just whatever framework is trending on Hacker News this week.

Studying algorithms, data structures, design patterns, and complexity can feel unproductive. It gives you a toolkit that makes you faster at everything: you recognise problems you’ve seen before and stop reinventing wheels.

Learn why popular frameworks and architectures exist, what problems they solved, and what they traded away. That’s how you judge new tools instead of chasing silver bullets.

And widen the net a bit: some ops, raw SQL if you only know ORMs, some functional programming if you only know OOP. Each new paradigm gives you another way around a wall.

Sharpen the axe. It saves hours on dead ends and avoids the big refactor later, which is the real productivity killer.

6. Use Tools, Including AI, But Keep Thinking

Good tools boost your output a lot. Use them for the dull parts, not to skip understanding the problem.

AI coding assistants are great for boilerplate in a framework you don’t know well, regexes, format conversions, test scaffolding, and the “how do I do X in Y again” questions that used to mean another StackOverflow tab.

But read what they produce. You’re still responsible for the code you ship, however it was written. If an assistant writes a complex SQL query, check that it does what you meant. If you don’t understand the output, that’s a red flag. Blind copy-pasting is how subtle bugs and security holes get in.

The other trap is over-automation. We love automating things, and usually that’s good. But don’t spend 4 hours automating a 5-minute monthly task. I’ve been guilty of this, tweaking dotfiles for an afternoon to save two shell commands. It’s fun, but be honest about whether it’s productive.

My rule of thumb: if it’s repetitive and you’ll keep doing it, automate it. If it’s rare, just do it. You can automate later if it becomes a pain.

7. Protect Your Time

This might be the most important one. Your most valuable asset as a developer is long, uninterrupted stretches of time to think and build.

That means setting boundaries. Two big offenders:

Meetings. Everyone has sat in a meeting that should have been an email. Atlassian’s widely shared estimate is that about half of meeting time is wasted [16]. Where you have a say, propose shorter meetings or async updates. If one drags, ask, “Can we agree next steps and get back to work?” People are usually relieved someone said it.

If you freelance, negotiate fewer meetings upfront: “To get the most done for you, I keep meetings to a weekly check-in and handle the rest over Slack or email.” Most clients are happy to pay for output rather than calendar time.

Notifications. Constant pings kill deep work. Close Slack or Teams for an hour if your culture allows, or at least mute the non-urgent channels. The cost isn’t the 30 seconds to read the message. It’s getting back into the problem afterwards.

Block “Focus” time in your calendar and treat it like a meeting with your code. Plenty of teams have rules like no meetings before noon. If yours doesn’t, you can often run that policy for yourself.

As your career progresses, protecting your time, for work and for life outside it, becomes most of the game. That might mean saying no, or pushing back on a deadline by explaining the trade-offs. You can usually turn time into money. Turning money back into time is a lot harder.

8. Nootropics and “Hacks”: Proceed with Caution

I’ve tried a fair few so-called nootropics: racetams, modafinil, assorted supplement stacks from Silicon Valley startups. The unsexy lesson is that homeostasis usually wins. If something gives you a big boost, your body tends to adapt, you need it to feel normal, and when you stop you dip below where you started.

I once ran a stack that made me feel like a genius for a month. The next month I was foggy without it. When I mixed that regimen with ordinary social drinking, I had odd memory lapses, like forgetting where I’d parked the car. That was my cue to stop.

Caffeine is the one that reliably works, as long as you manage the timing. Everything else comes with trade-offs. Everyone’s brain chemistry is different and I won’t tell you not to experiment, but be sceptical. Sleep, exercise, food, and stress management move the needle most. If those aren’t in place, no pill will save you. If they are, you probably won’t feel the need.

Nicotine deserves a special mention. Yes, it’s a stimulant, and some famous scientists and writers swore by it. It’s also very addictive and easily becomes a crutch with a nasty health bill attached. The same goes for heavy drinking or anything else that wrecks your sleep and the next day.

Your body and brain are the hardware. Treat them accordingly.

The Compound Effect

None of this is dramatic on any single day. A 1% improvement is invisible by Friday. The hard part is staying consistent long enough for it to add up. Once it does, it feels like momentum, and things get easier.

If I had to pick the habits that made the biggest difference for me, they’d be good sleep, regular exercise, focused work sessions, and continuous learning. None of them is a secret, which is exactly why most people skip them.

And keep it fun. The point of productivity isn’t to become a churn-and-burn coding robot. It’s to cut out the unnecessary grind so you enjoy the work and still have energy left for the people and things you care about outside it.

Got a productivity habit that made a real difference for you? Tell me. I’m still working on my own 1% a day.

Sources

1. Clear, J. (2018). Atomic Habits. Avery. Chapter 1 uses the 1.01^365 ≈ 37.78 illustration.

2. World Economic Forum / Business Insider – “The case for the 3-hour workday” (2017). Cites a UK survey of 1,989 office workers reporting an average of 2h53m of productive work in an 8-hour day.

3. Mark, G., González, V. M., & Harris, J. (2005). No task left behind? Examining the nature of fragmented work. CHI ‘05, 321–330. doi:10.1145/1054972.1055017

4. Wagner, U., Gais, S., Haider, H., Verleger, R., & Born, J. (2004). Sleep inspires insight. Nature, 427, 352–355. doi:10.1038/nature02223

5. Volkow, N. D., et al. (2012). Evidence that sleep deprivation downregulates dopamine D2R in ventral striatum in the human brain. Journal of Neuroscience, 32(19), 6711–6717. doi:10.1523/JNEUROSCI.0045-12.2012

6. Owen, G. N., Parnell, H., De Bruin, E. A., & Rycroft, J. A. (2008). The combined effects of L-theanine and caffeine on cognitive performance and mood. Nutritional Neuroscience, 11(4), 193–198. doi:10.1179/147683008X301513

7. Drake, C., Roehrs, T., Shambroom, J., & Roth, T. (2013). Caffeine effects on sleep taken 0, 3, or 6 hours before going to bed. Journal of Clinical Sleep Medicine, 9(11), 1195–1200. doi:10.5664/jcsm.3170

8. Huberman, A. (Huberman Lab Newsletter) – “Use Caffeine for Mental & Physical Performance” (2024). Recommends avoiding caffeine 8–10 hours before bedtime.

9. Ganio, M. S., et al. (2011). Mild dehydration impairs cognitive performance and mood of men. British Journal of Nutrition, 106(10), 1535–1543. doi:10.1017/S0007114511002005

10. Erickson, K. I., et al. (2011). Exercise training increases size of hippocampus and improves memory. PNAS, 108(7), 3017–3022. doi:10.1073/pnas.1015950108

11. Harvard Health Publishing – Komaroff, A. L., “Will vitamin D supplements keep me younger?” Harvard Health Letter. Discusses who benefits from vitamin D supplementation.

12. Manson, J. E., et al. (2019). Vitamin D supplements and prevention of cancer and cardiovascular disease. New England Journal of Medicine, 380, 33–44. doi:10.1056/NEJMoa1809944

13. Lieberman, D. Z., & Long, M. E. (2018). The Molecule of More. BenBella Books.

14. Kral, T. R. A., et al. (2022). Absence of structural brain changes from mindfulness-based stress reduction: Two combined randomized controlled trials. Science Advances, 8(20), eabk3316. doi:10.1126/sciadv.abk3316

15. Arnsten, A. F. T. (2009). Stress signalling pathways that impair prefrontal cortex structure and function. Nature Reviews Neuroscience, 10, 410–422. doi:10.1038/nrn2648

16. Atlassian – “You waste a lot of time at work” (infographic), widely cited in employee productivity roundups.

Most retail traders aka “dumb money” don’t diversify.

Contrastingly, professional portfolio managers diversify:

1. across markets;
2. across sectors;
3. across asset types;
4. across investment strategies.

Professional portfolio managers are more afraid of the occasional downswings and crashes than retails traders are. And rightly so. The best managers monitor their exposure and seek to re-balance it to according to their perceived optimum portfolio weights. They understand that diversification is key to minimizing risk.

The most popular, most used model for calculating the optimal portfolio weights is the Markowitz model and the Efficient Frontier.
This is now considered ancient technology.

There are 2 main issues with the Markowitz Model:

1. Markowitz is always overfitting the training data..

   Markowitz’s result was an optimization process for the weights that would have been profitable in the past. Markowitz model is not learning a set of rules for predicting the future. Instead, this model can be summed up as “Has this portfolio worked well in the past 10 years? Then it is sure to work well for the next 10!”. You can change the 10 year for whatever timestep you like - you’ll still be overfitting.

2. It finds the weights that maximize Sharpe ratio. Sharpe ratio sucks.

   “What?! But everybody is using Sharpe ratio!”

   Using Sharpe for showcasing portfolio metrics to clients is ok; using it for daily quant work is not ok. Sharpe ratio’s denominator is the standard deviation of returns, which is a good measure of variability, not a good measure of risk. Sharpe ratio penalizes large positive swings, penalizes accelerating returns, and fails to penalize deaccelerating returns. For experimental evidence, see: https://www.crystalbull.com/sharpe-ratio-better-with-log-returns/

Enter Machine Learning.

There are several ways we can frame the problem of portfolio optimization as a Machine Learning problem:

1. Reinforcement Learning: learning the optimal increase/decrease in portfolio weights.

2. Supervised Learning: learning the optimal portfolio weights for the next N days/months/years.

3. Unsupervised Learning: learning clusters of assets, according to their price and fundamentals similarity.

Today, I will focus on the latter problem. Learning groups of assets can identify how to diversify according to historical data.

We will be starting with the following dataset:

cd ..

import pandas as pd

data = pd.read_csv("data/stocks.csv")
data

Each line describes an instance. Each instance contains quantitative information about a given stock for a given year, as well as its GICS sector, industry_group, industry, and sub_industry.

We select the top50 industries with that contain the most instances:

top50 = data.gics_industry.value_counts().iloc[:50]
top50

Machinery                                         1379
Oil, Gas & Consumable Fuels                       1312
Chemicals                                          916
Specialty Retail                                   899
Energy Equipment & Services                        779
Aerospace & Defense                                634
Health Care Equipment & Supplies                   582
Electronic Equipment, Instruments & Components     485
Hotels, Restaurants & Leisure                      480
Food Products                                      463
Metals & Mining                                    452
Commercial Services & Supplies                     424
Health Care Providers & Services                   387
Containers & Packaging                             362
Textiles, Apparel & Luxury Goods                   302
IT Services                                        301
Construction & Engineering                         272
Professional Services                              270
Household Durables                                 255
Building Products                                  250
Auto Components                                    250
Media                                              241
Life Sciences Tools & Services                     226
Multiline Retail                                   223
Pharmaceuticals                                    221
Household Products                                 203
Electrical Equipment                               203
Diversified Consumer Services                      197
Trading Companies & Distributors                   194
Equity Real Estate Investment Trusts (REITs)       170
Food & Staples Retailing                           167
Diversified Telecommunication Services             166
Technology Hardware, Storage & Peripherals         160
Road & Rail                                        157
Leisure Products                                   149
Capital Markets                                    137
Personal Products                                  136
Automobiles                                        134
Paper & Forest Products                            125
Industrial Conglomerates                           125
Entertainment                                      118
Software                                           115
Marine                                             106
Tobacco                                            103
Beverages                                          101
Airlines                                            90
Construction Materials                              81
Real Estate Management & Development                80
Gas Utilities                                       65
Air Freight & Logistics                             63
Name: gics_industry, dtype: int64

And filter our data so that it only contains those GICS industries.

data = data[data.gics_industry.isin(top50.index)]

We then standardize the numerical information:

quantitative_vars = data.select_dtypes('number').columns
quantitative_data = data.loc[:, quantitative_vars]
data.loc[:, quantitative_vars] = (quantitative_data - quantitative_data.mean()) / quantitative_data.std()

We can now calculate and visualize the dissimilarity between industries, measured by the Maximum Mean Discrepancy between the samples of the different industries).

from qclustering import dissimilarity_matrix
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

dmatrix = dissimilarity_matrix(data, 'gics_industry')

fig, ax = plt.subplots(figsize=(30,30))
cbar_kws = cbar_kws={'shrink': .8, 'label':'Distance'}
sns.heatmap(dmatrix, ax=ax, vmin=0, square=True, cmap='Blues', cbar_kws=cbar_kws)
fig.axes[-1].yaxis.label.set_size(23)
fig.axes[-1].tick_params(labelsize=18)

We can also frame this as a Hierarchical Clustering problem and use MMD as a linkage metric between industries and clusters of industries:

from qclustering import hierarchical_clustering, plot_dendrogram
initial_clusters, linkage = hierarchical_clustering(data, 'gics_industry')
plot_dendrogram(initial_clusters, linkage, color_threshold=0.09, above_threshold_color='#CCCCCC');

Based on the chart above, we can see that some industries are being clustered with industries of different sectors. Therefor, the data indicates that we’re better off diversifying by industries than diversifying by sectors.

This concludes Part 1.

In Part 2, we will look at ML-based diversification strategies at compare their foward testing / out-of-sampling results, including some methods from the MlFinLab python package.

pip install jupyter_to_medium
jupyter bundlerextension enable --py jupyter_to_medium._bundler --sys-prefix

2. Get your integration token:

Get your integration token by going to your Medium Account’s Settings > Integration Token.

3. Set your integration token:

mkdir -p ~/.jupyter_to_medium # make directory if missing 
echo "YOUR-INTEGRATION-TOKEN" > ~/.jupyter_to_medium/integration_token

All done!

The next time you restart Jupyter Notebook, you will be able to deploy your notebooks straight to Medium!

PS: This medium article has been generated from this Jupyter Notebook using jupyter_to_medium.