I spoke in my last post about how the stock market is essentially a bet on the progress of humankind. I believe that companies out there will continue to innovate and provide value to individual customers and society in general, but let’s look at the statistics.
I believe that statistics is one of the most underutilized and underappreciated branches of math – in fact any field of study. You will hear people say things like “I don’t know why they made me study calculus in high school, I never use it in real life as an adult”. Fair enough, but you do use statistics – at least you should.
I remember a project at work where a some was asking me how much data they thought we should check. We had many thousands of records, so we couldn’t verify all of it manually. On the other hand, it seemed obvious that we couldn’t just check 10 random records and assume everything would be okay. I told them “Well it depends how sure we want to be”. They looked surprised and asked me “What do you mean?” I replied “Well, so you want to be 90% sure? 95% sure? 99.9% sure? I can estimate how many records we would need to check for each of those”. Despite having a college education, they were mystified by this.
The truth is that statistics can be applied almost anywhere. How many days will it rain next summer in Tokyo? How many car accidents and earthquakes will there be in 2027? And yes, how will the stock market do? Even large language models and other forms of AI are basically based on statistics. Which word is likely to follow the last one given the context? It’s used in the medical field, too. What are the chances you will develop cancer in the next 10 years? How does this change if you are a smoker?
Basically, the way we make statistical predictions is like this:
- We observe the behavior of a system and collect data
- We look at the characteristics of this data and create a model
- We use the model to predict future behavior
Statistics assumes in many cases that things are random, but most things are.
Diffusion
Take an example: You pump air into a tire. The average pressure and temperature increase. Technically, the pressure and temperature don’t need to be even – but in practice they are. Why? The air molecules bounce around chaotically and everything evens out in short order. The movement is random, but there are so many molecules that the transfer is nearly instantaneous. We don’t find situations where all of the air molecules are on one side of the tire – even though it is theoretically possible, it’s so unlikely that it never happens.
A similar thing happens if you place a drop of food coloring into a glass of pure water. It quickly spreads until after just a few seconds the entire volume of water has the same pale shade of color. Again, theoretically this need not be the case, but there are so many molecules of water and dye and so much thermal energy making them bounce around randomly that the randomness wins in short order.
If you were to make an equation to describe this, it would have a curve such that the chances of everything evening out almost perfectly would be very close to 100%. It will actually never be perfectly mixed since things are always in motion, but the chances of it getting very far away from perfect are very, very low.
The law of large numbers
Say we take something that doesn’t have millions of chances to interact, like a coin flip. A single coin flip should have almost a 50% chance of landing on heads, and a 50% chance of landing on tails. If you assume the coin is perfectly balanced and never lands on it’s side, then the percentage would actually be exactly 50%.
Still, if you flip a coin twice, there is a reasonable chance that you could get two heads in a row, or two tails in a row. In fact, there is a 25% chance.
When you get to the third flip, the chance drops to 12.5%, and with the fourth flip, it drops to 6.25%. By the time you get to 10 flips, the chance is only 0.09766%, or 1 in 1024. Make this 20 coin flips, and the chance is 1 in 1,048,576. So is it possible that you could flip the coin 20 times and have it land on heads every time? Sure. You’ll probably never see such a result, though. If you don’t believe me, just try it!
This is called the “Law of Large Numbers” and is related to “Reversion to the mean”. It’s simple: The more trials you have, the more likely you are to align with predictions based on the statistical model you have (assuming it’s correct).
A trip to the casino
The same concept applies when you are at the casino: You might play the slot machine one time and win the jackpot. It’s unlikely, but it could happen. You also might put coins into the machine for an hour and never get a single payout – but that’s equally unlikely. Casinos tune their machines so you win just often enough to keep playing and lose your money slowly, so they payout ratio might be something like 0.99 – in other words, you get back 99% of your money. This means sometimes you get back more than you put in, and sometimes you get back less – but on average it will be 99%.
The average doesn’t tell the whole story though. Imagine two different scenarios:
- For every 100 coins you put in, none of the coins pay out until you get to the 100th coin – then 99 coins come rushing out. This would have an average payout ratio of 99%, and also if you only look at trials of 100 coins, would always pay out 99% of what you put in. There would be zero variance.
- For any coin you put in, there is a 99% chance a single coin will come out – but more than one coin never comes out. You could very easily have the case that you put 100 coins in, and 100 coins come out, and also have the case where you only get 99 coins, or 98 coins, etc. As the number of coins goes down, that scenario becomes increasingly unlikely. The case where you get 0 coins back is technically possible – but practically impossible.
- For any coin you put in, any number of coins could come out. Maybe 0, maybe 1,000,000 – but the long term average is that you will get back 99% of your money. You might put in 1 coin and get back 100 sometimes, or you might put in 10 coins and get nothing. Both are very unlikely.
All of these scenarios are the same in that if you put in 1000 coins, you should expect to get 990 out on average, but the level of certainty is very different. In the first scenario, you are guaranteed to get 990 coins. In the last, you are actually unlikely to get exactly 990 coins. You might bet 800 this time, and 1100 next time, but if you did thousands of trials, the average would converge on 990 coins.
Clearly the “average” doesn’t express all there is to know, so we usually use a combination of the average and the variance, expressed as standard deviation. I won’t get into the details, but just know that the larger the standard deviation, the more variance you can expect.
One last example: Average height
We know that different people are different heights. If the average height of men is 175 cm, then that means if you measured the height of millions of men and averaged the results, you would get a result of 175 cm. Generally, the farther you get from this average, the less likely it is. For example, if you take a range of 170-180 cm, it may be that 85% of men fit into this bracket. If you change the numbers to 165-185 cm, perhaps 95% of men fit into the bracket.
In the case of height, extreme case are not really possible. We can say with confidence that no men are ever measured to be 10 cm or 3 m. Not only have we never measured such a person, but we know it to be biologically impossible. Such limitations don’t exist with things like coin tosses or diffusion, but when you are far enough away from the average, it doesn’t really matter.
The “Normal” distribution
I think everyone is familiar with the so-called “Bell curve” which was often used to adjust the scores when grading exams.
The idea is this, some people will always do better than average, and some will always do worse – but there will always be an average, and there will always be people who aren’t exactly average. The farther away you get from the average, the fewer people you expect to get that score.
For example, if the average score is 80%, you might expect that few people get 100% and 60%, and still fewer people get 0%.
Professors manipulated test scores to fit this distribution in order to account for differences in test difficulties. For example, if the average score was 95%, then they may decide the test was too easy, and so therefore 95% shouldn’t indicate an A, it should indicate a C.
Likewise, if the average test score was 60%, then 60% shouldn’t be a D, it should be a C.
Scaling the score linearly would probably work fine (and I suspect that’s what many professors did), but making it fit the normal distribution is theoretically better, since it is thought that test scores are in fact normally distributed when the sample size (number of test takers, in this case) is large enough.
This assumption may be incorrect, but it was believed to be true because so many phenomenon follow such curves.
A normal distribution is a statistical data distribution where data is evenly distributed around a central mean, and this happens most of the time in nature. It also seems to happen with most data from the stock market. That is, stock market data when viewed in volume seems similar to random patterns found in nature.
But what does standard deviation actually mean?
Well, if you assume bell curve mentioned above (which we normally do), then 68.2% of all data falls within one standard deviation of the average (mean). For two standard deviations, the number is 95.4%, and for three standard deviations, the number is 99.7%.
For a real world example: You might say the S&P has average annual nominal returns of 10% and a standard deviation of 15%. That means that based on this model, for any given year the expected return is 10%, but there is a 68.2% chance that the returns will fall between -5% and 25%.
Two standard deviations is 30%, so that means that there is a 95.4% chance that the return for any year will be between -20% and 40%.
Three standard deviation is 45%, so there is a 99.7% chance that returns will be between -35% and 55%.
That’s a pretty wild ride, but it’s a good indicator. There is only a 0.03% chance that you will lose more than 35% or gain more than 55% in any given year.
Put another way, standard deviation quantifies how far returns deviate from the average. The bigger the number, the more variance.
How valuable is a model based on past data?
I should note at this point, there are two ways to create a model:
- Create it from scratch, based on what you assert to be true. This is what we did for our coin flip experiment. We decided ahead of time that the chance on landing on heads for each coin flip was 50%
- Measure it from data in the real world. This is what we did for the men’s height experiment. We don’t know ahead of time what the average or standard deviation “should” be, we just collect data and analyze it to find these numbers.
With the stock market, we need to look at the actual data and come up with a model for practical purposes. There are limitations to this, the main one of which is: What time period should we use?
The time period connundrum
Looking at the S&P 500, there is data since 1926 – That means nearly 100 years of data! There is also 77 years of data for the Nikkei 225.
That’s a lot of data, and in general, the more data the better. For example, in determining people’s average height and the standard deviation in height, the more data we collect, the more accurate our model will be.
But… in many countries, the average height has been increasing over time.
Statistics from MEXT show that the average height of a 17 year old boy was 160.6 in 1926, and 170.8 in 2024. That’s over a 10 cm difference in average height in just 98 years.
What data should you use when creating a model for male height in Japan? Should you include all of the data since 1926? Should you include only more recent data? If so, how many years?
In the case of height, the trend rises from 1926 until around 1985, and then seems stable from there – so I would use data from 1985 or so until now. This is because in the case of height, we know there is a clear trend, and we also know that causes likely include things which aren’t likely to reverse – like better nutrition.
With the stock market, though, it’s less clear. For example, the average returns of the S&P 500 for the last 2 years have been well above 10% – but such a short term numbers tell us very little.
If you just look at the time period when COVID happened, the declines were rapid, and if you calculated standard deviation based on that alone, it would be 80%.
Likewise, if you calculated the standard deviation around the time of the Lehman Shock, it would be over 40%.
Which of these number is “right”? Well, probably none of them is exactly going to predict the future precisely – that’s the nature of random occurances.
The real question to ask yourself is “Has the basic nature of the market changed?”
Looking back, it seems clear that the advent electricity didn’t change things as drastically as one would think. In hindsight, neither did the Internet, 5G, of cryptocurrency. Businesses still exist: Manufacturing, farming, mining, you name it – they existed back then and will exist in the future. If something like smart phones or AI makes them more efficient, great. If not? Well they won’t be adopted.
The fundamental nature of business hasn’t changed. The equation is still Revenue – COGS = profit. Based on that, I see no reason not to use the full history of data at our disposal.
If we do that, then we can be fairly sure that average returns will stay somewhere above 7%, and the long term standard deviation is something like 15% – 17%.
Decline of the American Empire and USD
But what if you think America (or Japan) is in decline? What if you think the USD or JPY will be worthless in a decade.
Well, it’s certainly not impossible. In the past France, the UK, Spain and Japan had empires, but the influence of all has faded over time. Rome was once the center of the universe as far as anyone is concerned, but that’s clearly not the case now.
Japan has dropped in the GDP rankings recently, but that isn’t necessarily bad for Japanese companies. In fact, the falling Yen is good for exporters. Likewise, the situation in the United States is complicated to say the least. Various structural problems have been looming for decades, while the national debt continues to climb in both the US and Japan.
It’s not unthinkable that the relevance of Japanese and US companies fades over time with the rise of China, India, and Africa. In the short term, it’s also completely possible that the next blockbuster drug comes from a European or Korean drug company.
It’s true, investing outside of Japan shouldn’t mean just the US. The US may be the largest market in the world right now, but that may shift long term.
For example, the US had a GDP of $31.82 trillion USD in 2026, while the EU had $22.52 trillion. The means that the US economy was 41% larger than that of the EU – but the EU has actually been growing faster, so in a decade or so, the GDP of the EU may surpass that of the US.
All of this is pure speculation, of course, but that’s kind of the point.
Numerous companies were removed from the S&P 500 over the past 10 years, including Xerox, Tiffany & co, Western Union, and FLIR. At the same time, companies like NXP, Ceridian, and Enphase were added.
Enter the Global Index
Fortunes rise and fall, and just as different companies have their time in the spotlight, the same is probably going to be true for countries as well.
This is why you can invest in global index funds such as the eMAXIS Slim All World Equity All Country fund. The US currently makes up 64% of this fund, but this number will fluctuate with the fortunes of countries as time goes by. If the influence of Japan or the US wanes and developing markets continue to grow, you can participate in that success by holding such a fund. What this means in practice is that not only can you diversify your investment across companies and sectors, but even countries.
So why doesn’t everyone invest in global indexes? Well, in recent years the S&P 500 has beaten the global indexes.
- If you believe that the US will continue to outperform other countries, then you might be interested in investing in something like the S&P 500 or the Russel 2000.
- If you want to invest domestically, then there are many funds that track the Nikkei 225, TOPIX, and other benchmarks.
- If you want to diversify to the maximum extent possible and take advantage of the long term gain made by other countries, then something like an All Country fund might be a good idea.
Leave a Reply