Finding Patterns in Market Data

Finding Patterns in Market Data

| Rich Clifford | Blog
Finding patterns in market data? In 2019, a mathematician named Sarah Peluse established a proof that stated “almost all entries in the character table of the symmetric group are multiples of any given prime.” In other words, patterns are “…guaranteed to show up in every single sufficiently large collection of numbers, regardless of how the numbers are chosen.

There are patterns everywhere. That’s what she’s saying.

Select any set of numbers between 1 and 1000 and there must be a pattern to your selection. The question is… how large does a selection of numbers need to be before those patterns emerge? Or what portion of the larger set must be observed? 1%? 10%? 18.426%? There must be a number.

Peluse, while focusing on polynomial progressions, proved there must be a specific minimum number of selections made before patterns emerge. Your data set can’t be too small. It must be observable. And it needs to be representative of the whole (no number outside the set to throw you off).

In 1975, Endre Szemerédi proved patterns must exist in arithmetic progressions but was unable to prove the number of data points required to confirm a pattern. “Szemerédi basically said that complete disorder is impossible. No matter what set you take, there have to be little inroads of structure inside of it,” said Ben Green of Oxford.

In short, to find a pattern in anything, one must observe a minimum set of data of some defined size.

Why am I reading a blog about patterns and progressions and what does this have to do with the markets?

As Spinoza once said, “Nothing in Nature is random. A thing appears random only through the incompleteness of our knowledge.” Or… put another way: No market is random. A market appears random only through the incompleteness of the data being observed.

We’re often asked if we cover the crypto markets (this is not a blog about crypto… don’t worry). The short answer is “they are too small for us to trade.”  That’s true. But the more precise answer is “the data sets are too small for us to analyze.”

Our current data set for the S&P includes over 1.5 quadrillion points of data. That’s a one with fifteen zeros behind it (1,500,000,000,000,000). For the Dow, the data set is even larger with over 2 quadrillion points. For Gold, it’s 512 trillion (or half a quadrillion). Smaller markets like Soybean Oil include 60 trillion data points.

It’s a lot of data… and more than enough to develop fractals in search of infinitely repeating patterns.

By comparison, Bitcoin comes in at 12.5 trillion points of data… less than 0.806360% that of the S&P.
Ethereum has 63 billion points (0.004056% of the S&P).
XRP has 6.2 billion (0.000398% of the S&P).
And Cardano has 5.8 billion (0.000377% of the S&P).

The crypto markets are small and come with a short history. There simply is not enough data to structure a fractal… efficiently… yet.

Finding patterns in market data? And that brings us to our main point.

There are patterns in every market. But observing those patterns requires an enormous amount of data, math and physics.

On a weekly basis, we’re confronted with comments like:
“I can see patterns in the charts.” (No you cannot)
“I trade fractals in bar charts” (Fractals are not 2 dimensional)
“You just have to look at the charts and see the patterns.” (No, just no)

As Peluse proved, there must be a certain selection of data available before patterns emerge. Observing a 1 minute chart for 30 minutes gives a user 30 points of data. In that same 30 minutes, our programs have absorbed more than 47,581 points (in just the S&P). Even that quantity represents only 0.000003051% of the ever growing complete set … and the quantity of data when structuring a pattern means a lot.

“Observing” a pattern while looking at a chart is not only a wonderful way to lose money, but it’s nothing more than confirmation bias. If you’re bullish, what you “observe” is going to be bullish. If you’re bearish… what you see looks bearish.

“Finding patterns in market data” in charts is equivalent to filling up a glass with ocean water and concluding, “well, clearly there are no fish in this ocean.” The quantity and quality of data you absorb means a lot. Take a step back and consider how little of that data your current bar chart represents… and then ask yourself, what does it all mean? (did we do too far with that one?)

As stated earlier in this blog: In short, to find a pattern in anything, one must observe a minimum set of data of some defined size. Your bar chart (that one right in front of you), doesn’t have anything close to the amount of data needed to find a pattern.

Finding patterns is all we do as humans. It really is one of the only things we’re actually good at. But false positives and confirmation bias often lead us down the wrong path while the solution is often more data. If you want some help finding patterns in enormous amounts of data get in touch with our team at fractalerts.