During a walk through the ruins of the San Francisco Dam Disaster Site, about 65 km from downtown Los Angeles, with my archaeologist friend, John, we talked about the stained life of its builder and the age of the “Gentlemen Scientist.”
The San Francisco Dam was built between 1924 and 1926 to create a giant garage tank for the city of Los Angeles, California, through the Office of Water Works and Supply, now the Water and Energy Decompose. The decomposition under the direction of its chief general manager and leading engineer, William Mulholland. If you’ve ever noticed the vintage movie, “Chinatown,” William Mulholland is such a vital component of the Los Angeles story that they had to split it into two characters.
While a legend in his day, Mulholland is not a civil engineer by today’s standards. He was self-taught in his early days as a “sweet” to the water department. After a hard day’s work, Mulholland will examine the textbooks of mathematics, engineering, hydraulics and geology. This original story is the basis of the “Gentlemen Scientist” character: it devours all the curtains on a subject and then claims an understanding that would allow them to oversee a large company, despite any form of testing or certification.
If I went to NASA and said I was qualified to send humans to Mars because I read a lot of books about the area and used to build style rockets as a kid, they’d turn me off the property. In the days of Mulholland, this meant an ascent to run the department.
Mulholland is an integral component of Los Angeles history. While many of his early efforts literally replaced the Los Angeles landscape (he oversaw the design and structure of the Los Angeles Aqueduct, which brought water to much of the county), his lack of fashionable civil engineering “is one of the worst American civilians. “20th-century engineering disasters,” according to Catherine Mulholland, in her biography of William Mulholland, her grandfather.
Minutes earlier, on March 12, 1928, the dam failed catastrophically and the resulting flood caused the deaths of at least 431 people, however, some reports claim to be as high as a thousand. Even with the smallest number, the San Francisco Dam Cave remains the largest loss of life in California history. Only the 1906 earthquake and chimney in San Francisco killed more people.
The discussion with my friend that day made me aware of the search engine optimization activity and his collection of “Gentleman Scientists”.
Instead of building dams, our colleagues are looking to redesign complex search engine algorithms like Google through misconception practices to design benchmark methods backed by poor quality science.
For decades, legions of search engine optimization professionals have claimed to have “tested” other theories about Google’s algorithms through highly questionable practices. At first, that evidence referred to a self-proclaimed mad scientist who replaced a facet of a single internet page and then waited for the next Google Dance to see if his website was progressing in a search engine index. If it worked, they posted an article about the effects on a forum or on their websites. If the poster were popular enough, the search engine optimization network would reflect your new “hack” until Yahoo, Google or one of the first search engines told them to avoid or figure out how to prevent it from falling into their algorithms.
The first legends of search engine optimization were born from this type of activity.
Finally, corporations like Moz, Ahrefs and SEMrush have discovered tactics to reflect Google’s index, the “tests” or “studies” they have conducted have had a much more valid facet due to access to much larger knowledge sets. Google closed these theories with the old and appropriate reaction “Correlation does not amount to causation”; However, the maximum of these erroneous statements have survived under the banner of “Trust but verification”.
My long-standing position on this factor stems from the fact that Google’s multiple algorithms load from knowledge problems to create a World Wide Web index of billions of Internet pages. With something so sophisticated, are the professionals of maximum optimization of qualified search engines to “test” Google using our limited understanding of statistics?
With rare exceptions, which will actually be highlighted once this article is published, the maximum of other people running in search engine optimization are novice statisticians who, at best, have taken the typical courses and have retained more than the maximum. Some colleagues have a deeper understanding of statistics, but are not yet statistical or mathematical, but have acquired their mathematical talents in examining other sciences accustomed to less complex data. In most cases, the statistical systems they use are used to analyze surveys or media purchase forecasts. They are not for giant complex systems discovered in search engine algorithms and the data they organize.
I’ll be the first to admit that I’m not a mathematician or a statistician. I struggled with math in school long enough to complete my college studies and didn’t feel comfortable with everything before I graduated. Even then, in the popular elegance of trade statistics that other people suffered when searching for their MBA.
Just as when I worked with actual intellectual property lawyers for my article on the legality of Google’s Featured Snippets, I sought out an actual statistician. Most importantly, I needed someone who doesn’t work in the SEO space to avoid any observer bias, that is, someone who would subconsciously project their expectations onto the research.
My search led me to the statistician, Jen Hood. Jen studied mathematics and economics at Virginia’s Bridgewater College, and for most of the 15 years she has been working as a statistician. She was a data analyst for Volvo. Since 2019, she has been working as an analytics consultant at her company, Avant Analytics, mostly helping small businesses that wouldn’t usually have an in-house analyst.
We explained how the maximum of studies around search engine optimization were based on the concept of statistical correlation in our early discussions. Statistical correlation shows whether, and to what extent, variable pairs, such as the secure facets of an Internet page and the position of that page on Google search engine effect pages, are linked.
“The vast majority of statistical work, even forecasting the future, revolves around measuring correlation,” Jen says cautiously. “However, causation is incredibly difficult to prove.” Causation is the action of causing something to happen; that is, the real reason things work the way they do.
“Without knowing the details of how any of these companies create their metrics, I’m suspicious there’s a significant amount of confirmation bias occurring,” Jen continued. Confirmation bias happens when the person performing an analysis wants to prove a predetermined assumption. Rather than doing the actual work needed to confirm the hypothesis, they make the data fit until this assumption is proven.
To give Jen a better idea of how these corporations were generating her data, I shared some of the most popular search engine optimization studies in recent years. Some of the proclamations made in these studies have been refuted through Google several times over the years, others persist on Twitter, Reddit and Quora and are the topic of discussion about what appears to be a daily basis.
“The confirmation bias error looks like in those reference articles,” Jen says immediately. “This is not unusual in all the subjects where someone tells you how to gain an advantage.”
First, Jen reviewed an exam submitted through Rob Ousbey at Mozcon 2019, when she applied for Distilled (lately works for Moz) on the search engine optimization verification platform, then called Distilled ODN, now the spin-off of seekPilot. Among the theories presented that day, one stated that the effects of page 1 of the search engine’s effect pages are motivated more through engagement with those pages than through links. Jen begins to suspect immediately.
“With the data available, it’s hard to tell if Rob’s theory on the first page of effects is motivated by engagement and the following link-motivated effects are accurate,” Jen wrote after reviewing the presentation. “This concept that those are basically links [main search effects to page 2] is a bit given that there are so many points that go to the ranking.”
“The easy test would be: if you can rank on Page 1, especially the top of the page, without previously having any engagement, then the engagement is most likely driven by placement, not the other way around.”
I reached out to Will Critchlow, founder, and CEO of Distilled. He offered another study by a former colleague of Rob Ousbey, Tom Capper, that provided a deeper dive into the material that Rob presented back in 2019. “Tom approached this from a few different angles – but the short answer is no – this is not just because top results get more interaction because they are top results.”
“[Tom provided] other types of evidence,” Will continued, “one is that the links have a higher correlation with the decrease in seRP ratings than on the first page (and especially for higher volume keywords)”.
“Other evidence includes how grades replace when a question moves from a low-volume search expression to a number one term (for example, a very sharp volume),” Says Will, referring to a search for the search term, “Mother’s Day Flowers. ” “
“It keeps getting more interesting,” Jen writes after reviewing the new information. “These new [knowledge] are components of actual correlation values but in an absolutely smaller and much smaller pattern in UK knowledge: only 4,900 queries in two months.”
Before we continue, it’s crucial to understand how correlation studies are supposed to work.
There are multiple ways to measure the relationship, or correlation, between two factors. Regardless of the method, the numbers returned from these calculations measure between -1 and 1. A correlation of -1 means as one factor goes up, the other factor goes down every time. A correlation of 1 means as one factor goes up, the other factor goes up every time. A correlation of zero means there is no relationship – no predictable linear pattern, up/down, up/up, down/up, or otherwise.
“Most correlation coefficients (results) aren’t close to 1 or -1,” Jen clarifies. “Anything at +/-1 means that 100% of the variation is explained by the factor you’re comparing. That is, you can always use the first factor to predict what the second factor will do.”
While there is no rule that a correlation is strong, weak, or somewhere in between, there are accepted thresholds, which Jen describes. “Considering that we can have values for accounting factors without problems, such as the number of links on a website and the rating of that website on Google, the maximum correlation would be 0.7 to 1.0, moderate would be 0.3-0.7, and low would be 0-0.3”.
“Someone can just challenge those precise groups,” Jen acknowledges, “even though I made a mistake in the aspect of generosity by the strength of correlation.”
We’ll go back to the test. “Tom’s slides basically refer to a February 2017 presentation that he made about whether Google still wants links. There is also a referenced Moz exam that, at this stage, dates back five years” (Jen stops here to say, “By the way, I find it attractive that everyone recognizes that algorithms have undergone significant adjustments and yet refer to studies dating back two, three years or more.”
“In this, [Tom] examines the dating between the domain authority and the ratings,” referring to Moz’s metric, which is the cornerstone of incoming link reporting tools. “Provides the correlation between the domain authority and Google’s rating of a website: 0.001 for positions 1 to five and 0.011 for positions 6 to 10.”
“This that the domain authority is more strongly correlated with the search engine rating for positions 6 through 10, but both effects are very weak correlations,” Jen paused to make sure I understood.
“To put it simply, for positions 1 to five in Google results, the domain authority can be used for 0.1% of the SERP classification variation. For positions 6 to 10, this is 1.1% of the variation of the SERP classification “, clarifying its point.
“This shows that domain authority is not as important to high-level positions. However, the correlations for both are so incredibly weak that they make almost no sense,” Jen says enthusiastically through the discovery. At the same time, I know how many domain names and links are purchased and sold using this metric. “Elsewhere, it mentions 0.023 and 0.07 as correlation coefficients for authority and domain rating in the 10 most sensitive positions, which makes no sense since their past values are lower.”
Jen leads the explanation to close the cycle: “As this is the most technical support detail provided by the company, it is moderate to think that the correlations in the original exam you sent me are of a similar level.” In other words, even if we don’t have the numbers from Rob Ousbey’s original presentation, they have an equally weak correlation.
“The Mother’s Day exam is very anecdotal,” Jen continues, “The effects are attractive and raise doubts about the involvement this can have for other terms of study. However, this is a term from studies studied over a month. The content of this exam is sufficient to bring out universal implications”.
“Good for a sales pitch; bad for a statistical study,” says Jen. “During this time, I have not yet noticed anything that shows how they have shown that the most productive effects do not get more interaction because they are the most productive result.”
“There are many examples presented in other slides of the claims, but there is no in-depth study.” Jen refers to some of the other studies provided in Rob’s original presentation through Larry Kim, Brian Dean and Searchmetrics.
“[Larry Kim’s examination of the influence of click-through rate on ratings] suggests that a decrease in click-through rate leads to a decrease in ranking. However, this may be the lowest ranking with the lowest click-on click rate,” jen explains, illustrating an unusual fact. paradox with this kind of data. “I would completely expect a high correlation between the page rating and the click-through rate just because more people have the opportunity to participate.”
“Does the bounce rate the search position or vice versa?” Jen asks, moving to another slide that refers to an exam through Backlinko’s Brian Dean claiming that the bounce rate metric influences the position of search results. “I find it attractive that the story looks different if you actually access the source data.”
Jen refers to the original Backlinko exam in which the chart used in Rob’s presentation was drawn, which read: “Note that we are not suggesting that low rebound rates lead to higher grades. Google can use the bounce rate as a rating sign (although in the past it refused to do so). Or it may just be the fact that high-quality content helps keep others more engaged. Therefore, a decrease in bounce rate is a byproduct of high-quality content, which Google measures. “
He concludes: “As this is a correlation study, it is highly unlikely that you will only realize our knowledge,” demonstrating Jen’s interest in publishing these studies.
Jen concludes firmly: “The use of this chart is deliberately misleading.”
“[These studies are] just looking at one factor. With multiple algorithms in place, there must be many factors all working together. Each must have individual ratings that are weighted into a total for the specific algorithm and likely weighted again within the aggregating algorithm they use.” Jen states, mirroring something that Google’s Gary Illyes and John Mueller has said more than once at various conferences and on Twitter and something this publication’s own Dave Davies has recently discussed.
Because of this acknowledged complexity, some SEO studies have abandoned correlation methods entirely in favor of machine learning-based algorithms, such as Random Forest. A technique a 2017 investigation by SEMrush uses to propose top-ranking factors on Google, such as page traffic and content length. “This is a good approach to predict what’s likely to happen,” Jen writes after reviewing the SEMrush study and its explanation of its methodology, “but it still doesn’t show causation. It just says which factors are better predictors of ranking.”
Most of the research around search engines that is issued comes not from independent sources or educational institutions, but from companies selling tools to help you with SEO.
This kind of activity by a company is the ethical equivalent of Gatorade proving its claims of being a superior form of hydration for athletes by referencing a study conducted by The Gatorade Sports Science Institute, a research lab owned by Gatorade.
When I told Jen Hood how many studies she reviewed have resulted in new rules or completely new products, she was surprised that someone took those measurements or products seriously.
“Anyone claiming that they have a metric which mimics Google is asserting that they’ve established many cause-effect relationships that lead to a specific ranking on Google,” Jen wrote, referring to Moz’s Domain Authority. “What this should mean is that their metric consistently matches with the actual results. If I started a brand-new site or a brand-new page today and did everything that they say is an important factor, I should get a top ranking. Not probably rank high. If there’s a true match to the algorithms, the results should always follow.”
Jen provides a hypothetical example:
“Suppose I offer a service where I will tell you precisely where your website will be classified for a particular search term based on a statistic that I include in that service. I have a formula to calculate this metric to be able to do it for many other sites. If I can tell you precisely where it would be based on my formula 0.1% of the time, would you feel that my formula understands Google’s algorithms? If I raise that figure 1.1% of the time, could I now feel safe?
“It seems like all those studies [and products] seem to be doing,” says Jen. “Hide in enough statistical terms and main points to give the impression that it is much more significant.”
* * *
As Jen alluded to earlier, most studies of Google’s results are using a limited amount of data, but claiming statistical significance; however, their understanding of that concept is flawed given the nature of the very thing they are studying.
“Rand says he estimates that Jumpshot’s data contains ‘somewhere between 2-6% of the total number of mobile and desktop internet-browsing devices in the U.S., a.k.a., a statistically significant sample size,’” Jen is referring to a 2019 study by SparkToro’s Rand Fishkin that claims that less than half of all Google searches result in a click. “Rand would be right about statistical significance if the Jumpshot data were a truly random and representative sampling of all Google searches.”
“From what I can find, [Jumpshot] collected all its knowledge from users who were Avast antivirus,” referring to the now closed parent company of the service. “This set of users and their knowledge probably differs from all Google users. This means that the pattern provided through Jumpshot is not random and is probably not representative enough, an old sampling error commonly known as availability bias.”
“Non-context statistics deserve to be taken with a grain of salt. That’s why there are experts in analysis to ask questions and give context. What kind of questions do other people ask and how have they changed? Jen said, delving into the premise of studying.
“For example, other people who are looking for topics for which there is no additional price to access some other online page will probably not miss opportunities for those who miss clicks. Users without delay refine their search term because the set of rules did not capture the context of what they asked for? Jen suggested, or anything that Rand then clarified as a component of his statement as to why clicks on effects take place in more than the effects component. “Now we are increasingly nuanced, however, if Rand claims that clickless searches are bad, then there must be a context explaining why this could happen even in the absence of a [selection extract].”
* * *
If the concept of using data too thin to be accurate isn’t damning enough, there’s the problem that there’s no concept of peer review within the SEO industry. Most of these studies are conducted once and then published without ever being replicated and verified by outside sources. Even if the studies are replicated, they are done by the same people or companies as a celebrated annual tradition.
Of all the historical studies of the St. Francis Dam Disaster, one by J. David Rogers, Ph.D., Chair in Geological Engineering, Department of Geological Sciences & Engineering and professor at Missouri University of Science and Technology, stands out to me. He stated one of the critical reasons for the failure: “The design and construction being overseen by only one person.”
“Unless the results are life and death or highly regulated, we don’t normally see people doing the actual work required to show causation,” Jen Hood adds. “The only way to really show causation is to have a robust study that randomizes and controls for other factors on the proper scale. Outside of clinical drug testing, which normally takes years, it’s very uncommon to see this taking place.”
How the SEO industry conducts and presents its research is not how scientific studies have been administered since the 1600s. You don’t have to believe me. I’m not a scientist, but Neil deGrasse Tyson is.
“There is no truth that does not exist without experimental verification of that truth,” said Tyson in an interview with Chuck Klosterman for his book, “But What If We’re Wrong”. “And not only one person’s experiment, but an ensemble of experiments testing the same idea. And only when an ensemble of experiments statistically agrees, do we then talk about an emerging truth within science.”
The standard counter to this argument is just to state, “I never said this study was scientific.” If that’s so, why does this information get shared and believed with such conviction? This is the heart of the problem of confirmation bias, not just with the researchers but also with the users of that research.
“[I]f you really think about what you really actually know, it’s only a few things, like seven things, maybe everybody knows,” comedian, Marc Maron, is talking about the concept of knowledge in his stand-up special, “End Times Fun”. “If you actually made a column of things, you’re pretty sure you know for sure, and then made another column of how you know those things, most of that column is like, ‘Some guy told me.’”
“You know, it’s not sourced material, it’s just – it’s clickbait and hearsay, that’s all,” Maron continues. “Goes into the head, locks onto a feeling, you’re like, ‘That sounds good. I’m gonna tell other people that.’ And that’s how brand marketing works, and also fascism, we’re finding.”
Science has focused on understanding how the physical world has painted since the time of the aristocracy, that most other people now agree, was wrong about many things. Scientists want to make those efforts because there is no manual for our planet or anything else in the universe. We can’t make a stop at a random deity in the workplace and ask him why he made gravitational paints like he does.
But with Google and search engines, we have that access.
I hate to turn to “Because Google said so!” However, unlike the ultimate science, we can get notes from the working hours advertised by The Creator and occasionally from Twitter.
John Mueller’s next pre-year tweet was in reaction to another correlative review published through another search engine optimization company without any outdoor corroboration, claiming to have unlocked Google’s secrets with a limited amount of data.
He has also created complex algorithms on a giant scale: he knows it is never a single calculation with static multipliers. These things are complex and replaced over time. I locate those desirable reports, who would have the idea X? – However, I’m afraid other people think they are useful.
– ? John ? (@JohnMu) 28 April 2020
John Mueller and I express a very clear vision of the presentation of this kind of knowledge: “I’m afraid other people think they are useful,” that is, that this knowledge is not entirely useful and even potentially misleading.
The above came here after the studio’s author, Brian Dean, said the report “was more intended to shed some attention on how some of Google’s rating points might work.”
Claims like this are a popular variant of a typical mea culpa when a search engine optimization study examines is dismissed as incorrect. “I never said it was a Google rating factor, but there was a strong correlation,” implying that even if Google says it’s not valid, it can be a smart proxy for Google’s algorithm. After that, verbal exchange fails as search engine optimization professionals claim to have caught Google on some sort of disinformation crusade to protect their intellectual property. Even the slightest crack in their reaction is treated as if someone discovered that they were the souls of the search engine optimization professionals conquered to force their servers.
“I have no words about how much this has not become a challenge before,” Jen said in our last conversation. I tell you that at all times it has been a challenge and that at all times there have been other people like me who register to report it.
“There is no science forged with other people who know enough to be productive, harmful or downright deceptive,” she says, astonished by the concept. “A sweepstakes can do a bigger task than any of the studios I’ve noticed so far that expect a site to rank higher than another website.”
“The only way to statistically demonstrate that any individual metric that claims to recreate Google’s search algorithms is accurate is to do giant random testing over time, control variations, and random characteristic adjustments to minimize ranking,” jen explains. provide a solution that is incredibly remote to our industry. “This will have to be large-scale in many subjects, study styles, etc.”
“Even then, I suspect Google has common updates to algorithms of other amplitudes,” says jen, which I confirm. “Without a doubt, they have dozens or lots of engineers, programmers, analysts, etc. running on those algorithms, which means that if we take a snapshot in time of what we suspect of the algorithm, over time we fully test it, changed
At the end of the day, Jen says our industry doesn’t have the equipment we want to make those studies useful. “The mathematics of analyzing how Google’s index works is closer to astrophysics than predicting election results, however, these are the strategies that are used today closer to the last.”
* * *
I don’t need to make other people who publish these studies absolutely chatty. Their efforts obviously come from a fair search for discovery.
They gave it to me. It’s a laugh to play with all the knowledge you have and take a look to find out how something so confusing works.
However, there are well-known methodologies that reveal what are presented as theories with these studies, but only apply … Not at all.
When it comes down to it, these “Gentlemen Scientists” of SEO are trying to build a dam without a full understanding of engineering, and that’s just dangerous.
Sure, publishing yet another report claiming something is a ranking factor because of a high correlation won’t accidentally kill 400 people. It is undoubtedly wasting multitudes of their clients’ time and money by sending them on a wild goose chase.
More Resources:
Get our newsletter from SEJ founder Loren Baker on the latest industry news!
Jeff Ferguson is the spouse of Digital Amplitude, a Los Angeles-based virtual media advertising firm and anArray. [Read the full biography]
Develop your business with highly specific leads. Place your logo in front of consumers with the Adzooma market.