During a walk through the ruins of the San Francisco Dam Disaster Site, about 65 km from downtown Los Angeles, with my archaeologist friend, John, we talked about the stained life of its builder and the age of the “Gentlemen Scientist.”
The San Francisco Dam was built between 1924 and 1926 to create a giant garage tank for the city of Los Angeles, California, through the Office of Water Works and Supply, now the Water and Energy Decompose. The decomposition under the direction of its chief general manager and leading engineer, William Mulholland. If you’ve ever noticed the vintage movie, “Chinatown,” William Mulholland is such a vital component of the Los Angeles story that they had to split it into two characters.
While a legend in his day, Mulholland is not a civil engineer by today’s standards. He was self-taught in his early days as a “sweet” to the water department. After a hard day’s work, Mulholland will examine the textbooks of mathematics, engineering, hydraulics and geology. This original story is the basis of the “Gentlemen Scientist” character: it devours all the curtains on a subject and then claims an understanding that would allow them to oversee a large company, despite any form of testing or certification.
If I went to NASA and said I was qualified to send humans to Mars because I read a lot of books about the area and used to build style rockets as a kid, they’d turn me off the property. In the days of Mulholland, this meant an ascent to run the department.
Mulholland is an integral component of Los Angeles history. While many of his early efforts literally replaced the Los Angeles landscape (he oversaw the design and structure of the Los Angeles Aqueduct, which brought water to much of the county), his lack of fashionable civil engineering “is one of the worst American civilians. “20th-century engineering disasters,” according to Catherine Mulholland, in her biography of William Mulholland, her grandfather.
Minutes earlier, on March 12, 1928, the dam failed catastrophically and the resulting flood caused the deaths of at least 431 people, however, some reports claim to be as high as a thousand. Even with the smallest number, the San Francisco Dam Cave remains the largest loss of life in California history. Only the 1906 earthquake and chimney in San Francisco killed more people.
The discussion with my friend that day made me aware of the search engine optimization activity and his collection of “Gentleman Scientists”.
Instead of building dams, our colleagues are looking to redesign complex search engine algorithms like Google through misconception practices to design benchmark methods backed by poor quality science.
For decades, legions of search engine optimization professionals have claimed to have “tested” other theories about Google’s algorithms through highly questionable practices. At first, that evidence referred to a self-proclaimed mad scientist who replaced a facet of a single internet page and then waited for the next Google Dance to see if his website was progressing in a search engine index. If it worked, they posted an article about the effects on a forum or on their websites. If the poster were popular enough, the search engine optimization network would reflect your new “hack” until Yahoo, Google or one of the first search engines told them to avoid or figure out how to prevent it from falling into their algorithms.
The first legends of search engine optimization were born from this type of activity.
Finally, corporations like Moz, Ahrefs and SEMrush have discovered tactics to reflect Google’s index, the “tests” or “studies” they have conducted have had a much more valid facet due to access to much larger knowledge sets. Google closed these theories with the old and appropriate reaction “Correlation does not amount to causation”; However, the maximum of these erroneous statements have survived under the banner of “Trust but verification”.
My long-standing position on this factor stems from the fact that Google’s multiple algorithms load from knowledge problems to create a World Wide Web index of billions of Internet pages. With something so sophisticated, are the professionals of maximum optimization of qualified search engines to “test” Google using our limited understanding of statistics?
With rare exceptions, which will actually be highlighted once this article is published, the maximum of other people running in search engine optimization are novice statisticians who, at best, have taken the typical courses and have retained more than the maximum. Some colleagues have a deeper understanding of statistics, but are not yet statistical or mathematical, but have acquired their mathematical talents in examining other sciences accustomed to less complex data. In most cases, the statistical systems they use are used to analyze surveys or media purchase forecasts. They are not for giant complex systems discovered in search engine algorithms and the data they organize.
I’ll be the first to admit that I’m not a mathematician or a statistician. I struggled with math in school long enough to complete my college studies and didn’t feel comfortable with everything before I graduated. Even then, in the popular elegance of trade statistics that other people suffered when searching for their MBA.
Just as when I painted with genuine high-value asset attorneys for my article on the legality of Google’s featured statements, I looked for a genuine statistician. More importantly, I needed someone who did not paint in the search engine optimization area to avoid any observer bias, that is, someone who unconsciously assigned their expectations in the investigation.
My search led me to the statistician, Jen Hood. Jen studied mathematics and economics at Virginia’s Bridgewater College, and for most of the 15 years she has been working as a statistician. She was a data analyst for Volvo. Since 2019, she has been working as an analytics consultant at her company, Avant Analytics, mostly helping small businesses that wouldn’t usually have an in-house analyst.
We spoke about how most of the studies around SEO rely on the concept of statistical correlation during our first discussions. Statistical correlation shows whether – and how strongly – pairs of variables, such as certain aspects of a webpage and that page’s position in Google’s search engine result pages, are related.
“The vast majority of statistical tables, including forecasts for the future, revolve around measuring correlation,” Jen says cautiously. “However, the causal link is incredibly difficult to prove.” Causation is the act of provoking an event; that is, the genuine explanation of why things paint as they do.
“Without knowing the details of how any of these companies create their metrics, I’m suspicious there’s a significant amount of confirmation bias occurring,” Jen continued. Confirmation bias happens when the person performing an analysis wants to prove a predetermined assumption. Rather than doing the actual work needed to confirm the hypothesis, they make the data fit until this assumption is proven.
To give Jen a better idea of how these companies were producing their data, I shared some of the more popular SEO studies over the past few years. Some of the proclamations made in these studies have been disproven by Google multiple times over the years, others still linger on Twitter, Reddit, and Quora and get discussed on what feels like a daily basis.
“The confirmation bias error looks like in those reference articles,” Jen says immediately. “This is not unusual in all the subjects where someone tells you how to gain an advantage.”
First, Jen reviewed a study presented by Rob Ousbey at Mozcon 2019, back when he was working for Distilled (he currently works for Moz) on the SEO testing platform, then called Distilled ODN, now the spin-off SearchPilot. Of the various theories presented that day, one claimed that the results on Page 1 of search engine result pages are driven more by engagement with those pages than links. Jen gets suspicious immediately.
“With the information available, it’s hard to say if Rob’s theory about the first page of results is driven by engagement and subsequent results are driven by links is accurate,” Jen wrote after reviewing the presentation. “This idea that it’s mainly links [driving the search results for Page 2 onward] seems a bit strange given that there are so many factors that go into the ranking.”
“The simplest check would be this: if you can rank on page 1, especially on the most sensible page, without having had a commitment in the past, then the commitment has probably been decided by placing, not the other”
I contacted Will Critchlow, founder and CEO of Distilled. He proposed examining through a former colleague of Rob Ousbey, com.a.m. Tom Capper, who provided a deeper dive into the curtains Rob presented in 2019. “Tom has approached this from many other angles, but the short answer is no, it’s not just because the more productive effects generate more interaction, because those are the most productive effects.”
“[Tom’s study provided] various different kinds of evidence,” Will continued, “One is that links have a higher correlation with relative rankings lower down the SERPs than they do on the first page (and especially for high-volume keywords).”
“Other evidence includes how grades replace when a question moves from a low-volume search expression to a number one term (for example, a very sharp volume),” Says Will, referring to a search for the search term, “Mother’s Day Flowers. ” “
“It keeps getting more interesting,” Jen writes after reviewing the new information. “These new [knowledge] are components of actual correlation values but in an absolutely smaller and much smaller pattern in UK knowledge: only 4,900 queries in two months.”
Before proceeding, it is to perceive how correlation studies are intended to work.
There are several tactics for measuring dating, or correlation, between two things. Regardless of the method, the numbers returned through those calculations are between -1 and 1. A correlation of -1 means that when one thing increases, the other decreases time. A correlation of 1 means that when one thing increases, the other increases time. A correlation of 0 means that there is no dating, no predictable linear pattern, high/low, high/high, low/high or otherwise.
“Most correlation coefficients (results) are not close to 1 or -1,” says Jen. “Anything that’s at 1 means that one hundred percent of the variation is explained through what you’re comparing. In other words, you can use the first thing to wait for what the moment will do.”
While there’s no rule for saying a correlation is strong, weak, or somewhere in between, there are some generally accepted thresholds, which Jen describes. “Keeping in mind that we can have values that are +/-, for factors that are easily countable, such as the number of links a webpage has and that webpage’s ranking on Google, the high correlation would be 0.7-1.0, moderate would be 0.3-0.7, and weak would be 0-0.3.”
“Someone can just challenge those precise groups,” Jen acknowledges, “even though I made a mistake in the aspect of generosity by the strength of correlation.”
We’ll go back to the test. “Tom’s slides basically refer to a February 2017 presentation that he made about whether Google still wants links. There is also a referenced Moz exam that, at this stage, dates back five years” (Jen stops here to say, “By the way, I find it attractive that everyone recognizes that algorithms have undergone significant adjustments and yet refer to studies dating back two, three years or more.”
“In this, [Tom] examines the dating between the domain authority and the ratings,” referring to Moz’s metric, which is the cornerstone of incoming link reporting tools. “Provides the correlation between the domain authority and Google’s rating of a website: 0.001 for positions 1 to five and 0.011 for positions 6 to 10.”
“This that the domain authority is more strongly correlated with the search engine rating for positions 6 through 10, but both effects are very weak correlations,” Jen paused to make sure I understood.
“To put it simply, for positions 1 to five in Google results, the domain authority can be used for 0.1% of the SERP classification variation. For positions 6 to 10, this is 1.1% of the variation of the SERP classification “, clarifying its point.
“This is held up as proof that Domain Authority doesn’t matter as much for top positions. Yet the correlations for both are so extremely low as to be nearly meaningless,” Jen says excitedly by the discovery. At the same time, I consider how many domains and links are bought and sold using this metric. “Elsewhere, he mentions 0.023 and 0.07 as correlation coefficients for Domain Authority and ranking in top 10 positions, which doesn’t make sense with his earlier values both being lower.”
Jen leads the explanation to close the cycle: “As this is the most technical support detail provided by the company, it is moderate to think that the correlations in the original exam you sent me are of a similar level.” In other words, even if we don’t have the numbers from Rob Ousbey’s original presentation, they have an equally weak correlation.
“The Mother’s Day exam is very anecdotal,” Jen continues, “The effects are attractive and raise doubts about the involvement this can have for other terms of study. However, this is a term from studies studied over a month. The content of this exam is sufficient to bring out universal implications”.
“Good for a sales pitch; bad for a statistical study,” says Jen. “During this time, I have not yet noticed anything that shows how they have shown that the most productive effects do not get more interaction because they are the most productive result.”
“There are many examples presented in other slides of the claims, but there is no in-depth study.” Jen refers to some of the other studies provided in Rob’s original presentation through Larry Kim, Brian Dean and Searchmetrics.
“[Larry Kim’s examination of the influence of click-through rate on ratings] suggests that a decrease in click-through rate leads to a decrease in ranking. However, this may be the lowest ranking with the lowest click-on click rate,” jen explains, illustrating an unusual fact. paradox with this kind of data. “I would completely expect a high correlation between the page rating and the click-through rate just because more people have the opportunity to participate.”
“Does the bounce rate the search position or vice versa?” Jen asks, moving to another slide that refers to an exam through Backlinko’s Brian Dean claiming that the bounce rate metric influences the position of search results. “I find it attractive that the story looks different if you actually access the source data.”
Jen refers to the original Backlinko exam in which the chart used in Rob’s presentation was drawn, which read: “Note that we are not suggesting that low rebound rates lead to higher grades. Google can use the bounce rate as a rating sign (although in the past it refused to do so). Or it may just be the fact that high-quality content helps keep others more engaged. Therefore, a decrease in bounce rate is a byproduct of high-quality content, which Google measures. “
The statement concludes, “As this is a correlation study, it’s impossible to determine from our data alone,” thus proving Jen’s point of how inappropriate it is to publish these studies at all.
Jen concludes firmly: “The use of this chart is deliberately misleading.”
“[These studies are only one factor. With several sets of rules in place, many points have to be together. Each will have to have individual scores that are weighted overall for the express set of rules and probably backed down in the Jen says it reflects everything that Gary Illyes and John Mueller of Google have said more than once in meetings and on Twitter and everything Dave Davies of this post recently discussed.
Because of this identified complexity, some search engine optimization studies have abandoned correlation strategies in favor of device learning algorithms, such as Random Forest. A strategy used through SEMrush in 2017 to deliver top-tier points on Google, such as page traffic and content length. “This is a smart technique for predicting what is likely to happen,” Jen writes after reviewing the SEMrush exam and his explanation of his methodology, “but it still shows no causality. It only indicates which points are the highest predictors of the classification.” . »
Most engine search engines that are issued do not come from independent resources or educational institutions, but from corporations that promote SEO teams.
This kind of activity by a company is the ethical equivalent of Gatorade proving its claims of being a superior form of hydration for athletes by referencing a study conducted by The Gatorade Sports Science Institute, a research lab owned by Gatorade.
When I told Jen Hood how many studies she reviewed have resulted in new rules or completely new products, she was surprised that someone took those measurements or products seriously.
“Anyone who claims to have a metric that imitates Google claims to have established many cause-and-effect relationships that lead to an express rating on Google,” Jen wrote, referring to Moz’s domain authority. “This deserves to mean that your metric is consistently for genuine effects. If I started a new site with logo or a new logo page today and was doing everything they say is a vital factor, I deserve to get a higher rating. higher rank. If there is a genuine adjustment with the algorithms, the effects deserve to follow.”
Jen provides a hypothetical example:
“Suppose I offer a service where I will tell you precisely where your website will be classified for a particular search term based on a statistic that I include in that service. I have a formula to calculate this metric to be able to do it for many other sites. If I can tell you precisely where it would be based on my formula 0.1% of the time, would you feel that my formula understands Google’s algorithms? If I raise that figure 1.1% of the time, could I now feel safe?
“It seems like all those studies [and products] seem to be doing,” says Jen. “Hide in enough statistical terms and main points to give the impression that it is much more significant.”
* * *
As Jen referred to above, maximum studies of Google’s effects use a limited amount of data, but claim to have statistical significance; however, their concept is imperfect given the very nature of what they are studying.
“Rand says he estimates that Jumpshot’s data contains ‘somewhere between 2-6% of the total number of mobile and desktop internet-browsing devices in the U.S., a.k.a., a statistically significant sample size,’” Jen is referring to a 2019 study by SparkToro’s Rand Fishkin that claims that less than half of all Google searches result in a click. “Rand would be right about statistical significance if the Jumpshot data were a truly random and representative sampling of all Google searches.”
“From what I can find, [Jumpshot] collected all its knowledge from users who were Avast antivirus,” referring to the now closed parent company of the service. “This set of users and their knowledge probably differs from all Google users. This means that the pattern provided through Jumpshot is not random and is probably not representative enough, an old sampling error commonly known as availability bias.”
“Non-context statistics deserve to be taken with a grain of salt. That’s why there are experts in analysis to ask questions and give context. What kind of questions do other people ask and how have they changed? Jen said, delving into the premise of studying.
“For example, other people who are looking for topics for which there is no additional price to access some other online page will probably not miss opportunities for those who miss clicks. Users without delay refine their search term because the set of rules did not capture the context of what they asked for? Jen suggested, or anything that Rand then clarified as a component of his statement as to why clicks on effects take place in more than the effects component. “Now we are increasingly nuanced, however, if Rand claims that clickless searches are bad, then there must be a context explaining why this could happen even in the absence of a [selection extract].”
* * *
If the concept of using data too thin to be accurate isn’t damning enough, there’s the problem that there’s no concept of peer review within the SEO industry. Most of these studies are conducted once and then published without ever being replicated and verified by outside sources. Even if the studies are replicated, they are done by the same people or companies as a celebrated annual tradition.
Of all the historical studies of the St. Francis Dam Disaster, one by J. David Rogers, Ph.D., Chair in Geological Engineering, Department of Geological Sciences & Engineering and professor at Missouri University of Science and Technology, stands out to me. He stated one of the critical reasons for the failure: “The design and construction being overseen by only one person.”
“Unless the effects are life or death or highly regulated, we generally don’t see other people doing the actual paints necessary to demonstrate causation,” Hood adds. “The only way to show causation is to have a forged examination that randomizes and controls other points on the right scale. In addition to clinical drug trials, which usually take years, this is very rare.”
How the SEO industry conducts and presents its research is not how scientific studies have been administered since the 1600s. You don’t have to believe me. I’m not a scientist, but Neil deGrasse Tyson is.
“There is no truth that does not exist without experimental verification of that truth,” said Tyson in an interview with Chuck Klosterman for his book, “But What If We’re Wrong”. “And not only one person’s experiment, but an ensemble of experiments testing the same idea. And only when an ensemble of experiments statistically agrees, do we then talk about an emerging truth within science.”
The standard counter to this argument is just to state, “I never said this study was scientific.” If that’s so, why does this information get shared and believed with such conviction? This is the heart of the problem of confirmation bias, not just with the researchers but also with the users of that research.
“[I]f you really think about what you really actually know, it’s only a few things, like seven things, maybe everybody knows,” comedian, Marc Maron, is talking about the concept of knowledge in his stand-up special, “End Times Fun”. “If you actually made a column of things, you’re pretty sure you know for sure, and then made another column of how you know those things, most of that column is like, ‘Some guy told me.’”
“You know, it’s not sourced material, it’s just – it’s clickbait and hearsay, that’s all,” Maron continues. “Goes into the head, locks onto a feeling, you’re like, ‘That sounds good. I’m gonna tell other people that.’ And that’s how brand marketing works, and also fascism, we’re finding.”
Science has been about figuring out how the physical world works since the time of Aristotle, which most people agree now, was wrong about many things. Scientists must make these efforts because there’s no user’s manual for our planet or anything else in the universe. We can’t visit a random deity during office hours and ask why they made gravity work the way it does.
But with Google and the other search engines, we do have such access.
I hate to fall back on the “Because Google said so!” type argument for these things, but unlike most sciences, we can get notes from The Creator during announced office hours and occasionally, Twitter.
John Mueller’s next pre-year tweet was in reaction to another correlative review published through another search engine optimization company without any outdoor corroboration, claiming to have unlocked Google’s secrets with a limited amount of data.
He has also created complex algorithms on a giant scale: he knows it is never a single calculation with static multipliers. These things are complex and replaced over time. I locate those desirable reports, who would have the idea X? – However, I’m afraid other people think they are useful.
– ? John ? (@JohnMu) 28 April 2020
John Mueller and I express a very clear vision of the presentation of this kind of knowledge: “I’m afraid other people think they are useful,” that is, that this knowledge is not entirely useful and even potentially misleading.
The above came here after the studio’s author, Brian Dean, said the report “was more intended to shed some attention on how some of Google’s rating points might work.”
Claims like this are a popular variant of a typical mea culpa when a search engine optimization study examines is dismissed as incorrect. “I never said it was a Google rating factor, but there was a strong correlation,” implying that even if Google says it’s not valid, it can be a smart proxy for Google’s algorithm. After that, verbal exchange fails as search engine optimization professionals claim to have caught Google on some sort of disinformation crusade to protect their intellectual property. Even the slightest crack in their reaction is treated as if someone discovered that they were the souls of the search engine optimization professionals conquered to force their servers.
“I have no words about how much this has not become a challenge before,” Jen said in our last conversation. I tell you that at all times it has been a challenge and that at all times there have been other people like me who register to report it.
“There’s no solid science behind it with people knowing just enough to be dangerous at best or downright deceptive,” she says, amazed by the concept. “A coin flip can do a better job than any of the studies I’ve seen so far when it comes to predicting whether one site is going to rank higher than another website.”
“The only way to statistically prove that any individual metric claiming to recreate Google’s search algorithms is accurate is to do massive randomized testing over time, controlling for variation, and randomly assigning changes to be made to improve or decline in ranking,” Jen says, providing a solution that seems impossibly distant for our industry. “This needs to be on a large scale across many different topics, styles of searches, etc.”
“Even then, I suspect that Google has frequent algorithm updates of different magnitudes,” Jen supposes, which I confirm. “Undoubtedly, they have dozens or hundreds of engineers, programmers, analysts, and so on working on these algorithms daily, which means if we take a snapshot in time now of what we suspect the algorithm is, by the time we’ve fully tested it, it’s changed.”
At the end of the day, Jen says our industry doesn’t have the equipment we want to make those studies useful. “The mathematics of analyzing how Google’s index works is closer to astrophysics than predicting election results, however, these are the strategies that are used today closer to the last.”
* * *
I don’t need to make other people who publish these studies absolutely chatty. Their efforts obviously come from a fair search for discovery.
They gave it to me. It’s a laugh to play with all the knowledge you have and take a look to find out how something so confusing works.
However, there are well-known methodologies that reveal what are presented as theories with these studies, but only apply … Not at all.
In the end, search engine optimization “Gentlemen Scientists” are looking to build a dam without a complete understanding of engineering, and that’s just dangerous.
Of course, publishing some other report saying that anything is a question of rating due to a strong correlation will not kill 400 people. Without a doubt, this is a waste of time and money for your consumers by sending them in search of a wild goose.
More resources:
Get our newsletter from SEJ founder Loren Baker on the latest industry news!
Jeff Ferguson is the spouse of Digital Amplitude, a Los Angeles-based virtual media advertising firm and anArray. [Read the full biography]
Develop your business with highly specific leads. Place your logo in front of consumers with the Adzooma market.