During a walk through the ruins of the San Francisco Dam Disaster Site, about 65 km from downtown Los Angeles, with my archaeologist friend, John, we talked about the stained life of its builder and the age of the “Gentlemen Scientist.”
The San Francisco Dam was built between 1924 and 1926 to create a giant garage tank for the city of Los Angeles, California, through the Office of Water Works and Supply, now the Water and Energy Decompose. The decomposition under the direction of its chief general manager and leading engineer, William Mulholland. If you’ve ever noticed the vintage movie, “Chinatown,” William Mulholland is such a vital component of the Los Angeles story that they had to split it into two characters.
While a legend in his day, Mulholland is not a civil engineer by today’s standards. He was self-taught in his early days as a “sweet” to the water department. After a hard day’s work, Mulholland will examine the textbooks of mathematics, engineering, hydraulics and geology. This original story is the basis of the “Gentlemen Scientist” character: it devours all the curtains on a subject and then claims an understanding that would allow them to oversee a large company, despite any form of testing or certification.
If I went to NASA and said I was qualified to send humans to Mars because I read a lot of books about the area and used to build style rockets as a kid, they’d turn me off the property. In the days of Mulholland, this meant an ascent to run the department.
Mulholland is an integral component of Los Angeles history. While many of his early efforts literally replaced the Los Angeles landscape (he oversaw the design and structure of the Los Angeles Aqueduct, which brought water to much of the county), his lack of fashionable civil engineering “is one of the worst American civilians. “20th-century engineering disasters,” according to Catherine Mulholland, in her biography of William Mulholland, her grandfather.
Minutes earlier, on March 12, 1928, the dam failed catastrophically and the resulting flood caused the deaths of at least 431 people, however, some reports claim to be as high as a thousand. Even with the smallest number, the San Francisco Dam Cave remains the largest loss of life in California history. Only the 1906 earthquake and chimney in San Francisco killed more people.
The discussion with my friend that day made me aware of the search engine optimization activity and his collection of “Gentleman Scientists”.
Instead of building dams, our colleagues are looking to redesign complex search engine algorithms like Google through misconception practices to design benchmark methods backed by poor quality science.
For decades, legions of search engine optimization professionals have claimed to have “tested” other theories about Google’s algorithms through highly questionable practices. At first, that evidence referred to a self-proclaimed mad scientist who replaced a facet of a single internet page and then waited for the next Google Dance to see if his website was progressing in a search engine index. If it worked, they posted an article about the effects on a forum or on their websites. If the poster were popular enough, the search engine optimization network would reflect your new “hack” until Yahoo, Google or one of the first search engines told them to avoid or figure out how to prevent it from falling into their algorithms.
The first legends of search engine optimization were born from this type of activity.
Eventually, companies like Moz, Ahrefs, and SEMrush figured out ways to replicate Google’s index, the “testing” or “studies” they did got a lot more legitimate-looking because of the access to much larger data sets. Google would occasionally shut these theories down with the classic and appropriate, “Correlation does not equal causation” reply; however, most of these faulty proclamations lived on under the flag of “Trust but verify.”
My long-standing position on this factor stems from the fact that Google’s multiple algorithms load from knowledge problems to create a World Wide Web index of billions of Internet pages. With something so sophisticated, are the professionals of maximum optimization of qualified search engines to “test” Google using our limited understanding of statistics?
With rare exceptions, which will actually be highlighted once this article is published, the maximum of other people running in search engine optimization are novice statisticians who, at best, have taken the typical courses and have retained more than the maximum. Some colleagues have a deeper understanding of statistics, but are not yet statistical or mathematical, but have acquired their mathematical talents in examining other sciences accustomed to less complex data. In most cases, the statistical systems they use are used to analyze surveys or media purchase forecasts. They are not for giant complex systems discovered in search engine algorithms and the data they organize.
I’ll be the first to admit that I’m not a mathematician or a statistician. I struggled with math in school long enough to complete my college studies and didn’t feel comfortable with everything before I graduated. Even then, in the popular elegance of trade statistics that other people suffered when searching for their MBA.
Just as when I worked with actual intellectual property lawyers for my article on the legality of Google’s Featured Snippets, I sought out an actual statistician. Most importantly, I needed someone who doesn’t work in the SEO space to avoid any observer bias, that is, someone who would subconsciously project their expectations onto the research.
My search led me to the statistician, Jen Hood. Jen studied mathematics and economics at Virginia’s Bridgewater College, and for most of the 15 years she has been working as a statistician. She was a data analyst for Volvo. Since 2019, she has been working as an analytics consultant at her company, Avant Analytics, mostly helping small businesses that wouldn’t usually have an in-house analyst.
We spoke about how most of the studies around SEO rely on the concept of statistical correlation during our first discussions. Statistical correlation shows whether – and how strongly – pairs of variables, such as certain aspects of a webpage and that page’s position in Google’s search engine result pages, are related.
“The vast majority of statistical tables, including forecasts for the future, revolve around measuring correlation,” Jen says cautiously. “However, the causal link is incredibly difficult to prove.” Causation is the act of provoking an event; that is, the genuine explanation of why things paint as they do.
“Without knowing the main points of how one of these corporations creates their metrics, I suspect there is a significant verification bias,” Jen continued. Confirmation bias occurs when the user performing an investigation needs to present a default hypothesis. Instead of making the actual paintings needed to verify the hypothesis, they adjust knowledge until this assumption is fulfilled.
To give Jen a better idea of how these corporations were generating her data, I shared some of the most popular search engine optimization studies in recent years. Some of the proclamations made in these studies have been refuted through Google several times over the years, others persist on Twitter, Reddit and Quora and are the topic of discussion about what appears to be a daily basis.
“The confirmation bias error looks like in those reference articles,” Jen says immediately. “This is not unusual in all the subjects where someone tells you how to gain an advantage.”
First, Jen reviewed an exam submitted through Rob Ousbey at Mozcon 2019, when she applied for Distilled (lately works for Moz) on the search engine optimization verification platform, then called Distilled ODN, now the spin-off of seekPilot. Among the theories presented that day, one stated that the effects of page 1 of the search engine’s effect pages are motivated more through engagement with those pages than through links. Jen begins to suspect immediately.
“With the data available, it’s hard to tell if Rob’s theory on the first page of effects is motivated by engagement and the following link-motivated effects are accurate,” Jen wrote after reviewing the presentation. “This concept that those are basically links [main search effects to page 2] is a bit given that there are so many points that go to the ranking.”
“The easy test would be: if you can rank on Page 1, especially the top of the page, without previously having any engagement, then the engagement is most likely driven by placement, not the other way around.”
I contacted Will Critchlow, founder and CEO of Distilled. He proposed examining through a former colleague of Rob Ousbey, com.a.m. Tom Capper, who provided a deeper dive into the curtains Rob presented in 2019. “Tom has approached this from many other angles, but the short answer is no, it’s not just because the more productive effects generate more interaction, because those are the most productive effects.”
“[Tom provided] other types of evidence,” Will continued, “one is that the links have a higher correlation with the decrease in seRP ratings than on the first page (and especially for higher volume keywords)”.
“Other evidence includes how grades replace when a question moves from a low-volume search expression to a number one term (for example, a very sharp volume),” Says Will, referring to a search for the search term, “Mother’s Day Flowers. ” “
“It keeps getting more interesting,” Jen writes after reviewing the new information. “These new [knowledge] are components of actual correlation values but in an absolutely smaller and much smaller pattern in UK knowledge: only 4,900 queries in two months.”
Before we continue, it’s crucial to understand how correlation studies are supposed to work.
There are multiple ways to measure the relationship, or correlation, between two factors. Regardless of the method, the numbers returned from these calculations measure between -1 and 1. A correlation of -1 means as one factor goes up, the other factor goes down every time. A correlation of 1 means as one factor goes up, the other factor goes up every time. A correlation of zero means there is no relationship – no predictable linear pattern, up/down, up/up, down/up, or otherwise.
“Most correlation coefficients (results) aren’t close to 1 or -1,” Jen clarifies. “Anything at +/-1 means that 100% of the variation is explained by the factor you’re comparing. That is, you can always use the first factor to predict what the second factor will do.”
While there’s no rule for saying a correlation is strong, weak, or somewhere in between, there are some generally accepted thresholds, which Jen describes. “Keeping in mind that we can have values that are +/-, for factors that are easily countable, such as the number of links a webpage has and that webpage’s ranking on Google, the high correlation would be 0.7-1.0, moderate would be 0.3-0.7, and weak would be 0-0.3.”
“Someone can just challenge those precise groups,” Jen acknowledges, “even though I made a mistake in the aspect of generosity by the strength of correlation.”
We’ll go back to the test. “Tom’s slides basically refer to a February 2017 presentation that he made about whether Google still wants links. There is also a referenced Moz exam that, at this stage, dates back five years” (Jen stops here to say, “By the way, I find it attractive that everyone recognizes that algorithms have undergone significant adjustments and yet refer to studies dating back two, three years or more.”
“In this, [Tom] examines the dating between the domain authority and the ratings,” referring to Moz’s metric, which is the cornerstone of incoming link reporting tools. “Provides the correlation between the domain authority and Google’s rating of a website: 0.001 for positions 1 to five and 0.011 for positions 6 to 10.”
“This that the domain authority is more strongly correlated with the search engine rating for positions 6 through 10, but both effects are very weak correlations,” Jen paused to make sure I understood.
“To put it simply, for positions 1 to five in Google results, the domain authority can be used for 0.1% of the SERP classification variation. For positions 6 to 10, this is 1.1% of the variation of the SERP classification “, clarifying its point.
“This shows that domain authority is not as important to high-level positions. However, the correlations for both are so incredibly weak that they make almost no sense,” Jen says enthusiastically through the discovery. At the same time, I know how many domain names and links are purchased and sold using this metric. “Elsewhere, it mentions 0.023 and 0.07 as correlation coefficients for authority and domain rating in the 10 most sensitive positions, which makes no sense since their past values are lower.”
Jen brings the explanation full circle, “Since this is the backup detail, more technically focused, provided by the company, it seems like a reasonable leap to think that the correlations in the original study you sent me are of a similar level.” That is to say, while we don’t have the numbers for Rob Ousbey’s original presentation, they are most likely just as weak a correlation.
“The Mother’s Day exam is very anecdotal,” Jen continues, “The effects are attractive and raise doubts about the involvement this can have for other terms of study. However, this is a term from studies studied over a month. The content of this exam is sufficient to bring out universal implications”.
“Good for a sales pitch; bad for a statistical study,” Jen proclaims. “Meanwhile, I still haven’t seen anything that shows how they’ve proven that the top results don’t get more interaction because they are the top result.”
“There are many examples presented in other slides of the claims, but there is no in-depth study.” Jen refers to some of the other studies provided in Rob’s original presentation through Larry Kim, Brian Dean and Searchmetrics.
“[Larry Kim’s examination of the influence of click-through rate on ratings] suggests that a decrease in click-through rate leads to a decrease in ranking. However, this may be the lowest ranking with the lowest click-on click rate,” jen explains, illustrating an unusual fact. paradox with this kind of data. “I would completely expect a high correlation between the page rating and the click-through rate just because more people have the opportunity to participate.”
“Does the bounce rate the search position or vice versa?” Jen asks, moving to another slide that refers to an exam through Backlinko’s Brian Dean claiming that the bounce rate metric influences the position of search results. “I find it attractive that the story looks different if you actually access the source data.”
Jen refers to the original Backlinko exam in which the chart used in Rob’s presentation was drawn, which read: “Note that we are not suggesting that low rebound rates lead to higher grades. Google can use the bounce rate as a rating sign (although in the past it refused to do so). Or it may just be the fact that high-quality content helps keep others more engaged. Therefore, a decrease in bounce rate is a byproduct of high-quality content, which Google measures. “
He concludes: “As this is a correlation study, it is highly unlikely that you will only realize our knowledge,” demonstrating Jen’s interest in publishing these studies.
Jen strongly concludes, “The use of this graph is intentionally misleading.”
“[These studies are only one factor. With several sets of rules in place, many points have to be together. Each will have to have individual scores that are weighted overall for the express set of rules and probably backed down in the Jen says it reflects everything that Gary Illyes and John Mueller of Google have said more than once in meetings and on Twitter and everything Dave Davies of this post recently discussed.
Because of this identified complexity, some search engine optimization studies have abandoned correlation strategies in favor of device learning algorithms, such as Random Forest. A strategy used through SEMrush in 2017 to deliver top-tier points on Google, such as page traffic and content length. “This is a smart technique for predicting what is likely to happen,” Jen writes after reviewing the SEMrush exam and his explanation of his methodology, “but it still shows no causality. It only indicates which points are the highest predictors of the classification.” . »
Most engine search engines that are issued do not come from independent resources or educational institutions, but from corporations that promote SEO teams.
This type of activity of a company is the moral of Gatorade, which demonstrates its claims to be an incredible form of hydration for athletes by referencing an exam conducted through the Gatorade Sports Science Institute, a laboratory of studies owned by Gatorade.
When I told Jen Hood how many studies she reviewed have resulted in new rules or completely new products, she was surprised that someone took those measurements or products seriously.
“Anyone who claims to have a metric that imitates Google claims to have established many cause-and-effect relationships that lead to an express rating on Google,” Jen wrote, referring to Moz’s domain authority. “This deserves to mean that your metric is consistently for genuine effects. If I started a new site with logo or a new logo page today and was doing everything they say is a vital factor, I deserve to get a higher rating. higher rank. If there is a genuine adjustment with the algorithms, the effects deserve to follow.”
Jen provides a hypothetical example:
“Let’s say I offer a service where I’ll tell you exactly where your webpage will rank for a given search term based a metric I include in that service. I have a formula for calculating that metric so I can do it for many different sites. If I could accurately tell you where you’d rank based on my formula 0.1% of the time, would it seem like my formula has the Google algorithms figured out? If I upped that to 1.1% of the time, would you now feel confident?”
“That’s all these studies [and products] seem to be doing,” Jen explains. “Cloaking themselves in just enough statistical terms and details to make it seem like it’s much more meaningful.”
* * *
As Jen alluded to earlier, most studies of Google’s results are using a limited amount of data, but claiming statistical significance; however, their understanding of that concept is flawed given the nature of the very thing they are studying.
“Rand says he estimates that Jumpshot’s data contains ‘somewhere between 2-6% of the total number of mobile and desktop internet-browsing devices in the U.S., a.k.a., a statistically significant sample size,’” Jen is referring to a 2019 study by SparkToro’s Rand Fishkin that claims that less than half of all Google searches result in a click. “Rand would be right about statistical significance if the Jumpshot data were a truly random and representative sampling of all Google searches.”
“From what I can find, [Jumpshot] collected all its knowledge from users who were Avast antivirus,” referring to the now closed parent company of the service. “This set of users and their knowledge probably differs from all Google users. This means that the pattern provided through Jumpshot is not random and is probably not representative enough, an old sampling error commonly known as availability bias.”
“Non-context statistics deserve to be taken with a grain of salt. That’s why there are experts in analysis to ask questions and give context. What kind of questions do other people ask and how have they changed? Jen said, delving into the premise of studying.
“For example, other people who are looking for topics for which there is no additional price to access some other online page will probably not miss opportunities for those who miss clicks. Users without delay refine their search term because the set of rules did not capture the context of what they asked for? Jen suggested, or anything that Rand then clarified as a component of his statement as to why clicks on effects take place in more than the effects component. “Now we are increasingly nuanced, however, if Rand claims that clickless searches are bad, then there must be a context explaining why this could happen even in the absence of a [selection extract].”
* * *
If the concept of knowledge too thin to be accurate is not damning enough, there is a challenge that there is no concept of peer review in the search engine optimization industry. Most of these studies are done once and then published without being replicated and verified through external sources. Even if the studies are reproduced, they are conducted through the same people or corporations as a remarkable annual tradition.
Among all the ancient studies of the San Francisco Dam disaster, one through J. David Rogers, Ph.D., President of Geological Engineering, Department of Geological Sciences and Engineering and Professor at Missouri University of Science and Technology, distinguishes me. He indicated one of the main reasons for the failure: “Design and be supervised by a single person.”
“Unless the effects are life or death or highly regulated, we generally don’t see other people doing the actual paints necessary to demonstrate causation,” Hood adds. “The only way to show causation is to have a forged examination that randomizes and controls other points on the right scale. In addition to clinical drug trials, which usually take years, this is very rare.”
The way the search engine optimization industry conducts and presents its studies is not how clinical studies have been administered since the 17th century. You don’t have me. I’m not a scientist, but Neil deGrasse Tyson is.
“There is no fact that exists without the experimental verification of this fact,” Tyson said in an interview with Chuck Klosterman for his book, “But what if we’re wrong.” “And only one person’s experience, yet a set of reports that prove the same idea. And it is only when a set of experiments is statistically agreed that we are talking about an emerging fact within science.”
The opposite of that argument is simply to say, “I never said this would examine scientists.” If so, why is this data shared and viewed with such conviction? This is at the heart of the confirmation bias challenge, not only with researchers but also with the users of this research.
“[I]f you really think about what you really actually know, it’s only a few things, like seven things, maybe everybody knows,” comedian, Marc Maron, is talking about the concept of knowledge in his stand-up special, “End Times Fun”. “If you actually made a column of things, you’re pretty sure you know for sure, and then made another column of how you know those things, most of that column is like, ‘Some guy told me.’”
“You know, it’s not source material, it’s just, it’s bait and rumors, that’s all,” Maron continues. “Go to the head, it creates a feeling, you think” That sounds good. I’ll tell the others. “And that’s how logo marketing works, as well as fascism, we discover.”
Science has been about figuring out how the physical world works since the time of Aristotle, which most people agree now, was wrong about many things. Scientists must make these efforts because there’s no user’s manual for our planet or anything else in the universe. We can’t visit a random deity during office hours and ask why they made gravity work the way it does.
But with Google and search engines, we have that access.
I hate to fall back on the “Because Google said so!” type argument for these things, but unlike most sciences, we can get notes from The Creator during announced office hours and occasionally, Twitter.
John Mueller’s next pre-year tweet was in reaction to another correlative review published through another search engine optimization company without any outdoor corroboration, claiming to have unlocked Google’s secrets with a limited amount of data.
He has also created complex algorithms on a giant scale: he knows it is never a single calculation with static multipliers. These things are complex and replaced over time. I locate those desirable reports, who would have the idea X? – However, I’m afraid other people think they are useful.
– ? John ? (@JohnMu) 28 April 2020
John Mueller and I express a very clear vision of the presentation of this kind of knowledge: “I’m afraid other people think they are useful,” that is, that this knowledge is not entirely useful and even potentially misleading.
The above came here after the studio’s author, Brian Dean, said the report “was more intended to shed some attention on how some of Google’s rating points might work.”
Claims like this are a popular variant of a typical mea culpa when a search engine optimization study examines is dismissed as incorrect. “I never said it was a Google rating factor, but there was a strong correlation,” implying that even if Google says it’s not valid, it can be a smart proxy for Google’s algorithm. After that, verbal exchange fails as search engine optimization professionals claim to have caught Google on some sort of disinformation crusade to protect their intellectual property. Even the slightest crack in their reaction is treated as if someone discovered that they were the souls of the search engine optimization professionals conquered to force their servers.
“I’m perplexed how this hasn’t become an issue before,” Jen says during our final conversation. I tell her it’s always been an issue and that there have always been people like me who try and point out the problem.
“There is no science forged with other people who know enough to be productive, harmful or downright deceptive,” she says, astonished by the concept. “A sweepstakes can do a bigger task than any of the studios I’ve noticed so far that expect a site to rank higher than another website.”
“The only way to statistically prove that any individual metric claiming to recreate Google’s search algorithms is accurate is to do massive randomized testing over time, controlling for variation, and randomly assigning changes to be made to improve or decline in ranking,” Jen says, providing a solution that seems impossibly distant for our industry. “This needs to be on a large scale across many different topics, styles of searches, etc.”
“Even then, I suspect Google has common updates to algorithms of other amplitudes,” says jen, which I confirm. “Without a doubt, they have dozens or lots of engineers, programmers, analysts, etc. running on those algorithms, which means that if we take a snapshot in time of what we suspect of the algorithm, over time we fully test it, changed
In the end, Jen agrees that it appears our industry doesn’t have the tools we need to make these studies useful. “The mathematics of analyzing how Google’s index functions are closer to astrophysics than predicting election results, but that’s the methods used today are closer to the latter.”
* * *
I don’t need to make other people who publish these studies absolutely chatty. Their efforts obviously come from a fair search for discovery.
I get it. It’s fun to play with all the data they have at their disposal and try and figure out how something so complicated works.
However, there are well-known methodologies that reveal what are presented as theories with these studies, but only apply … Not at all.
In the end, search engine optimization “Gentlemen Scientists” are looking to build a dam without a complete understanding of engineering, and that’s just dangerous.
Of course, publishing some other report saying that anything is a question of rating due to a strong correlation will not kill 400 people. Without a doubt, this is a waste of time and money for your consumers by sending them in search of a wild goose.
More resources:
Get our newsletter from SEJ founder Loren Baker on the latest industry news!
Jeff Ferguson is the spouse of Digital Amplitude, a Los Angeles-based virtual media advertising firm and anArray. [Read the full biography]
Develop your business with highly specific leads. Place your logo in front of consumers with the Adzooma market.