The Dark Secret of Search Engines Find out what search engines are hiding from you FERNANDO UILHERME BARBOSA DE AZEVEDO Table of Content About the Author Introduction Chapter 1 - Understanding Search Engines Chapter 2 - A Basic History of Search Engines Chapter 3 - Search Engines in Numbers Chapter 4 - A Peek into Search Engine Manipulation Chapter 5 - Underhanded Tactics for On-Page SEO Chapter 6 - Off-Page SEO and the Value of Backlinks Chapter 7 - The Problem with Backlinks Chapter 8 - Search Engines using Artificial Intelligence Chapter 9 - The Dark World of Internet Marketing Final Words About the Author Fernando Uilherme Barbosa de Azevedo is an electronic, electrical and industrial engineer graduated from Pontifícia Universidade Católica of Rio de Janeiro. He is MBA graduate from Fundação Getúlio Vargas. He has been a programming instructor at Pontifícia Universidade Católica of Rio de Janeiro for 7 years. He published his first book “Macros for Excel hands on” by publisher Campus/Elsevier at age 27. The book is still sold in Brazil and Portugal. His first startup business won a prize from the Brazilian Federal Institutional FINEP. Coming to the United States in 2014, Fernando studied Web Development and Internet Technologies at University of California Santa Cruz - Silicon Valley Extension and also completed the “Innovation and Entrepreneurship Certification” from Stanford University. Fernando has been featured many times in news media and TV. He was interviewed as a specialist on his field by Forbes, The Entrepreneur, el Nuevo Herald and many other major Brazilian media companies. Today, Fernando runs 2 internet marketing companies in the United States and has clients in many countries. The companies offer services such as internet marketing, SEO, Online Reputation Managements, pen testing, systems audit, e- commerce, apps and other internet related activities. He is also a web development instructor for IronHack and weekly speaker at Radio Gazeta. Fernando considers himself an ethical hacker and thinks that the internet should be a safer place. By advocating against all the unethical activities online that are still present today, he hopes that our leaders and law makers can become aware of these threats and help create laws for a safer world. This book is dedicated to Fernando’s sister Christiane Monnerat. This book is a series of 5 books related to Internet Technologies and how the internet works. If you like this book, please check out other books from the same author. Introduction Search engines are the gateway to the internet. Google, Yahoo, Bing, DuckDuckGo and other help millions of people every minute. These companies got so big, their indexes algorithms gained a lot of complexity, and have created millions of new jobs for SEO (Search Engine Optimization) agencies, Digital Marketing and E-commerce. Search Engines are not fast and innovators like they used to be. They became the slow huge enterprises like following the path of IBM and Microsoft. Search Engines want to be relevant so you can keep using them and they can keep money running ads. Is the most relevant article for the big majority really the most important article for you? Are you really interested in the tabloid link- bait over-the-top headlines that most people click? If it rotates around keeping the ads running, is everything we search trying to sell us something? Is everything we search trying to understand our values so they can better influence us into buying? Meanwhile, outside of the Search Engines, SEO specialists work to manipulate their position on the Search Engines. Also, digital marketers have carefully created each piece of content for you, sometimes showing evidence for arguments on both sides. Privacy was not a concern till 2018 and companies were interested in increasing their sales by having our information and categorizing our social and economic aspects on search patterns and social medial information we give out. Search engines are evolving to something bizarre. There are arguments for everything you search. There are ads for everything we search. SEO specialists manipulate Search Engines in ways search engines can’t punish them. And even reputable websites are not free from biases. Chapter 1 - Understanding Search Engines To understand just how sophisticated a modern search engine really is, imagine that you need to do a book report on 644 million different books. They vary in size and in the number of pages they have. They also differ in the way they are formatted and in the language in which they’re written. Some of them are for entertainment while others contain functional information. Each one has a unique topic and each author has a unique take. Also imagine that the contents of these books change from day to day. When a new topic arises, a page is added to that book. You will need to add that new page to your report. This is similar to what search engines are doing to all contents in the internet. It would be impossible for one person to read books of that number. Before the digital age, for a person to find a book in a library, he first needs to go to the card catalogue area, look for the card of the book, take note of its location, and go to that portion of the library. Even if all the books are arranged and organized the right way, that person still needs at least 10 minutes to locate it and to bring it to the reading area. A search engine helps us with this problem. It saves us thousands of hours when it comes to searching for information. It does this by indexing all the information into a database. To present users with information that’s most relevant to their needs, the search engine must search far and wide to collect data from all searchable websites. While it may seem like a simple thing to do, it isn’t. A search engine that uses an indexing algorithm like Google uses computers to explore the internet. These computers are called crawlers bots, Googlebot, or simply just crawlers. They execute their programming by going to all the websites that they can explore, hopping from one website to another through links. In the process, they collect relevant data about the website, its content, its important features, and many others. It then indexes these data to a database so that they can pull up specific webpages in the future when someone needs them. In a nutshell, these crawlers read billions of webpages on a regular basis and keeps the index updated. As of the writing of this book, there are around 644 million active websites on the internet. Some of these websites have thousands of pages. Search engines organize all these in their indexes and serve the relevant content according to our search terms. With this in mind, we could break down the tasks of a search engine into three important steps: