The History of SEO: From Information Retrieval to AI Search

Introduction: SEO before it had a name

There was no single moment when SEO was invented. The field accumulated.

It inherited concepts from library science and information retrieval; technical constraints from the architecture of the Web; economic pressure from advertising markets; and adversarial behavior from the simple fact that visibility has value. The phrase search engine optimization only became visible in trade usage during the late 1990s.

The intellectual and technical problems behind it are much older.

This distinction matters. A narrow history begins with a webmaster adding keywords to a page.

A fuller history begins with a harder question: how can a machine determine which information deserves to appear first? Every major phase of SEO has been an answer to that question.

At first, relevance meant matching words in a query to words in a document. Later it incorporated links, anchor text, freshness, location, device context, language models, and assessments of quality.

In generative interfaces, the task now includes deciding which sources are worth retrieving, synthesizing, and citing inside an answer.

The method used here is historical and interpretive. Primary technical papers establish what early systems were designed to do.

Archival and institutional records establish dates and artifacts. Official search-engine statements document product changes, but they are treated as corporate sources rather than neutral adjudications of impact.

Academic work on search politics and Web spam supplies a broader frame: a ranking system does not merely organize information. It also distributes attention, opportunity, and authority.

Seen this way, the history of SEO is neither a parade of algorithm updates nor a morality play in which clever marketers repeatedly fool innocent machines. It is a feedback system.

Search engines create a model of relevance. Publishers learn the model and change their sites.

Some improvements genuinely help users; others simulate the signals without providing the underlying value. Engines then revise the model. The cycle repeats.

I. The intellectual prehistory: association, specificity, and retrieval

One useful starting point is Vannevar Bush’s 1945 essay “As We May Think”. Bush imagined a device called the memex that would let a person store records and connect them through associative trails.

The memex was not a search engine, and it should not be retroactively described as one. Its historical importance lies in a different idea: knowledge could be navigated through relationships rather than through a fixed hierarchy.

Hypertext would later give that idea a networked form.

Information retrieval supplied the mathematical counterpart. In 1972, British computer scientist Karen Spärck Jones published “A Statistical Interpretation of Term Specificity and Its Application in Retrieval”.

Her work formalized a durable intuition: a term that appears in almost every document is usually less discriminating than a term that appears rarely. This became foundational to inverse document frequency, a component of the TF-IDF family of methods.

Long before marketers debated keyword density, retrieval researchers were already asking how the distribution of words could help distinguish one document from another.

These two lineages—associative linking and statistical retrieval—would meet on the Web. Links supplied relationships. Text supplied lexical evidence. SEO eventually developed in the gap between them: the practical work of ensuring that a document could be accessed, interpreted, and selected by a retrieval system.

II. The Web creates a discovery crisis, 1989–1997

Tim Berners-Lee submitted his proposal for the World Wide Web at CERN in March 1989. According to CERN’s historical account, the first Web server and browser were running by the end of 1990.

On April 30, 1993, CERN made the Web software freely available, accelerating adoption beyond its original research setting.

Discovery was already a problem on the wider Internet. In 1989 and 1990, Alan Emtage at McGill University developed Archie, a system that indexed filenames available through public FTP archives.

McGill describes Archie as the first publicly distributed Internet search engine. The qualification is important: Archie searched pre-Web Internet resources, not Web pages.

It nevertheless demonstrated the value of automated indexing at a scale where manual knowledge of server locations no longer worked.

Early Web discovery mixed human curation and automated retrieval. Directories treated the Web as a catalog that editors could classify.

Crawlers treated it as a corpus that machines could traverse and index. Both models strained under rapid growth.

AltaVista, created at Digital Equipment Corporation in 1995, became a landmark of the crawler model. Its developers reported that the research project’s crawler was capable of millions of HTTP requests per day within weeks; the system emphasized fast crawling, full-text indexing, and rapid query response.

The Web was becoming too large to browse as a directory, and search was becoming infrastructure.

The crawler also created a new relationship between machine and publisher. In 1994, after automated agents overwhelmed a site, Martijn Koster proposed the Robots Exclusion Protocol.

A robots.txt file gave site owners a convention for telling compliant crawlers what not to access. Google’s later history of the protocol notes that it remained a de facto standard for twenty-five years before formal standardization work began.

Technical SEO can be located here as much as in any keyword tag: crawl access, server resources, and machine-readable instructions became conditions of visibility.

By 1997, a trade vocabulary was forming around the effort to improve placement in search results. The exact phrase “search engine optimization” appears in surviving industry records from that year.

Danny Sullivan’s retrospective research identified the phrase in a Search Engine Watch meta tag in May 1997 and in a July 1997 newsgroup sales pitch. This is evidence of emergence, not proof of a single inventor.

SEO was a name applied to a set of practices already developing across web design, submission services, content editing, and technical maintenance.

Then the rules changed.

III. Google, PageRank, and the redefinition of authority, 1998–2002

The decisive innovation of early Google was not that it found pages containing a query. Other engines did that.

It was that Google treated the structure of links as evidence about authority.

In their 1998 paper “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, Sergey Brin and Lawrence Page described a prototype that used hyperlink structure, PageRank, anchor text, and a scalable crawling and indexing architecture.

The indexed Web was not a controlled scholarly database; it was, in their words, an uncontrolled collection in which anyone could publish and attempt to influence retrieval.

PageRank changed the optimization problem. A page’s own text was no longer the whole case for relevance.

Other pages could make claims about it through links, and the importance of those linking pages mattered in turn. Anchor text provided external descriptions of a destination.

Links had always connected documents; now they also became votes in a computational system.

This was a conceptual breakthrough and an invitation to a market. Once links affected prominence, link acquisition became economically valuable.

Legitimate publicity, editorial citation, public relations, directories, reciprocal schemes, and eventually link farms all occupied the same strategic terrain. The distinction between earning an endorsement and manufacturing its appearance would become one of SEO’s defining conflicts.

At the same time, search acquired a commercial architecture. Daniel Fain and Jan Pedersen’s history of sponsored search identifies GoTo’s 1998 system as the first to combine query-matched advertising, ranked placement, click measurement, and advertiser charges into a durable model.

Paid listings were not SEO, but they changed the environment in which SEO operated. The same results page now allocated attention through two different logics: algorithmic relevance and auction-based sponsorship.

Search engine marketing became the umbrella under which organic optimization and paid search could be distinguished.

A final conceptual step came from Andrei Broder’s 2002 “A Taxonomy of Web Search”. Broder divided Web queries into informational, navigational, and transactional classes.

The taxonomy helped make explicit what effective optimization already implied: matching words is insufficient when users want different kinds of outcomes. Someone seeking an explanation, a known website, or a place to complete an action is expressing a different intent.

Search relevance had become behavioral as well as lexical.

IV. From optimization to adversarial information retrieval, 2003–2010

As search traffic became commercially meaningful, attempts to manipulate ranking systems became organized and repeatable. Hidden text, doorway pages, automated content, deceptive redirects, excessive cross-linking, and paid link networks were not random tricks.

They were strategies for separating a ranking signal from the quality it was intended to represent.

The research community named the problem directly. In 2005, Zoltán Gyöngyi and Hector Garcia-Molina published a taxonomy of Web spam for the first AIRWeb workshop.

They defined Web spam as actions intended to mislead search engines into ranking pages higher than deserved. The definition reveals why SEO cannot be reduced to rule-following.

Search engines do not fully disclose their models, and optimizers infer them from outcomes. The same observation can produce either a better document or a simulation of betterment.

This period is frequently described through named algorithm updates, especially Google’s Florida update in 2003. The update clearly occupies an important place in industry memory because many commercial sites reported severe visibility changes near the holiday season.

Yet the precise mechanism and scope are less firmly documented than later official announcements.

A responsible history therefore treats Florida as evidence of a new operational reality, not as a fully decoded technical event: ranking changes could suddenly alter business performance, and SEO required continuing adaptation rather than one-time configuration.

The emergence of rel="nofollow" in 2005 similarly demonstrated that Web standards could be modified in response to optimization behavior. Publishers received a way to mark certain links so they would not function as ordinary endorsements for ranking purposes.

Search engines were no longer passive maps of the Web. They were governors of an incentive system, designing rules that changed how links, content, and reputation circulated.

Academic criticism widened the frame. In “Shaping the Web: Why the Politics of Search Engines Matters”, Lucas Introna and Helen Nissenbaum argued that search-engine design raises political as well as technical questions.

Crawling, indexing, and ranking systematically make some material more visible and other material harder to find. Their point was not that relevance is impossible, but that every operational definition of relevance privileges certain properties and actors.

SEO developed inside that allocation system. To optimize a site was increasingly to understand the institutional criteria by which visibility was granted.

V. Quality, entities, and the mobile Web, 2011–2018

The early 2010s marked a public shift from isolated anti-spam actions toward broader quality evaluation. Google’s Panda update, first launched in 2011, attempted to reduce the visibility of low-quality sites while rewarding sites the company considered more useful.

In its international rollout announcement, Google stated that the initial change affected nearly 12 percent of English-language queries and that later international effects ranged from roughly 6 to 9 percent. These are company-reported figures, but the scale illustrates why content strategy became inseparable from SEO strategy.

Contemporaneous quality guidance asked whether content contained original reporting or analysis, whether it was written by someone with expertise, and whether a reader would trust it. The questions did not expose a ranking formula.

They shifted the professional standard. Producing pages for every keyword variation was becoming less defensible than publishing distinctive work that deserved to satisfy a user.

Penguin, announced in 2012 and later incorporated into Google’s core systems, addressed another legacy of PageRank: manipulative link practices. Google’s initial announcement said the change affected about 3.1 percent of English queries.

More important than the percentage was the direction. A link could no longer be treated simply as a countable asset.

Its context, origin, and pattern mattered, and aggressive attempts to manufacture authority could create risk rather than advantage.

Search was also moving beyond strings. In 2012 Google introduced the Knowledge Graph, described in its announcement as a system for understanding “things, not strings.”

At launch, the company said the graph contained more than 500 million objects and billions of facts and relationships. The historical significance is larger than the numbers.

Entity-based retrieval could connect people, places, works, organizations, and attributes even when the same words were ambiguous. Optimization consequently expanded toward consistent identity, structured data, authoritative corroboration, and semantic relationships.

Meanwhile, the device through which people searched was changing. Mobile search was not merely desktop search on a smaller screen.

It introduced location, touch interfaces, constrained bandwidth, voice input, and different expectations of immediacy. Google began experimenting with mobile-first indexing in 2016 and rolled it out more broadly in 2018, using the mobile version of a page for indexing and ranking.

The company declared the transition complete in 2023. The long rollout is revealing: search systems do not simply announce a new user reality; they must rebuild crawling, rendering, and evaluation around it.

For SEO, this period replaced a page-centered craft with a system-centered one. Content quality, link provenance, entity identity, structured markup, responsive design, performance, and local context could no longer be managed as separate specialties. They were different forms of evidence submitted to the same visibility system.

VI. Machine learning, page experience, and people-first publishing, 2019–2022

Machine learning had influenced search before 2019, but Google’s introduction of BERT made a particular change legible to the public. BERT processes words in relation to the words around them, helping a system interpret context that older approaches could miss.

In Google’s 2019 announcement, examples turned on prepositions and ordinary phrasing—the kinds of details people use naturally and crude keyword matching often discards.

The optimization lesson was not that keywords ceased to matter. Retrieval still requires language.

The lesson was that exact repetition became a weaker substitute for semantic completeness. A page could be useful because it answered the underlying need in natural terms, covered necessary relationships, and clarified ambiguous concepts.

Search engines were becoming better at evaluating the linguistic situation around a phrase, not just its frequency.

Technical experience also received a more formal vocabulary. In 2020, Google announced that Core Web Vitals would become part of its page-experience signals.

The original set addressed loading performance, interactivity, and visual stability. These metrics did not make good content optional, and they were never the whole ranking system.

Their importance was historical: part of the user’s experience after clicking became measurable evidence within search evaluation. SEO’s object was no longer only the document as indexed, but the page as experienced.

The 2022 helpful content update joined these strands. Google described an effort to reward original, people-first content written for an intended audience rather than content created principally to attract search traffic.

The company explicitly said that SEO remains useful when applied to people-first material. That distinction is central.

Optimization was being framed not as an alternative to editorial purpose but as a method for making purposeful work accessible.

The same logic appears in Google’s documentation on experience, expertise, authoritativeness, and trust, commonly abbreviated E-E-A-T. The documentation is often misread as naming a single measurable ranking factor. It does not.

Google states that its systems use a mixture of factors that can align with these qualities and that search quality raters do not directly control rankings.

Historically, E-E-A-T is better understood as an evaluative language: it articulates why provenance, demonstrated experience, clear authorship, evidence, and accountability matter in high-stakes information environments.

By the end of 2022, competent SEO had become recognizably interdisciplinary. It involved server behavior, information architecture, research, writing, digital public relations, user experience, analytics, and governance. The old boundary between “content” and “technical” SEO remained useful for staffing, but it described parts of one evidence system.

Now the search box is changing again.

VII. Generative search and the redistribution of visibility, 2023–2026

Generative AI introduced a new interface for an old institutional function. Instead of returning only a ranked list of documents, a search system can retrieve material, synthesize an answer, expose supporting links, and invite follow-up questions.

Google introduced its Search Generative Experience in 2023, launched AI Overviews broadly in the United States in May 2024, and expanded AI Mode in 2025. Other platforms developed their own answer engines and AI-assisted discovery products.

The surface changed dramatically, but the underlying problem did not disappear. A generative answer still depends on selection.

Systems must decide what to crawl, what to index, what to retrieve for a particular prompt, which sources can corroborate a claim, and how to represent uncertainty. In AI Mode, Google has described a “query fan-out” technique that issues related searches across subtopics and data sources.

This means visibility can be won or lost across a cluster of inferred questions, not only for one typed keyword.

For publishers, the result is a redistribution of visibility. A conventional organic result offers a title, snippet, and potential visit.

A generated answer may quote, cite, summarize, or absorb information before a visit occurs. Measurement therefore becomes more difficult.

Rank alone is insufficient when a brand can be mentioned without a click, cited for one subclaim, or excluded from a synthesis despite ranking for a related query.

This has prompted new labels—answer engine optimization, generative engine optimization, and large-language-model optimization. The labels identify real changes, but declarations that “SEO is dead” repeat a familiar historical mistake.

SEO has never been attached permanently to one interface. It survived the move from directories to crawlers, from lexical matching to link analysis, from desktop pages to mobile indexing, and from strings to entities and contextual language models.

Each transition enlarged the set of evidence required for visibility.

The generative era does not abolish SEO. It reveals SEO’s underlying subject: how institutions and machines decide which sources are worth representing.

A historically grounded practice for 2026 therefore emphasizes five properties. First is technical retrievability: a system cannot use what it cannot access or parse.

Second is semantic clarity: entities, claims, and relationships should be explicit. Third is demonstrable provenance: authorship, first-hand evidence, citations, and dates help readers and systems evaluate a source.

Fourth is corroborated authority: recognition across reputable sites, communities, and databases matters because no page establishes its own importance alone. Fifth is distinctive information: original data, analysis, images, experience, and expert judgment give a system a reason to retrieve one source instead of another paraphrase.

These principles also expose the political question raised by Introna and Nissenbaum a quarter-century earlier. If generative systems mediate more of what people know, the choice of sources and the form of synthesis allocate visibility even more directly.

Optimization cannot be ethically neutral. It can make excellent evidence easier to find, or it can industrialize the appearance of expertise.

The difference rests on whether the signals correspond to genuine value.

Watch the archive: Google’s 2023 video “25 Years in Search: The Most Searched” is a useful corporate memory of how search became embedded in public culture. It should be read as a commemorative primary source, not an independent history.

Here’s the pattern.

Conclusion: the history of relevance

The history of SEO begins before the phrase itself. It begins with attempts to organize expanding bodies of information, to distinguish a specific term from a common one, and to follow associative relationships between records.

The Web transformed those abstract problems into a public and commercial infrastructure. Crawlers made pages discoverable.

Text retrieval made them matchable. Hyperlink analysis made authority computational.

Advertising made query attention profitable. Spam made the system adversarial.

Mobile devices, entity graphs, language models, and experience metrics widened the definition of relevance. Generative interfaces now make retrieval partly invisible inside a composed answer.

Across these phases, one pattern persists. Search engines operationalize a theory of value; publishers respond to it; opportunists imitate its signals; and engines revise their theory.

SEO is the professional practice that developed inside this cycle. At its weakest, it chases the current proxy.

At its strongest, it aligns technical access, editorial usefulness, institutional credibility, and real audience need.

This history also corrects the idea that optimization is fundamentally about pleasing an algorithm. Algorithms are intermediaries.

The durable objective is to make knowledge findable, interpretable, and worthy of selection. That work can involve a crawl directive, a title, a link, an expert biography, a measurement plan, a fast mobile page, a dataset, or an original argument.

The artifact changes because the model of relevance changes.

In 2026, the most defensible SEO strategy is therefore historical in spirit. It does not assume that today’s interface is permanent.

It does not mistake a visible metric for the value the metric attempts to represent. It asks what evidence a search or generative system needs in order to retrieve a source responsibly—and what evidence a human reader needs in order to trust it.

The future of SEO belongs neither to mechanical keyword insertion nor to mass-produced synthetic prose. It belongs to organizations that can create something true, structure it clearly, demonstrate why it deserves confidence, and make it available wherever discovery occurs.

References

1. Brin, S., & Page, L. (1998). The anatomy of a large-scale hypertextual Web search engine. Computer Networks and ISDN Systems, 30, 107–117.
2. Broder, A. (2002). A taxonomy of Web search. SIGIR Forum, 36(2), 3–10.
3. Bush, V. (1945). As we may think. The Atlantic Monthly.
4. CERN. (n.d.). A short history of the Web; Where the Web was born.
5. Fain, D. C., & Pedersen, J. O. (2006). Sponsored search: A brief history. Bulletin of the American Society for Information Science and Technology, 32(2).
6. Google. (2012). Introducing the Knowledge Graph: Things, not strings.
7. Google. (2019). Understanding searches better than ever before.
8. Google. (2024). Generative AI in Search: Let Google do the searching for you.
9. Google. (2025). AI Mode in Google Search.
10. Google Search Central. (2011–2023). Documentation and announcements on site quality, Penguin, page experience, helpful content, and people-first content and E-E-A-T.
11. Gyöngyi, Z., & Garcia-Molina, H. (2005). Web spam taxonomy. First International Workshop on Adversarial Information Retrieval on the Web.
12. Introna, L. D., & Nissenbaum, H. (2000). Shaping the Web: Why the politics of search engines matters. The Information Society, 16, 169–185.
13. McGill University. (n.d.). Creation of the first Internet search engine.
14. Monier, L., et al. (1997). The AltaVista search engine. USENIX Annual Technical Conference.
15. Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21.

Editorial note: Dates and product descriptions were checked against primary or institutional records where available. Official search-engine statements are identified as company accounts.

The origin of the term “SEO” remains disputed and is therefore described as an emergence in late-1990s trade usage rather than attributed to a single inventor.