By the end of the lesson, you can tell crawling, indexing, and ranking apart, you know what robots.txt is for and what it is not for, you know the most important page-level signals (title, description, canonical address, structured data), and you can judge realistically what can and cannot be guaranteed in search.
Why do we need to deal with this?
Public web content is useful only if it is found. SEO (Search Engine Optimization) deals with the relation of crawling, indexing, and content that is useful to the user. SEO is not a collection of tricks: its basics are the same as the basics of good content and a good web page: an accessible, clearly edited, accurate page. The basics matter next to AI-based search interfaces too, and there is no guaranteed ranking or appearance.
How does a page reach the search results?
When someone searches for "how does DNS work", the search engine must first know that a page answering this exists. According to Google's documentation, search works in three phases:
Crawling: automated programs (robots, crawlers, see 1.3) download the content from the pages they have discovered: they follow links, or visit known addresses.
Indexing: the system processes the text, structure, language, links, and other signals of the page, and puts it in a large database, the index. According to Google, indexing is not guaranteed.
Serving and ranking: when someone searches, the system gives the pages it judges most relevant and of the best quality.
flowchart LR
P[Public page] --> F[Crawling]
F --> I[Indexing possible]
I --> R[Ranking per query]
R --> U[User's result]
It is worth separating the three steps. If the search engine cannot fetch the page, the content cannot be crawled. If it cannot interpret it or does not consider it worth including, it is not indexed. If it is indexable, but there are better, more reliable, or closer results for the query, it does not necessarily rank high. Google also stresses that it does not guarantee the crawling, indexing, and serving of any page, even if the page follows its guidelines. The exact ranking rules are not public and change over time; nobody can honestly promise first place.
Crawlability: what we want found should be reachable
The basis of crawlability is that the server responds normally for the public page (2.4). Internal course material placed behind a password or an interface tied to login cannot be treated like a public article: the goal is visibility that fits the intention, not every page has to be searchable.
Internal links help both the visitor and the robot understand which pages are related. If an important page can be reached only by a hard-to-reach path created late by JavaScript, that is a risk (6.2). Traditional, meaningful links with descriptive anchor text strengthen the navigation: instead of "click here", "a detailed explanation of HTTP status codes" is at once more usable and more informative (9.2).
robots.txt is not access control
robots.txt is a text file in the root of the website with which the site tells robots which URLs they may request. A real example: the robots.txt file of MDN (11 October 2026):
User-agent: * applies to every robot, the Disallow lines exclude the API and the media files from crawling, and Sitemap gives the address of the sitemap.
A sitemap is a list of the important public URLs; it helps, but is not mandatory, as search engines discover most pages through links. It does not replace good navigation and useful content.
Semantic content: first make it clear to people
Semantic HTML (3.2, 9.2) says "what this is", not just "how it looks". A real main heading is h1, a paragraph is p, navigation is nav, the main content is main. This helps people using a screen reader, maintainers, and machine processing too.
A good article has a clear title, a logical subheading hierarchy, a short introduction, and the actual answer to the question. It is not good because a keyword appears in it thirty times. Google's SEO starter guide also asks for useful, trustworthy content written for people, clearly, divided into sections. If someone asks, "What is the difference between a cookie and localStorage?", the useful page first answers briefly, then refines, gives an example, and notes the limits (7.3).
Metadata: short signals about the document
Metadata placed in the head complements the body text. According to the guide:
The title element should be unique and descriptive: it should express the content of the page accurately. "Web programming" is less telling than "HTTP status codes – Web Programming 1".
The meta description (<meta name="description">) is a short summary characteristic of the page. Search engines can use it, but are not obliged to display exactly this. It should be in human language and say what the reader can expect; it is not a heap of keywords.
The canonical address (<link rel="canonical">) helps when the same or very similar content is reachable at several URLs (for example because of filter parameters). According to Google this is a strong signal, but not an instruction: the search engine may decide otherwise. A redirect (301, 2.4) is a stronger signal, a sitemap a weaker one.
The alt text of images (9.1) helps the search engine too to understand the content of the image.
Our faulty and fixed course pages (9.2) illustrate all this nicely: Lighthouse rated the SEO score of the faulty page 73 (no title, no meta description, the image has no alt text), of the fixed page 100. The text of the "Click here" link, however, was not reported by the SEO audit either, so checking for a meaningful link text remains a human task. Most of the missing elements are not a "trick" but basic quality signals.
Structured data: facts in a machine-interpretable form
Structured data is a standardized markup with which the facts of a page can be described explicitly for machines. According to Google's documentation the supported formats are JSON-LD, Microdata, and RDFa; the recommended one is JSON-LD, which describes the data in a <script> element, separate from the visible text, and most search structured data uses the schema.org vocabulary.
A JSON-LD description of a course-opening event (for the fictional Kurzuspont, syntactically verified with JSON.parse):
{"@context":"https://schema.org","@type":"Event","name":"Web Programming 1 – opening lecture","startDate":"2026-09-18T10:00:00+02:00","eventAttendanceMode":"https://schema.org/OfflineEventAttendanceMode","location":{"@type":"Place","name":"Building A, Room 101"},"organizer":{"@type":"Organization","name":"Kurzuspont"}}
Structured data does not entitle automatically to a special appearance: according to Google it can only make a more useful result possible, and the requirements must be met. Its use is responsible if the marked-up information is also really to be found on the page for the user, accurate, and up to date. It is not right to write a fictitious rating, a non-existent stock, or a misleading price only into the machine markup: the information given to the machine should match what is given to the human.
AI-based search and content quality
AI-based search interfaces (answer engines, assistants) may select and cite content in different ways. One often hears "AIO" or a similar abbreviation for such optimization, but it is not a standard technical term, so we do not use it in this course. According to Google's own description, no special optimization, separate file, or separate schema.org markup is needed for Search's AI features (AI Overviews, AI Mode): the proven practices of SEO remain valid. The condition for appearing is that the page is indexed and can appear in search with a snippet; even when the conditions are met this is not a guarantee.
Practical consequences, according to Google's advice:
allow crawling (robots.txt and any CDN too), and have internal links;
the important content should be on the page in text form, not only in an image;
the structured data should match the visible text.
This is surprisingly close to the principles of good documentation: the page should answer clearly what it claims; use descriptive headings; separate facts, examples, and opinions; mark the author, the date, and, if relevant, the original source. In health, financial, legal, or educational questions the author's responsibility, timeliness, and the transparency of references are especially important. Such content does not mean "writing the text for a machine": the goal is natural, accurate, well-structured professional content.
Worked example: fixing a course page in Kurzuspont
Suppose the title of the Web Programming 1 page is "Information", the time appears on a poster uploaded as an image, the enrolment deadline can be read in two paragraphs with different dates, and there is no clear main heading. For the search engine, the student using a screen reader, and an AI assistant alike, it is uncertain what the authoritative information is. The fixed page:
Element
Before
After
title
"Information"
"Web Programming 1 – enrolment and dates, autumn 2026"
Main heading
none
h1: "Web Programming 1"
Time
in an image
as text: "18 September 2026, 10:00, Building A, Room 101"
Deadline
two different dates
a single one, with the date of the update
Enrolment
"Click here"
"Enrol in the Web Programming 1 course"
Structured data
none
a JSON-LD event, with data matching the visible text
This guarantees neither a search ranking nor an AI citation, but the content can be used more reliably by everyone concerned.
Common misconceptions
Statement
Clarification
"SEO is repeating keywords as many times as possible."
Excessive repetition does not replace a relevant, understandable answer.
"Indexing guarantees first place."
Getting into the index and ranking are different steps, and neither is guaranteed.
"The meta description always appears exactly."
The search engine may choose another snippet of text.
"Structured data surely gives a highlighted result."
It helps interpretation, but promises no form of appearance or ranking.
"robots.txt hides the page."
The URL of a blocked page can still appear; noindex or a password is needed.
"AI search needs special markup."
According to Google no special optimization is needed.
In brief
The three steps of search: crawling, indexing, serving/ranking; none is guaranteed.
robots.txt manages robot traffic, it does not hide: for secrets access control is needed, for exclusion noindex.
The title, description, canonical address, alt, and structured data (JSON-LD) help understanding, but guarantee no ranking.
The structured data should match the visible content.
"AIO" is not a standard concept; good, textual, structured, and accurate content is the basis.
Review questions
What is the difference between crawling, indexing, and ranking?
Why is robots.txt not suitable for hiding a secret page?
What does the canonical address do, and why is it a "signal" and not an "instruction"?
Why is it dangerous to put data into the structured markup that is not on the page?
Searchability and content quality
Objectives
By the end of the lesson, you can tell crawling, indexing, and ranking apart, you know what
robots.txtis for and what it is not for, you know the most important page-level signals (title, description, canonical address, structured data), and you can judge realistically what can and cannot be guaranteed in search.Why do we need to deal with this?
Public web content is useful only if it is found. SEO (Search Engine Optimization) deals with the relation of crawling, indexing, and content that is useful to the user. SEO is not a collection of tricks: its basics are the same as the basics of good content and a good web page: an accessible, clearly edited, accurate page. The basics matter next to AI-based search interfaces too, and there is no guaranteed ranking or appearance.
How does a page reach the search results?
When someone searches for "how does DNS work", the search engine must first know that a page answering this exists. According to Google's documentation, search works in three phases:
flowchart LR P[Public page] --> F[Crawling] F --> I[Indexing possible] I --> R[Ranking per query] R --> U[User's result]It is worth separating the three steps. If the search engine cannot fetch the page, the content cannot be crawled. If it cannot interpret it or does not consider it worth including, it is not indexed. If it is indexable, but there are better, more reliable, or closer results for the query, it does not necessarily rank high. Google also stresses that it does not guarantee the crawling, indexing, and serving of any page, even if the page follows its guidelines. The exact ranking rules are not public and change over time; nobody can honestly promise first place.
Crawlability: what we want found should be reachable
The basis of crawlability is that the server responds normally for the public page (2.4). Internal course material placed behind a password or an interface tied to login cannot be treated like a public article: the goal is visibility that fits the intention, not every page has to be searchable.
Internal links help both the visitor and the robot understand which pages are related. If an important page can be reached only by a hard-to-reach path created late by JavaScript, that is a risk (6.2). Traditional, meaningful links with descriptive anchor text strengthen the navigation: instead of "click here", "a detailed explanation of HTTP status codes" is at once more usable and more informative (9.2).
robots.txt is not access control
robots.txtis a text file in the root of the website with which the site tells robots which URLs they may request. A real example: therobots.txtfile of MDN (11 October 2026):User-agent: *applies to every robot, theDisallowlines exclude the API and the media files from crawling, andSitemapgives the address of the sitemap.A sitemap is a list of the important public URLs; it helps, but is not mandatory, as search engines discover most pages through links. It does not replace good navigation and useful content.
Semantic content: first make it clear to people
Semantic HTML (3.2, 9.2) says "what this is", not just "how it looks". A real main heading is
h1, a paragraph isp, navigation isnav, the main content ismain. This helps people using a screen reader, maintainers, and machine processing too.A good article has a clear title, a logical subheading hierarchy, a short introduction, and the actual answer to the question. It is not good because a keyword appears in it thirty times. Google's SEO starter guide also asks for useful, trustworthy content written for people, clearly, divided into sections. If someone asks, "What is the difference between a cookie and
localStorage?", the useful page first answers briefly, then refines, gives an example, and notes the limits (7.3).Metadata: short signals about the document
Metadata placed in the
headcomplements the body text. According to the guide:titleelement should be unique and descriptive: it should express the content of the page accurately. "Web programming" is less telling than "HTTP status codes – Web Programming 1".<meta name="description">) is a short summary characteristic of the page. Search engines can use it, but are not obliged to display exactly this. It should be in human language and say what the reader can expect; it is not a heap of keywords.<link rel="canonical">) helps when the same or very similar content is reachable at several URLs (for example because of filter parameters). According to Google this is a strong signal, but not an instruction: the search engine may decide otherwise. A redirect (301, 2.4) is a stronger signal, a sitemap a weaker one.alttext of images (9.1) helps the search engine too to understand the content of the image.Our faulty and fixed course pages (9.2) illustrate all this nicely: Lighthouse rated the SEO score of the faulty page 73 (no
title, no meta description, the image has noalttext), of the fixed page 100. The text of the "Click here" link, however, was not reported by the SEO audit either, so checking for a meaningful link text remains a human task. Most of the missing elements are not a "trick" but basic quality signals.Structured data: facts in a machine-interpretable form
Structured data is a standardized markup with which the facts of a page can be described explicitly for machines. According to Google's documentation the supported formats are JSON-LD, Microdata, and RDFa; the recommended one is JSON-LD, which describes the data in a
<script>element, separate from the visible text, and most search structured data uses the schema.org vocabulary.A JSON-LD description of a course-opening event (for the fictional Kurzuspont, syntactically verified with
JSON.parse):{ "@context": "https://schema.org", "@type": "Event", "name": "Web Programming 1 – opening lecture", "startDate": "2026-09-18T10:00:00+02:00", "eventAttendanceMode": "https://schema.org/OfflineEventAttendanceMode", "location": { "@type": "Place", "name": "Building A, Room 101" }, "organizer": { "@type": "Organization", "name": "Kurzuspont" } }Structured data does not entitle automatically to a special appearance: according to Google it can only make a more useful result possible, and the requirements must be met. Its use is responsible if the marked-up information is also really to be found on the page for the user, accurate, and up to date. It is not right to write a fictitious rating, a non-existent stock, or a misleading price only into the machine markup: the information given to the machine should match what is given to the human.
AI-based search and content quality
AI-based search interfaces (answer engines, assistants) may select and cite content in different ways. One often hears "AIO" or a similar abbreviation for such optimization, but it is not a standard technical term, so we do not use it in this course. According to Google's own description, no special optimization, separate file, or separate schema.org markup is needed for Search's AI features (AI Overviews, AI Mode): the proven practices of SEO remain valid. The condition for appearing is that the page is indexed and can appear in search with a snippet; even when the conditions are met this is not a guarantee.
Practical consequences, according to Google's advice:
robots.txtand any CDN too), and have internal links;This is surprisingly close to the principles of good documentation: the page should answer clearly what it claims; use descriptive headings; separate facts, examples, and opinions; mark the author, the date, and, if relevant, the original source. In health, financial, legal, or educational questions the author's responsibility, timeliness, and the transparency of references are especially important. Such content does not mean "writing the text for a machine": the goal is natural, accurate, well-structured professional content.
Worked example: fixing a course page in Kurzuspont
Suppose the title of the Web Programming 1 page is "Information", the time appears on a poster uploaded as an image, the enrolment deadline can be read in two paragraphs with different dates, and there is no clear main heading. For the search engine, the student using a screen reader, and an AI assistant alike, it is uncertain what the authoritative information is. The fixed page:
titleh1: "Web Programming 1"This guarantees neither a search ranking nor an AI citation, but the content can be used more reliably by everyone concerned.
Common misconceptions
robots.txthides the page."noindexor a password is needed.In brief
robots.txtmanages robot traffic, it does not hide: for secrets access control is needed, for exclusionnoindex.alt, and structured data (JSON-LD) help understanding, but guarantee no ranking.Review questions
robots.txtnot suitable for hiding a secret page?Learning resources