Website Wisdom · Grey Swan Designs

Robots.txt vs Sitemap.xml vs LLMs.txt

Three files. Three different purposes. One important part of building a website that search engines and emerging AI technologies can navigate more effectively.

Technical SEO Website Optimisation AI & Search Approx. 9 min read

When it comes to SEO, most business owners think about keywords, page titles, meta descriptions, backlinks and quality content. These are all important parts of a successful search engine optimisation strategy.

But there's another side of SEO that often goes unnoticed: the technical files that help automated systems discover, navigate and interpret your website.

You may have come across three filenames while researching website optimisation: robots.txt, sitemap.xml and the increasingly discussed llms.txt.

Although they are often mentioned together, they do different jobs. Understanding those differences can help you make better decisions about your website's technical SEO and its readiness for an increasingly AI-driven web.

Think of your website as a building. Robots.txt provides access instructions, sitemap.xml supplies the directory, and llms.txt offers a curated guide to the information inside.

01 / What is robots.txt?

The robots.txt file is a plain-text file located in the root directory of your website. Its primary purpose is to provide instructions to automated crawlers about which areas of your website they are allowed or not allowed to request.

Search engines such as Google use automated crawlers to discover and access website content. These crawlers follow links, request pages and collect information that can eventually be used in search results.

Robots.txt allows website owners to provide rules for these crawlers, helping manage which sections they should access.

What does robots.txt actually do?

  • Provides crawling instructions to search engine bots.
  • Can restrict crawlers from accessing specified directories or URLs.
  • Can identify different crawler user agents.
  • Can reference your XML sitemap.
  • Can help manage unnecessary crawler requests.

Example of a robots.txt file

Example configuration
User-agent: *
Allow: /

Disallow: /wp-admin/

Sitemap: https://www.example.com/sitemap_index.xml

In this example, the instructions allow all matching crawlers to access the website while asking them not to crawl the WordPress administration directory. The sitemap location is also provided.

Important: Robots.txt is not a security feature and should never be used to protect confidential information.

It also does not guarantee that a blocked URL will disappear from Google search results. A search engine may still know that a URL exists through external links. If a page must be kept out of search results, use appropriate indexing controls such as noindex or password protection.

You can read Google's official explanation here: Google Search Central – Robots.txt .

02 / What is sitemap.xml?

While robots.txt tells crawlers which URLs they can or cannot request, an XML sitemap serves a different purpose.

A sitemap.xml file is essentially a structured list of important URLs on your website. It helps search engines discover pages that you want them to consider for crawling and indexing.

Think of it as a directory of your website. Rather than relying entirely on crawlers finding every page through internal links, a sitemap provides a convenient reference to the URLs you consider important.

Why is an XML sitemap important?

⌕

Page discovery

Helps search engines discover important pages, particularly on large or newly established websites.

↻

Updated content

Can communicate when pages were last modified, helping search engines identify potentially updated content.

▤

Website organisation

Provides a structured overview of important website URLs and supported media content.

✓

Search monitoring

Allows you to submit and monitor sitemap processing through Google Search Console.

Example of an XML sitemap

Simplified XML example
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-11-13</lastmod>
  </url>

  <url>
    <loc>https://www.example.com/services/</loc>
    <lastmod>2026-11-10</lastmod>
  </url>

  <url>
    <loc>https://www.example.com/contact/</loc>
    <lastmod>2026-11-08</lastmod>
  </url>

</urlset>

This simplified example lists three website URLs and their last modification dates. Real sitemaps can contain many more entries and can also be separated into multiple files using a sitemap index.

Does every website need a sitemap?

Not necessarily, but it is generally a good idea. Google can often discover pages through internal links without a sitemap. However, sitemaps are particularly useful for larger websites, new websites, websites with complex structures and sites containing specialised media content.

If you're using WordPress with an SEO plugin such as Rank Math, your sitemap may already be generated automatically. A common location is:

Typical WordPress sitemap URL
https://www.yourwebsite.co.za/sitemap_index.xml

You can find out more in Google's documentation: Google Search Central – Sitemaps .

Remember: Submitting a sitemap does not guarantee that Google will crawl or index every URL listed in it. It is a discovery signal, not an indexing guarantee.

03 / What is llms.txt?

This is where things get interesting.

As AI-powered tools become increasingly common, people are changing the way they search for information. Instead of always typing a question into Google, users may ask AI assistants such as ChatGPT, Claude or other tools to explain a service, compare products or research a business.

This has created interest in a proposed website file called llms.txt.

The idea is straightforward: provide AI systems and compatible tools with a clean, structured overview of your website, your business and the most useful information available on your pages.

Unlike robots.txt, which focuses on crawler access instructions, llms.txt is intended to provide context and curated links to useful website information.

What might an llms.txt file contain?

  • A brief introduction to your business.
  • Your main products and services.
  • Important website pages and their descriptions.
  • Useful documentation and frequently referenced content.
  • Information that helps explain your website's structure and purpose.

Example of an llms.txt file

Simplified Markdown example
# Example Business

> Example Business provides professional
> website design and digital marketing services.

## About Us

We help businesses establish and grow
their online presence through websites,
SEO and digital marketing.

## Main Services

- [Website Design](https://www.example.com/web-design/)
- [SEO Services](https://www.example.com/seo/)
- [Website Hosting](https://www.example.com/hosting/)
- [Contact Us](https://www.example.com/contact/)

## Important Information

- [About Our Business](https://www.example.com/about/)
- [Frequently Asked Questions](https://www.example.com/faq/)
- [Our Services](https://www.example.com/services/)

This example provides a simple, readable overview of a business and points to its most relevant pages. The intention is to make it easier for compatible AI agents and tools to identify useful information without having to interpret an entire website's navigation and layout first.

Does llms.txt improve your AI search visibility?

This is where we need to separate the opportunity from the marketing hype.

The llms.txt proposal is an emerging convention, not a universally adopted web standard. Publishing one does not guarantee that ChatGPT, Claude, Gemini or any other AI platform will read it, recommend your business or cite your website.

Google has also explicitly stated that it does not use llms.txt for Google Search or its generative search features. According to Google, creating this file neither helps nor harms your Google rankings.

Our take: An llms.txt file can be a worthwhile, relatively simple addition to a website's technical setup, particularly where AI tools and agents support the convention. However, it should complement proper SEO fundamentals rather than replace them.

You can explore the original proposal here: The llms.txt proposal on GitHub .

For Google's current position, see: Google's Guide to Optimising for Generative AI Search .

04 / The difference between all three files

Now that we understand what each file does, let's put them side by side.

FileMain purposePrimary audience
robots.txt Provides instructions about which URLs crawlers may request. Search engine and other compliant web crawlers.
sitemap.xml Lists important website URLs to assist content discovery. Search engines and sitemap-compatible crawlers.
llms.txt Provides a curated, readable overview of website information. AI agents and tools that choose to support the proposal.

Robots.txt controls crawler instructions. Sitemap.xml helps crawlers discover pages. LLMs.txt provides a curated guide to your content.

05 / Do these files work together?

Absolutely. In fact, there is no reason to think of these files as competing alternatives. They address different aspects of how automated systems interact with your website.

A properly configured website can use all three files together.

How they complement one another

01

robots.txt

Provides crawler access instructions and can point crawlers towards your sitemap.

02

sitemap.xml

Provides a structured list of important URLs that search engines can discover.

03

llms.txt

Offers compatible AI tools a concise guide to your business and website content.

04

Good website structure

Makes your content easier to navigate through clear pages, internal links and meaningful information.

For example, your robots.txt file can include a reference to your XML sitemap, allowing crawlers to find its location.

Example robots.txt sitemap reference
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap_index.xml

Your llms.txt file can then provide a separate overview of your business and link to the pages that explain your services, products and expertise.

These files do not replace the need for quality website content, proper internal linking, good metadata, structured data or a technically sound website.

06 / How to check whether your website has these files

You don't necessarily need expensive SEO software to perform a basic check. You can start by opening the relevant file locations directly in your browser.

Check your robots.txt file

https://www.yourwebsite.co.za/robots.txt

Check your XML sitemap

https://www.yourwebsite.co.za/sitemap.xml

If you're using Rank Math on WordPress, your sitemap may instead be available at:

https://www.yourwebsite.co.za/sitemap_index.xml

Check your llms.txt file

https://www.yourwebsite.co.za/llms.txt

If a file doesn't load, it doesn't automatically mean that your website has an SEO problem. The file may be located elsewhere, generated dynamically, or simply not configured.

What should you look for?

  • Robots.txt: Check that important public pages aren't accidentally blocked.
  • Sitemap.xml: Check that important, indexable URLs are included and that the sitemap is accessible.
  • LLMs.txt: If you publish one, check that its business information and links are accurate and up to date.
  • Google Search Console: Check sitemap processing, indexing reports and crawl-related issues.

07 / Common technical SEO mistakes to avoid

Having these files is one thing. Configuring them correctly is another. Small mistakes can create unnecessary problems for your website.

Blocking important pages in robots.txt

Accidentally disallowing important sections of your website can prevent compliant crawlers from accessing their content. Always review your rules before publishing changes.

Including incorrect URLs in your sitemap

Your sitemap should focus on the URLs you actually want search engines to consider for indexing. Avoid including redirected, broken, duplicate or intentionally noindexed pages.

Assuming a sitemap guarantees indexing

A sitemap helps with discovery, but search engines still decide whether individual URLs should be crawled and indexed.

Using robots.txt as website security

Robots.txt is not an access-control mechanism. Sensitive information should be protected through authentication, appropriate server permissions and other genuine security measures.

Treating llms.txt as a guaranteed AI SEO shortcut

Publishing llms.txt does not automatically improve rankings or guarantee AI citations. It is important to understand its current limitations and treat it as an optional supporting convention.

Forgetting to maintain your files

Websites change. New pages are created, old services are removed and URLs are updated. Your technical files should reflect the website as it exists today, not how it looked several months ago.

08 / So, does your business website need all three?

Our recommendation is to get the fundamentals right first.

For most business websites, a correctly configured robots.txt file and a valid XML sitemap are sensible technical SEO foundations. They help search engines understand where they can crawl and which pages your business considers important.

An llms.txt file is an additional consideration for businesses interested in making their website information available in a more structured format for compatible AI tools.

But don't lose sight of what matters most: your website still needs valuable content, clear service descriptions, accurate business information, sensible page structures, fast loading times, mobile responsiveness and a good user experience.

The bigger picture: Search engine optimisation is not simply about adding files to your website. It's about making sure your website is technically accessible, well organised, informative and genuinely useful to the people looking for your services.

Don't just build a website that looks good. Build one that search engines can navigate, customers can understand and technology can interpret.

09 / Frequently asked questions

Does robots.txt improve SEO rankings?

Not directly. Robots.txt provides crawler access instructions. Correct configuration can help search engines access the appropriate parts of your website, but the file itself is not a ranking booster.

Is sitemap.xml still important for SEO?

Yes. Sitemaps remain a useful way to help search engines discover important URLs, especially on larger, newer or more complex websites. They do not guarantee indexing.

Does Google read llms.txt?

Google has stated that it does not use llms.txt for Google Search or its AI search features. The file may still be useful to other tools that choose to support the proposal.

Can llms.txt help my business appear in ChatGPT?

There is no guarantee. Publishing the file does not ensure that ChatGPT will read, reference or recommend your website. AI visibility depends on many factors, and the adoption of llms.txt is not universal.

Can I have all three files on my WordPress website?

Yes. They serve different purposes and can coexist. WordPress SEO plugins such as Rank Math can generate XML sitemaps and help manage robots.txt settings. An llms.txt file can be added separately if you choose to implement it.

Where should these files be located?

Robots.txt is expected at your website's root directory. An XML sitemap is commonly published at the root or referenced through a sitemap index. The llms.txt proposal uses a conventional root location, although it also supports scoped locations within a website.

10 / Is your website technically SEO-ready?

A website can look fantastic on the surface and still have technical issues behind the scenes. Incorrect crawling rules, missing sitemaps, broken links, duplicate pages and poorly structured content can all affect how effectively your website is discovered and understood.

That's why technical SEO should be part of your website strategy from the beginning, not something you only investigate when your search traffic starts declining.

At Grey Swan Designs, we believe that a successful website needs more than attractive visuals. It needs a solid technical foundation, useful content and a strategy that supports your business goals.

Let's get your website working smarter

Your website deserves better SEO.

Whether you need a new website, an existing website revamp or help improving your technical SEO, Grey Swan Designs can help you build a stronger online presence.

From website design and hosting to technical SEO and ongoing optimisation, we help businesses get more out of their websites.

Let's Talk About Your Website

Grey Swan Designs · Creative websites. Smarter SEO. Better digital experiences.