How to Write a Robots.txt File: 2026 Guide with Examples

Are Google's crawlers unable to reach your website's important pages, or are search engine bots wasting time on unnecessary URLs? The cause may be a misconfigured robots.txt file. The answer to how to write a robots.txt file isn't just a matter of adding a few commands; it comes down to correctly deciding which content should be crawled.

In this guide, we'll look at how robots.txt works, Googlebot rules, sample configurations for different types of websites, and common SEO mistakes. We'll also explain how to test your file in 2026 in line with Google's current guidelines.

What Is Robots.txt and Why Does It Matter for SEO?

Robots.txt is a plain text file located in a website's root directory that tells search engine crawlers which URL paths they can and cannot crawl. It works based on a standard called the Robots Exclusion Protocol (REP).

Before crawling a website, Googlebot checks the relevant rules in the robots.txt file. These rules help manage crawl access, but on their own they do not determine whether pages will appear in Google's index.

Core Functions of a Robots.txt File

• Crawl management: Controls search engines' access to specific directories and URL paths.
• Reducing unnecessary crawling: Can limit the crawling of search results, filter combinations and similar low-value URLs.
• Protecting server resources: Can help reduce unnecessary bot requests, especially on sites with a high volume of URLs.
• Specifying the XML sitemap: Can tell crawlers where your sitemap is located through the Sitemap directive.

Robots.txt is not a direct ranking factor. However, when it is misconfigured, it can block the crawling of important pages and negatively affect organic visibility.

Especially on large websites, evaluating crawl efficiency is one of the key parts of a comprehensive technical SEO optimization project.

The Difference Between Robots.txt and Noindex

Robots.txt and noindex are often confused. In reality, they work at different stages: robots.txt controls crawl access, while noindex controls whether a page is indexed in search results when it is properly detected.

Important: When a URL is blocked by robots.txt, Google may not be able to see the noindex directive on the page. For this reason, applying a robots.txt block to a page you want removed from the index at the same time is not a sound starting method.

How to Write a Robots.txt File: Step-by-Step Setup

You don't need any special software to create a valid robots.txt file. A plain text editor and access to your site's root directory are enough. However, you need to identify the target URLs before applying the rules in a live environment.

1. Identify the Areas to Crawl and to Block

First, review the URL structures on your website. Product, category, service and blog pages should generally be crawlable. Internal search results, certain filter combinations and technical action URLs, on the other hand, can be restricted depending on the situation.

Not every directory needs to be blocked automatically. For example, blocking the JavaScript or CSS files needed for products to display can make it harder for Google to render the page correctly.

2. Create the Robots.txt File

Open a text editor on your computer and save the file as robots.txt with UTF-8 encoding. Use a plain text editor rather than word processors like Word that add their own formatting.

To get started, you can use the following structure, which imposes no crawl restrictions:

User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml

In this example, the asterisk represents all crawlers. The empty Disallow line does not block any path. The sitemap URL is just an example; it should be replaced with the actual XML sitemap URL of your own site.

3. Upload the File to the Root Directory

Place the file you created in the root directory of your site's main host. For example, it must be accessible at https://www.example.com/robots.txt.

Placing the file in a subfolder is not enough. In addition, the www version, the non-www domain and different subdomains are each treated within their own scope. A file on HTTPS does not automatically manage HTTP URLs.

4. Check That the File Is Accessible

Open the robots.txt URL in your browser and verify that the text is displayed. Check that the server returns an HTTP 200 response, that the file does not redirect to an HTML error page, and that it contains the expected rules.

As a final step before going live, review the robots.txt report in Google Search Console. Also evaluate the crawlability of critical pages with the URL Inspection tool.

Robots.txt Commands and Usage Examples

The core fields Google supports in a robots.txt file are the User-agent, Disallow, Allow and Sitemap directives. Each command has a different purpose and matching behavior.

User-agent: Defines Which Bot Is Affected

The User-agent command specifies which crawler the following rules apply to. The asterisk represents the general group covering all bots, while Googlebot refers to Google's main web crawler.

User-agent: Googlebot
Disallow: /ozel-dizin/

This example prevents Googlebot from crawling the specified directory. However, the behavior of Google's other crawlers, which serve different purposes, may need to be evaluated separately.

Disallow: Blocks Specific URL Paths

The Disallow command specifies the URL paths you don't want crawled. For example, the structure below defines a crawl block for the /arama/ directory and the URLs under it.

User-agent: *
Disallow: /arama/

Before blocking a directory, check whether that area contains pages that bring in organic traffic or have the potential to. A single misused rule can affect a large number of important URLs.

Allow: Defines Exceptions Within Blocked Areas

The Allow command lets you grant access to specific URLs within a broader block. Google applies the more specific rule among matching rules; when conflicting matches are of equal length, Allow is preferred.

User-agent: *
Disallow: /dokumanlar/
Allow: /dokumanlar/genel-rehber.pdf

This structure blocks the documents directory in general while allowing the specified PDF file to be crawled.

Sitemap: Declares the XML Sitemap

The Sitemap line makes it easier for search engines to find your sitemap. This directive should use a full URL rather than a relative path.

Sitemap: https://www.example.com/sitemap.xml

Adding a URL to a sitemap does not mean that page will definitely be crawled or indexed. You should build a structure that is consistent with your robots.txt rules, canonical signals and page accessibility.

Wildcards and URL Matching

In Google's interpretation of robots.txt, the asterisk (*) can match any sequence of characters. The dollar sign ($) is used to indicate the end of a URL path.

User-agent: *
Disallow: /*?sort=
Disallow: /*.pdf$

The first rule targets example URLs where the sort parameter follows the path. The second rule blocks the crawling of URLs whose path ends in .pdf. Patterns like these should be tested with real URL examples before use.

Also keep in mind that Google does not support the Crawl-delay and Noindex directives in robots.txt files. Trying to manage crawl rate or indexing status with these commands will not produce the expected result.

Examples for WordPress, E-Commerce and Corporate Sites

There is no single ideal robots.txt template that works for every website. Rules should be shaped according to the content management system, the URL architecture and the resources search engines need to access.

Robots.txt Example for a WordPress Site

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap_index.xml

In this example, crawling of the WordPress admin directory is restricted, while the admin-ajax.php file, which can be used in some front-end functions, is allowed. The sitemap URL may vary depending on the SEO plugin or sitemap infrastructure you use.

In WordPress systems, robots.txt can also be generated virtually. Before deciding how to edit it, check whether the existing file is a physical file or an output generated by the application.

Robots.txt Example for an E-Commerce Site

User-agent: *
Disallow: /sepet/
Disallow: /odeme/
Disallow: /arama/
Disallow: /*?sort=
Sitemap: https://www.example.com/sitemap.xml

On e-commerce sites, a large number of filter, sorting and search URLs can be generated. Uncontrolled crawling of these URLs can cause bots to visit the same or very similar content over and over again.

However, blocking all filtered pages is not always the right move. For example, category URLs for brands, colors or product attributes that have search demand can be valuable for organic visibility.

For this reason, in e-commerce SEO optimization work, filter URLs should be classified first, and the crawling and indexing strategy should be defined afterward.

Robots.txt Example for a Corporate Website

User-agent: *
Disallow: /site-ici-arama/
Disallow: /gecici-raporlar/
Sitemap: https://www.example.com/sitemap.xml

On corporate sites, the priority is making sure that service, industry, case study and resource center pages are not blocked by mistake. If subdomains are used for multiple languages or countries, the robots.txt scope of each host should be reviewed separately.

The directory names in these examples are illustrative. In particular, the security of payment areas cannot be ensured with robots.txt; authorization, session management and the necessary server-side access controls should be used.

Most Common SEO Mistakes in Robots.txt Files

Mistakes in a robots.txt file may look minor, but they can affect large groups of URLs. There are critical risks that need to be checked, especially during site launches, infrastructure changes and migrations.

Accidentally Blocking the Entire Site from Crawling

User-agent: *
Disallow: /

This structure prevents compliant bots from crawling the entire site. It can be used intentionally in test environments, but leaving it on a live site can cause serious organic visibility problems.

Blocking CSS and JavaScript Resources

Google can use CSS and JavaScript resources when rendering pages. Blocking these resources wholesale can make it harder to render content correctly, especially content generated with JavaScript.

Instead of applying broad resource blocks without a security reason, evaluate the files that are truly unnecessary one by one.

Mistaking Robots.txt for a Privacy Tool

The robots.txt file is publicly accessible. Listing hidden admin paths in it does not hide them from the outside world; on the contrary, it can make those URLs easier to discover.

Authentication and authorization mechanisms should be used for private information. Robots.txt is not a firewall.

Overlooking URL Case Differences

Google's robots.txt path matching is case-sensitive. For this reason, /Katalog/ and /katalog/ may not be covered by the same rule.

Prepare your file according to your actual URL structure. Test different variations, especially in multilingual projects and those with custom URL redirects.

Ignoring Server Errors

A robots.txt file that constantly returns a server error can affect crawling. In the face of 5xx responses or connection problems, Google may temporarily pause or reduce crawling.

It is important to run these checks during site migrations and server changes. With regular SEO audits and reporting processes, faulty rules and access issues can be detected early.

How to Test Robots.txt and Track Performance

Creating the file is only the first stage of the process. You need to verify that it works correctly and then monitor crawl behavior.

Checking with Google Search Console

1. Open the relevant website property in your Google Search Console account.
2. Review the robots.txt report in the Settings section.
3. Check the version of the file Google sees and its fetch status.
4. Evaluate the crawl information in the URL Inspection tool for important URLs.
5. After making the necessary fixes, request a recrawl of the file and track the results.

The robots.txt report in Search Console provides information about the file's accessibility and the content Google sees. It should not be regarded as a current equivalent of the old standalone robots.txt Tester tool, which interactively tested each URL rule.

SEO Metrics to Track

After robots.txt changes, don't focus solely on an increase in organic traffic. First, verify that the target URLs are being crawled correctly and that no unexpected access blocks have appeared on important pages.

Pre-Launch Robots.txt Checklist

• Is the file located in the root directory of the correct main host?
• Is the file named robots.txt and encoded in UTF-8?
• Is there a rule that blocks the live site entirely?
• Are important service, product and category URLs crawlable?
• Are the CSS and JavaScript resources needed to render the page accessible?
• Do the Allow and Disallow rules match correctly on real URL examples?
• Is the sitemap URL valid, accessible and up to date?
• Was a backup of the file taken before the change?

Especially on large websites, putting file changes under version control makes it easier to roll back to the previous configuration if something goes wrong.

To verify Google's current technical requirements, you can also review the official robots.txt technical documentation.

Frequently Asked Questions

What happens if there is no robots.txt file?

Having a robots.txt file is not mandatory. When there are no other access restrictions, search engines can crawl the accessible URLs on your site. The file becomes important especially when you want to manage crawl access deliberately.

Does the robots.txt file affect Google rankings?

Robots.txt is not a direct ranking factor. However, accidentally blocking important pages can make it harder for Google to access those pages and evaluate their content. For this reason, it can have indirect SEO effects.

Can a page be removed from Google with robots.txt?

No, robots.txt alone does not guarantee removal from Google results. Google may list a blocked URL that is linked from other pages without showing any content. To remove a page from the index, consider using noindex on an accessible page or applying the appropriate HTTP status codes.

Where is the WordPress robots.txt file located?

In WordPress, the file can be viewed at the /robots.txt path of the site's root address. Some installations use a virtual output generated by WordPress instead of a physical file. How you edit it may vary depending on your server infrastructure and SEO plugin.

When does Google see robots.txt changes?

Google generally caches the robots.txt file and, under normal conditions, can refresh it roughly every 24 hours. This period may be longer when there are connection or server issues. You can request a recrawl of the updated file through Search Console, but immediate application is not guaranteed.

Conclusion: Manage Crawling with the Right Robots.txt Configuration

The core answer to how to write a robots.txt file is this: define the right User-agent groups, build your Disallow and Allow rules carefully, place the file in the root directory, and test it with real URLs.

In 2026, a successful robots.txt strategy should focus on limiting unnecessary URL crawling while keeping valuable content accessible. This approach is especially important for e-commerce and corporate sites with thousands of URLs.

Is your website's robots.txt file configured correctly? At Seo Performans, we evaluate crawl blocks, indexing issues and technical SEO risks together. For a site-specific analysis and an actionable improvement plan, you can explore our Technical SEO Optimization service.