{"id":213,"date":"2024-05-05T19:04:34","date_gmt":"2024-05-05T19:04:34","guid":{"rendered":"https:\/\/zalvis.com\/docs\/?p=213"},"modified":"2024-05-05T19:04:34","modified_gmt":"2024-05-05T19:04:34","slug":"how-to-create-a-robots-txt-file-in-cpanel","status":"publish","type":"post","link":"https:\/\/zalvis.com\/docs\/how-to-create-a-robots-txt-file-in-cpanel.html","title":{"rendered":"How To Create A Robots.txt File In CPanel"},"content":{"rendered":"<article id=\"general\">If you\u2019ve ever built your website, you may have heard of a robots.txt file and wondered, what is this file for? Well, you\u2019re in the right place! Below, we will review this file and why it\u2019s crucial.<\/article>\n<article>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_81 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/zalvis.com\/docs\/how-to-create-a-robots-txt-file-in-cpanel.html\/#What_Is_A_Robotstxt_File\" >What Is A Robots.txt File?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/zalvis.com\/docs\/how-to-create-a-robots-txt-file-in-cpanel.html\/#How_To_Create_The_Robotstxt_File\" >How To Create The Robots.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/zalvis.com\/docs\/how-to-create-a-robots-txt-file-in-cpanel.html\/#Examples_Of_Usage_And_Syntax_Rules\" >Examples Of Usage And Syntax Rules<\/a><\/li><\/ul><\/nav><\/div>\n<h3 id=\"what-is-a-robots-txt-file\"><span class=\"ez-toc-section\" id=\"What_Is_A_Robotstxt_File\"><\/span>What Is A Robots.txt File?<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>First of all, the <em><strong>robots.txt<\/strong><\/em> is nothing more than a plain text file (ASCII or UTF-8) located in your domain <strong>root\u00a0directory<\/strong>, which blocks (or allows) search engines to access certain areas of your site. The <em>robots.txt<\/em> contains a simple set of commands (or directives), and it\u2019s typically applied to restrict crawler traffic onto your server, thus preventing unwanted resource usage.<\/p>\n<p>Search engines use so-called <strong>crawlers<\/strong> (or bots) to index parts of a website and return those as search results. You might want specific sensitive data stored on your server to be inaccessible for web searches. The robots.txt file helps you do just that.<\/p>\n<p><strong>Note:<\/strong> Files or pages on your website are not entirely removed from crawlers if these files are indexed\/referenced from other websites. To protect your URL from appearing in Google search engines, you can password-protect the files directly from your server.<\/p>\n<\/article>\n<article id=\"creation\">\n<h3 id=\"how-to-create-the-robots-txt-file\"><span class=\"ez-toc-section\" id=\"How_To_Create_The_Robotstxt_File\"><\/span>How To Create The Robots.txt File<strong><br \/>\n<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>To create your <em>robots.txt<\/em> file (if not already existent), follow the following steps:<\/p>\n<p><strong>1.<\/strong> Log into your cPanel account<\/p>\n<p><strong>2.<\/strong> Navigate to <em>Files<\/em>\u00a0section and click on <strong>File Manager<\/strong><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-214\" src=\"https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7917.png\" alt=\"\" width=\"1536\" height=\"580\" srcset=\"https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7917.png 1536w, https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7917-300x113.png 300w, https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7917-1024x387.png 1024w, https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7917-768x290.png 768w\" sizes=\"auto, (max-width: 1536px) 100vw, 1536px\" \/><\/p>\n<p><strong>3.<\/strong>\u00a0 Browse File Manager to the website directory ( e.g public_html ) then Click on \u201c<strong>File<\/strong>\u201d\u00a0 &gt;&gt; Type in \u201crobots.txt\u201d\u00a0 &gt;&gt; Click on \u201c<strong>Create New File<\/strong>\u201d.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-215\" src=\"https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7918.png\" alt=\"\" width=\"1536\" height=\"759\" srcset=\"https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7918.png 1536w, https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7918-300x148.png 300w, https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7918-1024x506.png 1024w, https:\/\/zalvis.com\/docs\/wp-content\/uploads\/2024\/05\/IMG_7918-768x380.png 768w\" sizes=\"auto, (max-width: 1536px) 100vw, 1536px\" \/><\/p>\n<p><strong>4.<\/strong> Now, you are free to edit the content of this file by double-clicking on it.<\/p>\n<p><strong>Note: <\/strong>you can create only <strong>one <\/strong><em>r<\/em><em>obots.txt<\/em> file for each domain. Duplicates are not allowed on the same root path. Each domain or sub-domain must contain its own <em>robots.txt<\/em> file. <strong><br \/>\n<\/strong><\/p>\n<article id=\"examples\">\n<h3 id=\"examples-of-usage-and-syntax-rules\"><span class=\"ez-toc-section\" id=\"Examples_Of_Usage_And_Syntax_Rules\"><\/span>Examples Of Usage And Syntax Rules<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Usually, a <em>robots.txt<\/em> file contains one or more rules, each on its own separate line. Each rule blocks or allows access to a given crawler to a specified file path or the entire website.<\/p>\n<ul>\n<li>Block all crawlers (user-agents) from accessing the <em>logs<\/em> and <em>ssl<\/em> directories.<\/li>\n<\/ul>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">User-agent:*\r\nDisallow: \/logs\/\r\nDisallow: \/ssl\/<\/pre>\n<ul>\n<li>Block all crawlers to index the whole site.<\/li>\n<\/ul>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">User-agent: *\r\nDisallow: \/<\/pre>\n<ul>\n<li>Allow all user agents to access the entire site.<\/li>\n<\/ul>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">User-agent: *\r\nAllow: \/<\/pre>\n<ul>\n<li>Block indexation for the whole site from a specific crawler.<\/li>\n<\/ul>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">User-agent: Bot1\r\nDisallow: \/<\/pre>\n<ul>\n<li>Allow index to a specific web crawler and prevents indexation from others.<\/li>\n<\/ul>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">User-agent: Googlebot\r\nDisallow:\r\nUser-agent: *\r\nDisallow: \/<\/pre>\n<\/article>\n<ul>\n<li>Under <em><strong>User-agent<\/strong>:\u00a0<\/em> you can type in the specific crawler name. You can also include all crawlers simply by typing in the star (*) symbol. With this command, you can filter out all crawlers except AdBot crawlers, which you need to enumerate explicitly. You can find a list of all crawlers on the internet.<\/li>\n<li>Additionally, for the <strong><em>Allow<\/em><\/strong> and <strong><em>Disallow<\/em><\/strong>commands to work only for a specific file or folder, you must always include their names between \u201c<strong>\/<\/strong>\u201d.<\/li>\n<li>Notice how both commands are case-sensitive? It is especially relevant to know that the crawler agents\u2019 default setting is so that they can access any page or directory if not blocked by a <em>Disallow<\/em>: rule.<\/li>\n<\/ul>\n<p><strong>Note: You can find complete rules and syntax examples <a href=\"https:\/\/support.google.com\/webmasters\/answer\/6062596?hl=en&amp;ref_topic=6061961\" target=\"_blank\" rel=\"nofollow noopener\">here<\/a>.<\/strong><\/p>\n<\/article>\n<div class=\"raiiz69f32ddd24d18\" >If you enjoyed this article, then you\u2019ll love Zalvis's Cloud Hosting platform. Turbocharge your website and get 24\/7 support from our veteran team. Our world-class hosting infrastructure focuses on auto-scaling, performance, and security. Let us show you the Zalvis difference! <a href=\"https:\/\/zalvis.com\/\">Check out our services.<\/a><\/div><style type=\"text\/css\">\r\n@media screen and (min-width: 1201px) {\r\n.raiiz69f32ddd24d18 {\r\ndisplay: block;\r\n}\r\n}\r\n@media screen and (min-width: 993px) and (max-width: 1200px) {\r\n.raiiz69f32ddd24d18 {\r\ndisplay: block;\r\n}\r\n}\r\n@media screen and (min-width: 769px) and (max-width: 992px) {\r\n.raiiz69f32ddd24d18 {\r\ndisplay: block;\r\n}\r\n}\r\n@media screen and (min-width: 768px) and (max-width: 768px) {\r\n.raiiz69f32ddd24d18 {\r\ndisplay: block;\r\n}\r\n}\r\n@media screen and (max-width: 767px) {\r\n.raiiz69f32ddd24d18 {\r\ndisplay: block;\r\n}\r\n}\r\n<\/style>\r\n","protected":false},"excerpt":{"rendered":"<p>If you\u2019ve ever built your website, you may have heard of a robots.txt file and wondered, what is this file for? Well, you\u2019re in the right place! Below, we will review this file and why it\u2019s crucial. What Is A Robots.txt File? First of all, the robots.txt is nothing more than a plain text file [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-213","post","type-post","status-publish","format-standard","hentry","category-getting-started"],"_links":{"self":[{"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/posts\/213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/comments?post=213"}],"version-history":[{"count":2,"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/posts\/213\/revisions"}],"predecessor-version":[{"id":217,"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/posts\/213\/revisions\/217"}],"wp:attachment":[{"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/media?parent=213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/categories?post=213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zalvis.com\/docs\/wp-json\/wp\/v2\/tags?post=213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}