What is robots.txt and how to create robots.txt

-
Robots.txt is a text file which is used to tell search robots to which page of our site to visit and which pages not to. Search engines gives importance to robots.txt and follows according to it. robots.txt is not a way from preventing search engines from crawling our site, its just used to say " boss please do not visit this page. Only visit the pages which I said to. ". We have to locate this robots.txt file in the root of our site ex: domain.com/robots.txt.

how to create a robots.txt file:

Open a note pad and type the below given html code

User-agent: *

Disallow: /file-name.php
Disallow: /directory/

User-agents are search englines crawlers and disallow list the files and directories to be excluded form indexing.

enter the pages names and save the notepad as robots.txt and upload it to root of your site. Thats it. Hope this helped you in learning about robots.txt. If any thing missed please update it in comments.

Comments

  1. AnonymousDecember 24, 2010 at 7:59 PM

    This is best blog. Author had been publish many blog but everyone can not perfect have some mistake like as Some major crawlers support an Allow directive which can counteract a following Disallow directive

    ReplyDelete
  2. MySQL RepairFebruary 17, 2011 at 11:07 AM

    We can tell all spider to do not indexed our pages by a general code, the code is:

    # FULL access (All Spiders)
    User-agent: *
    Disallow: /file-name.php
    Disallow: /file-name.php
    Disallow: /file-name.php .....so on

    ReplyDelete

Post a Comment

Popular posts from this blog

Namaste Sada Vatsale - RSS Prayer

-
Image
image courtesy samvada.org Namaste Sada Vatsale is Rashtriya Swayamsevak Sangh's prayer. This prayer is in Sanskrit except last line which is in Hindi. It is compulsory to sing this prayer in all programs of Sangh. It was written by some unknown Sanskrit professor in guidance of Dr. K. B. Hedgewar and Madhav Sadashiv Golwalkar. Originally this poem was written in Sanskrit language. Later it was translated in Hindi also. Its original form in other Indian script as well as IOAT format is given as under: Devanagari नमस्ते सदा वत्सले मातृभूमे त्वया हिन्दुभूमे सुखं वर्धितोहम् । महामङ्गले पुण्यभूमे त्वदर्थे पतत्वेष कायो नमस्ते नमस्ते ।।१।। प्रभो शक्तिमन् हिन्दुराष्ट्राङ्गभूता इमे सादरं त्वां नमामो वयम् त्वदीयाय कार्याय बध्दा कटीयं शुभामाशिषं देहि तत्पूर्तये । अजय्यां च विश्वस्य देहीश शक्तिं सुशीलं जगद्येन नम्रं भवेत् श्रुतं चैव यत्कण्टकाकीर्ण मार्गं स्वयं स्वीकृतं नः सुगं कारयेत् ।।२।। समुत्कर्षनिःश्रेयस्यैकमुग्रं परं साधनं नाम वीरव्रतम् तदन्तः स्
Read more

Miklós Fehér - Soccer player who died during the match

-
Image
Miklós "Miki" Fehér (20 July 1979 — 25 January 2004) was an Hungarian footballer who played as a striker. On 25 January 2004, Fehér died of a cardiac arrest during a match between Vitória de Guimarães and his team Benfica in Guimarães, Portugal. On 25 January 2004, Benfica travelled to Guimarães to play against Vitória de Guimarães. The game was being broadcast live on television, and Benfica were leading 1–0. Fehér had just come on as a substitute, and assisted another player brought from the bench, Fernando Aguiar, for the match's only goal, but received a yellow card in injury time and suddenly bent forward, seemingly in pain. He then fell backwards to the ground. Members of both teams rushed immediately to aid Fehér before medical personnel arrived on the pitch. CPR was performed as match participants looked on in visible distress. An ambulance arrived on the pitch and Fehér was rushed to the hospital. His condition was covered by the Portuguese media througho
Read more

Search Engines Notice H1 Headings

-
What are H1 headings? An H1 heading is a prominent piece of text on a web page. It's like the headline of a newspaper or magazine article - it helps readers quickly understand what the web page is about. For example, if this email were a web page, "Search Engines Notice H1 Headings" would be its H1 heading. An H1 heading is plain text that lives between these HTML tags: How can H1 headings help my search engine rankings? Because headings are defined with HTML code, search engines understand that these pieces of text are of particular importance. You should include important keywords for a page in its H1 heading to tell search engines what the page is about, and help it rank higher in results for searches on those words. Do my website's pages have H1 headings? You can find out by looking at the HTML source code for every page on your site. When you're viewing the page, look for this option in your web browser. For example, in Firefox, you would clic
Read more