Introduction to robots.txt and sitemap.xml files.

1. Robots and Sitemap
By Naji El Kotob.
Information Technology Consultant
_______________________________
naji [@] DotNETHeroes.com
Review 1.0.1 - 13
.
Business Intelligence Group BIG SARL

2. by Naji El Kotob
Outlines
 SEO
 Robots.txt
 Sitemap.xml
 Q&A

3. by Naji El Kotob
SEO

4. by Naji El Kotob
What is SEO
 SEO is short for search engine optimization or
search engine optimizer.
 Search engine optimization is a methodology of
strategies, techniques and tactics used to
increase the amount of visitors to a website by
obtaining a high-ranking placement in the search
results page of a search engine (SERP)
-- including Google, Bing, Yahoo and other search engines.
Source | http://www.webopedia.com/TERM/S/SEO.html

5. by Naji El Kotob
Robots.txt

6. by Naji El Kotob
Robots.txt
 Robots.txt is the file that Search Engines
use to see what they should index.
 This file and site maps help make your
site easier to navigate by the bots and
let them know what is legal and what you
would rather not have the published in
their engines.

7. by Naji El Kotob
Robots.txt

8. by Naji El Kotob
Robots.txt Syntax
User-agent: *
Disallow:
Sitemap: http://dotnetheroes.com/sitemap.xml

9. by Naji El Kotob
Robots.txt Generator
http://tools.seobook.com/robo
ts-txt/generator/

10. by Naji El Kotob
Interactive Demo

11. by Naji El Kotob
Demo
 Design and Create a Dynamic Robots
Source | http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-file-to-an-asp-net-mvc-site/
Adding a Dynamic Robots.txt file to an ASP.NET MVC site
Robots.txt is required to allow search engines to properly index your
site, and more importantly not index it.
If you have a public-facing staging or preliminary site that you don’t
want to show up in Google results, you need to make sure that it
returns the correct robots.txt with the
Disallow: /
line to prevent indexing. However, manually adding robots.txt files
to staging and production environments as a manual process can be
improved with the process below – the same code can serve up a
locked down robots.txt in staging or internal URLs, and allow
indexing in production.

12. by Naji El Kotob
Demo (cont'd)
 You’ll also need to make sure that
runAllManagedModulesForAllRequests is
true in web.config as normally text files
bypass the ASP.NET pipeline:
<system.webServer>
<modules runAllManagedModulesForAllRequests="true">
</modules>
...
</system.webServer>

13. by Naji El Kotob
Demo (cont'd)
routes.MapRoute(
"Robots.txt",
"robots.txt",
new
{
controller = "Robots",
action = "RobotsText"
}
);

14. by Naji El Kotob
Demo (cont'd)
The create a new controller called “RobotsController” with a single action “RobotsText”. All requests to /robots.txt will go
here:
public class RobotsController : Controller
{
public FileContentResult RobotsText()
{
var contentBuilder = new StringBuilder();
contentBuilder.AppendLine("User-agent: *");
// change this to however you want to detect a production URL
var isProductionUrl = Request.Url != null &&
!Request.Url.ToString().ToLowerInvariant().Contains("elasticbeanstalk");
if (isProductionUrl)
{
contentBuilder.AppendLine("Disallow: /elmah.axd");
contentBuilder.AppendLine("Disallow: /admin");
contentBuilder.AppendLine("Disallow: /Admin");
contentBuilder.AppendLine("Sitemap: http://www.mysite.com/sitemap.xml");
}
else
{
contentBuilder.AppendLine("Disallow: /");
}
return File(Encoding.UTF8.GetBytes(contentBuilder.ToString()), "text/plain");
}
}

15. by Naji El Kotob
Sitemap

16. by Naji El Kotob
Sitemap
 Sitemap (XML) is a list of pages on your Website.
It is easiest way to give Google, and other search
engines, information about your site and may
speed up Google’s crawlers finding you.
 There are many advantages of submitting a
Sitemap to Google, especially if your site is new
and doesn’t have many backlinks.
Source | http://diywebjem.com/webdefinitions.html

17. by Naji El Kotob
Sitemap Format
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.example.com/</loc>
<lastmod>2005-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
Source | http://www.sitemaps.org/protocol.html

18. by Naji El Kotob
Sitemap Generators
 There are hundreds of online tools that
can be used to create an XML sitemap
that can be submitted to Google, Bing,
Yahoo and other
◦ E.g.
 http://www.xml-sitemaps.com/
 http://www.sitemapdoc.com/
 And more…

19. by Naji El Kotob
References
 http://geekswithblogs.net/jjulian/archive/2012/04/25/adding-robots.txt-
to-your-asp.net-mvc-3-applications.aspx
 http://tools.seobook.com/robots-txt/
 http://www.edandersen.com/2013/02/17/adding-a-dynamic-robots-txt-
file-to-an-asp-net-mvc-site/

20. by Naji El Kotob
QnA?

21. by Naji El Kotob
Thank You 
 Please send your feedback and
suggestions to:
◦ naji [@] DotNETHeroes.com
 Join us at
www.facebook.com/DotNETHeroes