Adding Bing.com as a User Agent to Robots.txt

Sometimes it’s the absolute simplest questions that you have to dig around for online!

This had me stumped for five or ten minutes, until I saw a comment on the Bing Twitter.

Which user agent / bot does Bing.com use to crawl your site?

The answer is msnbot.

To add msnbot to your robots.txt file, add the following:

User-agent: msnbot

You can then make any Allow or Disallow rules that you like.

Bing.com are also running a Webmaster Central, much like Google but seemingly cleaner and not as tool-packed.

Sascha Kimmel has some good info on navigating through this.

It should be interesting to see whether Bing.com can make a dent in Google’s 80% share, but it isn’t looking likely. Most say no, and if they are still using the msnbot technology, then I have to agree.

Update:

The official Bing page outlining more on their bots is here. Thanks to Benjamin for the link.