Welcome to WebHeadStart.org

Web Technologies

Sponsored By

WebHeadStart.org is currently in beta.
Please pardon our appearance as we work to provide you with the most comprehensive reference on today's web technologies.

Interested in advertising on WebHeadStart? Become an advertising partner today!

[WWW-HTML Mailing List Archive Home] [Messages By Thread] [Messages By Date]

Question about web spiders...

From: Peter Kupfer <peter.kupfer@sbcglobal.net>
Date: Sat, 25 Jun 2005 12:21:50 -0500
Message-ID: <42BD92AE.8050403@sbcglobal.net>
To: www-html@w3.org

I am not sure if this is the right list to be posting to, but out of the 
dozens of possible lists, this seems like the best one. I also didn't 
subscribe to the list, because I didn't see the link on the page I ended 
up at, <http://lists.w3.org/Archives/Public/www-html/ >, so if you could 
please cc me any responses I would appreciate it. :)

My questions is about web spiders. I use a service that spiders my site 
and creates a site map for me. I have links to my subdomains on my main 
page that I do not want the spider to follow. Their help told me to use 
the <nofollow></nofollow> tags so that it wouldn't follow those links. 
This works fine with the spider, but it is not W3 compliant.

The only information I have found anywhere is that I can add a meta tag 
that says nofollow and the spider won't follow any link on that page. 
However, this is not what I want. I just want the spider to not follow 
specific links on the page, but I want it to follow the rest.

Questions:

1) Is there another way to accomplish what I am trying to accomplish?
2) Does the W3C plan to implement the <nofollow> tag or anything like it 
in the near future?

I want to be standards compliant, but I also want to be able to tell a 
spider where it can and can not go.

Thank you.
-- 
Peter Kupfer
peschtra@yahoo.com
Received on Sunday, 26 June 2005 00:48:41 GMT
Valid XHTML 1.0! Valid CSS! Site Map | Privacy Policy | Terms of Use | WebHeadStart.org © 2005 All Rights Reserved.