[IUG] r2007 and robots.txt

["Welch, Andrew" <AWelch at auroragov dot org>]

Maybe I'm a little slow on the uptake for this one, but it appears that
r2007 has taken away our ability to change robots.txt without III
hand-holding. I used to be able to put a customized robots.txt one level
above /screens, using an FTP client; I'm actually still able to do that,
but the file I see when I navigate to
http://auroralibrary.org/robots.txt is not the one I put there.

 

The III helpdesk was able to change the file for me, and I learned that
the robots.txt file being served up is not accessible by the library,
even if it looks like it in an FTP client. They seemed to think that
this behavior has been in place for several releases, which isn't the
case. I had Google crawling our /screens directory as recently as the
beginning of last month (we upgraded to r2007 in mid-December).

 

I'd recommend to anyone who has customized their robots.txt file in the
past to navigate to it using your browser and verify that it's the right
one. I'm more than a little upset with III about this, not just because
the change was made with zero warning/documentation or that
functionality we used to have was taken away or that our existing file
was replaced by a III default file, but because anyone using a search
engine to find a page in our /screens directory in the past month wasn't
getting there. Given the number of Aurora Public Libraries there are in
this country, we need as much search result accuracy as we can get.

 

Andrew Welch

Sr. Library Systems Technician

Aurora Public Library

Aurora, CO 80012

http://auroralibrary.org/



--- StripMime Report -- processed MIME parts ---
multipart/alternative
  text/plain (text body -- kept)
  text/html
---