Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Years ago, Googlebot would send If-Modified-Since headers, and Apache would honor them.

I ran into this by chance when writing a wrapper to obfuscate e-mail addresses in mailing list archives. I didn't change the URL but had it served by a script instead of being a flat file. When it first went online, all of the robots kept crawling the files over and over. I finally made it supply the right mtime to Apache, which then did the right thing with the incoming IMS header, generating a 304 and not sending out new content.

It's possible this has regressed, but I would hope it hasn't.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: