Apache mod_rewrite broken?
I spent most of last night trying to figure out how to get URL rewrites working with Apache. What I wanted it to do was redirect anyone who tries to access robots.txt to a cgi script that prints out an allow list based on the robot that is browsing. But the only way I could get it to work was with a hard redirect (the server sends a 302 to the browser and the browser loads the new page) or a proxy statement.
The redirect doesn’t work for me because most robots don’t follow the redirect and assume you’re allowing them access to everything. The proxy didn’t work because it changes the IP address of the robot, so I can’t ban them properly. At the beginning of this adventure I thought about putting the commands in a .htaccess file but decided that was silly, it should work in the global config. After hours and hours of trying, I couldn’t get Apache to set the mime-type correctly so the browser didn’t know that it was loading a script, so instead it just downloaded the code of the script.
Finally I put the stuff in .htaccess, and fuck if it worked. This has got to be a bug, because rewrites to a static file (not CGI) works just fine in the config.
Popularity: 1% [?]
