The structure of one of our sites has many subdomains, millions of them (of course for a good reason). This means that the Google crawler must access the /ads.txt and /robots.txt files for each subdomain. Our server displays hundreds of requests for these files every second.
In this case, what would be the most efficient way to process these requests?
I am currently using this:
location = /robots.txt {
access_log off;
log_not_found off;
alias /home/sys/example.com/public/robots.txt;
}
location = /ads.txt {
access_log off;
log_not_found off;
alias /home/sys/example.com/public/ads.txt;
}
My intent here is to point the robot and display TXT files to the same location each time I request it, though I don't know if that makes any difference. Also disable logging with "access_log off". seemed like a good idea.
I think it is possible to return the contents of the txt file directly from nginx – would that be faster (less demanding for the nginx service)?
