Since the first day of Eurofurence last week, we are experiencing very slow response time.
This comes from a high number of requests being thrown at our server.
We are regularly informed about his happening by an external hosting service sending out emails when our most important URLs are not accessible.
The problem
They seem to start daily around midnight and go on until the late afternoon.
Requests come from a very large number of IP-Adresses.
Initially they where getting lost in the layered navigation of our categories being filtered by colours, sizes, species,…
Now they seem to have shifted to the search function. Doing complex product searches.
Both are already forbidden by robots.txt with an additional rate limit for requests.
These are being ignored.
Impact
Apart from the obvious impact of slowing down or preventing customers from using the shop or this blog,
this also limits us in using the backend of the shop and for our inhouse-tools to load data about products and sales for printing packaging, storage labels, invoices and postage.
It also limits payment providers from calling back our shop to inform us about the status of payments.
Our response
We are keeping an eye on the logs.
At first we improved server response times by optimising our PHP and Database settings.
Then we included „rel=nofollow“ to links in layered navigation, preventing web-scaping bots from following these links
We are currently investigating ways to rate limit access to the advanced search function on the server side.
This is an ongoing issue and we will update this blog posting.
Result
So far a complex and technologically expensive rate limiting of access to the advanced search function of the shop seems to keep the situation under control.