We are experiencing unusually high traffic and the reason seems to be ChatGPT trying to scrape the YiffyToys shop with all it’s filtering functionalities for multiple days now.
This also causes the cache of rendered pages to regularly overflow it’s quote and be automatically flushed.
We are investigating options for rate-limiting bot traffic.
Update
Most of the traffic (over 40Gb in a day) was caused be ChatGPT training it’s „AI“.
In total 2/3 of all web traffic came from bots.
We have blocked all the „AI“ training crawlers we could find and some „SEO optimisation“ bots that serve no purpose other then to waste bandwidth.
We have also rate-limited search engine crawlers.
The legitimate crawlers now get 4 sitemaps that are generated every day.
They list the last modification date of all listed pages, so crawlers don’t need to revisit them.
They prevent bots from getting lost in dynamic navigation.
(e.g. the extensive filtering options we offer, so you find the toys you want in our vast offering)
The sitemaps also only include the primary image of each product.