Kategorie: Behind the scenes

Since the first day of Eurofurence 30 last week, we are experiencing very slow response time.
This comes from a high number of requests being thrown at our server.
We are regularly informed about his happening by an external hosting service sending out emails when our most important URLs are not accessible.

The problem

They seem to start daily around midnight and go on until the late afternoon.
Requests come from a very large number of IP-Adresses.
Initially they where getting lost in the layered navigation of our categories being filtered by colours, sizes, species,…
Now they seem to have shifted to the search function. Doing complex product searches.
Both are already forbidden by robots.txt with an additional rate limit for requests.
These are being ignored.

Impact

Apart from the obvious impact of slowing down or preventing customers from using the shop or this blog,
this also limits us in using the backend of the shop and for our inhouse-tools to load data about products and sales for printing packaging, storage labels, invoices and postage.
It also limits payment providers from calling back our shop to inform us about the status of payments.

Our response

We are keeping an eye on the logs.
At first we improved server response times by optimising our PHP and Database settings.
Then we included „rel=nofollow“ to links in layered navigation, preventing web-scaping bots from following these links
We are currently investigating ways to rate limit access to the advanced search function on the server side.
This is an ongoing issue and we will update this blog posting.

Result

So far a complex and technologically expensive rate limiting of access to the advanced search function of the shop seems to keep the situation under control.

We should have done this a while ago but kept postponing it.
Currently we are doing a full inventory count. One manufacturer at a time.
Stretched out over multiple days.
A number of small discrepancies have been found so far.
Surprisingly mostly having 1 toy too many and not too few.

We are almost done now. Only 3 major manufacturers left to be counted.
Luckily we have some self-written mobile software that makes this a rather quick process.

Looks like the update to the shop software broke email.
This has just been fixed.
Sorry for the inconvenience.

Our web severs are getting hammered by bots.
Mostly ones scraping the entire web for „AI training“.

While blocking some bots,
we set „crawl delays“ for others to not come back as often.
It seems that while doing that we made a mistake.
Apparently all settings that are valid for ALL bots need to be repeated when setting something for one specific bot.

That meant that instead of just adjusting the crawl delay for Google,
we disallowed it to crawl the entire site.

We are back and have started the get an overview of the tasks ahead of us.
It’s always quite overwhelming to return from such a long trip and to start a new year too.

The truck shows that it has crossed the entire country twice.
We can’t get all that salt off right away because it has started snowing.
Orders made during the holidays have recently been shipped.
The physical mailbox is not as full as we feared.
Closing down the books for 2025 is waiting for some final documents.
Then we can start the year 2026.

TwinTails has found a way for us to place orders in USD. Not requiring the bad exchange rate of Shopify.

Exotic Erotics has already shipped but the toys haven’t arrived yet because USPS seems to be shipping them back and forth between Dallas and Indianapolis. So far this has happened 3 times.

More toys from Purple Leopard have been ordered.

HornyRam is preparing for our next order.

We will order with FlurbMe and Neotori soon.

Some Email server run by Docomo and KDDI Japan seem to reject a ton of spam from a Brazilian botnet that uses us as a faked sender address…
This slowly started in April but has massively increased in volume this month.

Background

Emails claiming to be from us must be signed with a secret, cryptographic key that only we have. The corresponding public key can be found in our domain name entry. So every email server can check that an email it receives is actually from us. (This is called DKIM.)
In addition we also list the IP addresses of all email servers that are allowed to send email with us as a claimed sender. (This is called SPF.)

How do we know all this?

These are summary reports sent back to us from email servers that get our emails.
(This is a mechanism called DMARC.)
Red means it checked a signature and found it missing or invalid.

The reports do contain the IP-address and hostname of the email-server trying to send.
Most of them are dynamic IP addresses of DSL customers in Brazil
So it’s a fair bet that some botnet is trying to send spam with us as a faked sender.

Looks like tonight 2:54 the fuse in the room where we have the NAS with all company data blew.
At 3:05 it reported 44min of runtime.
At 3:51 the battery was low and the NAS was shut down properly.
We came in early at 7:35 and are currently checking the situation…

We are experiencing unusually high traffic and the reason seems to be ChatGPT trying to scrape the YiffyToys shop with all it’s filtering functionalities for multiple days now.

This also causes the cache of rendered pages to regularly overflow it’s quote and be automatically flushed.

We are investigating options for rate-limiting bot traffic.

Update

Most of the traffic (over 40Gb in a day) was caused be ChatGPT training it’s „AI“.
In total 2/3 of all web traffic came from bots.
We have blocked all the „AI“ training crawlers we could find and some „SEO optimisation“ bots that serve no purpose other then to waste bandwidth.
We have also rate-limited search engine crawlers.

The legitimate crawlers now get 4 sitemaps that are generated every day.
They list the last modification date of all listed pages, so crawlers don’t need to revisit them.
They prevent bots from getting lost in dynamic navigation.
(e.g. the extensive filtering options we offer, so you find the toys you want in our vast offering)
The sitemaps also only include the primary image of each product.

For historical reason the old „colour“ attribute had two values for black.
„pitch black“ and „black“.
We have gone through hundreds of old products to convert all old „pitch black“ products to „black“ now and deleted the old value from the selection.

With so many products, house keeping in a shop like ours becomes a lot of work.