Howdy.

Today, the site had experienced severe issues, which resulted in about an hour of downtime, and a few hours more of degraded performance.
I apologize for this mishap. We are looking into this situation, and into ways to mitigate it in the future.

What went wrong?

As many of you know, the site is continually developed and improved.
We usually have weekly releases, typically on Wednesdays. That day was chosen because traffic to the site is usually lighter than normal.
Today, we were deploying a whole slew of changes, including a revamped replacements list UI, tweaks to the upload page, some quality of life improvements, and so on.
Some of these improvements required changes to the database table containing post version data. Those changes were not major – you likely would not have even noticed them if they did go through.

Needless to say, something went very wrong during the deployment process. I'm going to have to get a bit technical here.
Databases typically use indexes – data structures used to quickly locate entries without having to search through every row in a database table.
They are very useful, considering how much data is required to make the site work. The post versions table specifically contains almost 50 million entries.

Unfortunately, the indexes related to the post versions table ended up corrupted.
We are still looking into what exactly went wrong here. However, the fact remains that at that moment, post version lookups ended up slowing down to a crawl.
In fact, database requests to that table were not just slow – they were outright failing, as the requests were taking longer than the maximum timeout period allowed.

This really should not have been this much of a problem, all things considered. Not a lot of people are searching through post versions on a daily basis, after all.
That's where the other issue came in – the "recent tags" function. It's a feature of the post editing form. As the name implies, it suggests to you the tags that you recently added to other posts.
The person who wrote that functionality did not really think things through, though. Instead of only fetching data when needed, it caused every post page to do an extra database lookup searching for recently used tags.
Yes, even if you did not open the post editing form. Or even if you never did any tagging at all.

And so, when the post versions requests started to fail, they took post pages with them.
Furthermore, all of these hanging requests were causing an increased load on the servers – and soon enough, the rest of the site started lagging too.

What do we do now?

Currently, the "recent tags" functionality is disabled. We will be looking into updating it to avoid hammering the servers with pointless requests.
The underlying issue is still being investigated. The indexes had been rebuilt, and the post versions pages should work normally.
We will also be looking into improving the testing processes in order to catch issues like this before they hit the main site.

Once again, I apologize for the inconvenience.

database lookups for recent user actions... I'm not really knowledgeable on webdev, but is that even something that should ever be necessary? because, like, that seems like something that could be handled with cookies, no?

dba_afish said:
database lookups for recent user actions... I'm not really knowledgeable on webdev, but is that even something that should ever be necessary? because, like, that seems like something that could be handled with cookies, no?

That would be possible if we actively stored data in localStorage when you upload/edit posts (which I doubt we currently do), but on top of the natural limitations of that (only works with the same browser on the same device, can be wiped, etc), it would likely exclude aliased/implied tags (unless we specifically added those to the response - even more of a pain). Not impossible, but not as straightforward, durable, & consistent as a db query.

A lot cheaper on server-side processing, sure, but hindsight's 20/20. And that's assuming the person who wrote it was familiar with client-side browser tech, which isn't a guarantee. Also, we might face some legal hurdles with that (the standard Accept/Reject cookie disclaimer you see on everything, storing cookies for minors who lied to get past the click-wrap, just like YouTube did, etc); I doubt we'd get hit for it, & we'd almost certainly dodge COPPA specifically, but it might be better to proactively bring up.
Edit: I meant localStorage, it's been too long since I messed with JS.

dba_afish said:
database lookups for recent user actions... I'm not really knowledgeable on webdev, but is that even something that should ever be necessary? because, like, that seems like something that could be handled with cookies, no?

Not with cookies specifically, but yes.
It does seem like something that could be kept in localStorage, and retrieved whenever it's needed.
It would require more work to implement. There had been plans to port re621's SmartAlias's tag display to vanilla e621, and we might be able to integrate the recent tags with it.

But as it stands, the site used to fetch the last 150 post changes you made and extract the tags you used from there.
It's significantly easier to implement this way. A very straightforward lookup and sort.

It's mostly the fact that the database query would run on every post page load for every logged in user that baffled me.

I was wondering what happened to the recent tags. (Looking at my edit log, you might come to the conclusion that I occasionally do a little tagging here and there. ;) )

Thanks for letting us know.

What kind of database does e621 use? SQL or noSQL? If it's not secret i'd love to hear which one you decided to be better for the site. I'd expect SQL to work just fine but maybe i'm missing something.

nordicwerewolf said:
What kind of database does e621 use? SQL or noSQL? If it's not secret i'd love to hear which one you decided to be better for the site. I'd expect SQL to work just fine but maybe i'm missing something.

E621 is open source: https://github.com/e621ng/e621ng
the database is postgres

donovan_dmc said:
E621 is open source: https://github.com/e621ng/e621ng
the database is postgres

oh that's actually super cool!!
Definitely will take a look for this, thank you for the link :3