650thz.de Service Status @trashserver@social.tchncs.de

metalhead.club is still recovering ~ 330 GB of data. The process will probably take several hours. I'll keep you updated!

Looks like recovery was successful!

Still, some data was lost: A backup from 2021-09-02 01:00 am (UTC) was used. Users who registered during that day will need to re-register.

I'm sorry for any inconvenience!

First thought: Let's do some maintenance work for trashserver.net during the Christmas holidays!

Second thought: Better not. Don't frighten away all the new users who we're just introduced to XMPP by their Family members aka personal IT support.

🤷‍♂️

There seem to be issues with our data center IPv4 gateway. We'll try our best to get v4 connectivity back, soon.

IPv6 is unaffected.

Alle Dienste sind wieder online. ✅

trashserver.net und metalhead.club sollten in wenigen Minuten wieder zur Verfügung stehen.

Wegen eines Problems mit der Stromversorgung gibt es gerade einen Ausfall der 650thz.de Services.

An einer Lösung wird gearbeitet.

Sorry wegen der kurzen Unterbrechungen - Firewall will gewartet werden ;-)

Issue seems to be resolved, now.

Hint: Use 5.1.92.244 or 2a01:360:50e:1::244 for metalhead.club in /etc/hosts

.club TLD DNS resolvers seem to be unavailable right now :o

I'm just getting SRVFAIL, whatever resolver or domain I use for testing.

Unfortunately, this does also affect metalhead.club. Since I cannot take action here, please stand by 😔

I'm happy to announce that metalhead.club is available, again!

Restoring the backup worked quicker than I expected.

Indeed, throughput has increased to ~ 4 MB/s... which would mean: 22 hours to go.

Ok, 3,665,825 files need to restored. The speed is low. If we continue with avg 2.8 MB/s, the backup will be restored in 32 hours 🙄

Let's hope the throughput increases over time and my assumptions are wrong.

Looks like recovery was successful!

Still, some data was lost: A backup from 2021-09-02 01:00 am (UTC) was used. Users who registered during that day will need to re-register.

I'm sorry for any inconvenience!

metalhead.club is still recovering ~ 330 GB of data. The process will probably take several hours. I'll keep you updated!

As the storage pool was damaged too badly by memory corruption, we need to recover from external full backup.

I'm starting with trashserver.net, since this service is likely to be the most important one to most users (and has the biggest number of active users).

Recovery will continue to run over night. Let's hope for a working trashserver.net tomorrow!

Storage pool recovery is ongoing. This will take a while...

Some storage pools are not online. I'll do my best to get them running asap.