Dear users / customers
Tonight around 01 o'clock German time our internal monitoring systems started.
An initial investigation showed, also due to the number of systems that reported at once, that the cause was not with us.
After we informed the data center, we had to wait and see. What else is left in such a situation that lies outside our sphere of influence.
Later that night we received a report via Internal Status channels that there was a fire.
According to our knowledge, the fire brigade tried to extinguish the fire, but could not be brought under control at first.
Meanwhile, we have just 08:45 clock. The fire is extinguished and fortunately there were no injured persons according to current status!
We are in the process of drawing up an internal plan on how to proceed.
Currently, the communication level is at 0. The data center has cut all lines and is silent.
Acts quite communicative on social media so far.
For really urgent matters you can write to us at "firstname.lastname@example.org".
Before you write, keep in mind that we don't have any data for you to give out at this time.
Affected are all services which were issued with SBG / France / Strasbourg on the websites.
This currently affects about 60% of our network.
We are as displeased with this incident as you are.
It is difficult in our difficult times to deal with such an incident in addition.
For you, too, I'm sure! Stay calm and let us do our work.
Thanks a lot
Status Message: 10:41 AM
SBG1, SBG2, SBG3 & SBG4 will not be powered up today. The fires have been extinguished. However, SBG1 still needs to be cooled down. SBG2, SBG3 and SBG4 have been shut down in the meantime.
Most of the STREAMPANEL environments are located in SBG1 and SBG3.
Please do not worry about your data. We have secure backups of all components.
At the moment, however, we believe that there is no point in starting up new systems in other data centres if the emergency shutdown is likely to be lifted again tomorrow.
Status message: 16:38
We start our disaster recovery plan and begin with web hosting and the customer center. Due to the amount of data, the process will take until later in the night.
Tomorrow we'll see if we can get the data centers back up and running.
Status message: 23:38
1133 backups must now be imported. So that the websites alone are running again.
According to our plan, all websites and the customer center will be available again by 10:00 tomorrow morning.
Status message: Day 2 / 07:38 clock
All web hosting environments are up and running again. Not all websites are available again yet. Also DNS Conditioned it can and comes here to delays. Please be patient the restore is running!
Status message: Day 2 / 13:50 hrs
90% of the website are available again distributed on all hosts.
The remaining projects are in the high GigaByte range. We had put these in last place.
The customer center is also available again with restrictions. A few conditions are still missing, then we are also finished with it.
Status message: Day 2 / 16:03 hrs
We have now sent out circulars to all customers with active products so that everyone really knows.
Get in touch with us. The ticket system does not yet record your messages. In the next hour, the system will deliver all of them to us.
Status message: Day 3 / 08:59 o'clock
All backups to the web hosting have been restored. If there are any problems, write us! Unfortunately the backups do not save the PHP version. If you know how you can change it in the panel itself.
All others who have not written so far, please write us now. As soon as we have air, We also look again really all sides manually.
We have word from the data center that backups of the platform will arrive by March 19.
The time frame is unacceptable. We and you certainly don't want to wait until then. We'll start setting up a new environment now and get back to all cloud customers by noon.
The process is associated with a new setup due to the fire. Hostnames and ports can remain with almost all users.
We don't want to forcefully hold anyone. If you are unsure, look around on the market. Otherwise, we are happy about every user who wants to stay!
Towards afternoon the missing Teamspeak environments will follow.
If nothing unexpected happens today, we will be able to provide all products again today.
Please note that all STREAMPANEL developments are at a standstill until the backups arrive from the data center.
Status message: Day 3 / 09:18 hrs
The system will immediately start to receive your e-mails in the system again.
Also, it will automatically take its usual course. Please do not be alarmed if the system reminds or creates invoices in the current situation.
But please make sure that everything fits as it was and report us aufälligkeiten. Thank you!
Status message: Day 3 / 11:22 a.m.
The STREAMPANEL Free environment will not be restored for the time being and is on hold.
We ask for your understanding!
Status message: Day 3 / 14:49 hrs
The new cloud is ready and the news is going out now.
Please keep in mind that we need until 16 o'clock approx. to redirect the old hostnames accordingly.
Status message: Day 3 / 16:42 hrs
We served all cloud users with new servers.
Also the old hostnames now refer to the new environments.
If we have missed anything, please contact us.
We will now begin the process of restoring the Teamspeak environments.
Status message: Day 4 / 01:47 clock
It took longer than expected, unfortunately.
All Teamspeak customers have now also received a replacement system.
Status message: Day 4 / 13:36 hrs
Many of our requests are also related to the SHOUTcast tools.
Especially websites with integrations of the platform are currently suffering indirectly due to long loading times.
We will now start with the restore. Unfortunately, the current backup of the tools cannot be restored for some reason.
We do it by hand. Due to the amount of data of almost 80 GB, this takes time.
1.2 TB cover data are not restored. We wait in this case, then but rather on the backups of the data center. ^^
Status message: Day 4 / 16:55 hrs
The following products and integrations are currently still: Apps, Timed Inhale, Content Syndications, Safeguard.
All of the above integrations are dependent on the customer center.
Status message: Day 5 / 13:02 hrs
The SHOUTcast tools are giving us problems. We are still working on it.
Status message: Day 5 / 16:06 hrs
The SHOUTcast tools are still affected. Individual codes as well as players are already delivered again since yesterday. Other dependencies are still pending.
Status message: Day 5 / 20:43 hrs
We have a solution for the SHOUTcast tools. Until tomorrow, this should also have worked. We will now no longer update here and create new entries in the blog for important matters.