Downtime and missing data: status
ESL was offline yesterday between about 5:30pm and 2:30am. The site has come back online since then but there is some data missing. We know about it and we are right now busy recovering all data. PLEASE do not yet write any Support Tickets, eMails or posts to ask about your missing accounts, ladders...!
Please wait at least until tomorrow before you start asking us questions about any missing data. It is very likely, that your data will be restored by tomorrow anyway.
What has happened?
Since Saturday we had huge performance problems with our main database. We tried to fix these problems with several methods, but we were not satisfied with the results so far.
On Monday we were able to work on this problem with our whole team and found inconsistencies in the database's structure, which forced us to bring the ESL page offline at about 5:30pm.
We had to remove and restore all data, which was a total of 90 GB of SQL data. Due to the pure bulk of data, we had to stay offline for many hours until very late at night.
Today it turned out that several data have not been restored successfully. Unfortunately, there is no specific pattern that could help us to restore the missing data quickly. Data is missing in many different areas in the database. Sometimes there is a missing match, sometimes it is a league or a user account. To give you an idea of how many things are missing: We have to restore only about 300 accounts out of 2 Million.
The missing data will be restored with the aid of backups, but this might take the whole day.
In the meantime we'd like to apologise for any inconvenience caused! All Premium Members will get a new premium day after the backup as a compensation.
Meanwhile most of the missing data has been recovered, e.g. all leagues and useraccounts are back online.
diejule, Tuesday, 23/03/10 13:51
- info write comment not allowed