System files error on Brimestone

[James]

Dear Customers,

Some system files have got corrupted on the Windows shared hosting server Brimestone.specialservers.com - 217.112.82.19 which has affected the server OS. As a result the server is temporarily down our Windows admins are looking in to this and trying to get the server back online asap.

We do not have an ETA at the moment. Further information & updates will be posted in this thread once we have an update. Inconvenience caused is regretted. We request you to bear with us during this while.

[ajvb]

Sites have now been down now for over 24 hours which is a very long time indeed. There have been no updates on this thread as promised.

When the server is back I would like to know what the problem was, why it took so long to resolve & what steps are being taken to improve response times in the event on any future problems.

[James]

We have successfully managed to recover 100% MSSQL databases updated till 1st April 2008 and data recovery process is still in progress.

[NickJS]

We have successfully managed to recover 100% MSSQL databases updated till 1st April 2008 and data recovery process is still in progress.

Any idea when emails via my domain will work again?

[ajvb]

What about MySql dBs?

[PhillipJones]

ETA on MYSQL

could i have some more information as to when the MySQL databases will be fully restored because its been well over 30 hours now since i have had a working website.

[Administrator]

We had 2 options to restore this server and both seem to have trouble right now. RAID1 was setup on this server, due to which it is taking long time to restore data from the drives. Our Recovery software has managed to recover only 50 GB till now which is 50% of the total data.

We've got .psa backup generated by Plesk which won't restore due to some problems. This backup was taken on 30th March but it won't restore on newly configured server. Swsoft has assigned one of their senior technical person to help us with this problem and hopefully we will be able to restore backup from the psa backup file in next couple of hours.

It will take at least 12 hours more to restore this server and there is no other possible solution to get this sorted sooner. We will issue 3 months credit to all accounts hosted on this server and we apologize for this problem.

[Dan]

First major problem since last year, hopefully you will get it sorted

[Administrator]

First major problem since last year, hopefully you will get it sorted

RAID is suppose to prevent problems, but here it has created more problems as we cannot restore any data from the drives using normal process.

We will work out a completely new solution for backup of Windows Servers as we cannot rely on Swsoft for such problems. They have no clue of what they are doing and so far their Level 1 techs have given only links for their knowledgebase articles. I've had a word with our account manager in Swsoft and he assigned a Dedicated L2 tech to get this sorted.

We will keep this thread updated with status of recovery process.

[Nicolaus-sr]

Hello Customers,

We've successfully managed to recover the MSSQL databases, but the web data & MySQL databases are still under recovery.

If you have a copy of your data on your end, we can setup a new account for you quickly so that your website & emails will be working back.

We will update you as soon as the server is back up and running. We appreciate your complete patience & understanding into this issue.

[optrex]

We will issue 3 months credit to all accounts hosted on this server and we apologize for this problem.

I've not been affected by this outage, but I had similar issues with a previous host, which led to a 40 hour outage. I would just like to say that your compensation offer in this situation is very good.

[cage5150]

We will issue 3 months credit to all accounts hosted on this server and we apologize for this problem.

Yes I agree that this is quite a generous compensation offer -

[PhillipJones]

If you think about it the compensation is not that good at all i personal have lost irreplaceable information im going to have to spend hours repairing and reconstructing website database which will take hours of my time which is valuable in short supply. and this doesn't take into the account the hundreds of pound lost in sales and the money lost in sales from the e commerce websites i run from which are on this server. and the loss of the customers faith in our websites which have already placed orders and where waiting for them to be dispatched.

so no real it is not that generous compensation

[Dan]

If you think about it the compensation is not that good at all i personal have lost irreplaceable information im going to have to spend hours repairing and reconstructing website database which will take hours of my time which is valuable in short supply. and this doesn't take into the account the hundreds of pound lost in sales and the money lost in sales from the e commerce websites i run from which are on this server. and the loss of the customers faith in our websites which have already placed orders and where waiting for them to be dispatched.

so no real it is not that generous compensation

Apologies have already been issued, this isn't an easy fix and the team is working round the clock to resolve it. As stated swsoft isn't really much of a help so it's all down to the team in the end.

But you are based on such a large server that the data is taking never-ending hours to restore, along with the initial problems which are stated in the primary post.

[Administrator]

If you think about it the compensation is not that good at all i personal have lost irreplaceable information im going to have to spend hours repairing and reconstructing website database which will take hours of my time which is valuable in short supply. and this doesn't take into the account the hundreds of pound lost in sales and the money lost in sales from the e commerce websites i run from which are on this server. and the loss of the customers faith in our websites which have already placed orders and where waiting for them to be dispatched.

so no real it is not that generous compensation

Hello Phillip,

We maintain multiple backups of all our servers and all our servers have Hardware RAID1 configured to eliminate HDD failure problems. We never expected anything similar to happen as hardware RAID1 made it more complicated to recover data from drives, while plesk .psa backups failed to restore on new server with error message "archive is corrupt".

I'm equally upset as we tried to eliminate every possible option that could result in data loss, but we got to learn new problem this time. We will now maintain a clone of whole server using rsync and xcopy on our backup servers so we won't need to rely on plesk backup utility.