Open letter to John - MD of Webhosting

[IanJ0208]

We are now in day five of this debacle. Can you please give me a *definitive*, *precise* date and time when this madness is going to end and your systems are again going to be usable. I don't want any rubbish like 'the techs are looking into it', or 'it's fixed', I've had all that for, what seems like, as long as I can remember.

Having my private sites falling over (the current average - now that the madness is over [Raymond], and it's all been fixed [Pete and others] is about 20 minutes) is one thing, but now my company's websites are falling over - twice today, so far.

That happening is totally unacceptable so I think it would be a good plan for you to get your cheque book ready. The company *needs* its websites as sales tools for its worldwide agents. My MD is not going to tolerate having to pay sales personnel, in for example the USA, when they cannot sell anything. I am sure that you, as a businessman, will understand that attitude.

Our company strategy included installing Sugar CRM in the next few days and probably moving to a dedicated server later in the year. There is no way that I am going to install a CRM system on your machines. I can't even trust them with vanilla HTML, never mind sensitive and vital information, and I cannot, unfortunately, trust one word that your service personnel give out.

I've already recommended to sites that I webmaster for, but don't own the account or domain, that they move from your systems as quickly as possible. My own account expires next month so I can afford to wait for a few more days yet.

Anyway, I look forward to receiving your reply on when I, and others who are also suffering from the constant failures of your systems, can expect full and reliable restoration of the services we have paid for.

Regards

Ian

[sysadmin]

Originally Posted by IanJ0208

We are now in day five of this debacle. Can you please give me a *definitive*, *precise* date and time when this madness is going to end and your systems are again going to be usable. I don't want any rubbish like 'the techs are looking into it', or 'it's fixed', I've had all that for, what seems like, as long as I can remember.

Having my private sites falling over (the current average - now that the madness is over [Raymond], and it's all been fixed [Pete and others] is about 20 minutes) is one thing, but now my company's websites are falling over - twice today, so far.

That happening is totally unacceptable so I think it would be a good plan for you to get your cheque book ready. The company *needs* its websites as sales tools for its worldwide agents. My MD is not going to tolerate having to pay sales personnel, in for example the USA, when they cannot sell anything. I am sure that you, as a businessman, will understand that attitude.

Our company strategy included installing Sugar CRM in the next few days and probably moving to a dedicated server later in the year. There is no way that I am going to install a CRM system on your machines. I can't even trust them with vanilla HTML, never mind sensitive and vital information, and I cannot, unfortunately, trust one word that your service personnel give out.

I've already recommended to sites that I webmaster for, but don't own the account or domain, that they move from your systems as quickly as possible. My own account expires next month so I can afford to wait for a few more days yet.

Anyway, I look forward to receiving your reply on when I, and others who are also suffering from the constant failures of your systems, can expect full and reliable restoration of the services we have paid for.

Regards

Ian

Hi Ian,

I've posted here about what exactly happened with the web server on our DNP cluster: Server hacked for more than 24 hours

Regarding the applications pools which are constantly crashing & that your sites aren't working as expected, please provide me with one ticket ID which has all the problems mentioned into it & I'll personally myself get all of them fixed 100%. A major ASP.Net sites hosted on this server use .Net version 2, hence the rest which need ASP.Net version 4 will have to manually be set to the required version. Right now, we've setup additional application pools to avoid this issue, but if your website crashes one more time, please submit a support ticket with it's description & I'll have it sorted for sure.

[John]

Originally Posted by IanJ0208

We are now in day five of this debacle. Can you please give me a *definitive*, *precise* date and time when this madness is going to end and your systems are again going to be usable. I don't want any rubbish like 'the techs are looking into it', or 'it's fixed', I've had all that for, what seems like, as long as I can remember.

Having my private sites falling over (the current average - now that the madness is over [Raymond], and it's all been fixed [Pete and others] is about 20 minutes) is one thing, but now my company's websites are falling over - twice today, so far.

That happening is totally unacceptable so I think it would be a good plan for you to get your cheque book ready. The company *needs* its websites as sales tools for its worldwide agents. My MD is not going to tolerate having to pay sales personnel, in for example the USA, when they cannot sell anything. I am sure that you, as a businessman, will understand that attitude.

Our company strategy included installing Sugar CRM in the next few days and probably moving to a dedicated server later in the year. There is no way that I am going to install a CRM system on your machines. I can't even trust them with vanilla HTML, never mind sensitive and vital information, and I cannot, unfortunately, trust one word that your service personnel give out.

I've already recommended to sites that I webmaster for, but don't own the account or domain, that they move from your systems as quickly as possible. My own account expires next month so I can afford to wait for a few more days yet.

Anyway, I look forward to receiving your reply on when I, and others who are also suffering from the constant failures of your systems, can expect full and reliable restoration of the services we have paid for.

Regards

Ian

Hello Ian,

I'm personally not into giving "definite" time scales into something which is relatively unknown. It would seem quite a few of my technicians don't share that view though. So, no, no time scale will be given, but I believe "sysadmin" (the head of the Windows Team) believes the problems are fixed now and is inviting users to PM him if they have any ongoing technical problems.

I don't have too much I say about your decision not to deploy SugarCRM on our servers. We have around 2500 physical and tens of thousand virtual servers. 1, all be it a very important one, was hacked. This particular one was only as damaging as it was because of the shared nature of windows shared hosting. After speaking with the head of the windows team, I believe he accepts that he gave ease of client functionality too big a priority in this instance. He has now disabled a number of features he deems to be a problem, which may indeed break some legitimate websites. As unfortunate as that is, I would hope nobody would wish for us to put the server at risk. In terms of a Dedicated Environment, a different set of challenges are present there. However, considering the server isn't shared with other users, 95% of the potential problems are already avoided. In terms of application security, that would be something you have to take responsibility for. There isn't a server in the World which someone could categorically state will never get hacked. Of course, when that server is shared by thousands of websites, the risk goes up.

In regards to my cheque book, I personally can't stand the things! There will also be no money being paid directly to clients. As we have always done, any performance related credit due to downtime and lapses in our communication will be dealt with via a credit in a client's account. The sum hasn't been calculated yet, as I will be waiting on the final all clear before considering it. If a client wishes a refund instead of this credit, that will also be considered.

I'm aware that some people will have doubts about our service (rightly so!), but we can only prove ourselves by our ongoing service. However, I note some of the names floated around on this issue are actually resellers of ours who we provide servers and support for. Perception is an interesting thing., particularly since some of those were said to provide excellent service.

The past week or so has been a personal source of embarrassment for me. That mainly came from many of our technicians failing to adequately explain the situation to clients and in some cases trying to pass blame on to customers. I'm well aware of where our faults and strengths are and the faults are what we will continue to work at. Communication, just as it has been for some time, is my top priority.

[IanJ0208]

Originally Posted by John

Hello Ian,

I'm personally not into giving "definite" time scales into something which is relatively unknown. It would seem quite a few of my technicians don't share that view though. So, no, no time scale will be given, but I believe "sysadmin" (the head of the Windows Team) believes the problems are fixed now and is inviting users to PM him if they have any ongoing technical problems.

Yes, things would appear to be a lot more stable. My post, requesting a fix time was within seconds of some of my websites going down for the umpteenth time yesterday.

I don't have too much I say about your decision not to deploy SugarCRM on our servers.

While I am certainly not an expert on it, I doubt that I could deploy Sugar. I doubt that Sugar uses absolute paths, which would appear to be the only way open to us any more. I am certainly not going to spend hours and hours researching and I'm not going to deploy it to find that it's fallen over. It's a lot quicker and easier to move host.

After speaking with the head of the windows team, I believe he accepts that he gave ease of client functionality too big a priority in this instance. He has now disabled a number of features he deems to be a problem, which may indeed break some legitimate websites.

Yes, *any* website using .NET for file upload. You have completely knackered a complete Microsoft framework for a lot of people. Is is going to be the same problem for anyone using DNN? I suspect it might be.

I am sorry, but I am completely confused by this. There must be thousands of hosts around the planet employing .NET. I would think that there is a large proportion of them offering DNP. Why is it that Webhosting would appear to be the only one who can't get DNP configured correctly to be able to use the two together? As the .NET FileUpload component has been around for a number of years, and is probably used on many thousands of websites, I find it most strange that I can't find one posting on the asp.net forums about a host disabling the ability to use it.

As unfortunate as that is, I would hope nobody would wish for us to put the server at risk.

Of course not, but there must be better ways of ensuring security without breaking websites and without disabling an important part of a Microsoft framework. As I said previously, it would seem that no other hosting company has felt the need to do that. Why Webhosting?

There isn't a server in the World which someone could categorically state will never get hacked. Of course, when that server is shared by thousands of websites, the risk goes up.

I doubt there is anyone here who does not know, recognise and appreciate that. As I have said in other posts, I don't have any problem with equipment failures or even security breaches. I have a problem with being lied to.

In regards to my cheque book, I personally can't stand the things! There will also be no money being paid directly to clients. As we have always done, any performance related credit due to downtime and lapses in our communication will be dealt with via a credit in a client's account.

I wasn't talking about compensation. AFAIC, these things happen. I was talking about a refund for a cancelled account. Not my personal one, that expires next month anyway. But my company one. The one I recommended they move to your hosting so we could use .NET, AJAX and SQL Server. The one which has just been completely broken by your seeming inability to configure DNP in line with other hosting companies.

I'm aware that some people will have doubts about our service (rightly so!), but we can only prove ourselves by our ongoing service.

I think your problem is more likely to be one of trust. I, personally, am not going to be trusting your first-line techs any more. They have lied to me, time after time, they have told me it is all my fault, multiple times, and one was completely unable to recognise a basic issue with a database (see another of my posts re: table creation in schemas).

particularly since some of those were said to provide excellent service.

I've always considered you to give very good service, though your techs seem to have an overdeveloped 'it's your code' muscle.

That mainly came from many of our technicians failing to adequately explain the situation to clients and in some cases trying to pass blame on to customers.

I could understand if it was an occasional, isolated incident. But it wasn't. It was policy, ongoing and there was a constant furore both in chat, email and in the forums. Nothing was done about it, though there was plenty of opportunity. How long does it take to instruct your staff on policy? A couple of minutes probably.

Communication, just as it has been for some time, is my top priority.

'Top priority'? Now you are having a laugh. I even posted in the Suggestions forum several days ago that something be done about communications. People have been posting in these forums and in the forums on eUKHosts about communications, and the lack of them. The people here have been *shouting* for days and you (a) haven't bothered to listen, or (b) ignored them. Communication is a two-way thing. I don't suppose anyone even bothered to read my post. *That* is the level of communication in reality.

Regards

Ian

[John]

Originally Posted by IanJ0208

'Top priority'? Now you are having a laugh. I even posted in the Suggestions forum several days ago that something be done about communications. People have been posting in these forums and in the forums on eUKHosts about communications, and the lack of them. The people here have been *shouting* for days and you (a) haven't bothered to listen, or (b) ignored them. Communication is a two-way thing. I don't suppose anyone even bothered to read my post. *That* is the level of communication in reality.

Hello Ian,

Perhaps you misread my reply. As I stated, I'm aware communication in a crisis isn't our strong point. Infact, pretty damn weak would describe it and this is where both I and other managers are trying to address on a priority basis. Like you and others here, on Thursday I was told everything was fine and it was all down to client permissions originally. Now, if that communication is given directly to me, I can understand the frustration of clients on the receiving end of said communication while their websites were repeatedly attacked.

Yes, *any* website using .NET for file upload. You have completely knackered a complete Microsoft framework for a lot of people. Is is going to be the same problem for anyone using DNN? I suspect it might be.

I am sorry, but I am completely confused by this. There must be thousands of hosts around the planet employing .NET. I would think that there is a large proportion of them offering DNP. Why is it that Webhosting would appear to be the only one who can't get DNP configured correctly to be able to use the two together? As the .NET FileUpload component has been around for a number of years, and is probably used on many thousands of websites, I find it most strange that I can't find one posting on the asp.net forums about a host disabling the ability to use it.

This feature has been enabled on these servers for around two years without any incident before it. I'm not going to comment overly much on what other hosts are doing, but I'd say just because something hasn't happened before or frequently doesn't mean it isn't a problem. As sysadmin posted here, it was a combination of a few things.

I'll have a chat with him today to find out how much this feature was the cause of the problem and if we can do anything else in its place.

[IanJ0208]

Originally Posted by John

Hello Ian,

Perhaps you misread my reply.

Thanks for your replies.

Possibly, and if that's the case, I apologise.

This feature has been enabled on these servers for around two years without any incident before it.

It's a part of the .NET framework. IIS. I would expect it to be enabled. .NET doesn't work, as specified by M$, without it.

I'm not going to comment overly much on what other hosts are doing, but I'd say just because something hasn't happened before or frequently doesn't mean it isn't a problem.

It's obviously not a problem to anyone else.

I'll have a chat with him today to find out how much this feature was the cause of the problem and if we can do anything else in its place.

That would be good. I've just had my MD online to me. He's incandescent that we've moved from a perfectly good hosting service, that he liked using and understood, on my say-so, and now we are stuffed. Moving to plesk is not an option as they cannot lose their email. They cannot do any sort of transfer via Outlook (or whatever) as a lot of them are not competent to do it. Much that I wouldn't mind a round-the-world trip to do it for them, that's not going to happen.

The company creates Internet video. Previously, they had to use uploading through Youtube to check their edits. I promised them a nice upload/viewing system if they moved from the Linux system they were on to a .NET system. I've been with your systems for almost a year and have been, until this week, reasonably pleased with them and felt comfortable recommending the move. I've written the company the new upload/viewing system and now a week's work is down the tubes and I've got an MD after my spherical objects for recommending you in the first place.

Regards

Ian

[John]

Originally Posted by IanJ0208

It's obviously not a problem to anyone else.

Two weeks ago it wasn't a problem for us

[IanJ0208]

Originally Posted by John

Two weeks ago it wasn't a problem for us

Tell me. It's *specifically* why we moved to your systems. It still isn't a problem for you. It's a problem for *us*!

Does it also affect anyone using DNN? Anyone who uploads profile pictures or product pictures? We can't just dump all our stuff in root just to get our sites to work. We also can't use absolute file paths.

I promise you I am not trying to do anything cute in my code. It is one line of bog-standard, cookbook VB that takes a file from a remote machine and puts it into a specific folder. That is completely normal on any server on the planet, AFAIK. YMMV.

I assume that as retrieval of people's profile pictures or product pictures also usually depends on server path mapping, the display of those things is also broken. I wonder how many people realise that yet, if my assumption is correct. 'We haven't sold a lot of anything today. I wonder why'. Or can they read from folders, just not write to them? So they can display, for the moment, but can't change them or upload new ones.

Cheers

Ian

[sysadmin]

Originally Posted by IanJ0208

Tell me. It's *specifically* why we moved to your systems. It still isn't a problem for you. It's a problem for *us*!

Does it also affect anyone using DNN? Anyone who uploads profile pictures or product pictures? We can't just dump all our stuff in root just to get our sites to work. We also can't use absolute file paths.

I promise you I am not trying to do anything cute in my code. It is one line of bog-standard, cookbook VB that takes a file from a remote machine and puts it into a specific folder. That is completely normal on any server on the planet, AFAIK. YMMV.

I assume that as retrieval of people's profile pictures or product pictures also usually depends on server path mapping, the display of those things is also broken. I wonder how many people realise that yet, if my assumption is correct. 'We haven't sold a lot of anything today. I wonder why'. Or can they read from folders, just not write to them? So they can display, for the moment, but can't change them or upload new ones.

Hi Ian, I've asked for details on the file uploading queries here : Server hacked for more than 24 hours

Please update me the ticket ID which contains those details & I'll get that working if its not at the moment.

[craigedmonds]

Hi John,

One of my dedicated servers has been down (AGAIN) for 9 hours last night despite monitoring being in place.

See ticket number #OIW-423-82512

This is after I have been promised that this will not happen again.

I am going to a wedding this coming weekend with my wife. We are dropping our kids of at a friends so we can enjoy the first weekend away together for several years.

If my weekend is spoilt by my server going down for more than 15 minutes, I will be pulling all my business away from WHUK.

I am not joking this time. Obviously there is some serious issues with your windows dept and monitoring dedicated servers.

All I want is to go to bed at night knowing that my servers are being monitored.

Craig Edmonds

PS: I have been on live chat ( i will post transcript into the ticket for you) since 0824 GMT and the server has just come back online at 0853, thats nearly 30 minutes since I requested the reboot. That is insane. Your reboot system needs to be revised. I understand currently that your live chat people are sitting in India and the DC is in the UK. The live chat operator sends a reboot request by whatever means over to DC and DC then reboots server. Why would it take 30 minutes? I can understand 2 minutes maybe 5, but 30 minutes?

[John]

Hello Craig,

I believe Jenna is already dealing with this as she mentioned it to me this morning.

At this stage I have no idea why the monitoring didn't alert our people to a problem, as I know it was supposed to have been setup last time. This is something which will be investigated.

In regards to a reboot, it's very rarely a case of someone just going and flicking a switch. Usually a monitor and keyboard needs to be attached and then going through FSCK, which takes times.

By the way, I have no issue with you posting whatever you like on the forum, but there really is no need to open an old thread to get my attention. It just makes things harder for other people who use these forums to gain information. I'm therefore closing this thread. If you wish to open a new one, you are more than welcome to.