1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Major outage at our hosting company

Discussion in 'Forum Rules, Information and Announcements' started by oss, May 19, 2016.

  1. oss
    Offline

    oss Somewhere Staff Member

    Hi everyone.

    Just to explain that our hosting problem had a major problem yesterday afternoon with one of their Linux servers, the net result was that they lost the server and had to rebuild it from scratch.

    The reason the rebuild took so long is that we share this server with 1756 other websites and I believe the reconstruction is still ongoing.

    https://help.fasthosts.co.uk/app/answers/detail/a_id/2774

    Luckily we are back online otherwise you would not be reading this.

    The company we use is a highly respected web host with many years experience in the UK and essentially these things happen, it is outside our control I'm afraid.

    Some of you may lose images that were posted to the forum recently such as avatars and images pasted directly into posts, this will depend on just how old the server backup is.

    The server that hosts our database was not affected at any time during this incident it is hosted on another physical server and we back it up ourselves over the internet every night at 2 am.

    Anyway nice to see the site back up and running.
    • Like Like x 3
  2. Methersgate
    Offline

    Methersgate Well-Known Member Lifetime Member

    Thank you very much. It must have meant a lot of hard work.
  3. oss
    Offline

    oss Somewhere Staff Member

    Not for us we didn't do a thing :) all of it done by the hosting company :)
    • Like Like x 1
  4. aposhark
    Offline

    aposhark Well-Known Member Lifetime Member

    It's nice to be back :like: :)
  5. Dave_E
    Offline

    Dave_E Well-Known Member Trusted Member

    Took a long time.

    Do they use floppy disks for backup?
  6. oss
    Offline

    oss Somewhere Staff Member

    They've been rebuilding the storage volume for nearly 23 hours and if they were only 30% done at 3pm today that suggests that we got lucky and came in at around 50%.

    I guess there could be performance issues for a while if the storage is being intensively written to, however once most of our pages are cached we are just going to the database so maybe that will help avoid performance issues.
  7. florgeW
    Offline

    florgeW Lady Mod Senior Member

    Have we lost any vital/pinned info shared or published in our site?
  8. Markham
    Offline

    Markham Guest

    Good to see that we're back.
  9. aposhark
    Offline

    aposhark Well-Known Member Lifetime Member

    What is the approximate size of the forum's storage volume, Jim?
  10. oss
    Offline

    oss Somewhere Staff Member

    The static pages and image content take up about half a gig I think, that gradually expands when people paste images into a page, but obviously there is no cost to us for pictures hosted elsewhere like Flickr.

    The space we use is I think a fair bit of what we have the right to use but we are still a bit away from any pressure on storage space, the database is closer to our limits it is several hundred megabytes now.

    You should understand that we just occupy a folder on a Linux server, nearly 2000 other sites occupy their own folder on the same server, we are restricted from seeing the content of other users folders but we can see some of those that exist in the raw unix ext partition.

    The webserver itself is separate from the storage I expect the storage is on a NAS of some kind from the description they gave in the link I posted, it sounds like something basic failed resulting in the NAS having to be restored as well as the web server, but I really don't know.
    • Informative Informative x 1
  11. oss
    Offline

    oss Somewhere Staff Member

    The database is separate from the website, the database is backed up every night completely outside of Fasthosts, it goes to one of my servers and the backup is actually less than 2 minutes, I have over a months worth of complete database images at any one time.

    The database IS the forum, it is where all our posts live.

    The webserver contains the forum software but the pages are static i.e. they never change, the only bit that changes is the contents of the images folder and that only happens because people sometimes paste an image directly into a post, or when they upload an avatar image, when they do that it uses up our space but nothing else on the website actually changes, ever :)

    So we have multiple copies of the website files offsite as well but we do not back up the static website every night as there would be no point and also the security on Fasthosts is designed to make it hard for us to do that, basically they want is to pay for their backup service but as I said the data never changes so it would be a waste to pay for redundant backup.

    What might be a good idea is to look at adding failover at our next renewal.
    • Informative Informative x 1
  12. oss
    Offline

    oss Somewhere Staff Member

    Now they have a database issue on a server on the same network segment as ours :D

    I wouldn't like to be one of the tech support guys in their data centre right now :)
  13. Markham
    Offline

    Markham Guest

    I tried posting a message with an embedded image which the host would not accept, so we do still have problems.
  14. bigmac
    Offline

    bigmac Well-Known Member Trusted Member

    i thought it was all my fault
  15. oss
    Offline

    oss Somewhere Staff Member

    I just tried, no biggie, I'm not too worried about them not setting the folder permissions correctly yet, the error is coming from XenForo being unable to write to the file system, they might be going to run chmod script at the end of the restore, maybe they don't want anyone writing to the filesystem while they are still restoring.

    My guess of 50% earlier was oddly accurate I had not checked the restore status on the link I provided at that point in time, but on checking later they posted an update at about 6pm that the restore of the volume had reached about 50%, we are obviously back but other sites hosted on this server are no doubt still waiting and probably very upset.
    Last edited: May 20, 2016
  16. KeithAngel
    Offline

    KeithAngel 2063 Lifetime Member

    Spelling Davee:)
  17. Markham
    Offline

    Markham Guest

    So far today, the server has been offline three times today for around 10 - 15 minutes but has at least responded with a 'maintenance message'. But I think they must have replaced what was quite a fast machine with an old XT as it's taking rather long to get any response from it. Fasthost it isn't!
  18. oss
    Offline

    oss Somewhere Staff Member

    Performance has certainly been noticably poor today.
  19. aposhark
    Offline

    aposhark Well-Known Member Lifetime Member

    Freudian slip, Keith :lol:
  20. Markham
    Offline

    Markham Guest

    It would be if Dave actually wrote what Keith claims. But he didn't.

Share This Page