WaGuns.org :: View topic - We're BAAAAAAACCCCKK!

Pablo wrote:

Now I have a bunch of Facebook "friends", probably going to kill that account!

Massivedesign wrote:

Pablo wrote:

Now I have a bunch of Facebook "friends", probably going to kill that account!

For the record, I never sent you a friend request.

Pablo wrote:

Massivedesign wrote:

Pablo wrote:

Now I have a bunch of Facebook "friends", probably going to kill that account!

For the record, I never sent you a friend request.

This is true. But I was hoping. Broke my heart really.

Massivedesign wrote:

When we first started WaGuns (8 years ago) I backed up alllll the time. Never once needed it. As we grew, so did our DB. Our SQL and FIle structure exceeds 5GB. So backing that up daily to a local source would take a pretty massive amount of time and bandwidth. So, we relied on host backups.

This was, in reality, the perfect storm. When the server crashed, our mirror backup should have been able to be deployed, losing at most a day of data. But, the backup appliance had also failed.

The SAN (Storage Area Network) we were on had an uptime of 100% over 5.5 years. We are now on a new SAN with better specs and it'll be interesting to see the incentives they give to keep the customers who were impacted. Regardless of what happens, we will be looking at at least additional 3rd party data retention companies.

The impact of this, to our members, was too great for us to not take this as a warning to do better ourselves. While Steve and I run this site out of passion for the 2A, and make no money off of it, it's clear that WaGuns does have a large footprint in WA. When she went down, it hurt the 2A a little. We didn't like that.

Massivedesign wrote:

Author:	Massivedesign [ Fri Oct 12, 2018 12:43 am ]
Post subject:	We're BAAAAAAACCCCKK!
Holy... CRAP. I have no words, but I'm going to type anyway. As you know, we had a massive server failure that gave us a 7 day outage. Not fun, not fun at all. We could have been back online 2 days ago, but ony with data up to early Feb 2018. This wasn't okay and I pushed / asked nicely for the hosts to recover other data. The best they could do is October 5th. We will take it! I have some housekeeping to do this AM, then we will resume complete normal operations. The server may need to be restarted here and there too, but downtime should be minutes, not days!

Author:	heymagic [ Fri Oct 12, 2018 12:47 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
I missed you guys ( I'll have to get the sights adjusted....) what a weird deal. Probably some evil scheme of Bloombergs.

Author:	Pablo [ Fri Oct 12, 2018 12:58 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Now I have a bunch of Facebook "friends", probably going to kill that account!

Author:	Massivedesign [ Fri Oct 12, 2018 12:59 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Pablo wrote: Now I have a bunch of Facebook "friends", probably going to kill that account! For the record, I never sent you a friend request.

Author:	KeystoneCowboy [ Fri Oct 12, 2018 1:01 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
This is bullshit! I want my money back! At least get pro rated for down time, damn it!

WaGuns.org https://www.waguns.org/

We're BAAAAAAACCCCKK! https://www.waguns.org/viewtopic.php?f=32&t=94648	Page 1 of 9

Author:	Pablo [ Fri Oct 12, 2018 1:02 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Massivedesign wrote: Pablo wrote: Now I have a bunch of Facebook "friends", probably going to kill that account! For the record, I never sent you a friend request. This is true. But I was hoping. Broke my heart really.

Author:	Massivedesign [ Fri Oct 12, 2018 1:05 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Pablo wrote: Massivedesign wrote: Pablo wrote: Now I have a bunch of Facebook "friends", probably going to kill that account! For the record, I never sent you a friend request. This is true. But I was hoping. Broke my heart really. I was hoping you would be the Dom in this relationship.

Author:	Pablo [ Fri Oct 12, 2018 1:14 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Ok I'm out.

Author:	Massivedesign [ Fri Oct 12, 2018 1:15 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
When we first started WaGuns (8 years ago) I backed up alllll the time. Never once needed it. As we grew, so did our DB. Our SQL and FIle structure exceeds 5GB. So backing that up daily to a local source would take a pretty massive amount of time and bandwidth. So, we relied on host backups. This was, in reality, the perfect storm. When the server crashed, our mirror backup should have been able to be deployed, losing at most a day of data. But, the backup appliance had also failed. The SAN (Storage Area Network) we were on had an uptime of 100% over 5.5 years. We are now on a new SAN with better specs and it'll be interesting to see the incentives they give to keep the customers who were impacted. Regardless of what happens, we will be looking at at least additional 3rd party data retention companies. The impact of this, to our members, was too great for us to not take this as a warning to do better ourselves. While Steve and I run this site out of passion for the 2A, and make no money off of it, it's clear that WaGuns does have a large footprint in WA. When she went down, it hurt the 2A a little. We didn't like that.

Author:	TINCANBANDIT [ Fri Oct 12, 2018 1:17 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!

Author:	Barfly [ Fri Oct 12, 2018 1:20 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Massivedesign wrote: When we first started WaGuns (8 years ago) I backed up alllll the time. Never once needed it. As we grew, so did our DB. Our SQL and FIle structure exceeds 5GB. So backing that up daily to a local source would take a pretty massive amount of time and bandwidth. So, we relied on host backups. This was, in reality, the perfect storm. When the server crashed, our mirror backup should have been able to be deployed, losing at most a day of data. But, the backup appliance had also failed. The SAN (Storage Area Network) we were on had an uptime of 100% over 5.5 years. We are now on a new SAN with better specs and it'll be interesting to see the incentives they give to keep the customers who were impacted. Regardless of what happens, we will be looking at at least additional 3rd party data retention companies. The impact of this, to our members, was too great for us to not take this as a warning to do better ourselves. While Steve and I run this site out of passion for the 2A, and make no money off of it, it's clear that WaGuns does have a large footprint in WA. When she went down, it hurt the 2A a little. We didn't like that. Thank you for doing what you do! We ALL appreciate it.

Author:	delliottg [ Fri Oct 12, 2018 6:45 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Massivedesign wrote: When we first started WaGuns (8 years ago) I backed up alllll the time. Never once needed it. As we grew, so did our DB. Our SQL and FIle structure exceeds 5GB. So backing that up daily to a local source would take a pretty massive amount of time and bandwidth. So, we relied on host backups. This was, in reality, the perfect storm. When the server crashed, our mirror backup should have been able to be deployed, losing at most a day of data. But, the backup appliance had also failed. The SAN (Storage Area Network) we were on had an uptime of 100% over 5.5 years. We are now on a new SAN with better specs and it'll be interesting to see the incentives they give to keep the customers who were impacted. Regardless of what happens, we will be looking at at least additional 3rd party data retention companies. The impact of this, to our members, was too great for us to not take this as a warning to do better ourselves. While Steve and I run this site out of passion for the 2A, and make no money off of it, it's clear that WaGuns does have a large footprint in WA. When she went down, it hurt the 2A a little. We didn't like that. I have a batch script that automatically backs up our MySQL & PostGres databases on a nightly basis, and keeps a week of backups local (and our IT guys back up the drives they're stored on weekly to offsite). Theoretically in a catastrophic crash we'd only lose one day of data and we got to test that when we were hit with Wanna Cry virus a while back). Recovering from the virus took us about two days, but most of that was virus checking every single machine in the company manually. You're welcome to the script. It runs in Windows, but can be modified to run in Linux if that's your environment. Congratulations on getting the site back up with minimal losses. I saw signs of life earlier this morning when I could see directory structures again, but it was only in the "F"s.

Author:	MadPick [ Fri Oct 12, 2018 6:52 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
I just want to give Dan a huge “THANK YOU” for all that he’s done over the last week. He’s been working hard on the site, working hard twisting arms at the hosting company, and also keeping folks up to date via our Facebook page. Of course he has a family and a job too, and I appreciate that he took so much time over the last week to bring us back, and bring us back in good shape. Thank you, Dan. And to top it off ... TODAY IS HIS BIRTHDAY! Happy birthday to Massivedesign!

Author:	edogg [ Fri Oct 12, 2018 6:57 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Thanks for all the hard work, guys! I started to go through DTs. Sent from my iPhone using Tapatalk

Page 1 of 9	All times are UTC - 8 hours
Powered by phpBB® Forum Software © phpBB Group https://www.phpbb.com/

Author:	lionhrt [ Fri Oct 12, 2018 6:57 am ]
Post subject:	Re: We're BAAAAAAACCCCKK!
Good to have you back, thanks for going threw all the trouble to bring it back up. I didn`t realize how much time I spend here until I had to actually talk to the wife and my dogs are actually wanting to go to bed rather than play because I got got bored and wore them out daily.