|
|
|
It is currently Thu Apr 18, 2024 4:27 am
|
Author |
Message |
Massivedesign
Site Admin
Location: Olympia, WA Joined: Fri Mar 11, 2011 Posts: 38307
Real Name: Dan
|
Holy... CRAP. I have no words, but I'm going to type anyway.
As you know, we had a massive server failure that gave us a 7 day outage. Not fun, not fun at all. We could have been back online 2 days ago, but ony with data up to early Feb 2018. This wasn't okay and I pushed / asked nicely for the hosts to recover other data.
The best they could do is October 5th. We will take it! I have some housekeeping to do this AM, then we will resume complete normal operations. The server may need to be restarted here and there too, but downtime should be minutes, not days!
|
Fri Oct 12, 2018 12:43 am |
|
|
heymagic
Location: Elma Joined: Sun Sep 22, 2013 Posts: 420
Real Name: Gene
|
I missed you guys ( I'll have to get the sights adjusted....) what a weird deal. Probably some evil scheme of Bloombergs.
|
Fri Oct 12, 2018 12:47 am |
|
|
Pablo
Site Supporter
Location: Everson, WA Joined: Sun Jan 6, 2013 Posts: 28177
Real Name: Ace Winky
|
Now I have a bunch of Facebook "friends", probably going to kill that account!
_________________ Why does the Penguin in Batman sound like a duck?
Because the eagle sounds like a hawk.
|
Fri Oct 12, 2018 12:58 am |
|
|
Massivedesign
Site Admin
Location: Olympia, WA Joined: Fri Mar 11, 2011 Posts: 38307
Real Name: Dan
|
Pablo wrote: Now I have a bunch of Facebook "friends", probably going to kill that account! For the record, I never sent you a friend request.
|
Fri Oct 12, 2018 12:59 am |
|
|
KeystoneCowboy
Site Supporter
Location: Burlington Joined: Wed Aug 22, 2012 Posts: 5999
Real Name: Kyle
|
This is bullshit! I want my money back! At least get pro rated for down time, damn it!
_________________ Looking for: S&W Schofield 2x (.38/357) Coonan 1911 Nemo Omen JM Marlin 39M Tikka T3 Tactical(.308) BAR(.308)
|
Fri Oct 12, 2018 1:01 am |
|
|
Pablo
Site Supporter
Location: Everson, WA Joined: Sun Jan 6, 2013 Posts: 28177
Real Name: Ace Winky
|
Massivedesign wrote: Pablo wrote: Now I have a bunch of Facebook "friends", probably going to kill that account! For the record, I never sent you a friend request. This is true. But I was hoping. Broke my heart really.
_________________ Why does the Penguin in Batman sound like a duck?
Because the eagle sounds like a hawk.
|
Fri Oct 12, 2018 1:02 am |
|
|
Massivedesign
Site Admin
Location: Olympia, WA Joined: Fri Mar 11, 2011 Posts: 38307
Real Name: Dan
|
Pablo wrote: Massivedesign wrote: Pablo wrote: Now I have a bunch of Facebook "friends", probably going to kill that account! For the record, I never sent you a friend request. This is true. But I was hoping. Broke my heart really. I was hoping you would be the Dom in this relationship.
|
Fri Oct 12, 2018 1:05 am |
|
|
Pablo
Site Supporter
Location: Everson, WA Joined: Sun Jan 6, 2013 Posts: 28177
Real Name: Ace Winky
|
Ok I'm out.
_________________ Why does the Penguin in Batman sound like a duck?
Because the eagle sounds like a hawk.
|
Fri Oct 12, 2018 1:14 am |
|
|
Massivedesign
Site Admin
Location: Olympia, WA Joined: Fri Mar 11, 2011 Posts: 38307
Real Name: Dan
|
When we first started WaGuns (8 years ago) I backed up alllll the time. Never once needed it. As we grew, so did our DB. Our SQL and FIle structure exceeds 5GB. So backing that up daily to a local source would take a pretty massive amount of time and bandwidth. So, we relied on host backups.
This was, in reality, the perfect storm. When the server crashed, our mirror backup should have been able to be deployed, losing at most a day of data. But, the backup appliance had also failed.
The SAN (Storage Area Network) we were on had an uptime of 100% over 5.5 years. We are now on a new SAN with better specs and it'll be interesting to see the incentives they give to keep the customers who were impacted. Regardless of what happens, we will be looking at at least additional 3rd party data retention companies.
The impact of this, to our members, was too great for us to not take this as a warning to do better ourselves. While Steve and I run this site out of passion for the 2A, and make no money off of it, it's clear that WaGuns does have a large footprint in WA. When she went down, it hurt the 2A a little. We didn't like that.
|
Fri Oct 12, 2018 1:15 am |
|
|
TINCANBANDIT
Site Supporter
Location: Mohave Valley Arizona Joined: Fri May 20, 2011 Posts: 13371
Real Name: Casey
|
_________________Actor portrayal, Action figures sold separately, You must be at least this tall to ride, Individual results may vary, Sales tax not included, All models are over 18 years of age, upon approval of credit, Quantities are limited while supplies last, Some restrictions apply, Not available with other offers, At participating locations only, Void where prohibited, Above terms subject to change without notice, Patent pending.See my blog: http://tincanbandit.blogspot.com/
|
Fri Oct 12, 2018 1:17 am |
|
|
Barfly
Site Supporter
Location: Lacey Joined: Thu Nov 29, 2012 Posts: 2292
Real Name: Jim
|
Massivedesign wrote: When we first started WaGuns (8 years ago) I backed up alllll the time. Never once needed it. As we grew, so did our DB. Our SQL and FIle structure exceeds 5GB. So backing that up daily to a local source would take a pretty massive amount of time and bandwidth. So, we relied on host backups.
This was, in reality, the perfect storm. When the server crashed, our mirror backup should have been able to be deployed, losing at most a day of data. But, the backup appliance had also failed.
The SAN (Storage Area Network) we were on had an uptime of 100% over 5.5 years. We are now on a new SAN with better specs and it'll be interesting to see the incentives they give to keep the customers who were impacted. Regardless of what happens, we will be looking at at least additional 3rd party data retention companies.
The impact of this, to our members, was too great for us to not take this as a warning to do better ourselves. While Steve and I run this site out of passion for the 2A, and make no money off of it, it's clear that WaGuns does have a large footprint in WA. When she went down, it hurt the 2A a little. We didn't like that. Thank you for doing what you do! We ALL appreciate it.
|
Fri Oct 12, 2018 1:20 am |
|
|
delliottg
Site Supporter
Location: Duvall Joined: Thu Mar 17, 2011 Posts: 4604
Real Name: David
|
Massivedesign wrote: When we first started WaGuns (8 years ago) I backed up alllll the time. Never once needed it. As we grew, so did our DB. Our SQL and FIle structure exceeds 5GB. So backing that up daily to a local source would take a pretty massive amount of time and bandwidth. So, we relied on host backups.
This was, in reality, the perfect storm. When the server crashed, our mirror backup should have been able to be deployed, losing at most a day of data. But, the backup appliance had also failed.
The SAN (Storage Area Network) we were on had an uptime of 100% over 5.5 years. We are now on a new SAN with better specs and it'll be interesting to see the incentives they give to keep the customers who were impacted. Regardless of what happens, we will be looking at at least additional 3rd party data retention companies.
The impact of this, to our members, was too great for us to not take this as a warning to do better ourselves. While Steve and I run this site out of passion for the 2A, and make no money off of it, it's clear that WaGuns does have a large footprint in WA. When she went down, it hurt the 2A a little. We didn't like that. I have a batch script that automatically backs up our MySQL & PostGres databases on a nightly basis, and keeps a week of backups local (and our IT guys back up the drives they're stored on weekly to offsite). Theoretically in a catastrophic crash we'd only lose one day of data and we got to test that when we were hit with Wanna Cry virus a while back). Recovering from the virus took us about two days, but most of that was virus checking every single machine in the company manually. You're welcome to the script. It runs in Windows, but can be modified to run in Linux if that's your environment. Congratulations on getting the site back up with minimal losses. I saw signs of life earlier this morning when I could see directory structures again, but it was only in the "F"s.
_________________David Unique Treen
|
Fri Oct 12, 2018 6:45 am |
|
|
MadPick
Site Admin
Location: Renton, WA Joined: Sun Mar 13, 2011 Posts: 52031
Real Name: Steve
|
I just want to give Dan a huge “THANK YOU” for all that he’s done over the last week. He’s been working hard on the site, working hard twisting arms at the hosting company, and also keeping folks up to date via our Facebook page. Of course he has a family and a job too, and I appreciate that he took so much time over the last week to bring us back, and bring us back in good shape. Thank you, Dan. And to top it off ... TODAY IS HIS BIRTHDAY! Happy birthday to Massivedesign!
_________________SteveBenefactor Life Member, National Rifle AssociationLife Member, Second Amendment FoundationPatriot & Life Member, Gun Owners of AmericaLife Member, Citizens Committee for the Right to Keep and Bear ArmsLegal Action Supporter, Firearms Policy CoalitionMember, NAGR/NFGRPlease support the organizations that support all of us.Leave it cleaner than you found it.
|
Fri Oct 12, 2018 6:52 am |
|
|
edogg
Site Supporter
Location: Central FL Joined: Sun Apr 7, 2013 Posts: 3207
|
Thanks for all the hard work, guys! I started to go through DTs.
Sent from my iPhone using Tapatalk
|
Fri Oct 12, 2018 6:57 am |
|
|
lionhrt
Site Supporter
Location: Skagit county Joined: Sat Apr 21, 2012 Posts: 1389
Real Name: Dave
|
Good to have you back, thanks for going threw all the trouble to bring it back up. I didn`t realize how much time I spend here until I had to actually talk to the wife and my dogs are actually wanting to go to bed rather than play because I got got bored and wore them out daily.
_________________kf7mjf wrote: WaGuns Clue.
It was a Night Op in the library with an AK Pistol by TW...
|
Fri Oct 12, 2018 6:57 am |
|
|
|
Who is online |
Users browsing this forum: No registered users and 69 guests |
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot post attachments in this forum
|
|