Author Topic: Army.ca Status- The History of Outages & Dark Days (Read 123918 times)

Mike Bobbitt · « **Reply #25 on:** March 10, 2004, 09:49:00 »

Yesterday at around 1600 ET, the hard drive failed in the "primary" Army.ca server. This hardware has not yet been replaced, and we are currently running off of a backup system.

Unfortunately, any changes made between approximately 1000 ET and 1600 ET on 09 March 2004 were not backed up. That means that any posts, user registrations, etc made during this time have been lost.

The current setup is temporary, and as soon as the new hardware is in place, we‘ll be back up and running "for real" again. In the mean time, feel free to post normally, all changes made from here on out will be carried over to the new system.

I apologize for the inconvenience and the loss of information.

Cheers

George Wallace · « **Reply #26 on:** March 10, 2004, 11:44:00 »

Patrick H. · « **Reply #27 on:** March 10, 2004, 13:18:00 »

Good to have the site back, if there‘s anything I can do, don‘t hestitate to ask!

Peace

Che · « **Reply #28 on:** March 10, 2004, 13:59:00 »

*Breathing deeply in a paper bag*
*Hands trembling*
...I I...I...Came...And...it wasn‘t......No one......Error 404.....
*Hyperventalating*
Don‘t ever leave me again.....*SOB*

Seriously though, good to have the site back.

Mike Bobbitt · « **Reply #29 on:** March 10, 2004, 14:04:00 »

Hahah... Hey, if you thought *you* were flinching, you should have seen me!

Bruce Monkhouse · « **Reply #30 on:** March 10, 2004, 14:04:00 »

Talk about a long shift at work without army.ca!!
ZZZZZZZZZZZZZZZZZZZZzzzZZZZZz :cool:

RoadRunner · « **Reply #31 on:** March 10, 2004, 15:38:00 »

Che,
You are a hoot!

Lexi · « **Reply #32 on:** March 11, 2004, 08:42:00 »

Quote

Originally posted by Che:
[qb] *Breathing deeply in a paper bag*
*Hands trembling*
...I I...I...Came...And...it wasn‘t......No one......Error 404.....
*Hyperventalating*
Don‘t ever leave me again.....*SOB*

Seriously though, good to have the site back. [/qb]

Heh, that was me too. I was at home sick and I panicked when I realised that Army.ca wasn‘t there. I e-mailed everyone I knew and told them to try and get on Army.ca to see if it was only me.

Thank God it‘s back

Nerf herder · « **Reply #33 on:** March 11, 2004, 08:52:00 »

Quote

Originally posted by Lexi:
[qb]Heh, that was me too. I was at home sick and I panicked when I realised that Army.ca wasn‘t there. I e-mailed everyone I knew and told them to try and get on Army.ca to see if it was only me.
[/qb]

Yeah....tell me about it. To console poor young Lexi I even let her in on George‘s and my OTHER hideout....sorry George

Regards

Lexi · « **Reply #34 on:** March 11, 2004, 18:47:00 »

Quote

Originally posted by Franko:
[qb]
Quote
Originally posted by Lexi:
[qb]Heh, that was me too. I was at home sick and I panicked when I realised that Army.ca wasn‘t there. I e-mailed everyone I knew and told them to try and get on Army.ca to see if it was only me.
[/qb]
Yeah....tell me about it. To console poor young Lexi I even let her in on George‘s and my OTHER hideout....sorry George

Regards [/qb]

Aww, I feel special.

Well, all that matters is that Army.ca‘s back up again, and HOPEFULLY there won‘t be another failure for a while.

Has the new hardware been installed or are we still on a backup?

Mike Bobbitt · « **Reply #35 on:** March 11, 2004, 18:58:00 »

The outpouring of emotion is overwhelming.

We‘re still running on the backup hardware, which means two things:

1. There‘s no longer any backup. If we fail again. It becomes a big problem.

2. The bandwitdh on this connection is quickly going over the limit. This means we may start to incur "overage" chages soon.

I‘m trying to get the new hardware in place quickly, to resolve these issues, but it doesn‘t look like it‘ll happen until at least next week now.

Thanks for hanging in there...

Mike Bobbitt · « **Reply #36 on:** March 25, 2004, 17:25:00 »

Hi all,

It looks like the new hardware may be going into place shortly. That means I‘ll be taking the site down for a few hours to do the cutover. I‘m hoping it‘ll take place tonight, but if not possibly tomorrow.

Sorry for the short notice, I‘ll post here when everything‘s good to go again.

Cheers

Bruce Monkhouse · « **Reply #37 on:** March 25, 2004, 18:38:00 »

AWWWW Come on Mike I‘m on night shift, I need this tonight.

Mike Bobbitt · « **Reply #38 on:** March 26, 2004, 00:02:00 »

Well, we‘re back but not on the new hardware. Something went wrong and the new system never came up when it was rebooted.

I estimate that we‘ve racked up just short of $300 in usage charges this month already, and I was *really* hoping to get things cut over. Looks like another weekend spent tinkering.

Don‘t mean to whine, but I‘m at my wit‘s end right now...

Bruce Monkhouse · « **Reply #39 on:** March 26, 2004, 00:15:00 »

Hey Mike,
That name I mentioned before in the PM, could he be of any help? I happen to know he works for rye and cokes and I‘m just the guy to supply it. I could ask him for a small road trip to the hamlet.

Mike Bobbitt · « **Reply #40 on:** March 26, 2004, 07:15:00 »

Thanks Bruce, I appreciate it. The hardware is in BC, so once it goes "off the air" it‘s impossible for me to work with from here. I have to request assistance via e-mail from a guy doing me favours, so it can be slow at times. Troubleshooting some problems over the phone could be interesting too.

Hopefully it‘s nothing too serious, we‘ll see today hopefully...

Anyway, thanks for the offer!

Cheers

Mike Bobbitt · « **Reply #41 on:** March 26, 2004, 09:44:00 »

Good news! We‘re *finally* up and running on the new hardware!

I apologize again for the down time and the loss of data that occurred as part of this transition. The last couple weeks have been a challenge, and I‘ll be glad to have my weekends back!

Thanks to everyone who showed their support over the last month. It‘s been a long road, but we‘re back on the air and more stable than ever. I couldn‘t have done it without your support!

Cheers and thanks

Mike Bobbitt

P.S. The new hardware is a bit faster, so you may notice a bit of a speed-up.

Bruce Monkhouse · « **Reply #42 on:** March 29, 2004, 18:49:00 »

Just hoping this post will work as I seem to be all by myself here......

pte anthony · « **Reply #43 on:** March 29, 2004, 19:59:00 »

It worked

Mike Bobbitt · « **Reply #44 on:** March 29, 2004, 18:49:00 »

Well, we were out of commission all weekend, but we‘re back now.

Just my luck that we move onto the new hardware, and the network goes down for 3 days.

Anyway, we‘re back on the air and I‘m working to keep it that way.

Sorry for the inconvenience, but at least this time there was no data loss.

Cheers
Mike

Bruce Monkhouse · « **Reply #45 on:** March 29, 2004, 18:51:00 »

It must be because I logged back on.

I just got to work and logged on and WHAM....

Che · « **Reply #46 on:** March 29, 2004, 20:36:00 »

*twitching*
....b..b-b-b..back?
..w-w-w..we-re back n-n-now?

*gets off of floor*

I was fine really.

Mike Bobbitt · « **Reply #47 on:** March 29, 2004, 21:09:00 »

If you think *you* had a long weekend... I spent way too long trying to decide whether to cut over to the backup system (and lose some data) or just ride it out.

I rode it out and it took a little longer than I‘d hoped, but we lost nothing (except 1 year from my life expectancy...)

Mike Bobbitt · « **Reply #48 on:** April 07, 2004, 10:17:00 »

...and we‘re back again! This time it was a new problem - the system ran out of memory. I‘m going to try to track down what happened so I can make sure it doesn‘t happen again!

Anwyay, sorry for the inconvenience - again... We‘ll get this all worked out sooner or later.

Cheers
Mike

Mike Bobbitt · « **Reply #49 on:** April 14, 2004, 20:16:00 »

Ok, back again after a 20 hour outage. We went out at about midnight last night and came back at about 2000 ET tonight.

It‘s a good news bad news thing: the bad news is that our router seems to be consistently failing. The good news is that a new router has already been purchased, configured and shipped, so it should be in place within a week.

Thanks again for your patience, all!

News:

Author Topic: Army.ca Status- The History of Outages & Dark Days (Read 123918 times)

pte anthony