Home Menu

Menu


Reply
Thread Tools Display Modes
  #1  
Old Nov 12, 2009, 11:14 PM
DocJohn's Avatar
DocJohn DocJohn is offline
Founder & Your Host
Community Support Team
Chat Leader
 
Member Since: May 2001
Location: Greater Boston, MA
Posts: 13,806
Our apologies for the unexpected downtime today. What follows is a technical service update for those of you who are interested.

At approximately 3:00pm ET, we were notified by a log service on one of the database servers that one of the drives was experiencing sudden, unexpected failures. In an attempt to recover the drive, it failed at 4:30pm ET.

It was not a catastrophic failure, however, so we were able to make an additional backup of the most recent data from all forums (otherwise the data would've come from a 4-day old backup). This took about an hour.

We then shutdown the server and replaced the faulty drive. A full OS reinstall took about 2 hours, and the restore of all backup data and cleaning of the database tables took about another hour and a half.

As you know, we take our uptime here very seriously and worked immediately on resolving this issue as soon as the drive started failing. Given the way it failed, we were fortunate to not suffer from any data loss.

We are back up and running 100%. Please let me know if you notice any problems or unexpected errors on the forums. Thank you.

Again, we apologize for the unexpected downtime and appreciated your patience.

Best,
DocJohn
__________________
Don't throw away your shot.
Thanks for this!
Amazonmom, anna342, Anonymous273, Anonymous29357, Beholden, Christina86, DocClyde, Elysium, FooZe, Fox, gimmeice, googley, January, JayS, katheryn, lizardlady, lynn P., Michah, muffy, Naturefreak, opheliasorrow, pegasus, Puffyprue, Ratanddragon, Rhapsody, Rmdctc, sabby, Shangrala, shezbut, silentandscared, TheTrishgu, VickiesPath, wanttoheal

advertisement
  #2  
Old Nov 12, 2009, 11:16 PM
Rmdctc's Avatar
Rmdctc Rmdctc is offline
Poohbah
 
Member Since: Apr 2009
Location: CA
Posts: 1,415
Thanks for working so hard to get it back up fast.
__________________
I'm here to deal with my "issues".
Thanks for this!
Anonymous29357, wanttoheal
  #3  
Old Nov 12, 2009, 11:39 PM
sabby's Avatar
sabby sabby is offline
Moderator
Community Support Team
 
Member Since: Feb 2005
Location: Southwest of Northeast
Posts: 33,346
(((((((((((((((((((( DocJohn ))))))))))))))))))))))))

In the 5 years I have been a member here, I don't ever remember something like this happening. The longest down time I remember was last year when we migrated from the old platform to the current one. I think that is amazing, considering the size of this site.

Kudos to you and all others who worked so hard to get us up and running again! I have been through things like this in the work place and it can be so frustrating. I think ya'll did a bang up job and to have been able to retrieve everything from when the system failed....that's just icing on the cake!

Will you be sleeping in, in the morning???
Thanks for this!
Anonymous29357, lynn P., wanttoheal
  #4  
Old Nov 12, 2009, 11:49 PM
NuckingFutz's Avatar
NuckingFutz NuckingFutz is offline
Pet Lady of Psychcentral
 
Member Since: Sep 2005
Location: Las Vegas, NV
Posts: 4,564
I think the fix and the damage control was handled well from every aspect. A job well done.
Thanks for this!
Anonymous29357, wanttoheal
  #5  
Old Nov 12, 2009, 11:58 PM
Anonymous29368
Guest
 
Posts: n/a
No need to apologize, I really appreciate all the hard work done and consideration that you had in bringing PC back up as soon as you could
Thanks for this!
Anonymous29357, wanttoheal
  #6  
Old Nov 13, 2009, 12:01 AM
sunrise's Avatar
sunrise sunrise is offline
Legendary
 
Member Since: Jan 2007
Location: U.S.
Posts: 10,383
I am amazed (and grateful) at how on top of this you were and how fast the site is back up.

(I must admit, however, that it has been a very productive day for me. )
__________________
"Therapists are experts at developing therapeutic relationships."
Thanks for this!
Anonymous29357, wanttoheal
  #7  
Old Nov 13, 2009, 12:15 AM
Sabrina's Avatar
Sabrina Sabrina is offline
Wisest Elder Ever
 
Member Since: Feb 2006
Location: South Africa
Posts: 67,808
Luckily, with our time difference - I missed it all. Glad things are up and running. Well done to all involved.
__________________
Status Update on the Community

Crying isn't a sign of weakness. It's a sign of having tried too hard to be strong for too long.
Thanks for this!
wanttoheal
  #8  
Old Nov 13, 2009, 02:22 PM
pachyderm's Avatar
pachyderm pachyderm is offline
Legendary
 
Member Since: Jun 2007
Location: Washington DC metro area
Posts: 15,865
It was nice to have the online updates too.
__________________
Now if thou would'st
When all have given him o'er
From death to life
Thou might'st him yet recover
-- Michael Drayton 1562 - 1631
  #9  
Old Nov 13, 2009, 02:41 PM
Yoda's Avatar
Yoda Yoda is offline
who reads this, anyway?
 
Member Since: Oct 2006
Location: Appalachia
Posts: 9,968
Quote:
Originally Posted by DocJohn View Post

Again, we apologize for the unexpected downtime and appreciated your patience.

Best,
DocJohn

Rumor has it that it was DocJohn's cat that was responsible for the downtime.

Status Update on the Community
__________________
The purpose of life is not to be happy. It is to be useful, to be honorable, to be compassionate, to have it make some difference that you have lived and lived well. anonymous
Thanks for this!
Amazonmom, Anonymous29368, JayS, lynn P.
  #10  
Old Nov 13, 2009, 02:53 PM
VickiesPath's Avatar
VickiesPath VickiesPath is offline
Magnate
 
Member Since: Jul 2009
Location: Phoenix, AZ, USA
Posts: 2,779
In techie circles, there's a saying: There's only two kinds of hard drives. Those that have failed and those that are going to fail. Thanks to the techs at PC, they were prepared. Kudos to you!!!!!!
__________________
Status Update on the CommunityVickie
Thanks for this!
lynn P.
  #11  
Old Nov 13, 2009, 03:01 PM
Anonymous091825
Guest
 
Posts: n/a
((Doc John)))) you handled it very well. The up dates were great.
the updates let me know it was not my comp
well done
P.S. loved the opps part
  #12  
Old Nov 13, 2009, 04:29 PM
FooZe's Avatar
FooZe FooZe is online now
Administrator
Community Support Team
 
Member Since: Apr 2009
Location: west coast, USA
Posts: 26,679
I'll be curious to see if the new drive does away with some or all of the delays I'd gotten used to seeing between about 10:30 and midnight Pacific time (1:30 - 3 AM Eastern). I assume that's when backups and other system maintenance would be scheduled, and a failing drive might sometimes have been too busy doing retries to keep up with user requests for pages.

Thank you for the fast fix and especially for managing to preserve so much, maybe all, of the database.
  #13  
Old Nov 14, 2009, 11:15 PM
Yoda's Avatar
Yoda Yoda is offline
who reads this, anyway?
 
Member Since: Oct 2006
Location: Appalachia
Posts: 9,968
Now could you send some tech support my w@y for my vowel problem?
__________________
The purpose of life is not to be happy. It is to be useful, to be honorable, to be compassionate, to have it make some difference that you have lived and lived well. anonymous
  #14  
Old Nov 14, 2009, 11:18 PM
Trying & Caring Trying & Caring is offline
Member
 
Member Since: Jun 2009
Posts: 248
Thanks--Doc John--for everything!
Reply
Views: 854

attentionThis is an old thread. You probably should not post your reply to it, as the original poster is unlikely to see it.




All times are GMT -5. The time now is 12:44 AM.
Powered by vBulletin® — Copyright © 2000 - 2025, Jelsoft Enterprises Ltd.




 

My Support Forums

My Support Forums is the online community that was originally begun as the Psych Central Forums in 2001. It now runs as an independent self-help support group community for mental health, personality, and psychological issues and is overseen by a group of dedicated, caring volunteers from around the world.

 

Helplines and Lifelines

The material on this site is for informational purposes only, and is not a substitute for medical advice, diagnosis or treatment provided by a qualified health care provider.

Always consult your doctor or mental health professional before trying anything you read here.