Hi Everyone,

Our host has been experiencing some major problems for the last 12 hours. We are sorry for inconvenience but we're sure your employers appreciate the increase in productivity.

Has something changed recently? I used to be able to add icons on my replies - any ideas?

Don't know what's happened on your end, but when you finally able to add them again, you'll be so icono(c)lastic...

It appears that some buttons have stopped working in Safari. We'll look into it.

I've figured out the problem. I have not tested to see how many browsers this affects, but in Safari there seems to be an incompatibility between the twitter feed box and the reply icon set. I've disabled the twitter feed until there is a workaround. I know you are all crushed

We're having some server troubles today, sorry for the slowness. We have everyone in our corporate office working on the problem.

You have the dog working on it too?

We now have the dog on the problem. We are sorry for the continued problems. Some of our database queries are taking up to 91 seconds which is totally unacceptable.

curious, what are you guys running on?

hamster wheels, I warned them about not feeding them enough

We have are on a dedicated dual core 4Gigs ram, standard LAMP setup. Our db is on a shared DB server. We are looking at moving the DB onto the dedicated and putting nginx in front of apache to serve static content.

It looks like we have some of the issues worked out for the time being. Please don't hesitate to let us know if you run into performance issues.

These little babies will fix that.

Server is still really slow connecting for me and keeps logging me out.

another 503 as few minutes ago.

I've also noticed some performance issues. What's the database backend? Possible there's some indexing issues? I'm sure you have enough space but are the queries which bring up posts indexed correctly?

MySQL. We've had a DBA look into things. The problem is we have some pretty expensive queries, mainly the forum search query. I've been working on moving the forum search over to Solr which is insanely fast. We are also looking at moving the db onto a separate VPS or maybe a dedicated.

I've had to make a temporary change to our search function. The biggest range you can search now is just 1 year. I will change this back after we get a better handle on this.

It looks like we are back up!

I'll have more about this and what we are trying to do to rectify this tonight.

Hey Everyone,

As I'm sure everyone has noticed we've been having some problems the last week or so which culminated in today's downtime. There are two issues behind this.

First, our database for the forum has grown in size to the point where some of the queries are bogging down the database server. As a short term solution we are looking at getting our databases on some better hardware.

Second, our web host, who we've been with for 10 years, appears to be having problems keeping up with their growth. They have served us well (no pun intended), but it's time to find something that fits us better. Before these issues even came up I started looking at our options, and we are still evaluating those. I'm hoping to get our servers in Seattle to remove one weak link in this puzzle.

There is actually one last issue. There might be a misconception that there are several of us working full-time running cc.com, which is far from the truth. There are two of us right now, just Porter and myself, and we keep this up, dealing with host issues, tech support, help desk support, all the non-sense, in our spare time. So with that in mind I hope everyone can hang in here with us as we work through these server issues.

Thanks!

No worries. I certainly appreciate all the hard work that goes into maintaining the site, esp. since this isn't your real gig. When cc.com goes down, it just serves as a reminder that I need to get a life. I need that every now and again.

Things seemed to have stabilized a bit so we have upped the search range back up to 11 years. We are still working on getting onto some faster hardware.

We are sorry for the downtime this morning. We just moved our databases onto newer faster hardware but obviously there is something not right. We'll get it figured out. Thanks for the understanding.

Can people report any downtime or slowness to us, either here or in a PM. Performance seems to be pretty good right now but some feedback would be nice.

Now you're talking...nice speed boost.