SqWA News

Post Header

Published:
2021-11-27 18:49:47 -0500
Tags:

We had a short downtime (about 1pm Pacific to 3:45pm Pacific) but are back up and running. If you see anything odd, please tell me!

Comment

Post Header

Published:
2021-11-26 22:10:23 -0500
Tags:

Twice in the last two hours the server running SquidgeWorld bombed. I have no idea why, because there are no indicators. I've made some adjustments and we're back up. Please know there might be connectivity issues if this issue comes back.

Comment

Post Header

Published:
2021-11-24 17:02:47 -0500
Tags:

Yes, if you tried to get onto the archive since Sunday morning, you'll notice we weren't up and running. The reasons were many.

First, the failing SSD. When we rebooted after a kernel upgrade on Sunday, the SSD failed to boot. It would just not recognize itself as a bootable device. So after doing lots of things, I performed a "cloning" process to clone the data from the old, failing SSD to a new one. And that went perfect - until it didn't. See, instead of cloning just that SSD, it actually overwrote the second SSD for some reason, and altered what's called the UUIDs of both devices. The UUID is a unique identifier for a device - and it became corrupted on the second drive, so much that even if I could boot up the first replacement, I couldn't do anything with the system because it barfed at duplicate UUIDs.

That led to reinstalling. But beyond that, because of a few factors, I updated the level of code that is current for OTW's AO3. And then there were issue because the old code relied on certain specific items like elasticsearch 6.8, certain Java code levels, and certain Ruby gem levels that weren't available anymore and/or were outdated. So my update failed until I pulled all new code.

We should have been up by Tuesday morning at the latest. However, I shot myself in the foot (figuratively) because I overwrote some configuration items that I shouldn't have. This resulted in very mixed results - sometimes the configuration would work, sometimes not. And so this morning, bless zz9pzza because he sat with me for almost 2 hours until we realized that the wound was self inflicted. Yes, we should have been up about 30 hours ago.

So that's over and done with.

At this time, two things. First, I'm going to update my guide to installing OTW's code that created SquidgeWorld to include a bunch of new data. Second, the code that we're running now is a year newer than it was when we went down on Sunday. Most stuff is behind he scenes, but your user ID is now in your profile, and we have the "75 tag limit per work" code in production now. So there you go.

EDIT: As soon as we were up this afternoon, things went pearshaped again with the database. I just redid everything, and ::knock wood:: let's see if it stays up.

Comment

Post Header

Published:
2021-11-06 11:10:36 -0400
Tags:

We had a hard downtime earlier this week, and it turned out to be an issue with one of the SSDs (solid state drives, as opposed to hard disk drives) that SquidgeWorld runs on. It flailed about for a short period before settling down.

Because this is the second time that particular SSD has flailed, I've decided to pick up not only new spares, but an entirely new machine. I will be purchasing a new server, RAM, and SSDs, and before next Spring, we will be on this new hardware.

Comment

Post Header

Published:
2021-02-18 19:05:06 -0500
Tags:

We were down due to horrific storms here in the Pacific Northwest. No power, Internet, etc for 7 days. Questions? Ping me here or read my article at this link: https://squidgestatus.dreamwidth.org/81454.html

Comment

Post Header

Published:
2021-02-07 08:56:59 -0500
Tags:

Every website needs downtime from time to time for maintenance. Well, normally for SquidgeWorld, this means downtime on Sundays from 7am to 7:30am Pacific Time (15:00 to 15:30 UTC). So if you are on the site on Sundays during this time period and it suddenly goes down, you'll know it's because of downtime.

Comment

Post Header

Published:
2021-01-14 15:47:51 -0500
Tags:

We are back from an extended downtime!

Long story short: A power outage took us down

Long story: See, the more we hurt our environment, the more our environment is going to fight back against us. Here in the Pacific Northwest, we had a massive storm early Wednesday. Like torrential rains and 50MPH gusts. I wouldn't know, because I slept through it all (the power of melatonin, y'all!). Anyway, we lost power about 3am on Wednesday, and it's been out until about half an hour ago. It probably would have been longer, except the power line that was down was hanging over the middle of the street - COMPLETELY dangerous.

Here's the thing. Some people may yell at me when we go down, and I get it. But I can't afford to do all that I do in a datacenter. I have an amazing ISP, and for $100/month, we have a 1Gb/1Gb static fiber connection. So I put together a few machines, and host them in my garage. For the bandwidth that we use (between 750Gb and 1Tb per month) and the number of services we offer, it would cost me thousands of dollars per month. Why do you think The OTW does fundraising and comes away with hundreds of thousands of dollars per year? Sure, they're huge, and I'm just me. But if I tried to do this in a datacenter, I'd go broke. And Squidge.org has been around for 27 years come this Fall; I don't want to see it go away.

Keep track of us on Twitter for the latest!: SquidgeORG

Comment