SqWA News

Post Header

Published:
2021-11-24 17:02:47 -0500
Tags:

Yes, if you tried to get onto the archive since Sunday morning, you'll notice we weren't up and running. The reasons were many.

First, the failing SSD. When we rebooted after a kernel upgrade on Sunday, the SSD failed to boot. It would just not recognize itself as a bootable device. So after doing lots of things, I performed a "cloning" process to clone the data from the old, failing SSD to a new one. And that went perfect - until it didn't. See, instead of cloning just that SSD, it actually overwrote the second SSD for some reason, and altered what's called the UUIDs of both devices. The UUID is a unique identifier for a device - and it became corrupted on the second drive, so much that even if I could boot up the first replacement, I couldn't do anything with the system because it barfed at duplicate UUIDs.

That led to reinstalling. But beyond that, because of a few factors, I updated the level of code that is current for OTW's AO3. And then there were issue because the old code relied on certain specific items like elasticsearch 6.8, certain Java code levels, and certain Ruby gem levels that weren't available anymore and/or were outdated. So my update failed until I pulled all new code.

We should have been up by Tuesday morning at the latest. However, I shot myself in the foot (figuratively) because I overwrote some configuration items that I shouldn't have. This resulted in very mixed results - sometimes the configuration would work, sometimes not. And so this morning, bless zz9pzza because he sat with me for almost 2 hours until we realized that the wound was self inflicted. Yes, we should have been up about 30 hours ago.

So that's over and done with.

At this time, two things. First, I'm going to update my guide to installing OTW's code that created SquidgeWorld to include a bunch of new data. Second, the code that we're running now is a year newer than it was when we went down on Sunday. Most stuff is behind he scenes, but your user ID is now in your profile, and we have the "75 tag limit per work" code in production now. So there you go.

EDIT: As soon as we were up this afternoon, things went pearshaped again with the database. I just redid everything, and ::knock wood:: let's see if it stays up.

Comment

Post Header

Published:
2021-02-18 19:05:06 -0500
Tags:

We were down due to horrific storms here in the Pacific Northwest. No power, Internet, etc for 7 days. Questions? Ping me here or read my article at this link: https://squidgestatus.dreamwidth.org/81454.html

Comment

Post Header

Published:
2021-01-14 15:47:51 -0500
Tags:

We are back from an extended downtime!

Long story short: A power outage took us down

Long story: See, the more we hurt our environment, the more our environment is going to fight back against us. Here in the Pacific Northwest, we had a massive storm early Wednesday. Like torrential rains and 50MPH gusts. I wouldn't know, because I slept through it all (the power of melatonin, y'all!). Anyway, we lost power about 3am on Wednesday, and it's been out until about half an hour ago. It probably would have been longer, except the power line that was down was hanging over the middle of the street - COMPLETELY dangerous.

Here's the thing. Some people may yell at me when we go down, and I get it. But I can't afford to do all that I do in a datacenter. I have an amazing ISP, and for $100/month, we have a 1Gb/1Gb static fiber connection. So I put together a few machines, and host them in my garage. For the bandwidth that we use (between 750Gb and 1Tb per month) and the number of services we offer, it would cost me thousands of dollars per month. Why do you think The OTW does fundraising and comes away with hundreds of thousands of dollars per year? Sure, they're huge, and I'm just me. But if I tried to do this in a datacenter, I'd go broke. And Squidge.org has been around for 27 years come this Fall; I don't want to see it go away.

Keep track of us on Twitter for the latest!: SquidgeORG

Comment