Apologies to all for today’s performance issues and downtime.
Logging for our lemmy instance filled our main partition and brought everything to a halt. Our lemmy instance shares hardware with blahaj.zone, and the significant growth in lemmy over the last 10 days has caused the logs to grow in size exponentially! As a result of the logs filling up the main partition, both lemmy.blahaj.zone and blahaj.zone itself went down.
We’ve freed up some space, upgraded the hardware and spun everything up again, and it looks everything is back to normal.
Apologies for the inconvenience, and welcome back to everyone!
Servers going down because logs fill a disk is one common Sysadmin experience, achievement unlocked! 😄
Thanks for doing the work 👍
I mean, if we expected lemmy to grow in to an instance 3 times the size of blahaj.zone, we’d have done things differently from the start, but suffice to say, we didn’t expect Blahaj Lemmy to be anything but a bit of a sandbox for me to play with, so we made some initial choices that didn’t scale well with sudden unexpected growth :)
And that too is an achievement unlocked! 😄
Been there, done that, got some premature white hairs :P