• Buddahriffic@lemmy.world
          1·
          2 months ago

          Someone set up a script to automatically create daily backups to tape. Unfortunately, it’s still the first tape that was put in there 3.5 years ago, every backup since that one filled up failed. It might as well have failed silently because everyone who received the email with the error message filtered them to a folder they generally ignored.

  • umbrella@lemmy.ml
    1·
    2 months ago

    this week i sudo shutdown now our main service right at the end of the workday because i tought it was a local terminal.

    not a bright move.

    • LiveLM@lemmy.zipEnglish
      1·
      2 months ago

      Best thing I did was change my shell prompt so I can easily tell when it isn’t my machine

    • SavvyWolf@pawb.socialEnglish
      1·
      2 months ago

      There’s a package called molly-guard which will check to see if you are connected via ssh when you try to shut it down. If you are, it will ask you for the hostname of the system to make sure you’re shutting down the right one.

      Very usefull program to just throw onto servers.

    • Trainguyrom@reddthat.comEnglish
      1·
      2 months ago

      I was making after hours config changes on a pair of mostly-but-not-entirely redundant Cisco L3 switches which basically controlled the entire network at that location. While updating the running configs I mixed up which ssh session was which switch and accidentally gave both switches the same IP address, and before I noticed the error I copied the running config to the startup config.

      Due to other limitations and the fact that these changes were to fix DNS issues (and therefore I couldn’t rely on DNS to save me) I ended up keeping sshing in by IP until I got the right switch and trying to make the change before my session died due to dropped packets from the mucked up network situation I had created. That easily added a couple of hours of cleanup to the maintainence I was doing

    • naeap@sopuli.xyz
      1·
      2 months ago

      Never touch a running system
      Until you have a inviting hole in your system

      Nevertheless, I’m panicking every time I update my sever infrastructure…

  • Neuromancer@lemm.ee
    1·
    2 months ago

    In the old days some of the servers took at hour to reboot. That was stressful when you couldn’t ping it at an hour.

      • Neuromancer@lemm.ee
        1·
        2 months ago

        The more disk you had, the longer it took. It walked the scsi bus which took forever. So if you had more disk. It took even longer.

        Since everything was remote, you’d have to call hands and they weren’t technical. Also no cameras since it was the 90’s.

        Now when I restart a vm or container. I panic if it’s not back up in 10 minutes.

      • yhvr@lemm.ee
        1·
        2 months ago

        I’m at college right now, which is a 3 hour drive away from my home, where a server of mine is. I just have to ask my parents to turn it back on when the power goes out or it gets borked. I access it solely through RustDesk and Cloudflare Tunnels SSH (it’s actually pretty cool, they have a web interface for it).

        I have no car, so there’s really no way to access it in case something catastrophic happens. I have to rely on hopes, prayers, and the power of a probably outdated Pop!_OS install. Totally doesn’t stress me out I’ll just say I like to live on the edge :^)

  • Ignotum@lemmy.world
    0·
    2 months ago

    I have more than once typed shutdown instead of reboot when working on a remote machine… always fun

  • NastyNative@mander.xyzEnglish
    01·
    2 months ago

    Tbh there is nothing more taxing on my mental health than doing maintenance on our production servers.