Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.

Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.

Rolled back to the backup before I made it public and now I have a security checklist.

  • ohshit604@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    24
    ·
    edit-2
    1 day ago

    I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.

    • Valmond@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      1 day ago

      I’m interested, how do you expose your services (on your PC I assume) to the internet through wireguard? Is it theough some VPN?

      • Zanathos@lemmy.world
        link
        fedilink
        arrow-up
        8
        ·
        1 day ago

        Wireguard IS a VPN. He has somehow through his challenges of exposing services to the internet, exposed wireguard from his home to the internet for him to connect to. Then he can connect to his internal services from there.

        It’s honestly the best option and how I operate as well. I only have a handful of items exposed and even those flow through a DMZ proxy before hitting their destination servers.

      • ohshit604@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        VPN’s are neat, besides the fact they’re capable of masking your IP & DNS they’re also capable of providing resources to devices outside a network.

        A good example is the server at my work is only accessible on my works network, to access the server remotely without exposing it directly to the internet would be to use a VPN tunnel.

      • BCsven@lemmy.ca
        link
        fedilink
        arrow-up
        1
        ·
        19 hours ago

        Just a note, while wireguard makes its own VPN tunnel, it differs slightly in that it isn’t the typical VPN server with vpn connecting clients, it is more akin to a peer network. Each peer device gets it’s own “server” and “client” config section in the setup file. And you share the public keys between each peer before hand, and set the IPs to use.

  • ramius345@sh.itjust.works
    link
    fedilink
    arrow-up
    27
    arrow-down
    1
    ·
    edit-2
    2 days ago

    You should turn off ssh password logins on external facing servers at a minimum. Only use ssh keys, install fail2ban, disable ssh root logins, and make sure you have a firewall limiting ports to ssh and https.

    This will catch most scripted login attempts.

    If you want something more advanced, look into https://en.m.wikipedia.org/wiki/Security_Technical_Implementation_Guide and try to find an ansible playbook to apply them.

    • Oisteink@feddit.nl
      link
      fedilink
      arrow-up
      7
      ·
      2 days ago

      Just turn off password logins from anything but console. For all users. No matter where it runs.

      It becomes second to nature pretty fast, but you should have a system for storing / rotating keys.

      • Refurbished Refurbisher@lemmy.sdf.org
        link
        fedilink
        arrow-up
        4
        ·
        edit-2
        2 days ago

        How do I whitelist password logins? I only disabled password logins in SSHd and set it to only use a key.

        I also like to disable root login by setting its default shell to nologin, that way, it’s only accessible via sudo or doas. I think there’s a better way of doing it, which is how Debian does it by default when not setting a root password, but I’m not sure how to configure that manually, or even what they do.

        • Oisteink@feddit.nl
          link
          fedilink
          arrow-up
          3
          ·
          2 days ago

          Right - so console/tty login is restricted by pam and its settings. So disabling ssh root logins means you can still log in as root there.

          To lock root you can use passwd -l

          If locking root I would keep root shell so i could sudo to root.

          So my normal setup would be to create my admin user with sudo rights, set «PasswordAuthentication no» in sshd config and lock root with «sudo passwd -l root» Remember to add a pubkey to admin users authorizedkeys, and give it a secure but typable password

          My root is now only available through sudo, and i can use password on console. Instead of locking root you can give it secure typable password. This way root can log in from console so you dont need sudo for root access from console.

          It boils down to what you like and what risks you take compared to usable system. You can always recover a locked root account if you have access to single-user-mode or a live cd . Disk encryption makes livecd a difficult option.

        • Oisteink@feddit.nl
          link
          fedilink
          arrow-up
          3
          ·
          edit-2
          2 days ago

          To whitelist password logins in ssh you can match username and give them yes after you set no (for all). But i see no reason for password logons in ssh, console is safe enough (for me).

    • smiletolerantly@awful.systems
      link
      fedilink
      arrow-up
      2
      ·
      2 days ago

      Did all that, minus the no ssh root login (only key, obviously) plus one failed attempt, fail2ban permaban.

      Have not had any issues, ever

  • Punkie@lemmy.world
    link
    fedilink
    arrow-up
    172
    ·
    2 days ago

    Basic setup for me is scripted on a new system. In regards to ssh, I make sure:

    • Root account is disabled, sudo only
    • ssh only by keys
    • sshd blocks all users but a few, via AllowUsers
    • All ‘default usernames’ are removed, like ec2-user or ubuntu for AWS ec2 systems
    • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.
    • Services are only allowed connections by an allow list of IPs or subnets. Internal, when possible.

    My systems are not “unhackable” but not low-hanging fruit, either. I assume everything I have out there can be hacked by someone SUPER determined, and have a vector of protection to mitigate backwash in case they gain full access.

    • feddylemmy@lemmy.world
      link
      fedilink
      arrow-up
      71
      ·
      2 days ago
      • The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.

      Gosh I get unreasonably frustrated when someone says yeah but that’s just security through obscurity. Like yeah, we all know what nmap is, a persistent threat will just look at all 65535 and figure out where ssh is listening… But if you change your threat model and talk about bots? Logs are much cleaner and moving ports gets rid of a lot of traffic. Obviously so does enabling keys only.

      Also does anyone still port knock these days?

      • kernelle@0d.gs
        link
        fedilink
        arrow-up
        19
        ·
        2 days ago

        Also does anyone still port knock these days?

        Enter Masscan, probably a net negative for the internet, so use with care.

        • davidgro@lemmy.world
          link
          fedilink
          arrow-up
          7
          ·
          2 days ago

          I didn’t see anything about port knocking there, it rather looks like it has the opposite focus - a quote from that page is “features that support widespread scanning of many machines are supported, while in-depth scanning of single machines aren’t.”

          • kernelle@0d.gs
            link
            fedilink
            arrow-up
            4
            ·
            2 days ago

            Sure yeah it’s a discovery tool OOTB, but I’ve used it to perform specific packet sequences as well.

      • josefo@leminal.space
        link
        fedilink
        arrow-up
        4
        ·
        2 days ago

        Literally the only time I got somewhat hacked was when I left the default port of the service. Obscurity is reasonable, combined with other things like the ones mentioned here make you pretty much invulnerable to casuals. Somebody needs to target you to get anything.

  • kibiz0r@midwest.social
    link
    fedilink
    English
    arrow-up
    76
    ·
    2 days ago

    One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.

    “Hey, why is this Valheim server running like shit?”

    “Wtf is xrx?”

    “Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”

    So anyway, now I use NixOS.

    • pageflight@lemmy.world
      link
      fedilink
      English
      arrow-up
      17
      ·
      2 days ago

      Good point about a default deny approach to users and ssh, so random services don’t add insecure logins.

    • Tablaste@linux.communityOP
      link
      fedilink
      English
      arrow-up
      74
      arrow-down
      1
      ·
      edit-2
      2 days ago

      I published it to the internet and the next day, I couldn’t ssh into the server anymore with my user account and something was off.

      Tried root + password, also failed.

      Immediately facepalmed because the password was the generic 8 characters and there was no fail2ban to stop guessing.

      • lud@lemm.ee
        link
        fedilink
        arrow-up
        96
        ·
        2 days ago

        Don’t use passwords for ssh. Use keys and disable password authentication.

        • Voroxpete@sh.itjust.works
          link
          fedilink
          arrow-up
          57
          arrow-down
          4
          ·
          edit-2
          2 days ago

          More importantly, don’t open up SSH to public access. Use a VPN connection to the server. This is really easy to do with Netbird, Tailscale, etc. You should only ever be able to connect to SSH privately, never over the public net.

          • troed@fedia.io
            link
            fedilink
            arrow-up
            33
            arrow-down
            1
            ·
            2 days ago

            It’s perfectly safe to run SSH on port 22 towards the open Internet with public key authentication only.

                • DefederateLemmyMl@feddit.nl
                  link
                  fedilink
                  arrow-up
                  4
                  ·
                  1 day ago

                  A VPN like Wireguard can run over UDP on a random port which is nearly impossible to discover for an attacker. Unlike sshd, it won’t even show up in a portscan.

                  This was a specific design goal of Wireguard by the way (see “5.1 Silence is a virtue” here https://www.wireguard.com/papers/wireguard.pdf)

                  It also acts as a catch-all for all your services, so instead of worrying about the security of all the different sshds or other services you may have exposed, you just have to keep your vpn up to date.

                • designatedhacker@lemm.ee
                  link
                  fedilink
                  arrow-up
                  2
                  ·
                  1 day ago

                  Are you talking a VPN running on the same box as the service? UDP VPN would help as another mentioned, but doesn’t really add isolation.

                  If your vpn box is standalone, then getting root is bad but just step one. They have to own the VPN to be able to even do more recon then try SSH.

                  Defense in depth. They didn’t immediately get server root and application access in one step. Now they have to connect to a patched, cert only, etc SSH server. Just looking for it could trip into some honeypot. They had to find the VPN host as well which wasn’t the same as the box they were targeting. That would shut down 99% of the automated/script kiddie shit finding the main service then scanning that IP.

                  You can’t argue that one step to own the system is more secure than two separate pieces of updated software on separate boxes.

          • josefo@leminal.space
            link
            fedilink
            arrow-up
            4
            ·
            2 days ago

            Tailscale? Netbird? I have been using hamachi like a fucking neanderthal. I love this posts, I learn so much

            • jatone@lemmy.dbzer0.com
              link
              fedilink
              English
              arrow-up
              6
              ·
              edit-2
              2 days ago

              we’re probably talking about different things. virtually no distribution comes with root access with a password. you have to explicitly give the root user a password. without a password no amount of brute force sshing root will work. I’m not saying the root user is entirely disabled. so either the service OP is building on is basically a goldmine for compromised machines or OP literally shot themselves in the root by giving root a password manually. something you should never do.

              • satans_methpipe@lemmy.world
                link
                fedilink
                arrow-up
                2
                ·
                edit-2
                2 hours ago

                Yeah I was confused about the comment chain. I was thinking terminal login vs ssh. You’re right in my experience…root ssh requires user intervention for RHEL and friends and arch and debian.

                Side note: did you mean to say “shot themselves in the root”? I love it either way.

                • jatone@lemmy.dbzer0.com
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  1 hour ago

                  Side note: did you mean to say “shot themselves in the root”? I love it either way.

                  ssh its better with the typo. ;)

              • steventhedev@lemmy.world
                link
                fedilink
                arrow-up
                1
                ·
                2 days ago

                Many cloud providers (the cheap ones in particular) will put patches on top of the base distro, so sometimes root always gets a password. Even for Ubuntu.

                There are ways around this, like proper cloud-init support, but not exactly beginner friendly.

        • Björn Tantau@swg-empire.de
          link
          fedilink
          arrow-up
          8
          ·
          2 days ago

          Love Hetzner. You just give them your public key and they boot you into a rescue system from which you can install what you want how you want.

          • r00ty@kbin.life
            link
            fedilink
            arrow-up
            10
            ·
            2 days ago

            I think their auction servers are a hidden gem. I mean the prices used to be better. Now they have some kind of systrem that resets them when they get too low. But the prices are still pretty good I think. But a year or two ago I got a pretty good deal on two decently spec’d servers.

            People are scared off by the fact you just get their rescue prompt on auctions boxes… Except their rescue prompt has a guided imaging setup tool to install pretty much every popular distro with configurable raid options etc.

            • Björn Tantau@swg-empire.de
              link
              fedilink
              arrow-up
              7
              ·
              2 days ago

              Yeah, I basically jump from auction system to auction system every other year or so and either get a cheaper or more powerful server or both.

              • r00ty@kbin.life
                link
                fedilink
                arrow-up
                6
                ·
                2 days ago

                I monitor for good deals. Because there’s no contract it’s easy to add one, move stuff over at your leisure and kill the old one off. It’s the better way to do it for semi serious stuff.

      • cm0002@lemmy.world
        link
        fedilink
        arrow-up
        11
        ·
        edit-2
        2 days ago

        because the password was the generic 8 characters and there was no fail2ban to stop guessing

        Oof yea that’ll do it, your usually fine as long as you hardened enough to at least ward off the script kiddies. The people with actual real skill tend to go after…juicer targets lmao

        • Tablaste@linux.communityOP
          link
          fedilink
          English
          arrow-up
          11
          ·
          2 days ago

          Haha I’m pretty sure my little server was just part of the “let’s test our dumb script to see if it works. Oh wow it did what a moron!”

          Lessons learned.

      • stoy@lemmy.zip
        link
        fedilink
        arrow-up
        6
        ·
        2 days ago

        I ran a standard raspian ssh server on my home network for several years, default user was removed and my own user was in it’s place, root was configured as standard on a raspbian, my account had a complex but fairly short password, no specific keys set.

        I saw constant attacks but to my knowledge, it was never breached.

        I removed it when I realized that my ISP might take a dim view of running a server on their home client net that they didn’t know about, especially since it showed up on Shodan…

        Don’t do what I did, secure your systems properly!

        But it was kinda cool to be able to SSH from Thailand back home to Sweden and browse my NAS, it was super slow, but damn cool…

        • MonkeMischief@lemmy.today
          link
          fedilink
          arrow-up
          1
          ·
          edit-2
          1 day ago

          But it was kinda cool to be able to SSH from Thailand back home to Sweden and browse my NAS, it was super slow, but damn cool…

          That feels like sorcery, doesn’t it? You can still do this WAY safer by using Wireguard or something a little easier like Tailscale. I use Tailscale myself to VPN to my NAS.

          I get a kick out of showing people my NextCloud Memories albums or Jellyfin videos from my phone and saying “This is talking to the box in my house right now! Isn’t that cool!?” Hahaha.

          I’m almost glad I had to go that route. Most of our ISPs here in the U.S will block outgoing ports by default, so they can keep the network safe sell you a home business plan lol.

        • troed@fedia.io
          link
          fedilink
          arrow-up
          4
          ·
          2 days ago

          Why would a Swedish ISP care? I’ve run servers from home since I first connected up in … 1996. I’ve had a lot of different ISPs during that time, although nowadays I always choose Bahnhof because of them fighting the good fights.

          • stoy@lemmy.zip
            link
            fedilink
            arrow-up
            3
            ·
            2 days ago

            They probably don’t, unless I got compromised and bad traffic came from their network, but I was paranoid, and wanted to avoid the possibility.

      • JustEnoughDucks@feddit.nl
        link
        fedilink
        arrow-up
        6
        arrow-down
        1
        ·
        2 days ago

        Lol ssh has no reason to be port exposed in 99% of home server setups.

        VPNs are extremely easy, free, and wireguard is very performant with openvpn also fine for ssh. I have yet to see any usecase for simply port forwarding ssh in a home setup. Even a public git server can be tunneled through https.

        • MonkeMischief@lemmy.today
          link
          fedilink
          arrow-up
          1
          ·
          edit-2
          1 day ago

          Yeah I’m honest with myself that I’m a security newb and don’t know how to even know what I’m vulnerable to yet. So I didn’t bother opening anything at all on my router. That sounded way too scary.

          Tailscale really is magic. I just use Cloudflare to forward a domain I own, and I can get to my services, my NextCloud, everything, from anywhere, and I’m reasonably confident I’m not exposing any doors to the innumerable botnet swarms.

          It might be a tiny bit inconvenient if I wanted to serve anything to anyone not in my Tailnet or already on my home LAN (like sending al someone a link to a NextCloud folder for instance.), but at this point, that’s quite the edge case.

          I learned to set up NGINX proxy manager for a reverse proxy though, and that’s pretty great! I still harden stuff where I can as I learn, even though I’m confident nobody’s even seeing it.

          • JustEnoughDucks@feddit.nl
            link
            fedilink
            arrow-up
            2
            ·
            edit-2
            1 day ago

            Honestly, crowdsec with the nginx bouncer is all you need security-wise to start experimenting. It isn’t perfect security, but it is way more comprehensive than fail2ban for just getting started and figuring more out later.

            Here is my traefik-based crowdsec docker composer:

            services:
              crowdsec:
                image: crowdsecurity/crowdsec:latest
                container_name: crowdsec
                environment:
                  GID: $PGID
                volumes:
                  - $USERDIR/dockerconfig/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
                  - $USERDIR/data/Volumes/crowdsec:/var/lib/crowdsec/data/
                  - $USERDIR/dockerconfig/crowdsec:/etc/crowdsec/
                  - $DOCKERDIR/traefik2/traefik.log:/var/log/traefik/traefik.log:ro
                networks:
                  - web
                restart: unless-stopped
            
              bouncer-traefik:
                image: docker.io/fbonalair/traefik-crowdsec-bouncer:latest
                container_name: bouncer-traefik
                environment:
                  CROWDSEC_BOUNCER_API_KEY: $CROWDSEC_API
                  CROWDSEC_AGENT_HOST: crowdsec:8080
                networks:
                  - web # same network as traefik + crowdsec
                depends_on:
                  - crowdsec
                restart: unless-stopped
            
            networks:
              web:
                external: true
            

            https://github.com/imthenachoman/How-To-Secure-A-Linux-Server this is a more in-depth crash course for system-level security but hasn’t been updated in a while.

  • mlg@lemmy.world
    link
    fedilink
    English
    arrow-up
    47
    ·
    2 days ago

    Lol you can actually demo a github compromise in real time to an audience.

    Make a repo with an API key, publish it, and literally just watch as it takes only a few minutes before a script logs in.

  • Hozerkiller@lemmy.ca
    link
    fedilink
    English
    arrow-up
    46
    ·
    2 days ago

    I’ve gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn’t know better, instead of pure salt like another site I won’t mention.

    • Tablaste@linux.communityOP
      link
      fedilink
      English
      arrow-up
      31
      ·
      2 days ago

      I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.

      We all make mistakes!

      • LordCrom@lemmy.world
        link
        fedilink
        arrow-up
        16
        ·
        2 days ago

        There’s a 40 year I.T. veteran here that still suffers imposter syndrome. It’s a real thing I’ve never been able to shake off

        • pulsewidth@lemmy.world
          link
          fedilink
          arrow-up
          9
          ·
          2 days ago

          Just look at who is in the White House, mate - and not just the president, but basically you can pick anyone he’s hand-picked for his staff.

          Surely that’s an instant cure for any qualified person feeling imposter syndrome in their job.

    • LordCrom@lemmy.world
      link
      fedilink
      arrow-up
      11
      ·
      2 days ago

      If it’s public facing, how about dont turn on ssh to the public, open it to select ips or ranges. Use a non standard port, use a cert or even a radius with TOTP like privacyIdea. How about a port knocker to open the non standard port as well. Autoban to lock out source ips.

      That’s just off the top of my head.

      There’s a lot you can do to harden a host.

      • Faresh@lemmy.ml
        link
        fedilink
        English
        arrow-up
        2
        ·
        20 hours ago

        dont turn on ssh to the public, open it to select ips or ranges

        What if you don’t have a static IP, do you ask your ISP in what range their public addresses fall?

    • DefederateLemmyMl@feddit.nl
      link
      fedilink
      arrow-up
      3
      ·
      1 day ago

      Do not allow username/password login for ssh

      This is disabled by default for the root user.

      $ man sshd_config
      
      ...
             PermitRootLogin
                     Specifies whether root  can  log  in  using  ssh(1).   The  argument  must  be  yes,  prohibit-password,
                     forced-commands-only, or no.  The default is prohibit-password.
      ...
      
      
  • communism@lemmy.ml
    link
    fedilink
    arrow-up
    40
    ·
    2 days ago

    How are people’s servers getting compromised? I’m no security expert (I’ve never worked in tech at all) and have a public VPS, never been compromised. Mainly just use SSH keys not passwords, I don’t do anything too crazy. Like if you have open SSH on port 22 with root login enabled and your root password is password123 then maybe but I’m surprised I’ve never been pwned if it’s so easy to get got…

    • cmnybo@discuss.tchncs.de
      link
      fedilink
      English
      arrow-up
      28
      ·
      2 days ago

      By allowing password login and using weak passwords or by reusing passwords that have been involved in a data breach somewhere.

      • communism@lemmy.ml
        link
        fedilink
        arrow-up
        7
        ·
        2 days ago

        That makes sense. It feels a bit mad that the difference between getting pwned super easy vs not is something simple like that. But also reassuring to know, cause I was wondering how I heard about so many hobbyist home labs etc getting compromised when it’d be pretty hard to obtain a reasonably secured private key (ie not uploaded onto the cloud or anything, not stored on an unencrypted drive that other people can easily access, etc). But if it’s just password logins that makes more sense.

    • pageflight@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      ·
      2 days ago

      The one db I saw compromised at a previous employer was an AWS RDS with public Internet access open and default admin username/password. Luckily it was just full of test data, so when we noticed its contents had been replaced with a ransom message we just deleted the instance.

  • DaCrazyJamez@sh.itjust.works
    link
    fedilink
    arrow-up
    33
    ·
    2 days ago

    As a linux n00b who just recently took the plunge and set up a public site (tho really just for my own / selfhosting),

    Can anyone recommend a good guide or starting place for how to harden the setup? Im running mint on my former gaming rig, site is set up LAMP

  • Fair Fairy@thelemmy.club
    link
    fedilink
    arrow-up
    29
    ·
    1 day ago

    I’m confused. I never disable root user and never got hacked.

    Is the issue that the app is coded in a shitty way maybe ?

    • Xanza@lemm.ee
      link
      fedilink
      English
      arrow-up
      24
      ·
      1 day ago

      You can’t really disable the root user. You can make it so they can’t login remotely, which is highly suggested.

        • Xanza@lemm.ee
          link
          fedilink
          English
          arrow-up
          11
          ·
          24 hours ago

          There’s no real advantage to disable the root user, and I really don’t recommend it. You can disable SSH root login, and as long as you ensure root has a secure password that’s different than your own account your system is just as safe with the added advantage of having the root account incase something happens.

          • Possibly linux@lemmy.zip
            link
            fedilink
            English
            arrow-up
            2
            arrow-down
            3
            ·
            23 hours ago

            That wouldn’t be defense in depth. You want to limit anything that’s not necessary as it can become a source of attack. There is no reason root should be enabled.

            • Faresh@lemmy.ml
              link
              fedilink
              English
              arrow-up
              6
              ·
              20 hours ago

              I don’t understand. You will still need to do administrative tasks once in a while so it isn’t really unnecessary, and if root can’t be logged in, that will mean you will have to use sudo instead, which could be an attack vector just as su.

            • Xanza@lemm.ee
              link
              fedilink
              English
              arrow-up
              4
              arrow-down
              1
              ·
              22 hours ago

              Why do like, houses have doors man. You gotta eliminate all points of egress for security, maaaan. /s

              There’s no particular reason to disable root, and with a hardened system, it’s not even a problem you need to worry about…

      • MehBlah@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 day ago

        Another thing you can do under certain circumstances which I’m sure someone on here will point out is depreciated is use TCP Wrappers. If you are only connecting to ssh from known IP addresses or IP address ranges then you can effectively block the rest of the world from accessing you. I used a combination of ipset list, fail2ban and tcp wrappers along with my firewall which like is also something old called iptables-persistent. I’ve also moved my ssh port up high and created several other fake ports that keep anyone port scanning my IP guessing.

        These days I have all ports closed except for my wireguard port and access all of my hosted services through it.

    • cley_faye@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      1 day ago

      You can’t really disable it anyway.

      Hardening is mostly prevent root login from outside in case every other layer of authentication and access control broke, do not allow regular user to su/sudo into it for free, and have a tight grip on anything that’s executable and have a setuid bit set. I did not install a system from scratch in a long time but I believe this would be the default on most things that are not geared toward end-user devices, too.

  • Admiral Patrick@dubvee.org
    link
    fedilink
    English
    arrow-up
    25
    ·
    edit-2
    2 days ago

    And this is why every time a developer asks me for shell access to any of the deployment servers, I flat out deny the request.

    Good on you for learning from your mistakes, but a perfect example for why I only let sysadmins into the systems.

      • corsicanguppy@lemmy.ca
        link
        fedilink
        English
        arrow-up
        3
        ·
        2 days ago

        Please examine where devops allowed non-system people to be the last word on altering systems. This is a risk that needs block-letter indemnification or correction.

        It’s not that devops made ya lazy. I’ve been doing devops since before they coined the term, and it’s a constant effort to remind people that it doesn’t magically make things safe, but keeping it safe is still the way.

        • Tablaste@linux.communityOP
          link
          fedilink
          English
          arrow-up
          3
          ·
          2 days ago

          Ah not to discount devops, I mean that in a good way.

          Devops made me lazy in that for the past decade, I focus on just everything inside the code base.

          I literally push code into a magic black box that then triggers a rube goldberg of events. Servers get instanced. Configs just get magically set up. It’s beautiful. Just years of smart people who make it so easy that I never have to think about it.

          Since I can’t pay my devops team to come to my house, I get to figure it all out!

    • jatone@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      3
      ·
      edit-2
      2 days ago

      We have it at my company its just a very small group and we have to manually enable it for production and its through tools like teleport. Staging and the like its free game there for them for debugging, same infra through. Gives us best of all worlds

  • recklessengagement@lemmy.world
    link
    fedilink
    arrow-up
    25
    ·
    1 day ago

    This sounds like something everyone should go through at least once, to underscore the importance of hardening that can be easily taken for granted

  • phx@lemmy.ca
    link
    fedilink
    arrow-up
    21
    ·
    2 days ago

    Had this years ago except it was a dumbass contractor where I worked who left a Windows server with FTP services exposed to the Internet and IIRC anonymous FTP enabled, on a Friday.

    When I came in on Monday it had become a repository for warez, malware, and questionable porn. We wiped out rather than trying to recover anything.

  • nonentity@sh.itjust.works
    link
    fedilink
    arrow-up
    21
    ·
    2 days ago

    Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.

    The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.