I work at a medium size company with hundreds of Linux servers and none of them get updated. Because it’s more important that they keep running as they are than to have the latest updates. I bet this is very common for most companies.
There is nothing more important than security patches on a system.
I used to work at an FMI, which’s motto was “keep things stable”. Even the ciso department bought that crap.
Until we hired a white hat hacker. The only thing given was the name of the company. He managed to get into the building, access an employee’s workstation and install a root kit on one of the most important financial message tracking systems (you know, the one that instructs other systems to transfer money), using a security bug, which would have been patched if they kept a regular (security) update cycle. After shit hit the fan, many people were fired and an update cycle was introduced.
No system is important enough to not patch. And if you believe it is, you’re wrong.
Yeah, but that just takes way too much work. You think I really care about the company’s/bank’s money if I’m not getting paid enough for that job? Security patches can also introduce new problems, like x changes, so y doesn’t work, so the main app doesn’t work… and what, then I have to manually edit code, introduce the thing that x relied on so that y can work again?
I’m sorry, but this is not your average IT department’s job… or if it is, I expect a damn good compensation for it.
I’ve updated and rolled back snapshots because of shit like this… nah, not gonna try and figure out what the problem was… at least not for the salary I’m currently getting paid. If it burns, it burns, so be it.
I’d be surprised if you actually saw anything change from security updates tbh, I don’t think I’ve ever seen anything break from a quick patch.
Dist upgrades are when things might break, but they’re only once every few years. Leave them too long though and you may end up with compatibility issues if you need to make changes.
Fair enough if you’re not getting paid enough, the company should hire more people to stay on top of that though.
Exactly… because it’s tidious and time consuming… and I won’t get extra pay for it. Meanwhile I’m also expected to do everything else I do… sorry, just not gonna bother at all.
They have and that is why I don’t do them any more. Happened a few months ago in fact. Updated one of the Debian servers for one of webapps we have running (a black box piece of shit VM that’s stuck in 2010 I think)… suddenly, the app in the VM doesn’t work. The VM does start, but the app doesn’t work, just throws a 404. Why? Beats me, don’t have time to troubleshoot. Roll back a snapshot, everything works again. Conclusion, don’t update that.
See, around here, you don’t keep your job by messing around with things that already work. They work, period, why did even feel the need to mess with that 🤨. If that’s management’s view on security, fine, so be it 🤷.
I still file reports on things not being up to date, just so that if shit hits the fan, I’m not the one taking the fall for it.
“Way too much work” — if you ever said that where I work I’d fire you or not hire you in a heartbeat. An administrator’s role is not only to the stability of the system but the security too. You’re a hackers wet dream.
Cheapskates don’t get top of the line security hardening. Pay more now or suffer a breach and pay contractors $1000/hr to fix your broken shit because you paid minimum wage for an administrator position and wanted them to do 5 jobs at once.
No no no no, I meant what I said. Working 5 jobs AT ONCE (at one postion, one place)… AT THE SAME TIME. I do hardware, software, scripting, installing, configuration, maintenance (hardware and software… the whole shbang, DB included), Linux, Windows, BSD, servers, workstations (over 800 of them, and it’s me and 2 other guys under me!)… I even do freaking rig dusting! And ALL that, for 700 freaking euros a month! Not to mention pirating, that is also included cuz… they’re too cheap to pay for licenses for… well, anything really.
So, excuse me if I don’t care if the servers are up to date, mmmk.
One, you have no idea how much or little I’m getting paid. Two, you have no idea where I live and the struggles I have to face every day. Three, even if I do work “as expected”, I won’t get paid more (agan, you don’t know where I live).
It’s real easy to bitch about work ethics on a full stomach while getting back from work in a nice car with heated leather seats.
What I say still holds. If you ever uttered that sentiment in front of me or did not follow through on patching when asked you’d be out on your ass. Has nothing to do with your situation or what you’re making. Your righteous indignation on patching has no place in a business plain and simple.
Not trying to start an argument here but you sound very far removed from individual contributors, so maybe from your point of view it would simply look like adding it to a pile. More important than adding it to a pile is to make sure there’s systems in place to make sure OSs are patched. You wouldn’t be complaining to the IT/sysadmin guy about your servers’ vulnerability or patching schedules, you’d be talking to your cybersec department who’d have oversight. And if there’s a breach and your only defense is “I added it to the IT guys pile”, 100% you are getting fired as well.
I get your point, but capitalism is about doing the least amount of work to maximize your pay. And if the owners (who have the most skin ulin the game) don’t care about infrastructure then why should anyone else?
No, it doesn’t… because as I said, you have your needs (food, shelter, good car, nice place to live) met… you wouldn’t be talking like this if your place was a dump or you ate the cheapes shit on the market (cuz that’s what you can afford).
Every individual concern you’ve brought up in this thread is valid and not uncommon. The important part that @ramble81@lemm.ee is focusing on is that your mindset on how you approach these challenges is incredibly unprofessional.
Having more to do than you realistically have time to do is the reality of working in IT. Everything the business does ultimately comes back to IT at some point in the process, so we naturally have to work with every single branch of the business.
Being underpaid is a reality of work for most people in the modern world. The professional thing to do is to decouple your feelings of how you’re being paid from individual tasks or duties that are expected of you. “I’m not paid enough to deal with this” should be limited to tasks outside of your scope of work. If you’re ultimately not paid enough to do your job, complaining about individual tasks that are part of your job being bnove your paygrade is just saying you aren’t willing to do your job. I can’t tell you the best solution to your pay situation. Maybe changing jobs or even changing industries will help, but also changing your mindset can do wonders for your mental health. For example shifting to instead saying to yourself “I’m woefully underpaid but at least I’m working in IT and not at X” can greatly help ease the pain until you reach whatever milestone which does help improve your situation
What you need to do regarding your workload is have a conversation with your manager/superior about prioritization. You say “hey, I have this this this and this that need to get done right now and I can’t realistically do them all, what do you want me to prioritize and deprioritize” and if something important hasn’t been given priority in a long time (such as patches) you need to then push back and say “we haven’t been able to apply security patches in quite a while, I think we should reprioritize this so we can put some time into patching this week” this is how you manage a gigantic workload is by shifting priorities. The longer important maintenance tasks are ignored, the larger the impact and the harder it will be to complete the tasks
The important part that @ramble81@lemm.ee is focusing on is that your mindset on how you approach these challenges is incredibly unprofessional.
I know it is.
Now, ask me why that is.
The professional thing to do is to decouple your feelings of how you’re being paid from individual tasks or duties that are expected of you. “I’m not paid enough to deal with this” should be limited to tasks outside of your scope of work.
Lol 😂, no one has actually told me to keep the servers up to date, the only thing I was ever told was “keep shit running”. I’ve done updates on my own incentive, since I’m the senior IT engineer in the company. When things turn to shit after an update, hey, I’m rolling back a snapshot, I did not sign up for this 🙌.
And that is basically it 🤷… I get the same salary regardless if I do them or not. After a few failed ones, I just gave up. F it, not worth the time or the effort.
If you’re ultimately not paid enough to do your job, complaining about individual tasks that are part of your job being bnove your paygrade is just saying you aren’t willing to do your job.
Nope, I’m saying “Fuck you, pay me!”.
You think they care about updates and security? I’ve mentioned it a few times at meetings… “yeah, we’ll talk about that later”. OK 🤷. They obviously have no idea how fucked up things can get if you’re not up to date regarding security… but hey, they have been warned more than a few times 🤷.
What you need to do regarding your workload is have a conversation with your manager/superior about prioritization. You say “hey, I have this this this and this that need to get done right now and I can’t realistically do them all, what do you want me to prioritize and deprioritize” and if something important hasn’t been given priority in a long time (such as patches) you need to then push back and say “we haven’t been able to apply security patches in quite a while, I think we should reprioritize this so we can put some time into patching this week” this is how you manage a gigantic workload is by shifting priorities. The longer important maintenance tasks are ignored, the larger the impact and the harder it will be to complete the tasks
🤦… dude, you really have no idea where I live 😂🤣😂… otherwise you wouldn’t be saying this.
Things have been said more than once… I have asked, have pleeded for more personel… deaf ears. I have put it in writing, no use. Fine, then I just keep things running and that’s basically it 🤷.
Oh, and regarding workload, I already prioritize. The priority is to keep shit running, not to be up to date (obviously)… so, I just keep things running.
No no no, I am not getting promoted anyway… no promotions here. No one ever gets promoted, you don’t get bonuses, you don’t get overhours pay, you don’t get raise… so why bother 🤷.
The I can only recommend you to start automating everything you do, to make tour job easier and make more time to slack 😝
Start small, and build on that.
Try Ansible, it is easy and allows you to do some really cool stuff. It helped me migrate 500+ systems from KVM to vmware, where no commercial tool was able to help me…
Creating automation takes time. I have done it with certain things, but not everything. Some things are just way too complicated to easily create automation for them.
Though I do agree that things like Ansibel can help and I have been thinking about this for a while, but then I’d have to drop everything else I’m doing for like at least 2 weeks and do JUST that, nothing else… which is not an option currently.
Plus, there is always the risk of “shit not working” after an update… and frankly, we’ve had that before and again, I am not getting paid enough to deal with the backlash of things like that happeneing.
If it’s important that it keeps running then it should just be redundant and taking one node down for an update shouldn’t be an issue. I know this is wishful thinking for a lot of services but I refuse to be on call for something if the client can’t be bothered to make it redundant.
Jup same here. We have a colleague that constantly reminds everyone that we’re not properly patched (even running eol versions) but there’s always something to be done that’s a higher priority.
Exactly. Shit needs to just work, period. Why? Because otherwise, I’m the one getting 2AM calls… and I would be OK with that if I’m properly compensated for it… which I’m not.
See, building and configuring a staging environment also takes time and money… money which they are not willing to spend on something “for testing” and not in actual use. Plus, I’m not gonna get paid for doing that either, so why actually do it… to be honest, I would do it, even for free, but you gotta caugh up the money for the hardware man. I’ve been told “just use what you have in the scrap pile”… for what, a server 🤨? Are you serious? They barely spend any money on that even, why should I bother creating something as e staging environment.
Actually, I kinda am. Can’t really afford to spend a month or two without pay, so if I do find anything better (which I seriously doubt, every company here is more or less the same regarding IT practices), it would have to be a drop in replacement, which is also hard to do here (they’re gonna try and squeeze as much free labour as possible from you, so you’ll probably be stuck with a 200, 250 euro freelance salary for the next month or two, and as I said, I just can’t afford to do that right now, money is tight, got a family now).
Typically monthly or quarterly patching depending on severity and DMZ exposures. When log4j or shellshock hit it was patch once the patch was released and tested
If it’s a personal server that can manage being down for 15min or so. You could just setup auto updates with email if anything goes wrong and reboot off hours. Containers also make it less risky although it does fail to update every once in a great while.
The auto updates tools in Debian does allow you to specify security only or security and packages but not kernel. Mix and match, specify a version to stay on, include back ports, etc.
I work at a medium size company with hundreds of Linux servers and none of them get updated. Because it’s more important that they keep running as they are than to have the latest updates. I bet this is very common for most companies.
There is nothing more important than security patches on a system.
I used to work at an FMI, which’s motto was “keep things stable”. Even the ciso department bought that crap. Until we hired a white hat hacker. The only thing given was the name of the company. He managed to get into the building, access an employee’s workstation and install a root kit on one of the most important financial message tracking systems (you know, the one that instructs other systems to transfer money), using a security bug, which would have been patched if they kept a regular (security) update cycle. After shit hit the fan, many people were fired and an update cycle was introduced.
No system is important enough to not patch. And if you believe it is, you’re wrong.
Yeah, but that just takes way too much work. You think I really care about the company’s/bank’s money if I’m not getting paid enough for that job? Security patches can also introduce new problems, like x changes, so y doesn’t work, so the main app doesn’t work… and what, then I have to manually edit code, introduce the thing that x relied on so that y can work again?
I’m sorry, but this is not your average IT department’s job… or if it is, I expect a damn good compensation for it.
I’ve updated and rolled back snapshots because of shit like this… nah, not gonna try and figure out what the problem was… at least not for the salary I’m currently getting paid. If it burns, it burns, so be it.
I’d be surprised if you actually saw anything change from security updates tbh, I don’t think I’ve ever seen anything break from a quick patch.
Dist upgrades are when things might break, but they’re only once every few years. Leave them too long though and you may end up with compatibility issues if you need to make changes.
Fair enough if you’re not getting paid enough, the company should hire more people to stay on top of that though.
Usually you upgrade everything though, not just sec patches. And it’s a risk that something stops working, and nobody wants to spend time on that…
No, you update strictly only security patches.
Exactly… because it’s tidious and time consuming… and I won’t get extra pay for it. Meanwhile I’m also expected to do everything else I do… sorry, just not gonna bother at all.
They have and that is why I don’t do them any more. Happened a few months ago in fact. Updated one of the Debian servers for one of webapps we have running (a black box piece of shit VM that’s stuck in 2010 I think)… suddenly, the app in the VM doesn’t work. The VM does start, but the app doesn’t work, just throws a 404. Why? Beats me, don’t have time to troubleshoot. Roll back a snapshot, everything works again. Conclusion, don’t update that.
See, around here, you don’t keep your job by messing around with things that already work. They work, period, why did even feel the need to mess with that 🤨. If that’s management’s view on security, fine, so be it 🤷.
I still file reports on things not being up to date, just so that if shit hits the fan, I’m not the one taking the fall for it.
God I hope I don’t use any products from your company.
They make multimedia, so you’ve probably “used” it at some point… or not… depends how much TV you watch.
“Way too much work” — if you ever said that where I work I’d fire you or not hire you in a heartbeat. An administrator’s role is not only to the stability of the system but the security too. You’re a hackers wet dream.
The phrase “Fuck you, pay me” comes to mind.
Cheapskates don’t get top of the line security hardening. Pay more now or suffer a breach and pay contractors $1000/hr to fix your broken shit because you paid minimum wage for an administrator position and wanted them to do 5 jobs at once.
This guy gets it… and probably doesn’t live in the US, cuz he knows the term “work 5 jobs at once”.
Surely you meant the opposite? Working multiple jobs is a very USian thing. Now I’m curious, where are you from?
No no no no, I meant what I said. Working 5 jobs AT ONCE (at one postion, one place)… AT THE SAME TIME. I do hardware, software, scripting, installing, configuration, maintenance (hardware and software… the whole shbang, DB included), Linux, Windows, BSD, servers, workstations (over 800 of them, and it’s me and 2 other guys under me!)… I even do freaking rig dusting! And ALL that, for 700 freaking euros a month! Not to mention pirating, that is also included cuz… they’re too cheap to pay for licenses for… well, anything really.
So, excuse me if I don’t care if the servers are up to date, mmmk.
From Macedonia BTW.
Yeah if a company relies on some underpaid person from Macedonia to do an entire IT departments worth of crap then they deserve to get hacked lmao
One, you have no idea how much or little I’m getting paid. Two, you have no idea where I live and the struggles I have to face every day. Three, even if I do work “as expected”, I won’t get paid more (agan, you don’t know where I live).
It’s real easy to bitch about work ethics on a full stomach while getting back from work in a nice car with heated leather seats.
What I say still holds. If you ever uttered that sentiment in front of me or did not follow through on patching when asked you’d be out on your ass. Has nothing to do with your situation or what you’re making. Your righteous indignation on patching has no place in a business plain and simple.
Not trying to start an argument here but you sound very far removed from individual contributors, so maybe from your point of view it would simply look like adding it to a pile. More important than adding it to a pile is to make sure there’s systems in place to make sure OSs are patched. You wouldn’t be complaining to the IT/sysadmin guy about your servers’ vulnerability or patching schedules, you’d be talking to your cybersec department who’d have oversight. And if there’s a breach and your only defense is “I added it to the IT guys pile”, 100% you are getting fired as well.
I get your point, but capitalism is about doing the least amount of work to maximize your pay. And if the owners (who have the most skin ulin the game) don’t care about infrastructure then why should anyone else?
No, it doesn’t… because as I said, you have your needs (food, shelter, good car, nice place to live) met… you wouldn’t be talking like this if your place was a dump or you ate the cheapes shit on the market (cuz that’s what you can afford).
Every individual concern you’ve brought up in this thread is valid and not uncommon. The important part that @ramble81@lemm.ee is focusing on is that your mindset on how you approach these challenges is incredibly unprofessional.
Having more to do than you realistically have time to do is the reality of working in IT. Everything the business does ultimately comes back to IT at some point in the process, so we naturally have to work with every single branch of the business.
Being underpaid is a reality of work for most people in the modern world. The professional thing to do is to decouple your feelings of how you’re being paid from individual tasks or duties that are expected of you. “I’m not paid enough to deal with this” should be limited to tasks outside of your scope of work. If you’re ultimately not paid enough to do your job, complaining about individual tasks that are part of your job being bnove your paygrade is just saying you aren’t willing to do your job. I can’t tell you the best solution to your pay situation. Maybe changing jobs or even changing industries will help, but also changing your mindset can do wonders for your mental health. For example shifting to instead saying to yourself “I’m woefully underpaid but at least I’m working in IT and not at X” can greatly help ease the pain until you reach whatever milestone which does help improve your situation
What you need to do regarding your workload is have a conversation with your manager/superior about prioritization. You say “hey, I have this this this and this that need to get done right now and I can’t realistically do them all, what do you want me to prioritize and deprioritize” and if something important hasn’t been given priority in a long time (such as patches) you need to then push back and say “we haven’t been able to apply security patches in quite a while, I think we should reprioritize this so we can put some time into patching this week” this is how you manage a gigantic workload is by shifting priorities. The longer important maintenance tasks are ignored, the larger the impact and the harder it will be to complete the tasks
I know it is.
Now, ask me why that is.
Lol 😂, no one has actually told me to keep the servers up to date, the only thing I was ever told was “keep shit running”. I’ve done updates on my own incentive, since I’m the senior IT engineer in the company. When things turn to shit after an update, hey, I’m rolling back a snapshot, I did not sign up for this 🙌.
And that is basically it 🤷… I get the same salary regardless if I do them or not. After a few failed ones, I just gave up. F it, not worth the time or the effort.
Nope, I’m saying “Fuck you, pay me!”.
You think they care about updates and security? I’ve mentioned it a few times at meetings… “yeah, we’ll talk about that later”. OK 🤷. They obviously have no idea how fucked up things can get if you’re not up to date regarding security… but hey, they have been warned more than a few times 🤷.
🤦… dude, you really have no idea where I live 😂🤣😂… otherwise you wouldn’t be saying this.
Things have been said more than once… I have asked, have pleeded for more personel… deaf ears. I have put it in writing, no use. Fine, then I just keep things running and that’s basically it 🤷.
Oh, and regarding workload, I already prioritize. The priority is to keep shit running, not to be up to date (obviously)… so, I just keep things running.
Well, getting promoted is difficult when you watch Netflix or browse lemmy on company time
No no no, I am not getting promoted anyway… no promotions here. No one ever gets promoted, you don’t get bonuses, you don’t get overhours pay, you don’t get raise… so why bother 🤷.
The I can only recommend you to start automating everything you do, to make tour job easier and make more time to slack 😝
Start small, and build on that.
Try Ansible, it is easy and allows you to do some really cool stuff. It helped me migrate 500+ systems from KVM to vmware, where no commercial tool was able to help me…
Creating automation takes time. I have done it with certain things, but not everything. Some things are just way too complicated to easily create automation for them.
Though I do agree that things like Ansibel can help and I have been thinking about this for a while, but then I’d have to drop everything else I’m doing for like at least 2 weeks and do JUST that, nothing else… which is not an option currently.
Plus, there is always the risk of “shit not working” after an update… and frankly, we’ve had that before and again, I am not getting paid enough to deal with the backlash of things like that happeneing.
If it’s important that it keeps running then it should just be redundant and taking one node down for an update shouldn’t be an issue. I know this is wishful thinking for a lot of services but I refuse to be on call for something if the client can’t be bothered to make it redundant.
Jup same here. We have a colleague that constantly reminds everyone that we’re not properly patched (even running eol versions) but there’s always something to be done that’s a higher priority.
Exactly. Shit needs to just work, period. Why? Because otherwise, I’m the one getting 2AM calls… and I would be OK with that if I’m properly compensated for it… which I’m not.
Did you think of testing security updates on a staging environment before going in production with it, if you suspect in can break things?
I think there is no excuse to apply security fixes wich have a CVE number.
If you are on Debian stable unattended updates are not a problem.
See, building and configuring a staging environment also takes time and money… money which they are not willing to spend on something “for testing” and not in actual use. Plus, I’m not gonna get paid for doing that either, so why actually do it… to be honest, I would do it, even for free, but you gotta caugh up the money for the hardware man. I’ve been told “just use what you have in the scrap pile”… for what, a server 🤨? Are you serious? They barely spend any money on that even, why should I bother creating something as e staging environment.
This sounds so horrible, I would consider finding a better employer. I hope, you are not stuck with them.
Actually, I kinda am. Can’t really afford to spend a month or two without pay, so if I do find anything better (which I seriously doubt, every company here is more or less the same regarding IT practices), it would have to be a drop in replacement, which is also hard to do here (they’re gonna try and squeeze as much free labour as possible from you, so you’ll probably be stuck with a 200, 250 euro freelance salary for the next month or two, and as I said, I just can’t afford to do that right now, money is tight, got a family now).
Not at all.
Typically monthly or quarterly patching depending on severity and DMZ exposures. When log4j or shellshock hit it was patch once the patch was released and tested
If it’s a personal server that can manage being down for 15min or so. You could just setup auto updates with email if anything goes wrong and reboot off hours. Containers also make it less risky although it does fail to update every once in a great while.
All of that can also be tested in a preproduction environment as well, downtime is really a poor excuse for not patching
The auto updates tools in Debian does allow you to specify security only or security and packages but not kernel. Mix and match, specify a version to stay on, include back ports, etc.