Sign in

SadServers

@sadservers.com
427 followers 89 following 349 posts

SadServers.com - Practical Linux & DevOps Troubleshooting

PostsRepliesMedia
SadServers @sadservers.com · 30/09/2026
New AWS FinOps hands-on challenge "Auckland": S3 delete & archive old backups sadservers.com/scenario/auc...
aws cli
010
SadServers @sadservers.com · 29/09/2026
(Funny they included the cents in the figure). So Linux sysadmin / DevOps / Infra Eng / Cloud Eng / Platform Eng / SRE / flavour of the month whatever want to call it Eng will still be in demand 🙂
Anthropic
020
SadServers @sadservers.com · 28/09/2026
New SadServers AWS cli scenario "Istanbul": Delete unattached AWS EBS volumes , a pretty common cloud engineering task sadservers.com/scenario/ist...
aws cli
011
SadServers @sadservers.com · 27/09/2026
Waterloo Rocketry Reaches New Heights With World's Highest Amateur Liquid Bi-propellant Rocket. SadServers, another endeavour from Waterloo reaching new heights is happy to sponsor them! 🚀 www.waterloorocketry.com
Polaris launch
010
SadServers @sadservers.com · 27/09/2026
New SadServers scenario based on a real situation: needing to renew a TLS/SSL certificate and having many keys from different certificates without documentation of which key matched the certificate sadservers.com/scenario/sal...
sadservers.com
SadServers: "Salzburg": The TLS key that still matches - nginx, ssl Troubleshooting Lab
Linux Troubleshooting Labs Interview DevOps SRE
000
SadServers @sadservers.com · 23/09/2026
love.sadservers.com/t/46b41eb1-4...
I've been using SadServers as part of my hands-on DevOps learning path, and it's honestly one of the best tools I've found for building real troubleshooting instincts. SadServers drops you into an actual broken environment and makes you diagnose and fix it yourself. That's exactly the kind of pressure-tested practice that sticks.
000
SadServers @sadservers.com · 21/09/2026
Brave is a privacy-focused search engine with its own index. Glad to see more people coming to SadServers from it search.brave.com
brave
000
SadServers @sadservers.com · 20/09/2026
New optional SadServers UI for registered users (need to log in, Beta) pager.sadservers.com 🚀
000
SadServers @sadservers.com · 19/09/2026
"It's like KodeKloud but way better" love.sadservers.com/t/1385b955-4...
love.sadservers.com
Testimonial from Shea
Amazing build and platform! I will definitely share with my own students and use myself. I love how you built this! What an inspiration! It's like KodeKloud but way better in making sure you do the wo...
011
SadServers @sadservers.com · 19/09/2026
Tools in our SadServers stack that we love: - @tailscale.com : simple VPN - @sentry.io : application alerts, metrics and logs (especially vs datadog pricing) - @plausible.io: awesome Open Source Google Analytics alternative - Sqlite3: fast database in a file simplifies a lot of things
121
SadServers @sadservers.com · 18/09/2026
Four users so far have been able to solve the super-hard "Cadiz" SadServers scenario. Not for the faint of heart; here's the comment of one of those special users: love.sadservers.com/t/5d5b11fc-d...
love.sadservers.com
Testimonial from S Orekhivskyi
this is so good. i spent 8 days on cadiz in between exams. was pain and torture and i cried a bit but i did it and even made a script to automate it. worth every second if you love being desperate eno...
011
SadServers @sadservers.com · 13/09/2026
Sunday is for spicy takes. How to write a recruiter email engineers will actually answer (or whatever bait title you want, you get the point) docs.sadservers.com/blog/how-to-...
docs.sadservers.com
How to write a recruiter email engineers will actually answer
Brought to you by SadServers  – Hands-On Linux & DevOps; Real Challenges. Real Infra. Real Skills. (No AI tokens were spent in the creation of this post). I used to get a lot of emails from recruiters...
000
SadServers @sadservers.com · 11/09/2026
New #Linux SadServers scenario "Dakar": The half-installed package , what happens if an apt package is not fully installed? practice on a live server sadservers.com/scenario/dakar
sadservers.com
SadServers: "Dakar": The half-installed package - Packages, realistic-interviews Troubleshooting Lab
Linux Troubleshooting Labs Interview DevOps SRE
000
SadServers @sadservers.com · 11/09/2026
Last SadServers bulletin: list.sadservers.com/archive/0f93... We released 7 new scenarios in the last 4 weeks plus new features 😀
list.sadservers.com
SadServers Newsletter #22: more scenarios 🤔
000
SadServers @sadservers.com · 10/09/2026
Our hardest #Linux scenario yet, expecting very few people to solve it (AI agent couldn't after millions of tokens). Thinking of giving out a prize for defusing the bomb 💣 : sadservers.com/scenario/cadiz
defusing bomb
021
SadServers @sadservers.com · 09/09/2026
Canada just rebuilt 4 nuclear reactors at Darlington. 4 months EARLY. C$150M UNDER budget. They didn't work on the reactor but a copy of it, some "DevOps' practice here 😀 x.com/BenArendi/st...
x.com
Bensu.eth (@BenArendi) on X
They didn't practice on the reactor. They practiced on a copy of it. Ontario Power Generation (the state-owned company) built the world's first full-scale reactor mockup and had crews rehearse the fu...
010
SadServers @sadservers.com · 09/09/2026
SadServers scenario running in a browser sandbox using WebAssembly, thanks to vpod: vpod.sh/usecases/sad...
vpod.sh
sadservers × vpod
A SadServers troubleshooting scenario running live in a vpod sandbox — root shell, real Linux, no cloud.
001
SadServers @sadservers.com · 08/09/2026
New #kubernetes SadServers scenario "Kyoto" . Investigate and troubleshoot why a recent change is making an API fail, and fix it. sadservers.com/scenario/kyoto
k8s
021
SadServers @sadservers.com · 07/09/2026
Checklists are a great tool for DevOps/SRE professionals; a must-have for production procedures of any complexity. The book "The Checklist Manifesto" is a good read and the related podcast is just 1 hr tinyurl.com/rkhsenej
book cover
110
SadServers @sadservers.com · 05/09/2026
Docker healthchecks can be confusing, we explore them in this new SadServers scenario "Spa": The Docker Compose healthcheck that does not heal sadservers.com/scenario/spa
docker compose logo
111
SadServers @sadservers.com · 31/08/2026
New SadServers scenario "Hangzhou", a bit advanced NGINX QA configuration challenge sadservers.com/scenario/han...
000
SadServers @sadservers.com · 30/08/2026
New SadServers scenario "Nara": No ls on this system This is a fun challenge where we took out 'ls' and 'find', still there are at least 9 ways in this server to list and find a target file in a directory structure (and we learn something ourselves 😄) sadservers.com/scenario/nara
sadservers.com
SadServers: "Nara": No ls on this system - linux-other, bash Troubleshooting Lab
Linux Troubleshooting Labs Interview DevOps SRE
000
SadServers @sadservers.com · 26/08/2026
Not sure if to laugh or to cry; getting phishing attempts impersonating my own service 🫤
phishing attempt
020
SadServers @sadservers.com · 25/08/2026
We'd rather spend time adding tangible value to our users (new scenarios, new features) than doing maintenance. BUT it can be argued a security issue -even if unlikely - could affect users, so we upgrade to keep our bar of no dependencies with Critical or High vulnerabilities.
no critical or high vulns
010
SadServers @sadservers.com · 25/08/2026
It's been 35 years since Linus announced his Linux hobby project. It's a light one but I'm surprised not a lot of people have heard about his book talking about the early years. Same with Kernighan's book, which is a great read and anybody in the Unix/Linux world should read imho.
Kernighan and Torvalds memoirs
030
SadServers @sadservers.com · 24/08/2026
New pricing page. Allows for upgrades directly from the page while blocking things like a distracted user buying a second subscription (it has happened). Also ask and you shall receive: new individual "lifetime" purchase option for the super fans. sadservers.com/pricing
sadservers.com
SadServers: DevOps & SRE Labs | Job Interview Prep & Assessment
Linux Troubleshooting Labs Interview DevOps SRE
020
SadServers @sadservers.com · 24/08/2026
Have we invented a better command dependency tool (DAG) than "make"? it's from the 70s and I use it all the time (my cli is all 'git' and 'make' with the occasional 'docker')
020
SadServers @sadservers.com · 23/08/2026
"overnight success" 😀 SadServers has been updated every week in the last two years, either code or scenarios (weekends especially productive, one gap due to external emergency)
commit every week
140
SadServers @sadservers.com · 21/08/2026
When you take out playwright testing our of your Docker image 🤏
830->165MB
031
SadServers @sadservers.com · 18/08/2026
another case of retrying making things worse when there's a lot of traffic www.githubstatus.com/incidents/zk...
githubstatus.com
Incident with GitHub.com
GitHub's Status Page - Incident with GitHub.com.
010
SadServers @sadservers.com · 18/08/2026
Called this back in May. Copilot has different price tiers now with a high-cost one coming up.
cost tiers in copilot
011
SadServers @sadservers.com · 17/08/2026
😠 www.githubstatus.com guess we'll have to develop locally, like our ancestors did
github down
141
SadServers @sadservers.com · 12/08/2026
New #Linux practical SadServers challenge "Bergen": Port already in use (kind of giving a clue here) , how fast can you fix this issue? sadservers.com/scenario/ber...
010
SadServers @sadservers.com · 11/08/2026
Last SadServers newsletter went out list.sadservers.com/archive/3441... Sign up so you don't miss new scenarios and news list.sadservers.com/subscription...
list.sadservers.com
SadServers Newsletter: new challenges!
000
SadServers @sadservers.com · 11/08/2026
New Linux SadServers scenario "Vienna": The Vanishing Process, a systemd hands-on challenge sadservers.com/scenario/vie...
sadservers.com
SadServers: "Vienna": The Vanishing Process - systemd Troubleshooting Lab
Linux Troubleshooting Labs Interview DevOps SRE
000
SadServers @sadservers.com · 10/08/2026
Gitea is a self-hosted CI/CD platform. Check out SadServers latest practical challenge involving this tool: sadservers.com/labs/gitea/
gitea
000
SadServers @sadservers.com · 08/08/2026
Using Sentry for incident, logs and metrics. It's way cheaper than DataDog and the cost can be anticipated (you can cap it too).
010
SadServers @sadservers.com · 06/08/2026
What We Learned Running AWS Spot Instances in Production docs.sadservers.com/blog/what-we...
docs.sadservers.com
What We Learned Running AWS Spot Instances in Production
Most people think of AWS Spot Instances as cheap VMs that can disappear at any time. That’s true — but if Spot is a core part of your production infrastructure, there are a few less obvious details th...
040
SadServers @sadservers.com · 04/08/2026
If the AI support agent of a multibillion company like @stripe hallucinates a dashboard option and highlights in a statement the opposite of reality, imagine what the AI agents doing support on other sites are saying. (Unlike other sites at least with Stripe I can RTFM)
false statement
010
SadServers @sadservers.com · 01/08/2026
Classy and Canadian "not our fault but sorry anyways" tailscale.com/blog/hugging...
tailscale.com
Tailscale in the Hugging Face intrusion: The good news and the bad news
An AI agent used a stolen Tailscale auth key at Hugging Face. Workload identity federation, flow logs, and safer defaults could have reduced the risk.
072
SadServers @sadservers.com · 31/07/2026
Happy sysadmin day
000
SadServers @sadservers.com · 29/07/2026
New SadServers scenario (gunicorn + ninx + Flask) "Banha" : Intermittent 504 Gateway Time-Out errors sadservers.com/scenario/banha
sadservers.com
SadServers: "Banha" : Intermittent 504 Gateway Time-Out errors - gunicorn, nginx Troubleshooting Lab
Linux Troubleshooting Labs Interview DevOps SRE
011
SadServers @sadservers.com · 28/07/2026
New SadServers PostgreSQL challenge "Luxor" sadservers.com/scenario/luxor
postgresql
111
SadServers @sadservers.com · 27/07/2026
New SadServers scenario "Philadelphia": htop, btop and zombies 🧟‍♂️ sadservers.com/scenario/phi...
sadservers.com
SadServers: "Philadelphia": htop metrics disagree Troubleshooting Lab
Linux Troubleshooting Labs Interview DevOps SRE
032
SadServers @sadservers.com · 27/07/2026
AWS spot instances: - Within the same region, price can vary a lot among AZs (eg less than half). - An instance type that's more expensive on-demand may be cheaper as a spot instance. (eg t3 and t3a) - Spot capacity can suddenly disappear, need to prepare for it.
020
SadServers @sadservers.com · 22/07/2026
This brought so many memories, just released: The Java Story | The Official Documentary www.youtube.com/watch?v=ZqGS...
youtube.com
The Java Story | The Official Documentary
YouTube video by CultRepo
010
SadServers @sadservers.com · 17/07/2026
Almost had a heart attack but it's all good, some AI agent at AWS woke up with a sense of humor
aws-usage $2B
131
SadServers @sadservers.com · 07/07/2026
100 questions (and answers!) related to SadServers, did we miss anything? 😀 sadservers.com/faq
sadservers.com
FAQ | SadServers
Answers about SadServers scenarios, subscriptions, interviews, accounts, billing, and platform features.
000
SadServers @sadservers.com · 03/07/2026
Xin chào Việt Nam 🇻🇳
vietnam traffic
000
SadServers @sadservers.com · 27/06/2026
Latest SadServers newsletter is out: 7 new scenarios (including Ansible, Terraform and ELK stack ones) and new Lab section 💪 list.sadservers.com/archive/6009...
list.sadservers.com
New scenarios, New labs section 🚀
000