Sign in

nirik :fedora: :redhat:

@nirik.fosstodon.org.ap.brid.gy
12 followers 1 following 605 posts

Sysadmin shouting at clouds. #fedora #redhat [bridged from fosstodon.org/@nirik on the fediverse by fed.brid.gy ]

PostsRepliesMedia
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 8h
This timeline keeps having things where I think "am I missing something? in what way does this make sense at all" Example: datacenters in space. Yes, lets go to all the gigantic expense of sending hardware to orbit where the env is very unforgiving and where you can never service it.
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 9h
Well, I seem to have gotten way behind on shaing my daily stuff. Was a busy week, but to recap tuesday -> friday in my #fedora hypercube: * All the usual meetings, day to day stuff, etc. * Helped with updating/rebooting things early in the week * Outage thursday where we updated/rebooted the […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 29/09/2026
* Wrote up pr for www.fedoraproject.org handling ( forge.fedoraproject.org/infra/ticke… ) * Cleaned up some spam on the wiki ( forge.fedoraproject.org/infra/ticke… )
forge.fedoraproject.org
www.fedoraproject.org returns 403 on its own css when Referer is www, site renders unstyled
### Description of request every request to www.fedoraproject.org returns 403 when the Referer is https://www.fedoraproject.org/ (any path), pages included. the browser sends that when the page loads its own css, so the www site shows up as raw html, and clicking any internal link gets a 403 pag...
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 29/09/2026
Today in my #fedora pyramid: * Synced database servers again all around. Cleaned up backup crons until switchover. * Fedora Daily meeting - 2026-09-28 noon * Looked into a distrepo failure on a builder. Missing nfs mount. * Created swift-sig packaging group ( […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 28/09/2026
Just got what looks like a real email from amazon telling me: "Check if a message is really from Amazon. Just ask Alexa for Shopping to confirm before you click, call back or reply. Learn more." "Got a message claiming to be from Amazon? Before you click, call back, or reply — you can verify […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 26/09/2026
Another saturday, time for another recap of interresting #fedora infra things last week. This week: a fun interrelated issue with ansible and logs, rhel10 database server migration progress and an outage coming up next week […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 26/09/2026
Friday in the #fedora tree: * Restarted some signing stuff to get things flowing again. * Looked at why no compose report for rawhide, it was a connection problem to forge. * Started syncing database server data to rhel10 versions. * Investigated our opengraph problems. Fixed bodhi at least, and […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 26/09/2026
Thursday in my #fedora mandelbrot set: * Fedora Infrastructure weekly meeting - 2026-09-24 8am * Set a system git setting on rawhide-composer to try and fix a compose issue ( forge.fedoraproject.org/releng/comp… ) * Added some group sponsors to a group ( […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 24/09/2026
Yesterday in the #fedora spirograph: * generated updated list of provenpackagers without otp (just 4 left) * Poddlers bugzilla token not updated to refreshed one, redeployed it. * Looked at odd epel9 build issue with a perl package * Moved anubis to log at WARN reduce logs * Created pr to make […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 24/09/2026
And tuesday in my #fedora discus: * Many meetings * Worked with Greg to debug check-diff/log01 disk: was an ansible bug ( github.com/ansible/ansible/issues/8… ). Kudos for Greg finding that! * Installed db-datanommer01 in prod (rhel10) and rsynced data to it. * Looked at weird epel9 […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 24/09/2026
Monday in my #fedora rombus: * Finally migrated db-datanommer in staging to rhel10/postgres18/timescaledb2.30.0 * Cleaned up old transitory db-datanommer02/03 that were used for migration. * Did some reading up on restic/borg options for pull backups. * Looked into slow check-diff and massive […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 19/09/2026
Another saturday, another chance to clog #fedora planet with my ramblings. :) This week, a lot of description about upgrading our timescaledb database server in staging, a note about an upcoming outage and of course, scrapers […]
fosstodon.org
Original post on fosstodon.org
001
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 19/09/2026
Got behind posting here on my daily #fedora stuff. Wed/Thu/Fri: * F45 beta out, so infra freeze ended. Flurry of merging and deploying things that were blocked. * I did a bunch of playing around with how to upgrade our datanommer database servers. (Look for that in the long form blog post coming […]
fosstodon.org
Original post on fosstodon.org
000
Reposted by nirik :fedora: :redhat:
Alexander Bokovoy @abbra.mastodon.social.ap.brid.gy · 19/09/2026
Upgraded my static blog engine from Jekyll to Astro: vda.li/en/posts/2026/09/19/Blog-ref…. Used this as an opportunity to upgrade design and usability to myself as well, thanks to Claude. Hopefully, new editing capabilities will help to blog more often instead of fighting with […]
mastodon.social
Original post on mastodon.social
001
Reposted by nirik :fedora: :redhat:
Berkubernetus @fuzzychef.m6n.io.ap.brid.gy · 19/09/2026
For today's #Caturday, I bring you: Neighborhood Watch. #CatsOfMastodon
back view of a large black cat, sitting up and staring out the window around a curtain.  He's keeping track of what all the neighbors are up to.  He does not approve.
021
Reposted by nirik :fedora: :redhat:
Fedora Project @fedora.fosstodon.org.ap.brid.gy · 19/09/2026
What are your weekend plans? Maybe they will include messing around with beta testing? Check out the Fedora Linux 45 Beta for all of our editions and spins, including Fedora Asahi Remix! Beta announcement: fedoramagazine.org/announcing-fedor… Fedora Asahi Remix […]
fosstodon.org
Original post on fosstodon.org
2206
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 16/09/2026
Today in my #fedora heptagon: * Fedora 45 beta is out! Get it now! fedoraproject.org * Meetings and more meetings * More poking at db-datanommer01.stg upgrades. Managed to figure out how to upgrade it from rhel9/pgsql13/timescaledb-2.10.3 to rhel10/postgresql16/timescaledb-2.13.1 Still […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 16/09/2026
Monday in my #fedora complex polygon: * Checking on our openshift cluster subscriptions * Fedora Release engineering meeting - 2026-09-14 8am * Spam cleanup on devel list ( forge.fedoraproject.org/infra/ticke… ) * Grew os-control01 disk to 400G and fixed backups of it. ( […]
fosstodon.org
Original post on fosstodon.org
000
Reposted by nirik :fedora: :redhat:
warthog9 @warthog9.social.afront.org.ap.brid.gy · 14/09/2026
Ohhhh, hey look, after a much longer hiatus than intended, Gauss is back up and running and generating pretty graphs Fun fact: Fedora is by far the busiest distro in the FCIX mirror fleet right now. #FCIX #MicroMirror
Grafana dashboard view looking at the download statistics for the past 6hours or so from the FCIX mirroring fleet.
011
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 12/09/2026
A pretty short weekly recap of my #fedora doings this week. Including some talk about database upgrades and a note that Fedora 45 Beta is coming out next tuesday. www.scrye.com/blogs/nirik/posts/202…
scrye.com
misc fedora bits: second week of sep 2026
Another saturday, time for another recap of interesting things in longer form. ## RHEL10 database migrations I managed to migrate all but one database server in staging. No particular problems with those and they are all on postgresql 18 now with data page checksums enabled (well, except the single mariadb instance, and it's on 10.11.18). Hopefully I can get the production ones done in the next outage we have (probibly the week before final freeze). The last staging database server is more interesting: db-datanommer01.stg This is using timescaledb on top of postgres and is currently the rhel9 default version: 13. After looking around, I think this one is doable, but needs another step: * sync a data backup off the existing server * upgrade it to postgresql 16 * enable data page checksums * install a new rhel10 vm * sync the data to it * upgrade to 18 * install the correct timescaledb extension * take down the old instance and swap in the new I think this will all work, just an extra 13->16 jump. I will try it out in staging next week. I'm also a bit worried about how long it will take to generate all those checksums. The datanommer db's are really large. However, if we can do it I would love to as it's a very nice feature. Doing staging should at least give us an idea. ## Fedora 45 Beta is go for release next tuesday ( 2026-09-15 ) Looks like we are go for fedora 45 beta next week. I hope everyone will test it out and help us iron out any problems before the final release. As always, comment on the fediverse:
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 12/09/2026
Yesterday in my #fedora point: * Applied fix to rawhide ostree repo ( forge.fedoraproject.org/releng/tick… ) * Looking at what is using up space now on log01, was old 2025 logs * Deleted those logs (~400GB) * Fired another rawhide to make sure ostree fix worked (it seems […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 11/09/2026
Been using Sable for my matrix client lately ( github.com/SableClient/Sable ) It's available as a flatpak and android apk. It's a fork on cinny with a ton of things on top. Things I really like: * bookmarks! (mark a thing and you can find it later with one click) * multi account […]
fosstodon.org
Original post on fosstodon.org
100
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 11/09/2026
Today in the #fedora wedge: * Meetings and 1x1's (more than usual for a thursday) * F45 beta go/nogo: go for tuesday! * Found new replacement aarch64 machine and submitted pr to set it back up. * Last stable updates push for beta * Looked at broken signing token, figured out a way to revive it! […]
fosstodon.org
Original post on fosstodon.org
001
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 10/09/2026
Today in the #fedora sphere: * Our webhook2fedmsg app was tracebacking, not sure the cause/fix. Filed a ticket on it for others to look at. ( forge.fedoraproject.org/infra/ticke… ) * Migrated a bunch of stg db servers to rhel10: db-fas01.stg, db03.stg, db01.stg. * Setup a […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 09/09/2026
Monday was a holiday (thanks unions!) but I managed to setup a small solar panel to keep the battery charged in the car thats mostly parked. ☀️ Tuesday in my #fedora klein bottle: * Fixed ssh limits for our monitoring hopefully. * added 2 large packages to the heavybuilder channel * Fixed our […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 06/09/2026
Excellent, a real life test. Just had a ~25 second power outage here. The only reason I noticed it at all is that I have #homeassistant turning on a red light when the grid is down. (And I realized I needed a automation to turn it back off when the grid is back... forgot to setup that, oops) […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 05/09/2026
A september recap of my #fedora doings in longer form. This time including: Database migrations, fun with laptop battery replacement, Fedora 45 beta rc's (please test!) and note about authentication issues. www.scrye.com/blogs/nirik/posts/202…
scrye.com
misc fedora bits: first week of sep 2026
This week seemed to have a lot of small irq's all around. That said, I did make some progress on a few things: ## RHEL10 migrations: databases Next up on the migration train: databases. It turns out we already had one database server in staging moved to rhel10, but it was using the default postgresql 16 and since I am going to the trouble to migrate things to a new os, might as well move them to newer postgresql too. postgresql 18 makes data page checksums default. You can disable them at startup if you really want to, but they seem like a really good idea to enable. So, I took db-koji01.stg down and added checksums and it took around 45minutes. Not super great, but not nearly as bad as it could be. Then, the upgrade to 18 was super fast and painless. I did some work in our ansible repo to set up rhel10 hosts with postgresql18 to start with and created a db-fas02.stg to migrate that to. The rest of these servers will follow basically this pattern: * Setup a rhel10/postgresql18 02 vm for each server * Sync data from rhel9 one. * Stop rhel9 one (OUTAGE) * Sync data again from rhel9 one. * Enable data page checksums * Migrate from 16 to 18 * Profit I hope to get the staging ones all done next week and find any issues, then we can look at scheduling an outage after Beta freeze to do all the production ones. ## Laptop Battery replacement My lenovo slim7x battery was getting pretty pathetic. It was 2 years old and it's max full capacity was something like 35% of design full and sometimes it was now refusing to charge without a reboot. So, I ordered a new battery and installed it this last week. This laptop is anoying to work on because, while there are screws on the bottom panel, it's also held on by clips. So you have to pry it off and it sounds like you are breaking it while doing so. Anoying. Also on this laptop they routed all the cables around the battery. The battery has cable guides all around it. So, to replace the battery you have to move all those cables carefully and slide the old battery out and new one in, along with unplugging the bluetooth and wifi antennas, and disconnecting/connecting the battery connector. I did manage to do it, but it took more time that I thought it would. While I had the machine open I swapped the windows drive I had back in and updated the firmware (which there's no way to do from linux). It was quite a pain. windows took quite a while to notice that it was not 2024 and that it should check what _current_ updates were available. Did manage it in the end. I don't know what effect the newer firmware has, everything still works the same as far as I can tell. ## Fedora 45 beta rc's We have started making rc's for beta. That sure reads weird, but if you are able, please do test and file bugs. ## Authentication issues There's one weird auth issue I am aware of. Thats where services with keytabs are sometimes getting permission denied. It seems to be somehow some differing config on two of our ipa servers. Thats being tracked in https://forge.fedoraproject.org/infra/tickets/issues/13539 I am not aware of any other issues remaining. So, if you hit something, please do file a ticket and include enough information for us to try and fix it. ie, time/date, exactly what you were trying to login to, exactly what error message you got, etc. As always, comment on the fediverse:
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 05/09/2026
Weekend #homeassistant project: Picked up a Eureka J15 Ultra robot vac, going to install a brain parasite in it ( ie, valetudo). Wish me luck...
200
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 04/09/2026
Yesterday in my #fedora circle: - Cleaned up fedora-comps for sorting, missing packages - Looked a bunch at network connections/usage/bandwith related to a slow download report ( forge.fedoraproject.org/infra/ticke… ) - Fedora infra daily meeting - 2026-09-03 noon - […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 03/09/2026
This is amusing to me. My new battery in my laptop: energy-full: 72.03 Wh energy-full-design: 70 Wh
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 03/09/2026
Yesterday in my #fedora zone: * Meetings. Tuesday is meeting day. Some good ones tho! * refreshed a bunch of bugzilla api tokens * Updated/processed a number of tickets Today: * 1x1 with Fabian - 2026-09-02 8am * Grew prod test volume for new forge distgit testing ( […]
fosstodon.org
Original post on fosstodon.org
100
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 01/09/2026
Today in my ##fedora romboid: * Finally got the stuck F44 Flatpak updates-testing compose to go thru (thanks mattia for the patch!) * Some misc tickets and pr commenting/merging * Offline for a few hours to get fiber connected (see last toot). Pretty spiffy 1g symetrical... about 20x faster than […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 31/08/2026
How it started... how it's going,
A speedtest widget showing 69Mbps download and 15 Mbps upload 6.87 ms ping 0.27 jitter, and grade D for bufferbloat. Another speedtest widget showing 944 Mbs download and 941 Mbps upload with a 1.91ms ping, 0.32 jitter and grade A for bufferbloat.
100
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 29/08/2026
Just a quick saturday #fedora doings recap today, including: the last of the rhel10 migrations tracked, authentication woes over (perhaps?), and Fedora 45 beta infrastructure freeze. www.scrye.com/blogs/nirik/posts/202…
scrye.com
misc fedora bits: last week of aug 2026
Time for another saturday weekly recap in longer form. ## RHEL10 migrations Bunch more progress of various machines over the last week or two, and we are down to: * torrent01 (didn't manage to get this before beta freeze, will after) * mailman/lists ( https://forge.fedoraproject.org/infra/tickets/issues/13519 ) * database servers ( https://forge.fedoraproject.org/infra/tickets/issues/13517 ) * rabbitmq servers ( https://forge.fedoraproject.org/infra/tickets/issues/13383 ) Nice to finish this off soon. ## Authentication woes... over? I am hopefull that we have solved the last of our auth issues late this week. There's a lot of moving parts in our authentication stack: ipa servers on the backend, noggin on the frontend (accounts.fedoraproject.org), ipsilon for identity provider (id.fedoraproject.org). All of these have had various issues recently, but we have worked through them and I did some tuning of ipsilon on thursday that seems to have really helped. If you are still seeing any issue, please add exactly what you were trying to login to/do and time/date to our tracking ticket. ( https://forge.fedoraproject.org/infra/tickets/issues/13482 ) ## Fedora 45 Beta infrastructure freeze We are in beta freeze now, so hopefully that will keep things a bit quieter and we can catch up on work in staging and docs and other like things. As always, comment on the fediverse:
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 28/08/2026
yesterday in the #fedora flywheel: * Adjusted a bunch of things on ipslion (our idp) sssd, some ipa tuning settings and such to try and fix the last of the auth issues. I sure hope things are stable now, but time will tell. forge.fedoraproject.org/infra/ticke… * Meetings […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 28/08/2026
RE: fosstodon.org/@adamw/11717004824956… ditto!
fosstodon.org
000
Reposted by nirik :fedora: :redhat:
Lesley Carhart :unverified: @hacks4pancakes.infosec.exchange.ap.brid.gy · 27/08/2026
My boss just put it out there that I just flew across the planet, spoke at a stadium, organised a rager and closed out a bar, pushed code at 3am, and then woke up to give a 7a news interview before teaching a class.
216
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 26/08/2026
Today in my #fedora rombus: * Tons of meetings, it's meeting day. :) * Fixed various small pagure.io leftovers in our ansible repo/deployments. * Restarted chronyd on all machines when I realized they stopped talking to one of our time servers last week. ;( * Looked at and tweaked sssd.conf on […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 25/08/2026
Yesterday in #fedora infraland: * koji hubs got mobbed by a DDoS. Rebooted them, restarted postgresql, got all builders back checking in * Fixed logserver to not need awstats (it used 1 pl script from it). * swapped in the rhel10 logserver and retired the rhel9 one * Restarted sssd on […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 22/08/2026
Another saturday recap of the past week in #fedora land: This week: Mass updates/reboots/reinstalls done, pagure.io is now read-only, fedora 45 beta freeze coming up next tuesday and finally some laptop discussion […]
fosstodon.org
Original post on fosstodon.org
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 22/08/2026
Thursday and today in my #fedora square: * Helped out some with pagure.io read-only instance setup. * mass update/reboot/reinstall outage. A few problems, but overall a success. * Updated our openshift clusters * Cleaned up space on fedorapeople * Updated my work laptop a bunch.
000
Reposted by nirik :fedora: :redhat:
Eric The IT Guy @itguyeric.mastodon.social.ap.brid.gy · 21/08/2026
Rather not be posting this: I was let go from my professor role and the job search isn’t going well. Driving deliveries to keep things steady, about 40 days of runway left. Looking for TPMM, DevRel, or project work. Intros welcome, reposts help. #OpenToWork #Linux #OpenSource
132
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 19/08/2026
Catching up a bit on my recent #fedora truncated cone: * Monday I was off. Had a lovely time at the coast watching the sea lions bark and a great dinner. * yesterday was meetings day. 7:30am to about 2pm. * I also managed to update/reboot our entire staging infra and then update/reboot a bunch […]
fosstodon.org
Original post on fosstodon.org
010
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 17/08/2026
Hurray. Back yard faucett fixed. Water flowing to the 🍓 s and 🍎 trees and zuchini.
010
Reposted by nirik :fedora: :redhat:
Tom Gauld @tomgauld.bsky.social · 17/08/2026
My latest books cartoon for @theguardian.com
Title:
Who is Haunting The Library of Forbidden Tomes? 

Image: five panels showing characters in a gothic library 

22% - tormented souls, frozen between life and death (a ghost reading a book)
4% - dank, fetid horrors from the icy depths below (a black slug like creature with a single eyestalk reading a book)
15% - cold, lifeless bones possessed by a corrupt sorcery (a skeleton reading a book)
11% - frigid and malevolent ectoplasmic vapours (a shapeless apparition reading a book)
48% - regular readers happy to away from the heatwave
(A mother, small child and toddler in buggy, all reading books)
212156730
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 17/08/2026
Yesterday my droplet let me know (via #homeassistant ) that there was a 'unusual water event' happening. Nothing in the house... walk outside. yep. The outside faucet in the back yard decided to start spewing water. :( (35 gal before I shut it off) On the plus side it's a woodford model 19 and […]
fosstodon.org
Original post on fosstodon.org
100
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 15/08/2026
huh, one of these things doesn't seem like the others:
Legend for a map with "Amtrak Cascades Train" , "Seattle-Los Angles Train", "Bus" and... "Canada"
000
nirik :fedora: :redhat: @nirik.fosstodon.org.ap.brid.gy · 15/08/2026
Saturday long form recap of infra doings in #fedora This week includes: Fedora 45 branching from rawhide, download servers using BBR, rhel10 migrations, outage next week, beta freeze coming up and of course... scrapers […]
fosstodon.org
Original post on fosstodon.org
000