iiinova — where your digital self keeps evolving

ahmet alp balkan @ahmet.dev · 08/02/2026

@ericchiang.bsky.social hey question about your synctest blog: do you really need that "+ time.Millisecond" or is there any determinism that the goroutine will get to run 3 times without it? oblique.security/blog/go-sync...

110

ahmet alp balkan @ahmet.dev · 10/12/2025

@aphyr.woof.group.ap.brid.gy you thought LLM misquoting you was bad, no longer need to wait until LLM writes a jepsen article. sw.vtom.net/hn35/pages/9... (from sw.vtom.net/hn35/news.html)

sw.vtom.net

Jepsen: NATS 4.2

150

ahmet alp balkan @ahmet.dev · 03/12/2025

Turbopuffer (vector DB) architecture is a fairly interesting example of object-storage backed tiered DBs combined with data structure practices from DBs. turbopuffer.com/docs/archite... Here are the tradeoffs that makes it work: 567-labs.github.io/systematical...

050

Reposted by ahmet alp balkan

Murat (Distributolog) @muratdemirbas.bsky.social · 25/11/2025

[new blog post] Mitigating Application Resource Overload with Targeted Task Cancellation muratbuffalo.blogspot.com/2025/11/miti...

011

ahmet alp balkan @ahmet.dev · 25/11/2025

Full house from the KubeCon talk in Atlanta (500 people, holy sh*t)

0250

ahmet alp balkan @ahmet.dev · 09/11/2025

#kubecon: roughly 14% flights into atlanta cancelled today, even more predicted for sunday/monday. www.flightaware.com/live/cancell...

081

ahmet alp balkan @ahmet.dev · 09/11/2025

#kubecon talk finally coming together see you all there Thursday 11:45am or in the hallway kccncna2025.sched.com/event/27Fdd/...

2120

Reposted by ahmet alp balkan

squillace @squillace.bsky.social · 03/11/2025

I can't wait for the MOST cool, relaxed part of geeking out at @cncf.bsky.social #kubecon, which is actually hitting @rejekts.io!!! cloud-native.rejekts.io Join us if you're around #atlanta or at KubeCon!

cloud-native.rejekts.io

Cloud Native Rejekts

1124

Reposted by ahmet alp balkan

Heba Elayoty @helayoty.org · 06/10/2025

Most people think the Kubernetes scheduler makes scheduling decisions itself. Well, it doesn’t. The real power lies elsewhere. 🧵 Let’s talk about how the kube-scheduler is actually a plugin-first orchestration engine, not a decision-maker.

1124

ahmet alp balkan @ahmet.dev · 26/09/2025

KubeCon goers: what are the Atlanta hotels you're booking that's not next to the boring downtown/convention center area?

110

ahmet alp balkan @ahmet.dev · 05/08/2025

Anyone have recommended readings on any scalable k8s multi-cluster work API implementations? Planning to roll out a similar model on our own and I want to study the state of the art and the rationale behind the existing model.

230

ahmet alp balkan @ahmet.dev · 01/08/2025

#KubeCon schedule is out! Team LinkedIn will give 3 talks this year (myself included): 1️⃣ Making Application Rollouts Observable (lnkd.in/gmnM-8Z4) 2️⃣ How We Upgrade the Whole ML Training Fleet Bi-weekly(lnkd.in/gG_a_Etc) 3️⃣ Evicted! All the Ways Kubernetes Kills Your Pods (lnkd.in/ggm9HKZF)

0101

ahmet alp balkan @ahmet.dev · 30/07/2025

Our team at LinkedIn will give 3 talks at the #KubeCon 2025! Looks like we'll be meeting in Atlanta again. 👋🏼 Stay tuned.

0100

ahmet alp balkan @ahmet.dev · 29/07/2025

Does anyone know why all cloud-native open source ecosystem uses pull model to expose their /metrics (instead of pushing the metrics)? Pull model is really poor when the metric cardinality exposed on a single endpoint increases.

820

Reposted by ahmet alp balkan

madhavjivrajani.bsky.social @madhavjivrajani.bsky.social · 09/07/2025

We’ve been systematically evaluating the scalability of our fleet’s control planes, and the recent work on storage layer scalability in the Kubernetes community is going to prove pivotal as we scale further! Check out what we’ve been upto for the past couple of weeks ♥️

031

ahmet alp balkan @ahmet.dev · 09/07/2025

As we test Kubernetes clusters beyond the supported 5,000+ nodes limit at LinkedIn, we're finding out 'List API' calls are a bottleneck in cluster reliability. So I'm publishing a deep dive about why this is a problem and what you can do about them. ahmet.im/blog/kuberne...

1257

ahmet alp balkan @ahmet.dev · 23/06/2025

Open sourcing a new kubectl plugin called `kubectl node-resource` that queries allocations/utilizations for all/subset of the Kubernetes nodes. It has optimizations for large clusters. We've needed this in our set up to quickly answer questions. github.com/ahmetb/kubec...

14312

ahmet alp balkan @ahmet.dev · 23/06/2025

Glad to have @madhavjivrajani.bsky.social join us at LinkedIn compute team this summer. I’m constantly impressed by the depth of his kube-apiserver expertise.

2131

ahmet alp balkan @ahmet.dev · 29/05/2025

DAYS SINCE ETCD BACKUPS SAVED OUR ASS IN PRODUCTION: 0 If you manage your Kubernetes clusters and think you won’t lose your etcd, you’re almost certainly wrong. It takes years for the disaster to happen and you’ll be glad you have invested in building a backup/restore system.

1133

ahmet alp balkan @ahmet.dev · 11/05/2025

@kelseyhightower.com hi Kelsey, what was your toto model again? can’t find after the twitter nuke ☹️

120

Reposted by ahmet alp balkan

Ronak Nathani @ronaknathani.com · 16/04/2025

@ahmet.dev and I shared our approach at #KubeConEU25 couple weeks ago with ~500 attendees (though the credit belongs to the entire team behind the platform). Great to see it resonate! Watch here: youtu.be/dDkXFuy45EA?...

youtu.be

From Metal To Apps: LinkedIn’s Kubernetes-based Compute Platform - Ahmet Alp Balkan & Ronak Nathani

YouTube video by CNCF [Cloud Native Computing Foundation]

021

ahmet alp balkan @ahmet.dev · 04/04/2025

Finally delivered our talk at #KubeCon to 500 attendees! Slides are up at kccnceu2025.sched.com/event/1txGQ/ and recording will be coming soon. If you wanna chat, I'm at the conference today.

0160

ahmet alp balkan @ahmet.dev · 31/03/2025

Here we go again, this time in London. #kubecon maintainer summit

0150

ahmet alp balkan @ahmet.dev · 27/03/2025

🇬🇧 Looking forward to seeing y'all at our #KubeCon talk about how we're building a bare-metal compute platform at LinkedIn to run thousands of services on high-6-digit number of machines! Schedule: sched.co/1txGQ

1202

Reposted by ahmet alp balkan

Abdel Sghiouar @boredabdel.bsky.social · 27/03/2025

Our discussion for the @kubernetespodcast.com with @ahmet.dev and @ronaknathani.bsky.social of is finally live 🥳 I had a great time talking to these awesome people about how to leverage #k8s inside LinkedIn and what they learned along the way 🙌 Take a listen 👇 #KubernetesPodcast #k8s

022

ahmet alp balkan @ahmet.dev · 27/03/2025

Looking forward to seeing y'all at #KubeCon London next week! (Who's going?)

260

ahmet alp balkan @ahmet.dev · 27/02/2025

New article on my blog that explains all the various ways Kubernetes can evict your workloads. This wasn't all documented in one place so I'm hoping this helps the teams operating with Kubernetes in production. You'll also find a cheatsheet in the article. ahmet.im/blog/kuberne...

1369

Reposted by ahmet alp balkan

Abdel Sghiouar @boredabdel.bsky.social · 15/02/2025

Selam Türkiye 🇹🇷 Our Google /Dev/Cloud event is coming to Istanbul on March 13 🥳. We will have a full day of technical content and hands-on workshops about Google Cloud, GenAI, Cloud Runtimes, and more. Organized by the Cloud DevRel Team for developers 👨‍🏫 👩‍🏫 ⌨️ .

cloud.google.com

Google /dev/cloud day Istanbul

Looking to boost your AI and cloud skills? Join us for Google /dev/cloud day Istanbul! Learn, connect, and get hands-on experience. Register today.

153

ahmet alp balkan @ahmet.dev · 08/02/2025

I might _attempt_ to write a blog about all the different things that causes Kubernetes evicts pods. There are so many, and every single one works differently.

2311

ahmet alp balkan @ahmet.dev · 30/01/2025

What are some of the footguns you observe people keep adding to systems software? My pet peeves are: - environment variables for critical configs - default values for critical configs - assuming a value for a missing config in the code - soft-failing, trying to be smart around errors

8130

ahmet alp balkan @ahmet.dev · 23/01/2025

Come work with me! We operate several hundred thousand machines that make up our compute fleet, and we orchestrate stateless/stateful/batch/GPU workloads on our Kubernetes infrastructure. (Seattle/Bay Area only, happy to refer if interested!) www.linkedin.com/jobs/view/sr...

linkedin.com

LinkedIn hiring Sr. Staff Software Engineer, Systems Infrastructure (Kubernetes) in Mountain View, CA | LinkedIn

Posted 4:37:49 PM. Company Description LinkedIn is the world’s largest professional network, built to create economic…See this and similar jobs on LinkedIn.

1133

ahmet alp balkan @ahmet.dev · 22/01/2025

Published a blog about common mistakes beginners make developing Kubernetes controllers (especially using controller-runtime) and how to write scalable and production-grade controllers. ahmet.im/blog/control...

37118

ahmet alp balkan @ahmet.dev · 18/01/2025

📝 Drafting an article about Kubernetes controller development mistakes and anti-patterns we see beginners make based on what we learned over the past couple of years. I hope to publish on Tuesday. Anything you wanna add?

5232

ahmet alp balkan @ahmet.dev · 18/01/2025

Xiaohongshu is an amazing app. I never used TikTok, but this app isn’t about dance videos to begin with. There’s actually a lot of useful content here, and you get to converse with Chinese users with auto-translated comments. Fantastic to peek behind the propaganda about PRC.

140

ahmet alp balkan @ahmet.dev · 16/01/2025

Should Kubernetes controllers+CRDs managing resources in the external world (non-k8s APIs) try to fix drifts (such as out of band modifications made in the external API) periodically –or assume they're the only actor, and wait for the next modification to reconfigure? This makes scaling harder.

5140

ahmet alp balkan @ahmet.dev · 14/01/2025

Our KubeCon EU talk is accepted! It was a rejection for the North America earlier, but I'm glad we got in this time. Lots of exciting stuff to share soon.

1340

Reposted by ahmet alp balkan

ahmet alp balkan @ahmet.dev · 08/01/2025

Kubernetes community: We’re looking for a new teammate to work on low-level Kubernetes infrastructure primitives at LinkedIn. If you’re interested in high-scale bare-metal compute, please reach out to me. (The role is Bay Area & Seattle only.) www.linkedin.com/jobs/view/41...

linkedin.com

LinkedIn hiring Sr. Staff Software Engineer, Systems Infrastructure (Kubernetes) in Mountain View, CA | LinkedIn

Posted 4:37:49 PM. Company Description LinkedIn is the world’s largest professional network, built to create economic…See this and similar jobs on LinkedIn.

0188

ahmet alp balkan @ahmet.dev · 08/01/2025

linkedin.com

LinkedIn hiring Sr. Staff Software Engineer, Systems Infrastructure (Kubernetes) in Mountain View, CA | LinkedIn

Posted 4:37:49 PM. Company Description LinkedIn is the world’s largest professional network, built to create economic…See this and similar jobs on LinkedIn.

0188

ahmet alp balkan @ahmet.dev · 19/12/2024

Published my reading between the lines of OpenAI Kubernetes postmortem: ahmet.im/blog/openai-... Many lessons to learn here, IMO it comes down to: * unintentional coupling of data plane/control plane * not being able to shield k8s API from load * inherent risk of asynchronous service announcements

ahmet.im

Notes on OpenAI Kubernetes outage

Last week, OpenAI has suffered a several hours long outage and published a detailed postmortem about it. Highly recommend reading it. These technical reports are usually a gold mine for all large-scal...

2304

ahmet alp balkan @ahmet.dev · 12/12/2024

Something early career software engineers struggle with is the code quality of the tests. The tests ought to be readable and maintainable like your main code; not an unmaintainable copy pasta mess. Invest in tools to make your tests concise.

3100

ahmet alp balkan @ahmet.dev · 01/12/2024

Is “disable retweets” a feature here, yet? I follow a lot of people for their opinions and to stay up to date with their work; not what they repost. 😞

481

ahmet alp balkan @ahmet.dev · 01/12/2024

TIL the Pacific salmon doesn’t have enough energy to swim back to the ocean after spawning. One of the local parks smells like rotten flesh. 🥴

130

ahmet alp balkan @ahmet.dev · 26/11/2024

Still surprised nobody has done a “find my Twitter follow list on Bluesky” tool. How are you converting Twitter followers to this platform, or are you building a new following from scratch?

360

ahmet alp balkan @ahmet.dev · 25/11/2024

Submitted a talk to #KubeCon EU today! 🤞 🤞

1350

Reposted by ahmet alp balkan

ahmet alp balkan @ahmet.dev · 15/11/2024

go.bsky.app/MHYb3B9

0132

ahmet alp balkan @ahmet.dev · 22/11/2024

The most impressive technical feat I’ve seen at #KubeCon this year by far was the Gödel scheduler by ByteDance. A completely different scheduler design with sharding and optimistic concurrency achieving 2,000 pods/sec. Blog: www.cncf.io/blog/2024/04... Talk: youtu.be/PiUAdclow_4?...

cncf.io

Gödel Scheduler open-sourced: a unified scheduler for online and offline workloads

Member post by ByteDance Since its open-source release in 2014, Kubernetes has rapidly become the de facto standard for container orchestration. The infrastructure team at ByteDance adopted Kubernetes...

05111

Reposted by ahmet alp balkan

Justin Garrison @justingarrison.com · 19/11/2024

Kubernetes isn't the only orchestrator that exists. Here's some features I wish it had youtu.be/9N9IOpyl3v8

youtu.be

What Kubernetes should learn from other Orchestrators

YouTube video by Justin Garrison

9629

ahmet alp balkan @ahmet.dev · 18/11/2024

Newest addition to my home office set up is the BenQ Halo light with both desk and back lighting. This will make the dark days of Seattle fall bearable.

3160