iiinova — where your digital self keeps evolving

Yuan Tang @terrytangyuan.xyz · 27/08/2026

The project was created as a part of the inference benchmarking and metrics standardization effort in the Kubernetes Serving Working Group, and seeks to standardize tooling and metrics for measuring inference performance across the Kubernetes and model server communities.

100 1

Yuan Tang @terrytangyuan.xyz · 27/08/2026

Excited to share that our accompanying paper for the Inference Perf project (under Kubernetes SIGs) has been published in the Journal of Open Source Software! Paper: joss.theoj.org/papers/10.21... Repo: github.com/kubernetes-s...

joss.theoj.org

Inference Perf: A Benchmarking Tool for GenAI Inference

Chandrasekar et al., (2026). Inference Perf: A Benchmarking Tool for GenAI Inference. Journal of Open Source Software, 11(124), 10314, https://doi.org/10.21105/joss.10314

130 3

Yuan Tang @terrytangyuan.xyz · 24/08/2026

👉 Check out the August newsletter here: inferenceops.substack.com/p/state-o… 👉 Subscribe to get future issues in your inbox: inferenceops.substack.com 🚀 Thanks to everyone who subscribed so far!

000 0

Yuan Tang @terrytangyuan.xyz · 24/08/2026

Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem that our teams at Red Hat AI are contributing to, including updates from projects like vLLM, KServe, @llm-d.ai, @kubernetes.io, and beyond.

200 2

Yuan Tang @terrytangyuan.xyz · 24/08/2026

📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗔𝘂𝗴𝘂𝘀𝘁 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁!

110 1

Yuan Tang @terrytangyuan.xyz · 21/08/2026

I'm thrilled to share that I've been invited to become Omnigent's first external maintainer! It's remarkable that the project is only two months old and I feel incredibly fortunate to contribute at such an early stage and collaborate with the Databricks team and broader community.

140 4

Yuan Tang @terrytangyuan.xyz · 12/08/2026

I’m speaking at #KubeCon + #CloudNativeCon North America, November 9-12 in Salt Lake City, Utah! Check out our session to learn more about the work we are doing at the CNCF TAG Workloads Foundation and meet TAG leads in person! events.linuxfoundation.org/kubecon-clou...

050 5

Yuan Tang @terrytangyuan.xyz · 19/07/2026

My talk focused on something I believe doesn't get discussed enough: maintaining successful open source projects isn't just about writing code. It's about building healthy communities, making sustainable decisions, and creating projects that can thrive long after the initial excitement.

140 4

Yuan Tang @terrytangyuan.xyz · 19/07/2026

Today he's a maintainer of LightGBM and has become a respected contributor across the open source ecosystem. Seeing that journey come full circle was one of my favorite moments of the conference.

130 3

Yuan Tang @terrytangyuan.xyz · 19/07/2026

Another highlight was reconnecting with my former colleague, @jameslamb.bsky.social. About ten years ago, I introduced James to the world of open source and helped him make his first contribution. It was incredibly rewarding to catch up and see how far he's come since then.

110 1

Yuan Tang @terrytangyuan.xyz · 19/07/2026

What made it especially memorable was that my session was immediately followed by @hadley.nz’s talk on lessons from his 20 years of experience - in the very same room. Glad that we get to catch up after ten years!

110 1

Yuan Tang @terrytangyuan.xyz · 19/07/2026

I had the opportunity to give my talk, “Building for the Road Ahead: Transferable Lessons from the Front Lines of Open Source Maintenance”, where I shared lessons I've learned from over 10 years of maintaining and leading open source projects.

100 1

Yuan Tang @terrytangyuan.xyz · 19/07/2026

Attending @scipyconf.bsky.social this year was a refreshing experience in more ways than one. #SciPy2026 Thanks to organizers, speakers, volunteers, and everyone I had the chance to meet. Looking forward to the next one! Photos with @hadley.nz @jameslamb.bsky.social @tracykteal.bsky.social

160 6

Reposted by Yuan Tang

SciPy Conference 2026 @scipyconf.bsky.social · 09/07/2026

🌱 Talk Spotlight: @terrytangyuan.xyz shares what years on the open source front lines have taught him about sustainable maintenance. Hard-won lessons for the whole community. #SciPy2026 🔗 scipy2026.scipy.org

031 3

Yuan Tang @terrytangyuan.xyz · 09/07/2026

Excited to be speaking at #SciPy2026 in Minneapolis next week! If you're attending, I'd love to connect. Come say hi after the talk or find me during the conference. Let's chat about open source, AI infrastructure, or whatever's on your mind. Full schedule: www.scipy2026.scipy.org/schedule

12190 19

Yuan Tang @terrytangyuan.xyz · 06/07/2026

📖 Part 3 - Blueprints & Troubleshooting: developers.redhat.com/articles/2026… Which of these blueprints is closest to your current setup? What would you change?

001 1

Yuan Tang @terrytangyuan.xyz · 06/07/2026

📖 Part 1 - Core Concepts & Scaling Dimensions: developers.redhat.com/articles/2026…

100 1

Yuan Tang @terrytangyuan.xyz · 06/07/2026

We also cover the troubleshooting recipes that come up most in production: TPOT spikes from KV fragmentation, MoE hot-expert imbalance, KV transfer bottlenecks in disaggregated fleets, and speculative decoding drift.

110 1

Yuan Tang @terrytangyuan.xyz · 06/07/2026

- Long-context RAG - Batch pipelines - Model-as-a-Service AI grids - Sovereign-to-cloud burst - Edge on a single workstation GPU

110 1

Yuan Tang @terrytangyuan.xyz · 06/07/2026

That's the core of Part 3 of our 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗔𝗜 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 series, written with Fatih Nar, Yuchen Fama, and Greg Pereira: six deployment blueprints, each matched to a real traffic shape: - Chat & copilots

110 1

Yuan Tang @terrytangyuan.xyz · 06/07/2026

Reach for multi-tenant AI grids when they're serving one model to one team. The rule we keep coming back to: add one mechanism at a time, only when measurement shows the simpler setup has run out.

110 1

Yuan Tang @terrytangyuan.xyz · 06/07/2026

Most teams over-engineer their inference stack from day one. They disaggregate before measuring. Add speculative decoding before concurrency stabilizes.

491 9

Yuan Tang @terrytangyuan.xyz · 30/06/2026

Part 3 covering deployment blueprints and troubleshooting recipes is coming soon - follow along to catch it. Read Part 2: developers.redhat.com/articles/2026…

021 2

Yuan Tang @terrytangyuan.xyz · 30/06/2026

Round-robin leaves cache hits on the table. Co-authored with Fatih E. Nar, Yuchen Fama, and Greg Pereira.

100 1

Yuan Tang @terrytangyuan.xyz · 30/06/2026

But watch out - acceptance rates collapse under constrained decoding (JSON mode, tool calls), so measure before enabling on tool-calling traffic. One insight that keeps coming up: cache-aware routing via @llm-d.ai is what turns disaggregation from a checkbox into a working system.

100 1

Yuan Tang @terrytangyuan.xyz · 30/06/2026

they're not the same thing), and when FP8/FP4 quantization pays off. - 𝗦𝗽𝗲𝗰𝘂𝗹𝗮𝘁𝗶𝘃𝗲 𝗗𝗲𝗰𝗼𝗱𝗶𝗻𝗴: EAGLE 3.1 now extends gains into long-context regimes with 2x longer acceptance length than EAGLE-3.

100 1

Yuan Tang @terrytangyuan.xyz · 30/06/2026

We share how to measure whether the prefill-to-decode imbalance in your traffic justifies the split, with 25-40% cost reductions on chat and RAG workloads in our benchmarks. - 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿e: Tiering across HBM, DRAM, and NVMe with LMCache, the difference between prefix sharing and KV reuse (

100 1

Yuan Tang @terrytangyuan.xyz · 30/06/2026

Part 2 dives into the three optimization levers that deliver most of the cost and latency improvements once your parallelism layout is set: - 𝗣/𝗗 𝗗𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻: Not a feature to toggle on - it's a deployment topology.

100 1

Yuan Tang @terrytangyuan.xyz · 30/06/2026

Part 2 of our 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗔𝗜 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 series is now live on Red Hat Developer: 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘈𝘥𝘷𝘢𝘯𝘤𝘦𝘥 𝘋𝘦𝘱𝘭𝘰𝘺𝘮𝘦𝘯𝘵 𝘗𝘢𝘵𝘵𝘦𝘳𝘯𝘴. In Part 1, we covered prefill/decode phases and the 5D parallelism framework.

492 9

Yuan Tang @terrytangyuan.xyz · 26/06/2026

Follow along if you're interested in the infrastructure behind serving large models at scale. Read it here: developers.redhat.com/articles/2026…

000 0

Yuan Tang @terrytangyuan.xyz · 26/06/2026

becoming unavoidable as models push past 200K+ token contexts - Practical configuration trade-offs across different hardware budgets This is Part 1 of a three-part series. We'll share the remaining parts in future posts.

100 1

Yuan Tang @terrytangyuan.xyz · 26/06/2026

In this post, my co-authors and I break down: - Why prefill and decode need to be understood as fundamentally different workloads - The 5D parallelism framework (tensor, pipeline, expert, data, and context parallelism) that governs how models are distributed across GPUs - How context parallelism is

100 1

Yuan Tang @terrytangyuan.xyz · 26/06/2026

The decode phase is memory-bandwidth-bound, generating tokens one at a time. Batching strategies that optimize one phase degrade the other and and that tension shapes every architecture decision downstream.

100 1

Yuan Tang @terrytangyuan.xyz · 26/06/2026

Excited to share Part 1 of our blog series on Red Hat Developer: 𝘋𝘦𝘴𝘪𝘨𝘯𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘊𝘰𝘳𝘦 𝘊𝘰𝘯𝘤𝘦𝘱𝘵𝘴 𝘢𝘯𝘥 𝘚𝘤𝘢𝘭𝘪𝘯𝘨 𝘋𝘪𝘮𝘦𝘯𝘴𝘪𝘰𝘯𝘴. LLM inference is two workloads pretending to be one. The prefill phase is compute-bound, processing entire prompts in parallel to populate the KV cache.

262 6

Yuan Tang @terrytangyuan.xyz · 24/06/2026

Eitan Geiger, Francisco Arceo, Pete Cheslock, Jooho Lee, Pierangelo Di Pilato, Ran Pollak, Nir Rozenbaum, Yuan Tang, Wentao Ye

000 0

Yuan Tang @terrytangyuan.xyz · 24/06/2026

👉 Check out the June newsletter here: inferenceops.substack.com/p/state-o… 👉 Subscribe to get future issues in your inbox: inferenceops.substack.com 🚀 Thanks to everyone who subscribed so far! Kudos to all contributors to this edition!

101 1

Yuan Tang @terrytangyuan.xyz · 24/06/2026

The response has been incredible - we’ve gained over 𝟭𝟱𝟬𝟬 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲𝗿𝘀! Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from projects like vLLM, KServe, @llm-d.ai, @kubernetes.io, and beyond.

100 1

Yuan Tang @terrytangyuan.xyz · 24/06/2026

📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗝𝘂𝗻𝗲 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! We recently launched our newsletter publicly after sharing it internally at Red Hat AI for over a year.

221 2

Yuan Tang @terrytangyuan.xyz · 19/06/2026

I’ve had a great experience using Omnigent and have been collaborating with the Databricks team to land a few features and fixes, including: - Kubernetes & OpenShift deployment - NVIDIA OpenShell sandbox integration - Claude on Vertex AI auto-detection - Podman support

210 2

Yuan Tang @terrytangyuan.xyz · 12/06/2026

We will be hosting Cloud Native AI + Inference Day again as a co-located event with KubeCon North America! @cncf.io Looking forward to your proposals (by June 21) and sponsorship (by September 21)! events.linuxfoundation.org/kubecon-clou...

events.linuxfoundation.org

Cloud Native AI + Inference Day | LF Events

As we step into the era of rapid AI advancements, organizations are grappling with an unprecedented array of challenges. The rise of Large Language Models (LLMs), the development of Graph RAGs…

140 4

Yuan Tang @terrytangyuan.xyz · 31/05/2026

Thank you to everyone who has been part of the journey. I look forward to continuing to learn, connect, and support the community.

010 1

Yuan Tang @terrytangyuan.xyz · 31/05/2026

Many of the opportunities I've had, including open source leadership roles, conference speaking engagements, collaborations, and even career moves, started with a simple message or conversation there.

110 1