stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 07/04/2026@jauntywk.bsky.social you posted back in the day about SlateDB making a very fine streaming system: I wanted to notify you - it is already WIP - github.com/opendata-oss... 120
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 24/03/2026finance.yahoo.com/markets/stoc... wow that's wildfinance.yahoo.comOpenAI Guarantees 17.5% Minimum Returns to Private Market Investors: ReutersOpenAI is sweetening the pot in its battle to win private equity co-investments, guaranteeing 17.5% returns and early access to cutting-edge models 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 23/01/2026I sat down with Denis Magda to talk about Postgres. He literally just published a book called “Just Use Postgres”. In this conversation, that turned out to be super fun, we talked about everything from what MySQL got wrong to the recent PG dev explosion Watch here👇 www.youtube.com/watch?v=LxQy...youtube.comjust use postgres - the podcast (/w Denis Magda)YouTube video by 2 Minute Streaming 020
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 01/01/2026Stay tuned on our YouTube Premiere tomorrow: ✅ www.youtube.com/watch?v=pJQ7... PS: Happy New Year! 💥🎉youtube.comtansu - pluggable stateless kafka (written in rust)YouTube video by 2 Minute Streaming 020
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 01/01/2026- low-level performance optimizations including in-depth flame graphs analysis - the need for proxies in Kafka - Kafka's java bias - a lot more I know it may sound like "yet another Kafka", but it's the one I'm genuinely most excited about. Trust me there's good reason why 🙌 200
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 01/01/2026The chat is full of nerd-snipes, where we talk about basically everything, including: - the characteristics and value prop of Tansu - our combined decade and a half experiences with Kafka - schemas and today's complexity around them - SQL + Streaming - Rust for building low-level systems 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 01/01/2026You can tell he's incredibly passionate about programming and Kafka from the super-engaging conversation we had. If you’d like to hear a no-bs techincal engineering conversation - watch me tomorrow as I interview him for 3 hours and 30 minutes straight. 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 01/01/2026It's the story of a one-man-show - Peter Morgan - a super-productive solo Rust developer with the desire to build a world-class ergonomic Kafka broker UX that's optimized for developers. 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 01/01/2026Have you heard about Tansu? Tomorrow, I will post the longest-ever podcast about it. It’s an open-source (Apache), stateless, leaderless, single-binary Kafka broker supporting pluggable storage backends (S3, PostgreSQL, SQLite) with built-in schema registry and support for open-table formats 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 27/12/2025Arguing by quoting the raw numbers of posts taken down is very naive because in social media there's always a very small tail of high-performing posts. 000
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Data retention in disk/memory is decoupled from whether the message was consumed or not. Unlike certain message queues, this ensures availability is unaffected (can’t run out of disk/memory) from how consumers behave 000
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025A leader/follower replication model does not require strong distributed consensus between too many nodes to serve traffic. 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Partitions act as a sharding mechanism. A single partition usually does not push more than a few MB/s. 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025A Kafka cluster scales by horizontally adding more nodes (called brokers). 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025The append-only nature of the log (i.e no updates/deletes) means that reads do not require a quorum to serve fresh data. There are also no locks between concurrent reads or writes, hence no contention. 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Kafka heavily leverages the OS file system cache (page cache) to serve reads fast from memory 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Physical disk writes are heavily batched on the server-side. The fact that Kafka writes linear blocks to disk and fsync is not used allows the OS to batch up data in memory (page cache), coalescing the data into larger IO operations 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025- since the synchronous write path is literally writing to memory. Producers can also be configured to not block on this replication consensus when writing (acks=1 vs acks=all). 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Writes by default require consensus that every follower replica has acknowledged the write before returning, but Kafka works with fsync off by default and therefore writes to disk asynchronously, without blocking on the physical disk write. This provides near memory-like performance when well-tuned- 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025The broker does not touch the data, its a dumb pipe - messages aren’t even validated for their schema, hence compressed data isn’t decompressed on the server side 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Compression is usually also done on the client, and these larger batches compress much more favorably. This leads to larger network packets, larger sequential disk operations, contiguous memory blocks 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Writes are batched on the client-side. Producers batch multiple messages into one "record-batch", hence increasing throughput per request. 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025The system supports two main very simple and performant operations: - Produce, which appends to the end of the log - Consume, which reads sequentially starting from any particular offset 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 18/11/2025Why Kafka performs so well 👇 🧵 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 11/11/2025The Android feels choppy/flow and the touch also doesn't feel super responsive 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 11/11/2025github.com/tansu-io/tansu This is something to keep an eye on. Should be doable to add within a pg extension and bundle Kafka into Postgres? 000
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 11/11/2025Daylight is amazing. I'm really looking forward to a v2 (whenever it comes) that's a bit more refined 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 11/11/2025I mean, yeah - it's not a technical piece, it's a financial article. When I hear the Kafka community, I think about engineering talk more than anything else 000
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 10/11/2025With greater competition and commercialization, it seems bound to happen. I'm of the opinion we're bound to see consolidation in the space soon, because there's too many companies chasing too little of a market: bigdata.2minutestreaming.com/p/event-stre... 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025Then I don't see why a relational database can't maintain a pending queue too. Just process the lowest-id job 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025What are you referring to? My intuition is rather that discussion has somewhat died down, in general. 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025the data bible dropping soon 051
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025re: fairness - I admit I'm not as familiar with. Can you teach me how some of these systems ensure fairness? Do they refuse to give messages to performant workers and hold them off for slower ones? 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025There is some fundamental misunderstanding here. pgmq is not based on top of SQS. It only provides API parity with it. No messages actually go to SQS... Latency-wise, my tests showed single-digit write and read. 99% use cases don't need less. 200
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025I really don't know. I haven't evaluated it yet. 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025ok, so what is missing? > move data fairly, fast, and reliably It is fast. It is reliable. It is not fair (the simple implementation). Is fairness the missing piece? (Just want to streamline the discussion) 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025100%. I'm replying in facts. I simply perceived your intro/outro belittling the Postgres argument to clickbait to reach front-page of HN as such, hence fought back. 000
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025The problem with building that is that... it exists! github.com/tansu-io/tansugithub.comGitHub - tansu-io/tansu: Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta LakeApache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake - tansu-io/tansu 151
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025Pub-sub doesn't exist (much). So yeah, unless somebody builds the `pgmq` equivalent and makes it work nice enough, it would be extra work that probably isn't justified. It strongly depends on how much of that "complicated system" you truly need though! 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 08/11/2025I disagree with your take that it's "complicated". You can write a queue/pub-sub in an hour. We have to separate the two, though: - queues exists in battle-tested libraries like pgmq and the older pgq So this is already built. Unless you want complex routing a-la Rabbit, its fine to use Postgres! 100
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 03/11/2025I think the blog makes a lot of straw mans and misses the point. I definitely agree with some points made there and remain reasonable - but I think that under my mentioned constraints, PG is likely the right choice I posted a rebuttal here if you care enough to read github.com/gunnarmorlin...github.com"You Don't Need Kafka, Just Use Postgres" Considered Harmful - Gunnar Morling · gunnarmorling discussions.morling.dev · Discussion #336"You Don't Need Kafka, Just Use Postgres" Considered Harmful - Gunnar Morling Looking to make it to the front page of HackerNews? Then writing a post arguing that "Postgres is enough", or why "you ... 321
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 03/11/2025Well... that would pretty easily be alleviated by just running a separate instance, right? I am not advocating for co-locating it. In any case, I think one would need to expose some dumbed down API for pub-sub users to avoid them doing more than just using the pub-sub, at least in the apps. 210
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 03/11/2025What scale are we talking about here? 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 02/11/2025Doesn't sound like a bad idea at all. When you've got close to a billion users, might as well connect them! 120
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 02/11/2025Just use Postgres until it breaks 🧘♂️ 010
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 02/11/2025Thanks, it's mine. It comes off this post topicpartition.io/blog/postgre... where I focus on how PG solves so many problems at smaller scale 110
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 26/10/2025What has changed since? 010
stanislavkozlovski.bsky.social @stanislavkozlovski.bsky.social · 26/10/2025This was posted 15 years ago (part 1/2) 100
