Sign in

Sam Willis

@samwillis.uk
1.1K followers 283 following 146 posts

Building sync and streams Databricks & Neon. Prev co-founder @electric.ax - acquired. Started PGlite.dev, co-creator of Tanstack DB @tanstack.com.

PostsRepliesMedia
Sam Willis @samwillis.uk · 19/08/2025
I've got my ticket! Who else is going to @syncconf.bsky.social?
020
Sam Willis @samwillis.uk · 15/08/2025
Dev tools coming soon to @tanstack.com DB! The best client side state libs have brilliant dev tools, this is even more important for DB as you store so much state in it. The DB dev tools will give you full visibility of all collections and live queries. 🚀🎉
180
Sam Willis @samwillis.uk · 23/07/2025
Perfect lunch stop with the family after a 3hr hike! 😊
1840013
Sam Willis @samwillis.uk · 14/07/2025
PGlite (WASM Postgres) now gets over 1 million downloads a week! 🤯🚀 Absolutely incredible to see how people have embraced what we have built. Awesome work by the team!
16311
Sam Willis @samwillis.uk · 04/07/2025
Home for the next few weeks 😁 Love the 🏔️ in the summer!
140
Sam Willis @samwillis.uk · 09/06/2025
Last month I spoke @pgconf.dev about @pglite.dev, how we built it and where we plan to take it 🎉 You can now watch my talk at the link below 🚀🐘
060
Sam Willis @samwillis.uk · 10/05/2025
Getting notified of AirPods/Tags traveling with you is smart - I like it. But it's absurd that you can't click a button that says "these are my wife's, they will often travel with me, I don't need to know this!" - or rather "ignore forever"
110
Sam Willis @samwillis.uk · 07/05/2025
One week till my talk @pgconf.dev! Very excited to be visiting Montreal 🇨🇦 Give me a shout if you want to meet up a chat @pglite.dev, @electric-sql.com, Sync Engines or Local-first app development.
0153
Sam Willis @samwillis.uk · 04/05/2025
Sunday mornings are for family reading-in-a-coffe-shop sessions 📚
0130
Sam Willis @samwillis.uk · 28/04/2025
Got to enjoy the British summer while it's actually happening... Moving my office to the garden for the week.
030
Sam Willis @samwillis.uk · 24/04/2025
PGlite is now based on Postgres 17 🎉🚀 v0.3, which we just released, is a big step forward for the project, and the foundation for a lot of exciting things coming soon!
0213
Sam Willis @samwillis.uk · 09/04/2025
Can your Chat AI app make todo lists? It was fun adding this feature to the demo & really shows how a sync engine ensures your UI is kept up to date with agent side effects. PS: that token stream is multi-user via Postgres! 🤯
030
Sam Willis @samwillis.uk · 04/03/2025
I'm really excited to be presenting PGlite and the work it's taken to bring Postgres to WASM and the browser at @pgconf.dev in May 🎉🐘
1161
Sam Willis @samwillis.uk · 03/03/2025
Incremental streaming joins scale proportional to the update size, not the total dataset size. With D2TS I've found it possible to do 10s thousands updates a second with almost no change in latency as the dataset size increases. 📈🚀
082
Sam Willis @samwillis.uk · 03/03/2025
Incremental view maintenance is seen as a black art. The tools available, such as differential dataflow, are complex, with a steep learning curve. I want to change that with D2TS! It has a simple pipeline api, with full type inference at every step 🚀
const graph = new D2({ initialFrontier: 0 });

const issuesInput = graph.newInput<[string, Issue]>();
const usersInput = graph.newInput<[string, User]>();
const commentsInput = graph.newInput<[string, Comment]>();

const commentCounts = commentsInput.pipe(
  map(([_key, comment]) => [comment.issue_id, 1]),
  concat(issuesInput.pipe(map(([_key, issue]) => [issue.id, 0]))),
  reduce((values) => {
    let count = 0;
    for (const [num, diff] of values) {
      count += num * diff;
    }
    return [[count, 1]];
  })
);

issuesInput.pipe(
  map(([_key, issue]) => [issue.user_id, issue]),
  join(usersInput.pipe(map(([_key, user]) => [user.id, user]))),
  join(commentCounts),
  map(([_key, [[issue, user], commentCount]]) => [
    issue.id,
    {
      ...issue,
      user_id: user.id,
      username: user.username,
      user_email: user.email,
      user_full_name: user.full_name,
      comment_count: commentCount || 0,
    },
  ]),
  consolidate(),
  output((msgs) => {
    console.log(msgs);
  })
);
0113
Sam Willis @samwillis.uk · 28/02/2025
In the above video I'm using D2TS (github.com/electric-sql...) to join three change streams coming from the Electric sync engine: Issues, Users, and Comments. For the comments, it's calculating the count per issue. This is all done incrementally!
const graph = new D2({ initialFrontier: 0 });

const issuesInput = graph.newInput<[string, Issue]>();
const usersInput = graph.newInput<[string, User]>();
const commentsInput = graph.newInput<[string, Comment]>();

const commentCounts = commentsInput.pipe(
  map(([_key, comment]) => [comment.issue_id, 1]),
  concat(issuesInput.pipe(map(([_key, issue]) => [issue.id, 0]))),
  reduce((values) => {
    let count = 0;
    for (const [num, diff] of values) {
      count += num * diff;
    }
    return [[count, 1]];
  })
);

issuesInput.pipe(
  map(([_key, issue]) => [issue.user_id, issue]),
  join(usersInput.pipe(map(([_key, user]) => [user.id, user]))),
  join(commentCounts),
  map(([_key, [[issue, user], commentCount]]) => [
    issue.id,
    {
      ...issue,
      user_id: user.id,
      username: user.username,
      user_email: user.email,
      user_full_name: user.full_name,
      comment_count: commentCount || 0,
    },
  ]),
  consolidate(),
  output((msgs) => {
    console.log(msgs);
  })
);
140
Sam Willis @samwillis.uk · 28/02/2025
Introducing D2TS - a TypeScript implementation of Differential Dataflow I've been building 🚀 Enabling incremental computation on change streams and streaming joins, both on the server and client. Perfect for use with the @electric-sql.com sync engine! 🧵
3479
Sam Willis @samwillis.uk · 24/02/2025
Light weight in-memory Postgres instances with PGlite are perfect for unit tests. We recently added a `.clone()` method to make it super easy for have a clean state for each test. Just setup the schema and fixtures once, then clone!
import { PGlite } from '@electric-sql/pglite'
import { beforeAll, beforeEach, describe, it } from 'vitest'
import { allMigrations } from './migrations'
import { loadTestData } from './testData'

describe('my tests', () => {
  let db: PGlite
  
  beforeAll(async () => {
    db = await PGlite.create()
    for (const migration of allMigrations) {
      await db.exec(migration)
    }
    await loadTestData(db)
  })
  
  beforeEach(async () => {
    // Clone the database for each test
    const dbClone = db.clone()
    await setDatabaseMock(dbClone)
  })

  it('foo', () => {
    // ...
  })

  it('bar', () => {
    // ...
  })
})
0317
Sam Willis @samwillis.uk · 19/02/2025
This video is a great dive into building a local first app & the journey @t3.gg took to choosing Dexie + MySQL. www.youtube.com/watch?v=3gVB... @electric-sql.com is perfect for syncing into Dexie from a Postgres backend 🐘 Being "protocol first" (http+json), you can use anything for the client store.
0173
Sam Willis @samwillis.uk · 13/02/2025
The reality of the pride versioning scheme...
081
Sam Willis @samwillis.uk · 10/02/2025
Our Postgres in WASM project, @pglite.dev, has hit 1/2 million downloads a week! 🤯 We (@electric-sql.com) have a lot planned for (native... 😉) PGlite this year, exciting times ahead!
1446
Sam Willis @samwillis.uk · 05/02/2025
Started making it.
Had a breakdown and wrote a database.
Bon appetite.
2551
Sam Willis @samwillis.uk · 02/02/2025
A few years ago I build a multiplayer Sudoku app after my wife and I found ourselves enjoying racing each other over a coffee in the mornings. Our Daughter now plays it with us 😁
0252
Sam Willis @samwillis.uk · 29/01/2025
The top two commands in my VSCode command pallet are usually format and sort lines. Always fun when you hit sort lines when aiming for format...
130
Sam Willis @samwillis.uk · 01/01/2025
My wife @keirawillis.bsky.social had an awesome 2024, she had two books published 📕📗🥳 Amazon UK have put a promo on the first in the series, for a limited time it's just 99p 🎉 If you like crime thrillers, the 80s, and ghosts... do check it out! www.amazon.co.uk/gp/aw/d/B0CS...
Screen shot of Amazon with Tib Street Ballroom on promo.

--

How do you solve an unsolvable murder? Ask the victim...

In January 1986, newly-engaged Marnie Driscoll is found dead in her parents' kitchen. With no witnesses, it seems as though the circumstances of her death will remain a mystery.

Six months later, high-flying Detective Inspector Andrew Joyce’s career takes an unexpected detour when he finds himself unwillingly transferred to an obscure department within Greater Manchester Police, known as the Ballroom.

The Ballroom team employs unorthodox methods to crack previously unsolved cases, and Joyce, a sceptic by nature, must find a way to work with Peggy Swan, a reclusive ex-socialite with a unique talent: she can communicate with the dead.

Joyce soon discovers that Marnie’s death, initially dismissed as an opportunistic act of violence, actually seems to be a carefully orchestrated murder.

It will take both Joyce’s skill as an investigator and Peggy’s connection to her new ghostly charge to navigate the web of secrets surrounding the case and bring closure to Marnie’s tragic story before the killer can strike again.
050
Sam Willis @samwillis.uk · 28/12/2024
Nested CSS syntax landed in all browsers just over a year ago, but it doesn't work in the style attribute... However, with just 474 bytes of JS you can do this!
<script>
// In your <head>
(t=>{var r,i,a,d,l="",s={},u=document,b=1,n=e=>{(i=e.getAttribute?.(t))&&(d=s[i
]??=b,a=t+d,e.removeAttribute(e[t]),e.setAttribute(a,""),e[t]=a,d==b)&&(
l+=`[${a}]{${i}}`,b++)};new MutationObserver(e=>{e.map(e=>{
"childList"==e.type?e.addedNodes.forEach(e=>{n(e),[...e.querySelectorAll?.(
`[${t}]`)||[]].map(n)}):n(e.target)}),l&&((r=u.createElement("style")
).innerHTML=l,u.head.appendChild(r),l="")}).observe(u,{attributeFilter:[t],
childList:!0,subtree:!0})})("x-style");
</script>

<a
  href="/something-awsome"
  x-style="
    color: red;
    font-size: 30px;
    &:hover {
      background-color: yellow;
    }
    & > span {
      color: green;
      &::after {
        display: inline;
        content: '!';
      }
    }
  "
>
  Click here for <span>Something Awsome</span>
</a>
1201
Sam Willis @samwillis.uk · 16/12/2024
Sync extends reactive one-way dataflow all the way to your main database. Achieving this *with your existing API* isn't hard with a sync engine. Use your API, with all that custom business logic, for writes. Then just swap out the fetch for sync!
060
Sam Willis @samwillis.uk · 11/12/2024
Realtime replication from a single Postgres server to *one million* clients? With consistent latency and flat memory usage? The @electric-sql.com sync engine can do that 🤯 This is it doing 960 writes/minute!
0130
Sam Willis @samwillis.uk · 10/12/2024
Exciting day, @electric-sql.com is now in Beta 🚀 Electric makes realtime replication between Postgres & *any* client side store possible! (And we've scaled it to 1 million connections 🤯) Here it is syncing 100k rows to a @pglite.dev app:
6296
Sam Willis @samwillis.uk · 01/11/2024
While technically correct I expected it to do better than this...
Prompt: Who was the first object store to offer S3 compatibility and when?
ChatGPT 4o: Amazon Simple Storage Service (S3), introduced by Amazon Web Services (AWS) in March 2006, was the first object storage service to offer S3 compatibility.   This service established the S3 API as a de facto standard for object storage. Subsequently, other storage solutions adopted S3 compatibility to integrate seamlessly with applications designed for Amazon S3. For instance, MinIO, launched in March 2016, is a high-performance object storage system that is API compatible with Amazon S3.  
140
Sam Willis @samwillis.uk · 31/10/2024
Family Halloween Lego 👻🦇🧙🏻🧙🏼🧙🏻‍♀️🪦🧟‍♂️
050
Sam Willis @samwillis.uk · 25/10/2024
With PGlite + ElectricSQL sync you can build an issue tracker handling 100k issues + 500k comments, synced from Postgres to PGlite in the browser! Super fast UI updates with all writes going though the local database, and Postgres FTS in PGlite for search 🔥
1244