Sander Dieleman @sedielem.bsky.social · 25/06/2025The frequencies are log-spaced, so historically, 10k was plenty to ensure that all positions can be uniquely distinguished. Nowadays of course sequences can be quite a bit longer. 100
Sander Dieleman @sedielem.bsky.social · 25/06/2025Note also that getting this number slightly wrong isn't that big a deal. Even if you make it 100k instead of 10k, it's not going to change the granularity of the high frequencies that much because of the logarithmic frequency spacing. 100
Mario @mnslarcher.bsky.social · 25/06/2025Thanks @sedielem.bsky.social! I remember this reason for PE. What I’m unsure about is whether it still applies when encoding a timestep as 0-1. In the Flux code it’s first mapped to 0–1000, so I guess your point about needing enough granularity holds. 1/2 100
