iiinova — where your digital self keeps evolving

Horace He

@chhillee.bsky.social

832 followers 60 following 13 posts

@PyTorch "My learning style is Horace twitter threads" - @typedfemale

Posts Replies Media

Reposted by Horace He

Yaron Minsky @yminsky.bsky.social · 11/12/2024

@chhillee.bsky.social's talk at Jane Street is now up! youtu.be/139UPjoq7Kw?...

youtu.be

Building Machine Learning Systems for a Trillion Trillion Floating Point Operations

YouTube video by Jane Street

0317

Reposted by Horace He

Mike Smith @mjjsmith.com · 02/12/2024

Getting different attention masks working for AstroPT (a proto-foundation model for astronomy github.com/Smith42/astr...), so much nicer to do it with Flex Attention vs custom CUDA kernels -- thank you for releasing it to the world 🫡

github.com

GitHub - Smith42/astroPT: Transformer for galaxy images (and general astronomy)

Transformer for galaxy images (and general astronomy) - Smith42/astroPT

041

Horace He @chhillee.bsky.social · 01/12/2024

I judge social networks by how many FlexAttention users I can find on each one, and by that metric, Bluesky is doing pretty good!

1501

Horace He @chhillee.bsky.social · 25/11/2024

If you'd like to influence what features the PyTorch distributed team work on in torchtitan (e.g. MoE, multimodal, context parallelism, etc.), go made your voices heard here!

github.com

Vote on new features! · pytorch torchtitan · Discussion #693

Hi torchtitanists, Thank you for your interests in torchtitan! Please upvote on what features you would like to see next, and add one if it's not already there. We'll try to prioritize on the most ...

0111

Horace He @chhillee.bsky.social · 23/11/2024

First thought: Seems kinda "FlexAttention-y": bsky.app/profile/sungkim.bsky.socia… Second thought: oh cool, they're already using FlexAttention! it's a nice usage of the `or_masks` and `and_masks` API - I think they do (causal & sliding_window) | (register_mask)

090