Andi @andimara.bsky.social · 21/05/2025Read the blog: huggingface.co/blog/nanovlmhuggingface.conanoVLM: The simplest repository to train your VLM in pure PyTorchWe’re on a journey to advance and democratize artificial intelligence through open source and open science. 010 0
Andi @andimara.bsky.social · 21/05/2025Train your Vision-Language Model in just two commands: > git clone github.com/huggingface/... > python train.pygithub.comGitHub - huggingface/nanoVLM: The simplest, fastest repository for training/finetuning small-sized VLMs.The simplest, fastest repository for training/finetuning small-sized VLMs. - huggingface/nanoVLM 110 0
Andi @andimara.bsky.social · 21/05/2025New Blog📖✨: nanoVLM: The simplest way to train your own Vision-Language Model in pure PyTorch explained step-by-step! Easy to read, even easier to use. Train your first VLM today! 130 0
Andi @andimara.bsky.social · 14/05/2025Link: webml-community-smolvlm-realtime-webgpu.static.hf.space/index.htmlwebml-community-smolvlm-realtime-webgpu.static.hf.spaceCamera Interaction App 110 0
Andi @andimara.bsky.social · 14/05/2025Real-time SmolVLM in a web browser with transformers.js. All running locally with no installs. Just open the website. 131 0
Andi @andimara.bsky.social · 08/04/2025If you’re into efficient multimodal models, you’ll love this one. Check out the paper: huggingface.co/papers/2504....huggingface.coPaper page - SmolVLM: Redefining small and efficient multimodal modelsJoin the discussion on this paper page 010 0
Andi @andimara.bsky.social · 08/04/2025📱 Real-world Efficiency: We've created an app using SmolVLM on an iPhone 15 and got real-time inference directly from its camera! 🌐 Browser-based Inference? Yep! We get lightning-fast inference speeds of 40-80 tokens per second directly in a web browser. No tricks, just compact, efficient models! 130 0
Andi @andimara.bsky.social · 08/04/2025🌟 State-of-the-Art Performance, SmolVLM comes in three powerful yet compact sizes—256M, 500M, and 2.2B parameters—each setting new SOTA benchmarks for their hardware constraints in image and video understanding. 100 0
Andi @andimara.bsky.social · 08/04/2025✨ Less CoT, more efficiency: Turns out, too much Chain-of-Thought (CoT) data actually hurts performance in small models. They dumb ✨ Longer videos, better results: Increasing video length during training enhanced performance on both video and image tasks. 120 0
Andi @andimara.bsky.social · 08/04/2025✨ System prompts and special tokens are key: Introducing system prompts and dedicated media intro/outro tokens significantly boosted our compact VLM’s performance—especially for video tasks. 100 0
Andi @andimara.bsky.social · 08/04/2025✨ Pixel shuffling magic: Aggressively pixel shuffling helped our compact VLMs "see" better, same performance with sequences 16x shorter! ✨ Learned positional tokens FTW: For compact models, learned positional tokens significantly outperform raw text tokens, enhancing efficiency and accuracy. 120 0
Andi @andimara.bsky.social · 08/04/2025✨ Smaller is smarter with SigLIP: Surprise! Smaller LLMs didn't benefit from the usual large SigLIP (400M). Instead, we use the 80M base SigLIP that performs equally well at just 20% of the original size! 110 0
Andi @andimara.bsky.social · 08/04/2025Here are the coolest insights from our experiments: ✨ Longer context = Big wins: Increasing the context length from 2K to 16K gave our tiny VLMs a 60% performance boost! 110 0
Andi @andimara.bsky.social · 08/04/2025Today, we share the tech report for SmolVLM: Redefining small and efficient multimodal models. 🔥 Explaining how to create a tiny 256M VLM that uses less than 1GB of RAM and outperforms our 80B models from 18 months ago! huggingface.co/papers/2504....huggingface.coPaper page - SmolVLM: Redefining small and efficient multimodal modelsJoin the discussion on this paper page 2218 0
Andi @andimara.bsky.social · 17/03/2025SmolDocling is available today 🏗️ 🔗 Model: huggingface.co/ds4sd/SmolDo... 📖 Paper: huggingface.co/papers/2503.... 🤗 Space: huggingface.co/spaces/ds4sd... Try it and let us know what you think! 💬huggingface.cods4sd/SmolDocling-256M-preview · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science. 060 0
Andi @andimara.bsky.social · 17/03/2025At only 256M parameters, SmolDocling outperforms much larger models on key document conversion tasks: 🖋️ Full-page transcription: Beats models 27× bigger! 📑 Equations: Matches or beats leading models like GOT 💻 Code recognition: We introduce the first benchmark for code OCR 110 0
Andi @andimara.bsky.social · 17/03/2025What makes it unique? 📌 Handles everything a document has: tables, charts, code, equations, lists, and more 📌 Works beyond scientific papers—supports business docs, patents, and forms 📌 It runs with less than 1GB of RAM, so running at large batch sizes is super cheap! 110 0
Andi @andimara.bsky.social · 17/03/2025How does SmolDocling beat models 27× bigger? SmolDocling transforms any document into structured metadata with DocTags, being SOTA in: ✅ Full-page conversion ✅ Layout identification ✅ Equations, tables, charts, plots, code OCR 110 0
Andi @andimara.bsky.social · 17/03/2025🚀 We just dropped SmolDocling: a 256M open-source vision LM for complete document OCR! 📄✨ Lightning fast, process a page in 0.35 sec on consumer GPU using < 500MB VRAM ⚡ SOTA in document conversion, beating every competing model we tested (including models 27x more params) 🤯 But how? 🧶⬇️ 33310 0
Andi @andimara.bsky.social · 05/03/2025Extremely bullish on @CohereForAI's Aya Vision (8B & 32B) - new SOTA open-weight VLMs - 8B wins up to 81% of the time in its class, better than Gemini Flash - 32B beats Llama 3.2 90B! - Integrated on @hf.co from Day 0! Check out their blog! huggingface.co/blog/aya-vis...huggingface.coA Deepdive into Aya Vision: Advancing the Frontier of Multilingual MultimodalityWe’re on a journey to advance and democratize artificial intelligence through open source and open science. 010 0
Andi @andimara.bsky.social · 31/01/2025And that was why we didn't release this before. It's live research code. Most gets rewritten fairly often, and some parts have been the same for years. It works, it manages to produce SOTA results at 256M and 80B sizes, but it's not production code. Go check it out: github.com/huggingface/...github.comsmollm/vision at main · huggingface/smollmEverything about the SmolLM2 and SmolVLM family of models - huggingface/smollm 131 0
Andi @andimara.bsky.social · 31/01/2025And it also has a bunch of bugs like this one in our modeling_vllama3.py file. We start from a pretrained LLM, but for some reason the weights of the head are not loaded from the model. I still don't know why :( 230 0
Andi @andimara.bsky.social · 31/01/2025The codebase is full of interesting insights like this one in our dataset.py file: How do you get reproducible randomness in different processes across different machines? Start different random number generators based on a tuple (seed, rank)! 230 0
Andi @andimara.bsky.social · 31/01/2025Post training, you can run the evaluation on all of these tasks by running: sbatch vision/experiments/evaluation/vloom/async_evals_tr_346/run_evals_0_shots_val_2048 . slurm 130 0
Andi @andimara.bsky.social · 31/01/2025Launching the training for SmolVLM 256M is as simple as: ./vision/experiments/pretraining/vloom/tr_341_smolvlm_025b_1st_stage/01_launch . sh Then we use wandb to track the losses. Check out the file to find out details! 120 0
Andi @andimara.bsky.social · 31/01/2025Fuck it, today we're open-sourcing the codebase used to train SmolVLM from scratch on 256 H100s 🔥 Inspired by our team's effort to open-source DeepSeek's R1, we are releasing the training and evaluation code on top of the weights 🫡 Now you can train any SmolVLM—or create your own custom VLMs! 2255 0
Andi @andimara.bsky.social · 23/01/2025Links :D Demo: huggingface.co/spaces/Huggi... Models: huggingface.co/collections/... Blog: huggingface.co/blog/smolervlmhuggingface.coSmolVLM - a Hugging Face Space by HuggingFaceTBDiscover amazing ML apps made by the community 040 0
Andi @andimara.bsky.social · 23/01/2025SmolVLM upgrades: • New vision encoder: Smaller but higher res. • Improved data mixtures: better OCR and doc understanding. • Higher pixels/token: 4096 vs. 1820 = more efficient. • Smart tokenization: Faster training and better performance. 🚀 Better, faster, smarter. 110 0
Andi @andimara.bsky.social · 23/01/2025We have partnered with IBM 's Docling to build amazing smol models for document understanding. Our early results are amazing. Stay tuned for future releases! 120 0
Andi @andimara.bsky.social · 23/01/2025Our models are integrated into ColiPali, delivering SOTA retrieval speeds with performance rivaling models 10x their size. 🏃♂️💨 SmolVLM makes it faster and cheaper to build searchable databases. Real-world impact, unlocked. 110 0
Andi @andimara.bsky.social · 23/01/2025Smol but mighty: • 256M delivers 80% of the performance of our 2.2B model. • 500M hits 90%. Both beat our SOTA 80B model from 17 months ago! 🎉 Efficiency 🤝 Performance Explore the collection here: huggingface.co/collections/... Blog: huggingface.co/blog/smolervlm 1162 0
Andi @andimara.bsky.social · 23/01/2025Introducing the smollest VLMs yet! 🤏 SmolVLM (256M & 500M) runs on <1GB GPU memory. Fine-tune it on your laptop and run it on your toaster. 🚀 Even the 256M model outperforms our Idefics 80B (Aug '23). How small can we go? 👀 1487 0
Reposted by AndiLewis Tunstall @lewtun.bsky.social · 16/12/2024We outperform Llama 70B with Llama 3B on hard math by scaling test-time compute 🔥 How? By combining step-wise reward models with tree search algorithms :) We're open sourcing the full recipe and sharing a detailed blog post 👇 410821 0
Andi @andimara.bsky.social · 06/12/2024Do you want to try Llama 3.3? huggingface.co/chat/huggingface.coHuggingChatMaking the community's best AI chat models available to everyone. 031 0
Reposted by AndiTom Elliot @tomelliot.net · 03/12/2024Just been messing with SmolVLM - visual language model that is _Smol_. 2B params, does some amazing stuff, with little memory, quickly. I'll post a couple of examples below. Super cool stuff from @merve.bsky.social & @andimara.bsky.social! 161 0
Reposted by AndiLoubna Ben Allal @loubnabnl.hf.co · 30/11/2024📬 Summarize and rewrite your text/emails faster, and offline! Check @andimara.bsky.social's Smol Tools for summarization and rewriting. It uses SmolLM2 to summarize text and make it more friendly or professional, all running locally thanks to llama.cpp github.com/huggingface/...github.comsmollm/smol_tools at main · huggingface/smollmEverything about the SmolLM & SmolLM2 family of models - huggingface/smollm 1112 0
Reposted by Andimerve @merve.bsky.social · 28/11/2024it only takes a single CLI command to kick-off a Direct Preference Optimization fine-tuning run on SmolVLM huggingface.co/blog/smolvlm... you're welcome 1794 0
Andi @andimara.bsky.social · 29/11/2024Do you want to help us build SmolVLM2? We are hiring interns! Remote roles from EMEA or US. apply.workable.com/huggingface/... 130 0
Reposted by AndiAndi @andimara.bsky.social · 26/11/2024Let's go! We are releasing SmolVLM, a smol 2B VLM built for on-device inference that outperforms all models at similar GPU RAM usage and tokens throughputs. SmolVLM can be fine-tuned on a Google collab and be run on a laptop! Or process millions of documents with a consumer GPU! 410422 0
Reposted by Andimerve @merve.bsky.social · 27/11/2024The authors of ColPali trained a retrieval model based on SmolVLM 🤠 TLDR; - ColSmolVLM performs better than ColPali and DSE-Qwen2 on all English tasks - ColSmolVLM is more memory efficient than ColQwen2 💗 Find the model here huggingface.co/vidore/colsm... 4738 0
Reposted by AndiLucie-Aimée Kaffee @frimelle.bsky.social · 28/11/2024🚀 Excited to share that our provocation paper "Evaluations Using Wikipedia without Data Contamination: From Trusting Articles to Trusting Edit Processes" has been accepted at the NeurIPS workshop Evaluating Evaluations! 🌐📚 #NeurIPS2024 #evaleval #AIEvaluation 1204 0
Reposted by Andimerve @merve.bsky.social · 27/11/2024It's pretty sad to see the negative sentiment towards Hugging Face on this platform due to a dataset put by one of the employees. I want to write a small piece. 🧵 Hugging Face empowers everyone to use AI to create value and is against monopolization of AI it's a hosting platform above all. 2945570 0
Andi @andimara.bsky.social · 27/11/2024I'm not following. What brand? What meetings? I'm just a researcher sharing my research 👉👈 100 0
Reposted by AndiLeandro von Werra @lvwerra.bsky.social · 26/11/2024Some people are pushing models to the top right of the plot following the scaling laws, others push them to the top left and make them faster and cheaper! We need both! 1111 0
Andi @andimara.bsky.social · 26/11/2024We compared a bit against llama 11B, the smallest vision capable version. Of course llama 11B is better, but honestly not that by much, I was expecting more. This model also compares well against idefics 3, which is built on llama 3.1 8B 130 0
Andi @andimara.bsky.social · 26/11/2024Check out more! Demo: huggingface.co/spaces/Huggi... Blog: Model: huggingface.co/HuggingFaceT... Fine-tuning script: github.com/huggingface/... 010 0
