iiinova — where your digital self keeps evolving

Santiago Viquez @santiviquez.com · 11/03/2025

What’s the best way to track the progress of my book, The Little Book of ML Metrics? 1️⃣ Visit the book’s repo: github.com/NannyML/The-... 2️⃣ Download the latest digital WIP version. 3️⃣ Start reading while I keep writing. It gets updated every time I push new changes.

120

Santiago Viquez @santiviquez.com · 09/02/2025

It's happening! Join us next week to ask Sebastian Raschka anything! 📅 Date: February 11th ⏰ Time: 10:00 AM – 11:00 AM EST 📍 Register: lu.ma/evqa4rct

011

Santiago Viquez @santiviquez.com · 21/01/2025

Performance Estimation methods are a step forward in solving the real problem. Proud to be part of this team!

030

Santiago Viquez @santiviquez.com · 21/01/2025

Big kudos to my colleagues Jakub and Wojtek for their work on this! I’ll be honest, even at the risk of sounding biased. There are many ML monitoring companies out there, but none of them are solving the real problem. Most monitor the data, not the models.

100

Santiago Viquez @santiviquez.com · 21/01/2025

Super proud to work at a place that values open science. Four years ago, at NannyML, we invented the first version of Confidence-Based Performance Estimation. Today, a paper about it was published in JAIR. JAIR: jair.org/index.php/ja... ArXiv: arxiv.org/abs/2407.08649

121

Santiago Viquez @santiviquez.com · 18/01/2025

Took me over an hour to fully understand the computation behind the Pair Confusion Matrix. Hopefully, it’ll take you a lot less after reading my explanation in "The Little Book of ML Metrics" www.nannyml.com/metrics?via=...

020

Santiago Viquez @santiviquez.com · 18/01/2025

You can just do many things. Yesterday was my first day at culinary school!

020

Santiago Viquez @santiviquez.com · 16/01/2025

Don’t overthink it. Embrace the cringe.

020

Santiago Viquez @santiviquez.com · 16/01/2025

If people don’t think what you do is cringe, then you’re not pushing hard enough. Every person you admire was once considered cringe by someone. A Writer, YouTuber, Founder, Musician, you name it. They all got to where they are because they constantly shared their work with the world. Constantly.

130

Santiago Viquez @santiviquez.com · 13/01/2025

We’re deciding what book to read next in the "AI from Scratch" study group. So far, we have these two: 1. AI Engineering by Chip Huyen 2. Hands-On Generative AI with Transformers and Diffusion Models by Omar Sanseviero and gang Any other suggestions?

130

Reposted by Santiago Viquez

Eugene Yan @eugeneyan.com · 12/01/2025

• Work hard • Keep learning • Cherish loved ones • Find people who inspire you • Be kind & egoless • Eat healthy, exercise, sleep well • Read & write • Practice gratitude & meditate • Be present • Enjoy food & nature • Don’t sweat the small stuff • Smile =)

3221

Santiago Viquez @santiviquez.com · 12/01/2025

First AI from Scratch session of 2025! A big thanks to @carloscapote.bsky.social and Michael Erasmus for their excellent explanations in today's meeting.

041

Santiago Viquez @santiviquez.com · 10/01/2025

Forgot to share the news, but here it is: Our NannyML open-source package reached 2,000 GitHub stars! 🌟 Slowly but steadily 💪

010

Santiago Viquez @santiviquez.com · 09/01/2025

Another one from the book. Log Loss (aka cross-entropy loss)! --- If you're interested in more metric descriptions like this one, check out the book I'm writing: The Little Book of ML Metrics. GitHub Repo: github.com/NannyML/The-... Pre-order the book:https://www.nannyml.com/metrics

010

Santiago Viquez @santiviquez.com · 09/01/2025

Ideally before the end on Q2 2025

010

Santiago Viquez @santiviquez.com · 08/01/2025

Please recommend me your recommender system metric 😂

010

Santiago Viquez @santiviquez.com · 08/01/2025

Here are the ones I have so far: • MRR: Mean Reciprocal Rank • ARHR@k: Average Reciprocal Hit-Rank at K • nDCG@K: Normalized Discounted Cumulative Gain • Precision@k • Recall@k • F1@K • Average Recall@k • Average Precision@k • MAP: Mean Average Precision

130

Santiago Viquez @santiviquez.com · 08/01/2025

Which ranking metrics am I missing? In the coming weeks, I'll be working on the ranking chapter for "The Little Book of ML Metrics", and I want to make sure I'm not missing any popular ranking/recsys metrics.

230

Santiago Viquez @santiviquez.com · 07/01/2025

Every time you say "garbage in, garbage out" an ML model dies.

140

Santiago Viquez @santiviquez.com · 02/01/2025

6. "Large Language Models: A Deep Dive—Bridging Theory and Practice" (Kamath et al., 2024): amzn.to/3VZO6ct This one is probably too long and expensive, but I want to get it haha. Anything else to add?

amzn.to

Large Language Models: A Deep Dive: Bridging Theory and Practice

Amazon.com: Large Language Models: A Deep Dive: Bridging Theory and Practice: 9783031656460: Kamath, Uday, Keenan, Kevin, Somers, Garrett, Sorenson, Sarah: Books

000

Santiago Viquez @santiviquez.com · 02/01/2025

4. "Pen & Paper Exercises in Machine Learning"(Gutmann, 2022): arxiv.org/pdf/2206.13446 I’m not sure if I’ll go through the whole book, but it looks fun—maybe also for a study group?

arxiv.org

100

Santiago Viquez @santiviquez.com · 02/01/2025

ML Books I'll Be Reading in 2025 📚 1. "AI Engineering: Building Applications with Foundation Models" (Huyen, 2024): amzn.to/4gtQgJo We’ll probably read it in the study group "AI from Scratch."

182

Santiago Viquez @santiviquez.com · 30/12/2024

7. "Be able to dedicate myself to personal projects and make money doing so."Okay, maybe not a lot of money yet, but I’ve done some. 8. "Be creative and empathetic." 9. "Work at Webflow." At the time, Webflow was my dream company. 10. "Be grateful for every step." 11. "Exercise."

010

Santiago Viquez @santiviquez.com · 30/12/2024

4. "Be more present for my family." 5. "Improve my relationship with my friends." 6. "Live abroad with Kathu and Maple."When I wrote this, I didn’t know we’d move to Italy.

110

Santiago Viquez @santiviquez.com · 30/12/2024

Translation 1. "Find a job where I feel genuinely passionate and can contribute and grow professionally." I’m so happy to have found NannyML. 2. “Be constant in continuing to learn about Data Science and keep creating personal projects." 3. "Keep building a circle of love with Kathu and Maple."

100

Santiago Viquez @santiviquez.com · 30/12/2024

During the pandemic—specifically, on May 7, 2020—I wrote some goals on a piece of paper. I folded it, stored it in my wallet, and forgot about it. Today, I found it and realized I’ve accomplished all of them.

110

Santiago Viquez @santiviquez.com · 29/12/2024

I wrote a retrospective about my 2024, reflecting on all the amazing things that happened to me—and the not-so-amazing ones. Feeling grateful and extremely excited about 2025! www.santiviquez.com/blog/2024-re...

santiviquez.com

2024 Year in Review

A review of the year 2024. What I did, what I learned, and what I want to do in 2025.

010

Santiago Viquez @santiviquez.com · 23/12/2024

If you're feeling generous and want to buy me a Christmas present while also getting yourself one, hit that pre-order button! 😂 Just kidding—sharing this would mean the world to me too 🫶 📔About the book: www.nannyml.com/metrics?via=...

nannyml.com

The Little Book of ML Metrics

The book every data scientist needs on their desk. Metrics are arguably the most important part of data science work, yet they are rarely taught in courses or university degrees. Even senior data scie...

010

Santiago Viquez @santiviquez.com · 20/12/2024

Yep, it belongs within MLOps The main difference is that MLOps is often more focused on the engineering side of the model, not the science part. Data scientists design and train the models. Why don’t they continue monitoring them? Instead, this task is handed off to engineers.

020

Santiago Viquez @santiviquez.com · 20/12/2024

Approaching everything from a scientific perspective—not just the engineering one we're used to.

010

Santiago Viquez @santiviquez.com · 20/12/2024

Hear me out. Post-deployment data science. The part of data science that focuses on models after they have been deployed. - Checking if the model is delivering value. - Continuously estimating model performance. - Understanding performance issues and fixing them.

140

Santiago Viquez @santiviquez.com · 17/12/2024

If the reconstruction error is bigger than an threshold, we say that the structure learned by PCA no longer represents the underlying structure of the initial data. Indicating that there is MULTIVARIATE data drift in the production data.

000

Santiago Viquez @santiviquez.com · 17/12/2024

We can then use the learned compressor/decompressor to transform any production data and measure its reconstruction error.

100

Santiago Viquez @santiviquez.com · 17/12/2024

We came up with a method for this. It’s called Data Reconstruction with PCA. This method uses PCA to compress the feature space into a latent space. The algorithm then decompresses the latent space data and reconstructs it with some error. This error is called the reconstruction error.

100

Santiago Viquez @santiviquez.com · 17/12/2024

In this example, we only have two variables, so it’s easy to see the change. But what if we have a model with 10, 50, or 100 features? For that, we need algorithms capable of learning the relationships between many features and detecting when they change substantially.

100

Santiago Viquez @santiviquez.com · 17/12/2024

The distributions of X and Z didn't change independently, but we do see sudden changes in the relationship between them. This is called Multivariate Data Drift, and no traditional univariate drift detection method can detect it.

100

Santiago Viquez @santiviquez.com · 17/12/2024

We can say that both look like normal distributions. If we run univariate drift detection methods such as Jensen-Shannon, Kolmogorov-Smirnov or any other, we wouldn't be alerted by those minimal changes. But if we look at their joint distribution, the story is different.

100

Santiago Viquez @santiviquez.com · 17/12/2024

Univariate data drift doesn't show the full picture. Take a look at this demo created by my colleague @anopsy.bsky.social There are two univariate distributions (top and right) which remain almost unchanged during the whole process.

141

Santiago Viquez @santiviquez.com · 16/12/2024

Yesterday I forgot to post about our study group meeting 😅 It was an amazing one! @carloscapote.bsky.social walked us through Chapter 5: Pretraining on Unlabeled Data. Next week, we’ll take a short break, but we’ll be back after the holidays to finish Chapters 6 and 7 💪

031

Santiago Viquez @santiviquez.com · 14/12/2024

Amazing. Looking forward to Sunday's call. Let me know if you need any help from my side.

000

Reposted by Santiago Viquez

Carlos Capote 🇮🇨 @carloscapote.bsky.social · 13/12/2024

I've almost completed the preparations for the session about pretraining. For the moment, I'm pretty happy with the results. Sebastian's book is a source of information and inspiration. Now I've so many ideas about how to inspect an LLM to understand it better. 🎉 github.com/elcapo/llm-f...

github.com

llm-from-scratch/chapter-5.ipynb at main · elcapo/llm-from-scratch

Implementation of an LLM from scratch following Sebastian Raschka's book. - elcapo/llm-from-scratch

151

Santiago Viquez @santiviquez.com · 12/12/2024

F1-score often takes all the credit. But what F1-score doesn't want you to know is that it wouldn't be so popular without its big brother, F-beta. Check out other metrics at: github.com/NannyML/The-...

031