iiinova — where your digital self keeps evolving

Hannes mentioned this in Joe Reis's podcast and I still don't understand.. “And it's, by the way, it is, again, I won't name names, but it is completely common for analytical data management systems to ignore things like primary key constraints.” Would love to have more elaboration on this

120

But go spend couple of hours searching for this problem in any Data publication and you will finds virtually nothing

000

However, all these manual processes still need to be reported (as clean data) and integrated into existing systems (as structured data) by someone or a dedicated team. Quite often, this falls to the Data team.

100

Chris @chris.blue · 16/12/2024

Unexplored Topic: Most startups (at least in Indonesia) currently take a Business-Market Fit-first approach, keeping all processes manual until they generate revenue, before committing precious resources (like engineering bandwidth) to properly productize their solutions.

100

Reposted by Erald David

Last post of the year! S3, Parquet, Iceberg, and @duckdb.org are a great way to get customers their data.

materializedview.io

S3 Is the New SFTP

Customers want their data. A customer data lake is a great way to give it to them.

6332

Erald David @eraldds.bsky.social · 17/12/2024

I didn't notice any comment praising about the book that you just picked so I will be the first person to say it: Data and Reality is great. Perfect book! I went from "Huh, quite weird to choose old book for this" to "God, if I read this book early of my career, I can save so much time"

110

Reposted by Erald David

Jenna Jordan @jennajordan.me · 05/11/2024

The first book would have to be Data & Reality by Bill Kent (2nd ed of course). After that there are many potential options but I’ve wanted to have a book club for Data & Reality for so long that it has to be the first one. We could call it the “Data Philosophizing” book club or something.

12452

Reposted by Erald David

Andy Pavlo @andypavlo.bsky.social · 05/12/2024

For my last class this semester, I tried to cram our Advanced Database course into one lecture. We cover the following database systems in 60min: youtu.be/fr5lIchF6pw • Google Dremel / BigQuery • Snowflake • Amazon Redshift • Yellowbrick • Databricks Photon • @duckdb.org • TabDB

youtu.be

#25 - BigQuery + Snowflake + Redshift + Databricks + DuckDB (CMU Intro to Database Systems)

YouTube video by CMU Database Group

612018

Reposted by Erald David

Vicki @vickiboykis.com · 06/12/2024

Fantastic opportunity for ppl in #mlsky #databs #stats etc to write their own perspectives on how machine learning and engineering have changed over the course of your careers (see original link in the thread), especially last 5 years. Would love to read all of these posts!

4293

Erald David @eraldds.bsky.social · 01/12/2024

I just completed "Historian Hysteria" - Day 1 - Advent of Code 2024 #AdventOfCode adventofcode.com/2024/day/1

000

I filed this case to "I know the concept already but still surprised with the instantiation" folder No wonders why Einstein loves compounding

000

This also hammer the points how things can quickly change when you can "seize" small win and compounds it. "Yeah you know that city where grandpa used to go to check the price of corn? Pack up your suit man, we gonna bet against Argentina's Peso"

100

Jeremy Howard @howard.fm · 28/11/2024

It's crazy to think that Chicago become the center of modern finance because their geographic advantage and .....cow? What does analogy that fit into this case? On top of my head: it's like knowing your local IKEA become 3 Michelin star restaurant in the future... because they can cook meatball

110

Reposted by Erald David

FYI, here's the entire code to create a dataset of every single bsky message in real time: ``` from atproto import * def f(m): print(m.header, parse_subscribe_repos_message()) FirehoseSubscribeReposClient().start(f) ```

1944262

Olivier Grisel @ogrisel.bsky.social · 27/11/2024

Woahh idk about this. Very useful

100

Reposted by Erald David

I recently shared some of my reflections on how to use probabilistic classifiers for optimal decision-making under uncertainty at @pydataparis.bsky.social 2024. Here is the recording of the presentation: www.youtube.com/watch?v=-gYn...

A high-level summary diagram taken from the slides linked below. It shows the interplay of two main components: a probabilistic model and decision maker or planner.

Probabilistic predictions of an underfitting polynomial classifier on a noisy XOR task and the corresponding under-confident calibration curve.

Probabilistic predictions of an overfitting polynomial classifier and the resulting overconfident calibration curve on the same noisy XOR problem.

Simulation study to show the relative lack of stability of hyperparameter tuning when using hard metrics such as Accuracy or soft yet not probabilistic metrics such as ROC AUC compared to a strictly proper scoring rule such as the log-loss.

15219

Reposted by Erald David

Martin Kleppmann @martin.kleppmann.com · 26/11/2024

Interesting critique of the problems with peer review. Unfortunately it fails to offer a proposal for what might be better. I'm reminded of the old adage “democracy is the worst form of government, except for all the others.” Likewise for peer review. www.experimental-history.com/p/the-rise-a...

experimental-history.com

The rise and fall of peer review

Why the greatest scientific experiment in history failed, and why that's a great thing

910616

Reposted by Erald David

feynon @ankeshbharti.com · 26/11/2024

Finished reading the book, last two chapters on the philosophical aspects of technology worship are great, and the material provides vivid conceptual clarity on bubble dynamics. However, overall, the book could have been much shorter.

021

Erald David @eraldds.bsky.social · 27/11/2024

Let's compare notes! Let's flood Bluesky with optimism and vision

110

Reposted by Erald David

Jeremy Howard @howard.fm · 27/11/2024

I'm glad @hf.co is doing this. It brings down the barriers to allow more people to benefit from AI, rather than keeping it exclusively in the realm of deep pocketed giant companies. AI can help open the gates, to allow regular people to do things they couldn't do before. (Which can be threatening!)

916813

Erald David @eraldds.bsky.social · 25/11/2024

"Process-first" vs "Data-first" people is a really useful dichotomy. This also explain numerous occasion when me and one of my SE friend has. I kept thinking "How could you miss this important data point, this doesn't makes sense".. until he answered "Yes but what process generate that data?"

010

Erald David @eraldds.bsky.social · 25/11/2024

The first 3 days joining Bluesky, I kept talking to myself "Oh wow, I didn't know this person already in here" This is the way

020

"The other kind of bubble is a filter bubble. Participants in filter bubbles wall themselves off from opinions they disagree with and become increasingly convinced that their viewpoints reflect the one true way to understand the world."

030

(3) On 2 kind of bubbles "One kind, the classic speculative financial bubble, involves speculators egging one another on....until the only justification for asset prices is the expectation that someone else will be willing to pay even more...."

100

.... "After leaving comments in the margins, von Braun would circulate the entire annotated collection of notes within the organization. Through this informal system, everyone was able to tap into the organization’s collective knowledge and contribute solutions to each other’s problems."

100

(2) On Information Sharing Mechanism "In the case of Apollo, handwritten notes functioned as information-aggregation mechanisms. In von Braun’s system of “Monday notes,” engineers and technicians were required to identify the most salient issues and submit a single-page note...."

100

On Apollo's unique management system "NASA developed, implemented, and perfected an approach known as systems management. Leaders with systemic vision avoid micromanaging and are better able to accelerate a team’s overall progress.

100

The number of banger quotes in 'Boom' by Byrne Hobart is staggering Few of my favorites (a note to myself)

210

Cedric Chin @handle.invalid · 21/11/2024

A good side effect where you're working at Ops Excellence is that everyone, including Data team, shift their focus to find lever. This shift our incentive from "Okay, what tools that we want to try this month?" to "For Problem X, we should try Y and Z, and Data can check whether its working or not"

010

Reposted by Erald David

One thing I’ve learnt about The Practice is that it takes time for other controllable levers to show up — and sometimes you’re limited by how rich your understanding of your causal model of your process is. This is limited by experience, instrumentation, and creativity.

152