Sign in

Julien Hurault

@hachej.bsky.social
122 followers 298 following 45 posts

Freelance Data | Weekly Data Eng. Newsletter 📨 juhache.substack.com - 4k+ readers

PostsRepliesMedia
Julien Hurault @hachej.bsky.social · 13/12/2024
Indeed it s not simple unfortunately..just the a way to get started quickly atm .
060
Julien Hurault @hachej.bsky.social · 13/12/2024
❤️❤️
040
Julien Hurault @hachej.bsky.social · 13/12/2024
For iceberg catalog hard to find a simpler setup..
110
Julien Hurault @hachej.bsky.social · 13/12/2024
Nice! Can you orchestrate lambda or ecs tasks that way?
120
Julien Hurault @hachej.bsky.social · 13/12/2024
Just use Pyiceberg with AWS Glue, probably the fastest way to get started.
120
Julien Hurault @hachej.bsky.social · 06/12/2024
In term of volume of data exchange over the marketplace? No idea
010
Julien Hurault @hachej.bsky.social · 06/12/2024
SF sales rep told me that markeplace was THE feature that helped a lot to convert
120
Julien Hurault @hachej.bsky.social · 06/12/2024
Saw it lot in finance!
120
Julien Hurault @hachej.bsky.social · 06/12/2024
Multi pipelines will probably get tricky no?
100
Julien Hurault @hachej.bsky.social · 06/12/2024
Dlt + duck + evidence + @blef.fr's baby?
130
Julien Hurault @hachej.bsky.social · 06/12/2024
For those lost in GCP terminology: I wrote a summary of Iceberg integration in GCP a couple of weeks ago: juhache.substack.com/p/gcp-and-ic...
juhache.substack.com
GCP & Iceberg
Ju Data Engineering Weekly - Ep 77
000
Julien Hurault @hachej.bsky.social · 06/12/2024
New blog post: Building a 0$ Data Distribution System. juhache.substack.com/p/0-data-dis...
juhache.substack.com
0$ Data Distribution
Ju Data Engineering Weekly - Ep 78
040
Julien Hurault @hachej.bsky.social · 04/12/2024
30 million rows is just one month of data, right?
000
Julien Hurault @hachej.bsky.social · 02/12/2024
check catalog.boringdata.io/dashboard/in...
catalog.boringdata.io
Home
020
Julien Hurault @hachej.bsky.social · 01/12/2024
Do you support swiss banks by any chance?
020
Julien Hurault @hachej.bsky.social · 30/11/2024
" bash / make knowledge, a single instance SQL processing engine (DuckDB, CHDB or a few python scripts), a distributed file system, git and a developer workflow (CI/CD)" what s your best option to orchestate sql models in such setup?
100
Reposted by Julien Hurault
Javi Santana @javisantana.bsky.social · 30/11/2024
Some learnings after helping +50 companies in high performance data engineering projects javisantana.com/2024/11/30/l...
javisantana.com
javisantana.com
43812
Julien Hurault @hachej.bsky.social · 30/11/2024
Super good thx! "immutable workflow + atomic operation" 100%!
110
Julien Hurault @hachej.bsky.social · 30/11/2024
ATTACH url_to_your_dabase.duckdb;
110
Julien Hurault @hachej.bsky.social · 30/11/2024
Oh yeah wiki.postgresql.org/wiki/Don%27t...
030
Julien Hurault @hachej.bsky.social · 30/11/2024
S3 to Snow in the same aws region is free no?
000
Julien Hurault @hachej.bsky.social · 30/11/2024
Niiiice, your view is doing a read_parquet(*) on their bucket? Or do you copy the data?
000
Julien Hurault @hachej.bsky.social · 29/11/2024
1 Docker container embedding app code + SQLite DB → live chat app with 10k simultaneous users. youtu.be/0rlATWBNvMw
youtu.be
DHH discusses SQLite (and Stoicism)
YouTube video by Aaron Francis
010
Julien Hurault @hachej.bsky.social · 29/11/2024
Where is the data stored, then? In DuckDB itself? So, if you have a 1GB dataset, does that mean you’ll share a single .duckdb file containing the entire dataset? Or either a view pointing to parquet files: CREATE VIEW... as read_parquet(*.parquet) ?
000
Julien Hurault @hachej.bsky.social · 29/11/2024
Do you see DuckDB as a format? For me: • Parquet = Standard storage format • Iceberg = Standard metadata format • DuckDB = One possible distribution vector
100
Reposted by Julien Hurault
Jake Gold @jacob.gold · 11/11/2024
Yup, there are almost fifteen million SQLite databases on Bluesky’s PDS servers. It’s wildly efficient and simple but not without trade offs of course. Makes sense for this use case in large part because each users atproto repository is self contained, with links to other repos, like a website.
2396
Julien Hurault @hachej.bsky.social · 28/11/2024
just do both -> catalog.boringdata.io/dashboard/in...
catalog.boringdata.io
Home
110
Reposted by Julien Hurault
Ben Lindsay @benjlindsay.com · 27/11/2024
Here's what I put in my ~/.zshrc file to make sure my virtualenv autoactivates when I move to a directory with a .venv file. Works well for me so far. Do the rest of you do something like this? #Python #DataBS
A screenshot of .zshrc code with the following content:

# Function to auto-activate virtual environment
function auto_activate_virtualenv() {
    if [ -d ".venv" ]; then
        source .venv/bin/activate
    fi
}

# Hook the function to the 'chpwd' event (triggered when you change directories)
autoload -U add-zsh-hook
add-zsh-hook chpwd auto_activate_virtualenv

# Also call the function for the initial directory when the terminal starts
auto_activate_virtualenv
8372
Julien Hurault @hachej.bsky.social · 28/11/2024
Prediction: poeple will monetize custom feeds github.com/bluesky-soci...
github.com
GitHub - bluesky-social/feed-generator: ATProto Feed Generator Starter Kit
ATProto Feed Generator Starter Kit. Contribute to bluesky-social/feed-generator development by creating an account on GitHub.
000
Reposted by Julien Hurault
Nikhil Benesch @benesch.bsky.social · 26/11/2024
Something interesting is brewing in Iceberg-on-S3 land. 👀 lists.apache.org/thread/v7x65... cc @eatonphil.bsky.social
lists.apache.org
3295
Julien Hurault @hachej.bsky.social · 26/11/2024
Many vendors do not display their price on their landing page... Probably for that reason: can t be reverted.
110
Julien Hurault @hachej.bsky.social · 25/11/2024
Head for brittany not Paris :) en.m.wikipedia.org/wiki/Brittany
en.m.wikipedia.org
Brittany - Wikipedia
000
Julien Hurault @hachej.bsky.social · 25/11/2024
Building a data pipeline = 50% bringing data from A to B at time t 50% making pipeline fixing easy at time t+1
100
Julien Hurault @hachej.bsky.social · 24/11/2024
New blog post: GCP & Iceberg open.substack.com/pub/juhache/...
000
Reposted by Julien Hurault
@levelsio @levels.io · 24/11/2024
Now more than ever
5131916
Julien Hurault @hachej.bsky.social · 24/11/2024
Is there a starter pack for data engineering here on bluesky?
320
Julien Hurault @hachej.bsky.social · 23/11/2024
merged with location.foursquare.com/products/pla...
location.foursquare.com
Places
Foursquare's Places allows developers to unlock points of interest data (POI) with precision and in rich detail - all stored in your POI database. Learn more.
010
Julien Hurault @hachej.bsky.social · 23/11/2024
2010 — 2017: ML = pip install scikit-learn 2017 — 2023: ML = pip install torch 2023 — : ML = pip install requests
000
Julien Hurault @hachej.bsky.social · 22/11/2024
FINALLY! 🎉 aws.amazon.com/blogs/comput... No more endless searching for support with intrinsic docs.aws.amazon.com/step-functio...
lnkd.in
LinkedIn
This link will take you to a page that’s not on LinkedIn
000
Julien Hurault @hachej.bsky.social · 22/11/2024
Already gone here in GVA 😢
100
Julien Hurault @hachej.bsky.social · 19/11/2024
app.snowflake.com/marketplace/...
app.snowflake.com
Snowflake
000
Julien Hurault @hachej.bsky.social · 19/11/2024
Yes cool, focus on cyber atm right?
000
Julien Hurault @hachej.bsky.social · 19/11/2024
World wide address data? app.snowflake.com/marketplace/...
app.snowflake.com
Snowflake
100
Julien Hurault @hachej.bsky.social · 19/11/2024
cool, thanks !
000
Julien Hurault @hachej.bsky.social · 19/11/2024
Some datasets, probably yes. The Snowflake Marketplace experience is cool: any dataset just one SELECT away, but it’s locked inside the Snowflake ecosystem. I love the idea of taking that concept outside of Snowflake—having any dataset available with just one ATTACH command.
110
Julien Hurault @hachej.bsky.social · 19/11/2024
What are the cost for this setup? Storage + compute to bring the data to the bucket? Distribution via R2 should be free no?
010
Julien Hurault @hachej.bsky.social · 19/11/2024
I like the vibes here:)
110
Julien Hurault @hachej.bsky.social · 19/11/2024
What about dumping Snowflake Marketplace free listings to R2 ? @jakthom.bsky.social @ssp.sh @tobilg.com
110
Julien Hurault @hachej.bsky.social · 19/11/2024
The Snowflake unbundling continues: • Snowflake Storage → Iceberg • Snowflake Marketplace → Cloudflare R2 + DuckDB :)
210
Julien Hurault @hachej.bsky.social · 18/11/2024
Nice!
000