iiinova — where your digital self keeps evolving

Reposted by TEGNicholas.bsky.social

Ketan Joshi @ketanjoshi.co · 03/09/2025

"[Matt Yglesias] can write whole essays claiming that fracking is good and we need fossil fuel friendly energy policies, dismissing progressives as childish, while never engaging with the scientific literature on the consequences of climate change"

currentaffairs.org

Matt Yglesias Is Confidently Wrong About Everything

The Biden administration’s favorite centrist pundit produces smug pseudo-analysis that cannot be considered serious thought. He ought to be permanently disregarded.

835968

TEGNicholas.bsky.social @tegnicholas.bsky.social · 27/08/2025

This reads like the writings of climate denialists - constant strawmanning, shifting of goalposts, selection bias in research quoted, and conspiracy theorizing.

120

Reposted by TEGNicholas.bsky.social

Kyle Barron @kylebarron.dev · 04/08/2025

New blog post on Obstore, fast, multi-provider cloud storage access for Python: developmentseed.org/blog/2025-08...

developmentseed.org

We Needed Better Cloud Storage for Python so We Built Obstore — Development Seed

Obstore solves the friction we kept hitting in cloud-native workflows.

032

Reposted by TEGNicholas.bsky.social

Fernando Pérez @fernandoperez.org · 24/07/2025

Open source, open science for earth, climate and geospatial science? Coming to #AGU25? Build tools in #Python @jupyter.org? Submit an abstract for this session and come meet us and like minded scientists!

094

Reposted by TEGNicholas.bsky.social

Emily Atkin @emorwee.bsky.social · 21/07/2025

Will Alpine co-wrote Microsoft's manifesto on how AI will be a powerful force for good for climate change In a new interview, Alpine disavows the manifesto, saying he believes Microsoft used his work to distract from the much larger climate harms the company enables through contracts with Big Oil

heated.world

He helped Microsoft build AI to help the climate. Then Microsoft sold it to Big Oil.

A former Microsoft project manager reveals how the tech giant is using AI to help Big Oil drill—and how he and his partner are now pushing for change.

19554

Reposted by TEGNicholas.bsky.social

Henri F. Drake @henrifdrake.bsky.social · 30/06/2025

This is fucking insane. Closing these NOAA labs would obliterate our ability to observe, understand, and forecast the Earth System, from weather systems tomorrow to sea levels 50 years from now.

013473

Reposted by TEGNicholas.bsky.social

Dr. Jessie Christiansen @aussiastronomer.bsky.social · 29/06/2025

NASA is being told to cancel 19 *active* missions to save $6B, which looks to be less than the ICE *hiring/retention* budget going forward. I need people to let that sentence sink into their bones for a minute.

7241772102

Reposted by TEGNicholas.bsky.social

Michele Tobias, PhD @micheletobias.bsky.social · 28/10/2024

I've been adding new accounts to the Open Source Geospatial starter pack. Who else wants on or off? #gischat #geosky go.bsky.app/PGYLmPG

356921

TEGNicholas.bsky.social @tegnicholas.bsky.social · 22/05/2025

I would have suggested Max Jones, Aimee Barciauskas, or Lindsey Nield, but they don't seem to be on BlueSky, so you could instead add @jhamman.bsky.social , @rabernat.bsky.social , or myself.

100

Reposted by TEGNicholas.bsky.social

Adam Sobel @adamsobel.bsky.social · 21/05/2025

It's outrageous that NASA GISS, one of the best earth & space science labs in the world, is being kicked out of its Columbia home. The outstanding scientists who work there can't say that publicly, but I can. And so can you --- call your reps, esp. (but not only) if you live in NYC or NY state.

06135

Reposted by TEGNicholas.bsky.social

Earthmover @earthmover.io · 14/05/2025

𝐻𝑜𝑤 𝑑𝑜𝑒𝑠 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝑎𝑣𝑜𝑖𝑑 𝑟𝑒𝑑𝑢𝑛𝑑𝑎𝑛𝑡 𝑠𝑡𝑜𝑟𝑎𝑔𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑑𝑎𝑡𝑎 𝑣𝑒𝑟𝑠𝑖𝑜𝑛𝑠? Icechunk stores only new or changed chunks for each version —no redundant copies or rewrites. You get instant time travel, branching, and efficient updates, all with negligible storage overhead. More: bit.ly/3F1XFST

earthmover.io

Icechunk: Efficient storage of versioned array data - Earthmover

We recently got an interesting question in Icechunk’s community Slack channel (thank you Iury Simoes-Sousa for motivating this post): I’m new to Icechunk. How is the storage managed for redundant info...

024

TEGNicholas.bsky.social @tegnicholas.bsky.social · 12/05/2025

Excellent post by Brian Davis laying out why doing "Open Science" for data-driven workflows is almost impossible in practice, at least without much better data pipeline tools.

050

Reposted by TEGNicholas.bsky.social

Abrahm @abrahm.bsky.social · 24/04/2025

The proposed cuts to NOAA cold have profound consequences not just for climate change, but for our national security and the entire economy. Here's what I learned: www.propublica.org/article/trum...

propublica.org

White House Proposal Could Gut Climate Modeling the World Depends On

Potential funding cuts for NOAA and its research partners threaten irreparable harm not only to climate research but to American safety, competitiveness, and national security.

7574204

TEGNicholas.bsky.social @tegnicholas.bsky.social · 23/04/2025

It's fun to work with real hardcore software engineers like @functionth.bsky.social who can teach you about database consistency and transactions and all that Scientific data infrastructure should be built on solid foundations like this instead of on piles of janky code written by postdocs...

030

Reposted by TEGNicholas.bsky.social

Mike Mahoney @mikemahoney218.com · 19/04/2025

the fact that I've never once thought about making a range request, and yet make them constantly for extremely targeted data pulls, is absolutely an invisible technical miracle

141

TEGNicholas.bsky.social @tegnicholas.bsky.social · 17/04/2025

Of course- I really wrote this article for my past self! I wish someone had explained this cloud science stuff to me earlier.

020

TEGNicholas.bsky.social @tegnicholas.bsky.social · 17/04/2025

It’s also important for understanding what problem VirtualiZarr solves. I’ve given this explanation to many people in the past (including at @cworthy.bsky.social), so I hope that this article can serve as a useful reference the next time someone wonders what @zarr.dev actually is.

020

TEGNicholas.bsky.social @tegnicholas.bsky.social · 17/04/2025

I wrote the article I wish I could have read back when I first heard of Zarr and cloud-native science back in 2018. This explains how object storage and conventional filesystems are different, and the key properties that make @zarr.dev work so well in cloud object storage.

140

TEGNicholas.bsky.social @tegnicholas.bsky.social · 16/04/2025

5/ Almost all organisations working with scientific array data have this kind of data delivery issue, even if it's just internally. Whilst the Flux integrations today are established geospatial standards, you also see similar patterns in other fields such as Neuroscience.

000

TEGNicholas.bsky.social @tegnicholas.bsky.social · 16/04/2025

4/ Flux's architecture is auto-scaling, so once turned on there is no need to worry about how many users are hitting the data. As it's not a stateful server like THREDDS, it won't catch fire under pressure. This is what "Cloud-Native" architectures for scientific data look like.

100

TEGNicholas.bsky.social @tegnicholas.bsky.social · 16/04/2025

3/ Your downstream scientists, GIS users, analysts, and external users can all now forget about file formats! They just keep using the same GUI or tool or script that they prefer, and don't need any other services or copies of the data made bespoke for them - Flux does that on-demand!

100

TEGNicholas.bsky.social @tegnicholas.bsky.social · 16/04/2025

2/ Flux bridges this chasm. It sits in between your data and the consumers, springing up at a moment's notice to provide subsets of data however your users prefer it.

100

TEGNicholas.bsky.social @tegnicholas.bsky.social · 16/04/2025

1/ Flux solves the impedance mismatch between geospatial data providers and consumers. Providers want to manage data lakes stored in cloud-optimized formats like Zarr, but consumers want their applications to keep being fed data in ways they already understand.

130

Reposted by TEGNicholas.bsky.social

Chelsea Harvey @chelseaeharvey.bsky.social · 11/04/2025

Hard to overstate this plan's reach, which touches nearly every aspect of NOAA's work - dissolving its research arm, gutting climate science, diminishing sat observations, boosting fossil fuels. With amazing colleagues Daniel Cusick and @scottpwaldman.bsky.social : www.politico.com/news/2025/04...

politico.com

White House outlines plan to gut NOAA, smother climate research

The agency’s Office of Oceanic and Atmospheric Research would be “eliminated as a line office,” according to a memo from the Office of Management and Budget.

15739

TEGNicholas.bsky.social @tegnicholas.bsky.social · 10/04/2025

You could also do this for arbitrarily large scientific array datasets using Xarray + Icechunk + R2/Tigris juhache.substack.com/p/0-data-dis...

juhache.substack.com

0$ Data Distribution

Ju Data Engineering Weekly - Ep 78

001

Reposted by TEGNicholas.bsky.social

Earthmover @earthmover.io · 09/04/2025

📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...

earthmover.io

Exploring Icechunk scalability: untangling S3's prefix story | Earthmover

We show Icechunk can scale to extremely high concurrency levels, and explain how it achieves this in modern object stores.

254

TEGNicholas.bsky.social @tegnicholas.bsky.social · 03/04/2025

Several times some database comp sci nerd has suggested to me that you could just do everything in array land using tabular database tools. Whilst they are technically correct that you _could_, this article convincingly shows why you _should not_ - that would be horribly inefficient.

110

Reposted by TEGNicholas.bsky.social

Ryan Abernathey @rabernat.bsky.social · 28/03/2025

I am so excited to share this new project with NASA and @developmentseed.org . By applying Icechunk and the "virtualization" approach, we can make Petabytes of archival data accessible to high performance analytics. Unlocking NASA's cloud archives will have immense downstream benefits for the world.

23412

TEGNicholas.bsky.social @tegnicholas.bsky.social · 28/03/2025

Very proud to be working on enabling this - the potential improvements for NASA (and eventually all fields of science) are enormous, benefiting users and internal staff.

040

Reposted by TEGNicholas.bsky.social

Earthmover @earthmover.io · 24/03/2025

Register to attend our team's workshop at @cloudnativegeo.org! The workshop will teach attendees how to use the @zarr.dev data format model for cloud-native geospatial datacube analysis. 🗓️ Apr 30-May 2 📍Snowbird, Utah 🔗 Register here: 2025-ut.cloudnativegeo.org

041

TEGNicholas.bsky.social @tegnicholas.bsky.social · 20/03/2025

With Flux all the data engineers serving climate and weather data for maps, timeseries, and scientific analysis will no longer need to run their own servers

000

Reposted by TEGNicholas.bsky.social

Wah @robotpirateninja.com · 19/03/2025

You nailed it a while back.

615136

TEGNicholas.bsky.social @tegnicholas.bsky.social · 17/03/2025

The ATproto docs are actually really interesting, and a really good example of teaching like the original tweet describes atproto.com

atproto.com

AT Protocol

010

TEGNicholas.bsky.social @tegnicholas.bsky.social · 17/03/2025

As a non-US-citizen living in NYC on an O-1 visa, previously at Columbia University, who has been outspoken on Climate Change, I must say this kind of authoritarian repression has me worried. It makes me much more likely to take my skills and USCIS-approved "Exceptional Ability" to another country.

151

Reposted by TEGNicholas.bsky.social

William B. Fuckley @opinionhaver.bsky.social · 10/03/2025

if you find yourself reluctant to support the guy at Columbia because you find some of the politics he espoused distasteful or even abhorrent, you should know that this is exactly the reaction that they hoped for in choosing him as a target for unlawful detention based on political speech.

212180984737

Reposted by TEGNicholas.bsky.social

Leah Greenberg ❌👑 @leahgreenberg.bsky.social · 06/03/2025

lotta people trying very very hard to win the 2024 election instead of adapting to the rapidly shifting environment

311385170

Reposted by TEGNicholas.bsky.social

Andrew Lilley Brinker @alilleybrinker.com · 01/03/2025

Banger.

Withoutboats on Hacker News saying: "Everyone who reads Hacker News should understand that the content your consuming is usually from one of these kinds of people: a) dilettantes, who don't have a deep understanding of the technology; b) cranks, who have some axe to grind regarding the technology; c) evangelists, who are here to promote some other technology. The people who actually drive the technologies that shape our industry don't usually have the time and energy to post on these kinds of things, unless they get so angry about how their work is being discussed, as I am here."

1154

Reposted by TEGNicholas.bsky.social

David Ho @davidho.bsky.social · 01/03/2025

All websites that depend on NOAA data should display something like this. earth.nullschool.net

Weather and climate data shown on this website and countless others are at risk.

The National Oceanic and Atmospheric Administration (NOAA) is the U.S. agency responsible for global weather forecasting, hurricane prediction, ocean observation, and many other services vital to public safety. Its satellites, supercomputers, and research teams provide essential data that help us understand our planet and protect lives.

On February 27, the new U.S. administration initiated mass firings at NOAA. These actions are unethical and deeply disruptive to the talented scientists and engineers who dedicate themselves to the public good. The firings, along with expected budget cuts, have serious implications for the availability and quality of weather forecasts produced by the United States. They must be reversed immediately.

Much of the data on this website is downloaded directly from NOAA's servers. In this environment of uncertainty, access could be disrupted at any time. While I'll strive to keep all features on this website functional and switch to alternative data sources if necessary, some datasets have no substitute if they go offline.

If this concerns you, speak up. Share on social media. And if you're in the U.S., contact your representatives.

- Cameron

222177866

TEGNicholas.bsky.social @tegnicholas.bsky.social · 27/02/2025

We now have a public slack channel for VirtualiZarr development and questions, feel free to join! earthmover-community.slack.com/archives/C08...

earthmover-community.slack.com

Slack

054

TEGNicholas.bsky.social @tegnicholas.bsky.social · 27/02/2025

Helping the clients is mostly just fixing bugs and answering questions about the OSS Pangeo stack, which is what I was already doing anyway by watching the Xarray issue tracker every day, but now it's actually my main job instead of a side project!

120

TEGNicholas.bsky.social @tegnicholas.bsky.social · 27/02/2025

Hah! No, it's short for "forward deployed engineer", which is a term used in other startups to basically mean "engineer tasked with helping out a specific client". For my role it basically just means that I help clients whilst also doing development.

120

TEGNicholas.bsky.social @tegnicholas.bsky.social · 26/02/2025

I joined Earthmover! Excited to focus on building tools for sharing scientific data full-time. This is a natural continuation of my work on open-source projects over the last ~6 years, which itself was motivated by seeing the same frustrations in every field of science I've ever worked in.

1131

TEGNicholas.bsky.social @tegnicholas.bsky.social · 20/02/2025

Relatively simple but very important quality-of-life improvement when accessing big climate datasets such as ERA5 and CMIP.

020

Reposted by TEGNicholas.bsky.social

Ryan Abernathey @rabernat.bsky.social · 14/02/2025

Next week I'll be giving a webinar on one of my favorite parts of Icechunk--the ability to apply git-style version control to Zarr arrays and Groups. This capability makes data workflows so more safe, robust, and reproducible. Come check it out! (Registration link below.)

063

TEGNicholas.bsky.social @tegnicholas.bsky.social · 13/02/2025

Recording of the talk + lively community discussion is up! youtu.be/GZvJ0H89G0A If you're interested in these data cataloging questions then please also sign up to receive updates here forms.gle/yJY4Xp1Gvjd6...

youtu.be

FROST: Federated Registry Of Scientific Things

YouTube video by Pangeo

021

Reposted by TEGNicholas.bsky.social

Pangeo @pangeo.io · 12/02/2025

Join us today, February 12 at 4 PM EST for @tegnicholas.bsky.social's presentation and moderated discussion on FROST: Federated Registry of Scientific Things! Everyone is welcome to show up via the zoom link - hope to see you there! discourse.pangeo.io/t/pangeo-sho...

discourse.pangeo.io

Pangeo Showcase: "FROST: Federated Registry Of Scientific Things" (Feb 12, 2025)

Title: “FROST: Federated Registry Of Scientific Things” Speaker: Tom Nicholas (ORCID: 0000-0002-2176-0530) When: Wednesday, February 12, 2025 at 4 PM EST Where: Launch Meeting - Zoom Abstract: C...

011