Reposted by Ben Schneider
Nic ( @niccrane.bsky.social) taught us all about FREAKING PARQUET FILES on the #DSLab, which are wayyyyy cooler than I realized.
Parquet files:
Preserve data types
Are fast & small
Are column-oriented so you can read just 1 col if wanted!
& Positron displays'em out-the-box!
▶️ youtu.be/2mQ4kc9J9uQ
youtu.be
What is a Parquet file anyway? Why is Parquet good? | Nic Crane | Data Science Lab
YouTube video by Posit PBC
![“The contrast is outrageous! There’s this sex bomb in front, and this, this creature—I don’t know what else to call him—this fuzzy, violent, nasty creature in back who only got pleasure out of making people and other Muppets unhappy. A creature of impulse, let’s put it that way. And when I watch myself—’cause I’m usually so critical—but when I watch myself, I just laugh at me. What I love about it is that my character’s trying so hard to be sexy and sultry. And this horrible creature in back of me is acting up. [Laughter] See? I can’t stop laughing. It tickles me to death.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:zijpzw75j26esk5z2oilexan/bafkreiftmihi2643qz3l4h2dwggjulxsnbmyrmthxo4lqu5cvowep2hxta)
![use anyhow::bail;
use arrow_extendr::FromArrowRobj;
use extendr_api::prelude::*;
use geo_traits::{CoordTrait, PointTrait};
use geoarrow::array::{GeoArrowArrayAccessor, PointArray};
#[extendr]
fn geoarrow_centroid(x: Robj) -> anyhow::Result<Doubles> {
let res = PointArray::from_arrow_robj(&x)?;
let (sum_x, sum_y, n) = res.iter_values().filter_map(|xi| xi.ok()).fold(
(0.0_f64, 0.0_f64, 0usize),
|(sx, sy, n), v| {
let (x, y) = v.coord().unwrap().x_y();
(sx + x, sy + y, n + 1)
},
);
if n == 0 {
bail!("cannot compute centroid of empty array");
}
let res = [sum_x / n as f64, sum_y / n as f64]
.map(Rfloat::from)
.into_iter()
.collect::<Doubles>();
Ok(res)
}](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:63atwz67mlrdquqnasarutmr/bafkreifvhwk7acara7a64jwglprbjgszw5vd7dmfox55qpyhhe4c5i3sry)
![library(wk)
library(geoarrow)
devtools::load_all()
#> ℹ Loading extendrtest
pnts <- wk::xy(rnorm(100, -180, 180), runif(100, -90, 90))
x <- geoarrow::as_geoarrow_array(pnts)
geoarrow_centroid(x)
#> [1] -182.711392 -2.806439
sf::st_as_sfc(pnts) |>
sf::st_combine() |>
sf::st_centroid()
#> Geometry set for 1 feature
#> Geometry type: POINT
#> Dimension: XY
#> Bounding box: xmin: -182.7114 ymin: -2.806439 xmax: -182.7114 ymax: -2.806439
#> CRS: NA
#> POINT (-182.7114 -2.806439)](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:63atwz67mlrdquqnasarutmr/bafkreifb7wrdfujkxs356k5dj6peukdmtj2hekllcpbheckr4uhihc4r5a)
