Sign in

Library Innovation Lab

@harvardlil.bsky.social
2.5K followers 247 following 42 posts

A crowd of coders, lawyers, librarians, designers, & tinkerers building tools like Perma.cc & Caselaw Access Project at the Harvard Law School Library. Where @institutionaldatainitiative.org got started. 🌐 lil.law.harvard.edu

PostsRepliesMedia
Reposted by Library Innovation Lab
Jennifer Chapman @jelizachapman.bsky.social · 01/09/2026
JJ Dearborn & I wrote a blog post about the difficult, complex, but essential work, we're doing as fellows at the @harvardlil.bsky.social in helping research & develop resilient public data infrastructure. I'm very honored to be part of this project lil.law.harvard.edu/blog/2026/09...
lil.law.harvard.edu
Global Stewardship of Public Data: A Research & Development Initiative for Resilient Public Data Infrastructure | Library Innovation Lab
The Library Innovation Lab is growing knowledge and community by bringing library principles to technological frontiers.
185
Library Innovation Lab @harvardlil.bsky.social · 10/06/2026
Open-source tools power nearly every web archive, including Perma.cc. From bot defenses to shrinking funding, sustaining this critical preservation infrastructure can't be an afterthought. Takeaways from #iipcWAC26 keynote: lil.law.harvard.edu/blog/2026/06... #WebArchiveWednesday
lil.law.harvard.edu
Sustaining Open-Source Web Archiving Infrastructure: Takeaways from IIPC Conference 2026 | Library Innovation Lab
The Library Innovation Lab is growing knowledge and community by bringing library principles to technological frontiers.
021
Library Innovation Lab @harvardlil.bsky.social · 10/06/2026
Announcing WARCbench: an open-source command-line tool for exploring, analyzing, transforming, and extracting data from WARC files—built on a decade of Perma.cc experience to help archivists wrangle real‑world, messy web archives. #WebArchiveWednesday lil.law.harvard.edu/blog/2026/06...
lil.law.harvard.edu
WARCbench: A Swiss Army Knife for WARC Processing | Library Innovation Lab
The Library Innovation Lab is growing knowledge and community by bringing library principles to technological frontiers.
011
Library Innovation Lab @harvardlil.bsky.social · 04/06/2026
In partnership with dataindex.us, LIL's #PubDataProject Team investigates solutions for detecting and monitoring changes in data over time at scale. Read more about our methods in our newest blogpost: lil.law.harvard.edu/blog/2026/06...
a man made of binoculars and other eye equipment
041
Library Innovation Lab @harvardlil.bsky.social · 04/06/2026
In partnership with dataindex.us, LIL's #PubDataProject Team investigates solutions for detecting and monitoring changes in data over time at scale. Read more about our methods in our newest blogpost: lil.law.harvard.edu/blog/2026/06...
dataindex.us
010
Library Innovation Lab @harvardlil.bsky.social · 04/06/2026
In partnership with dataindex.us, LIL's #PubDataProject Team investigates solutions for detecting and monitoring changes in data over time at scale. Read more about our methods in our newest blogpost: lil.law.harvard.edu/blog/2026/06...
dataindex.us
030
Library Innovation Lab @harvardlil.bsky.social · 02/06/2026
New blogpost from the #PublicDataProject! Project Lead Molly Hardy reflects on our #AI benchmarking exercise, which grew out of our partnership with @dataindex.us and is part of our efforts to build a data monitoring tool. Read more on the LIL blog: lil.law.harvard.edu/blog/2026/06...
lil.law.harvard.edu
An Automated Data Monitoring Toolkit and the AI Benchmarking Exercise at the Public Data Project | Library Innovation Lab
The Library Innovation Lab is growing knowledge and community by bringing library principles to technological frontiers.
042
Library Innovation Lab @harvardlil.bsky.social · 15/12/2025
When might digital provenance matter? Could we imagine it being used to right past wrongs, to return objects to their rightful places, to restore justice? Our Public Data Project's @mollyhardy.bsky.social reflects on copying gov data & principles of provenance lil.law.harvard.edu/blog/2025/12...
lil.law.harvard.edu
Replication of Government Datasets and the Principles of Provenance | Library Innovation Lab
As part of our Public Data Project, LIL recently launched Data.gov Archive Search. In this post, we consider the importance of provenance for large, replicat...
062
Reposted by Library Innovation Lab
Antoine @ant.bsky.social · 19/11/2025
If you'd like an informative and interesting conversation to enjoy try "Inside Harvard’s Data.gov Archive – A Conversation with Jack Cushman" @jed.co interviewing Jack Cushman @harvardlil.bsky.social about Data.gov archiving on @source.coop Now live www.youtube.com/watch?v=XYMb...
youtube.com
086
Reposted by Library Innovation Lab
EOT End Of Term Web Archive @eotarchive.org · 19/11/2025
Our EOT2024 partner @harvardlil.bsky.social was interviewed by @jed.co on @source.coop about archiving government data. Listen & learn how 300,000+ federal datasets are archived for posterity Inside Harvard’s Data.gov Archive, A Conversation with Jack Cushman: www.youtube.com/watch?v=XYMb...
052
Reposted by Library Innovation Lab
Source Cooperative @source.coop · 20/11/2025
If you missed our conversation with Jack Cushman from @harvardlil.bsky.social‬, you can catch up now. We discussed the Data.gov Archive and the challenge of preventing federal data loss – it's about more than just web pages. youtube.com/live/XYMbQru...
youtube.com
Jack Cushman: Inside Harvard’s Data.gov Archive (EP 3)
YouTube video by Great Data Products
155
Reposted by Library Innovation Lab
infoDOCKET @infodocket.bsky.social · 22/11/2025
Workshop Report: "Resilience in Times of Crisis - Strengthening Open Science Against Geopolitical Pressures" (via Research Group Information Management" at Humboldt-Universität zu Berlin) infomgnt.org/posts/2025-1... @datarescueproject.org @harvardlil.bsky.social #libraries #openscience
031
Library Innovation Lab @harvardlil.bsky.social · 31/10/2025
And join Jack live on @source.coop's Great Data Products podcast on 11/19 to hear how we're making cultural memory collections easier to access and harder to delete. greatdataproducts.com/housekeeping...
greatdataproducts.com
Upcoming Episode: Inside Harvard's data.gov Archive
How the Harvard Library Innovation Lab is preserving and making Data.gov datasets discoverable using BagIt and static search.
043
Library Innovation Lab @harvardlil.bsky.social · 31/10/2025
Podcast: Jack Cushman joins “Pioneers & Pathfinders” to discuss libraries shaping legal tech, digital preservation, and realities of legal AI. Listen: www.seyfarth.com/news-insight...
seyfarth.com
Pioneers and Pathfinders: Jack Cushman
Today, we’re joined by Jack Cushman, director of the Harvard Library Innovation Lab, where he and his team are reimagining how library principles can shape the future of legal technology. Jack is a…
131
Library Innovation Lab @harvardlil.bsky.social · 15/10/2025
This series is part of our work investigating not just the technical, but the human and societal routes to long-term digital preservation. You can also check out @maxy.bsky.social's Century-Scale Storage on our website. lil.law.harvard.edu/century-scal...
lil.law.harvard.edu
Century-Scale Storage
If you had to store something for 100 years, how would you do it?
020
Library Innovation Lab @harvardlil.bsky.social · 15/10/2025
One answer from Frank Cifaldi who is the Founder and Director of the Video Game History Foundation: "I would have the best copyright lawyers in the country figuring out how we can actually make this work." lil.law.harvard.edu/generational...
lil.law.harvard.edu
Frank Cifaldi | Library Innovation Lab
InterviewerIf you were given unlimited funding to design a system for storing and preserving digital information for at least a century, what would you do?
120
Library Innovation Lab @harvardlil.bsky.social · 15/10/2025
One answer from Rebecca Frank who is an Assistant Professor at the University of Michigan School of Information: "The glib answer is the money itself would be the solution." lil.law.harvard.edu/generational...
lil.law.harvard.edu
Rebecca Frank | Library Innovation Lab
InterviewerIf you were given unlimited funding to design a system for storing and preserving digital information for at least a century, what would you do?
111
Library Innovation Lab @harvardlil.bsky.social · 15/10/2025
One answer from Amelia Acker who is an Associate Professor in the School of Communication & Information at Rutgers: "I probably wouldn't build a system. I'd build a bureaucracy." lil.law.harvard.edu/generational...
lil.law.harvard.edu
Amelia Acker | Library Innovation Lab
InterviewerIf you were given unlimited funding to design a system for storing and preserving digital information for at least a century, what would you do?
120
Library Innovation Lab @harvardlil.bsky.social · 15/10/2025
One answer from Che-Wei Wang and Taylor Levy, aka CW&T, who are designers, fabricators, and artists: "Wait, only a hundred years?" lil.law.harvard.edu/generational...
lil.law.harvard.edu
Che-Wei Wang & Taylor Levy | Library Innovation Lab
InterviewerIf you were given unlimited funding to design a system for storing and preserving digital information for at least a century, what would you do?
100
Library Innovation Lab @harvardlil.bsky.social · 15/10/2025
LIL fellow @maxy.bsky.social asked 14 scholars, archivists, designers, business leaders & engineers: "If you were given unlimited funding to design a system for storing and preserving digital information for at least a century, what would you do?" Their answers: lil.law.harvard.edu/generational...
lil.law.harvard.edu
Generational Data Interviews | Library Innovation Lab
14 Designs for Digital Preservation in 2025
1126
Library Innovation Lab @harvardlil.bsky.social · 08/10/2025
Our Public Data Project Lead @mollyhardy.bsky.social writes about public data preservation and how it’s complicated by the artificial contemporary distinction between science and the humanities @scholarlykitchen.bsky.social
scholarlykitchen.sspnet.org
Guest Post - Rethinking Disciplinary Data Regimes - The Scholarly Kitchen
Between a political policy environment focused on defunding and deleting data collections – an environment in which little can be trusted – and an onslaught of new AI tools that feed indiscriminately ...
002
Library Innovation Lab @harvardlil.bsky.social · 03/07/2025
Join our team! LIL is looking for a Product and Research Manager to help create, shape, and execute on our portfolio of open knowledge projects. PRMs work across every piece of the LIL ecosystem, from software experimentation to convening of events. Learn more at careers.harvard.edu/job/product-...
careers.harvard.edu
Product and Research Manager
032
Library Innovation Lab @harvardlil.bsky.social · 01/04/2025
Thrilled to share that @maxy.bsky.social's "Century-Scale Storage" was nominated for a Webby Award! You can vote in the "Best Individual Editorial Feature" category here: vote.webbyawards.com/PublicVoting...
vote.webbyawards.com
Vote for the best of the internet
I just voted in The Webby People's Voice Awards and checked my voter registration.
0103
Reposted by Library Innovation Lab
Greg Leppert @leppert.me · 12/03/2025
As the @institutionaldatainitiative.org expands its mission, we’re announcing a collaboration with @bpl.boston.gov to develop AI-driven tools capable of accelerating new digitization at libraries across the world, starting at the Boston Public Library. institutionaldatainitiative.org/posts/using-...
institutionaldatainitiative.org
Using AI to Accelerate Digitization at Boston Public Librarys
Today, as part of our mission expansion, we’re announcing a collaboration with BPL to develop AI-driven tools capable of accelerating new digitization of large collections at libraries across the worl...
11710
Reposted by Library Innovation Lab
Greg Leppert @leppert.me · 05/03/2025
I'm pleased to announce we're expanding our mission at the @institutionaldatainitiative.org with an open call for institutional collaborators, new digitization at Harvard Law School Library, and additional support to advance this work. institutionaldatainitiative.org/posts/open-c...
institutionaldatainitiative.org
Expanding Our Mission: An Open Call for Collaborators
Today, we’re pleased to announce an open call for institutional collaborators as new support expands the research capacity of the Institutional Data Initiative.
1119
Library Innovation Lab @harvardlil.bsky.social · 19/02/2025
Ed Summers at Stanford wrote this great deep dive of how and why we designed our data.gov archiver the way we did. Thanks for digging in, Ed, this is excellent. inkdroid.org/2025/02/17/n...
inkdroid.org
Bagging data.gov
1186
Library Innovation Lab @harvardlil.bsky.social · 06/02/2025
We just launched a 16TB archive of every dataset that has been available on data.gov since November. This will be updated day by day as new datasets appear. It can be freely copied, and we're sharing the code behind it to help others make their own archives of data they depend on.
lil.law.harvard.edu
Announcing the Data.gov Archive | Library Innovation Lab
Today we released our archive of data.gov on Source Cooperative. The 16TB collection includes over 311,000 datasets harvested during 2024 and 2025, a complet...
4219011000
Reposted by Library Innovation Lab
Lynda M. Kellam #DataRescue @lyndamk.bsky.social · 03/02/2025
Penn is getting a lot of questions about Data Refuge. That effort no longer exists, but several efforts are currently active. I've created a doc from what I & others have suggested. I'll update as I hear more. Feel free to share or suggest: docs.google.com/document/d/1...
docs.google.com
Data Rescue Efforts
Data / Website Rescue Efforts End of Term Crawl - The main coordinated effort to archive websites, but datasets have been more of a challenge. EDGI - They have been focused on environmental data. A ...
37643
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
What are we all missing? Anything you can't get by clicking from link to link like EOT, or downloading datasets directly from data.gov. If there's things you care about preserving that fit that description, that's where to focus.
1101
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Another limited but vital effort is @eotarchive.org. They collect a huge amount from .gov domains every four years, and make it discoverable through @archive.org
eotarchive.org
End of Term Web Archive
The End of Term Web Archive is a collaborative initiative that collects, preserves, and makes accessible United States Government websites at the end of presidential administrations.
0184
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Our collection from data.gov is limited: if an entry points directly to the data, such as a csv, we have the data. If it points to an html landing page, we just have the landing page. This means many, many datasets are not included. What we have from data.gov adds up to 15 or 20TB.
data.gov
Data.gov Home - Data.gov
1182
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Speaking of telling someone, here’s our update: we have copies of all metadata from data.gov, and all of the dataset URLs it points to (shallow crawl); all federal Github repositories with issues, comments, etc.; and articles from PubMed.
data.gov
Data.gov Home - Data.gov
1160
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Third, tell someone. Archive.org is one good place to store public data for discovery, and we at LIL will consider storing and signing data in some cases as well. Just posting data somewhere search engines can find is good too.
archive.org
Internet Archive: Digital Library of Free & Borrowable Texts, Movies, Music & Wayback Machine
0132
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
FOIA requests are another great way to scale up — check out @muckrock.com to get started.
0101
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Next, scale up. If you’re a programmer (or can team up with one), write a python script to download a full collection — say, everything from the data portal of a given government website. Run it yourself, and share it so we libraries can use it too.
0112
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
If you’re a data scientist, good news — your work isn't just downloading data and publishing about it, but also keeping safe copies!
090
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
To keep access to stuff you care about: first just make a copy. Use ArchiveWeb.page to click around and download all the parts of a website you’re interested in. We like the desktop version to avoid capturing login cookies or extensions, but the browser extension is good too.
archiveweb.page
ArchiveWeb.page
0124
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Public data gets taken down all the time. Everyone needs to understand that there are no complete copies — it belongs to all of us, and we paid for it, but it is too large for anyone to copy. This is why libraries work together to preserve stuff.
0191
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Why trust us? We’re the web archiving folks behind Perma.cc and the open data folks behind Case.law. Starting late last year we moved into collecting online government datasets, since it can be hard to get the datasets researchers need from web archives.
perma.cc
Websites change. Perma Links don't.
Perma.cc helps scholars, journals, courts, and others create permanent records of the web sources they cite.
03110
Library Innovation Lab @harvardlil.bsky.social · 31/01/2025
Last year we started a project to download and preserve public data. lil.law.harvard.edu/blog/2025/01... Since saving public data is in the news today — but is always needed — let’s talk about what you can do to help.
lil.law.harvard.edu
Preserving Public U.S. Federal Data | Library Innovation Lab
17420193
Library Innovation Lab @harvardlil.bsky.social · 22/01/2025
Could a tool like this help you overcome both language and knowledge barriers when exploring large collections of information? How might LLMs help people access and understand legal information that is either in a foreign language or requires specialized knowledge?
000
Library Innovation Lab @harvardlil.bsky.social · 22/01/2025
In this case study, @matteocargnelutti.dev and @kristimukk.bsky.social (in collaboration with Betty Queffelec, University of Western Brittany) investigate how such a tool might help non-French speakers of varying expertise ask questions in English to explore French law.
100
Library Innovation Lab @harvardlil.bsky.social · 22/01/2025
What insights emerge when a librarian, a software engineer, and a legal scholar come together to experiment with Retrieval Augmented Generation (RAG) to explore over 800,000 French legal articles 🇫🇷? Blog post: lil.law.harvard.edu/blog/2025/01... Case study: lil.law.harvard.edu/open-french-...
Title "Open French Law RAG" overlaid on llama dressed in a French lawyer robe in front of an abstract background.
123
Reposted by Library Innovation Lab
katie mackinnon @ktmac.bsky.social · 19/12/2024
I’ve been thinking so much lately about data storage, formats, inscription, archives & deep time for the #dataloss project. So cool to see this piece come out from @harvardlil.bsky.social on CENTURY-SCALE STORAGE by @maxy.bsky.social (ty @louravn.bsky.social!) lil.law.harvard.edu/century-scal...
lil.law.harvard.edu
Century-Scale Storage
If you had to store something for 100 years, how would you do it?
1145
Library Innovation Lab @harvardlil.bsky.social · 13/12/2024
"We picked a century scale because most physical objects can survive 100 years in good care. It is attainable, and yet we selected it because the design of mainstream digital storage mediums are nowhere close to even considering this mark." lil.law.harvard.edu/century-scale-storage
lil.law.harvard.edu
Century-Scale Storage
If you had to store something for 100 years, how would you do it?
064
Library Innovation Lab @harvardlil.bsky.social · 12/12/2024
lil.law.harvard.edu/blog/2024/12...
lil.law.harvard.edu
Library Innovation Lab Announces the Launch of Institutional Data Initiative to Expand Access to Knowledge | Library Innovation Lab
Cambridge, MA – Harvard Law School’s Library Innovation Lab (LIL) is proud to announce the official launch of the Institutional Data Initiative (IDI), a grou...
100
Library Innovation Lab @harvardlil.bsky.social · 12/12/2024
More news! @institutionaldatainitiative.org officially launched today. This massive effort was conceived at LIL by Greg Leppert, who's now leading the charge as executive director. Our mission to bring library principles to technological frontiers is embedded in IDI's purpose. Read more below!
132
Reposted by Library Innovation Lab
Jared Holt @jaredlholt.bsky.social · 11/12/2024
This is such a cool piece and presented in a beautiful format
1165
Library Innovation Lab @harvardlil.bsky.social · 11/12/2024
Happy to be here! 👋
010
Reposted by Library Innovation Lab
Maxwell Neely-Cohen @maxy.bsky.social · 11/12/2024
Here it is, the longest piece of nonfiction I’ve ever written, CENTURY-SCALE STORAGE. The kind folks at @harvardlil.bsky.social let me investigate the present and future of digital storage, and how we steward and preserve our most treasured cultural objects lil.law.harvard.edu/century-scal...
lil.law.harvard.edu
Century-Scale Storage
If you had to store something for 100 years, how would you do it?
1418572