David Gasquez
@davidgasquez.com
5.4K followers 1.4K following 1.2K posts
Data @ Protocol Labs. Open Data, Open Source, Open Protocols. Walks taker. Progressive Metal enjoyer. davidgasquez.com
Data @ Protocol Labs. Open Data, Open Source, Open Protocols. Walks taker. Progressive Metal enjoyer. davidgasquez.com
.
Products that own your context can force you to use their agent... and, I've never seen a good hosted agent!
Organizations need to own their context, and the way to do so effectively already exists and is well understood.
## Context as Data Infrastructure
Have you noticed the [shape of the diagrams](https://storage.googleapis.com/gweb-cloudblog-publish/images/WorkspaceIntelligence.max-2200x2200.png) that every company is adding to their "intelligence" products? I see that and it seems like I'm looking at a Fivetran or dbt product page in 2018. That is because they're the same thing!
Most organizations will need to build and maintain a model-agnostic knowledge base ([aka ontology, company brain, ...](https://x.com/DBredvick/status/2078150905078206789)) for the same reasons they maintain a data warehouse. A curated and normalized layer helps both humans and agents make sense of all the structure that outlives any particular model, harness, tool, or product. Context engineering is [that same work](https://x.com/JoshARosen/status/2084693306722705629): extracting, filtering, curating, modeling, and publishing artifacts to help the organization make better decisions.
For now, it seems there isn't a great set of tools (or [modern context stack](https://x.com/davidgasquez/status/2082895547677954466)) built for this purpose. Something like Fivetran and dbt optimized for the new sources (Slack, Drive) and transformations (transcription, summarization, text extraction from a slide deck, ...).](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:4z5i7njrld66ew36htufcwry/bafkreib2ouip7e2l64iwylfhe6cuqer4zrye7sajdxqjsaeshxqjid3o4e)
![❯ curl -s "https://huggingface.co/datasets/datania/aemet/raw/main/valores-climatologicos/2025/07/16.json" | jq '.[] | select(.nombre == "GRANADA AEROPUERTO")'
{
"fecha": "2025-07-16",
"indicativo": "5530E",
"nombre": "GRANADA AEROPUERTO",
"provincia": "GRANADA",
"altitud": "560",
"tmed": "30,3",
"tmin": "18,8",
"horatmin": "05:08",
"tmax": "41,8",
"horatmax": "16:09",
"dir": "11",
"velmedia": "2,2",
"racha": "10,3",
"horaracha": "16:30",
"sol": "13,6",
"presMax": "955,3",
"horaPresMax": "Varias",
"presMin": "951,0",
"horaPresMin": "16",
"hrMedia": "29",
"hrMax": "60",
"horaHrMax": "Varias",
"hrMin": "13",
"horaHrMin": "16:02"
}
~](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:4z5i7njrld66ew36htufcwry/bafkreiczhspvkw77o77ksfx7gi33eaaxgmbum2ib3wrxvlrgadjilo2ohe)
![❯ uvx linkweaver --help
usage: linkweaver [-h] [--list-links] [--xml-cat] [-x COMMAND] [-v] [--retries N] [-q] [--dry-run] [--no-color] [-f] [input_files ...]
Extract URLs from markdown files and save each link as individual markdown files in resource folders.
By default, existing files are skipped to save time and bandwidth.
positional arguments:
input_files One or more markdown files to process
options:
-h, --help show this help message and exit
--list-links, -l List all unique URLs found in the files (no downloading)
--xml-cat Concatenate files with their resource folders in XML structure
-x, --exec COMMAND Execute a shell command on the markdown output before saving (e.g., 'llm -t mdclean')
-v, --verbose Enable verbose output with detailed progress information
--retries N Number of retry attempts for failed URL fetches (default: 3)
-q, --quiet Minimize output (opposite of verbose, but still show errors/warnings)
--dry-run Show what would be done without actually doing it
--no-color Disable colored output
-f, --force Force redownload of files even if they already exist
Examples:
linkweaver notes.md # Process file, skip existing resources
linkweaver --force notes.md # Force redownload all resources
linkweaver --list-links notes.md # List all URLs found
linkweaver --xml-cat notes.md # Concatenate with resources
linkweaver -x 'llm -t clean' notes.md # Process with custom command
linkweaver --retries 5 notes.md # Use 5 retry attempts for failed URLs
linkweaver --quiet --dry-run notes.md # Preview actions quietly
linkweaver -q -v notes.md # Quiet mode overrides verbose
linkweaver --no-color notes.md # Disable colored output
Common workflows:
linkweaver --list-links *.md | head -10 # Preview first 10 URLs](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:4z5i7njrld66ew36htufcwry/bafkreibaegi4bgm4cwvnrbe3iqho4lpjlbmzkc4oddwimts74czkishbiq)
. Remove filler words.
- [Principles to keep in mind when writing documentation](https://mkaz.blog/misc/notes-on-technical-writing/).
- The purpose of technical writing is to help users accomplish tasks as quickly and effectively as possible.
"""
)
model = llm.get_model("gpt-4.1-mini")
chain_response = model.chain(
"""
Where is "You’re keen to share your excitement" mentioned in the sources?
""",
tools=[read_url],
# after_call=print,
fragments=[source_note],
system="You are a helpful research assistant. ALWAYS read ALL the links with `read_url`. Once you've readed all the links in the note, help the user with their questions.",
)
for chunk in chain_response:
print(chunk, end="", flush=True)](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:4z5i7njrld66ew36htufcwry/bafkreihatryfaocnskvt7mivayu277khuloa4z7lttblvsgduk4uas5en4)
![Since I’ve been doing Machine Learning projects recently (Kaggle-style competitions), I’ve developed a few extra things I do 1 on those projects.
Document the features you’re currently using and potential features you’d like to explore.
Keep track of open questions and EDA tasks you’d like to explore.
Maintain a DATA_DICTIONARY.md or similar with the terms you’re using. Context is king.
Rely on a functional pipeline style and modular feature engineering. Use “micro pipeline” scripts to generate features. If your dataset has granularity at the “user” level, create different scripts for different kinds of features that generate the feature_name.csv files. You can join them all later. This is one of the things I use a lot. Almost all the features I’ve added to my projects start like “Write a script that computes the user’s […]. Save as feature_name.csv with user, feature_name as headers.”
Track experiments. I’m bad at this one as I haven’t figured out a simple enough way to do this reliably. Ideally, keep track of input features, model parameters, preprocessing, postprocessing, local evaluation, and remote evaluation.
Have a temporary folder (listed in .gitignore) for the LLM to write small scripts and experiments. Here is where you’d have things like inspect_csv.py or other useful scripts that can give the LLMs context on the actual data (e.g., check columns, stats, …)](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:4z5i7njrld66ew36htufcwry/bafkreiep6cwutgspbvgunxvhcn5uiea3ng4eqbjit7f4qlqgiiyeltsroy)
![This code.
import duckdb
c = duckdb.connect()
c.sql("""
CREATE SECRET http (TYPE http, EXTRA_HTTP_HEADERS MAP {'Accept-Encoding': 'gzip'});
""")
failed_tables = []
total_tables = len(tablas)
for i, table in enumerate(tablas):
# Si el directorio no existe, lo creamos
table_dir = ine_dir / "tablas" / str(table["Id"])
table_dir.mkdir(exist_ok=True, parents=True)
# Copiamos el CSV a Parquet
try:
c.sql(f"""
copy (
from read_csv(
'https://www.ine.es/jaxiT3/files/t/en/csv_bdsc/{table["Id"]}.csv',
delim=';',
ignore_errors=true,
normalize_names=true,
null_padding=true,
parallel=true,
strict_mode=false,
compression='gzip'
)
)
to 'ine/tablas/{table["Id"]}/datos.parquet' (
format parquet,
compression 'zstd',
parquet_version v2,
row_group_size 1048576
);
""")
except Exception as e:
print(f"Error processing table {table['Id']}: {str(e)}")
failed_tables.append(table["Id"])](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:4z5i7njrld66ew36htufcwry/bafkreihwsjtfstuirvdsbuwt7h6r7h2idyh3lzqiamqnm3nv3zh7eun6sy)
