Sign in

Python4DataScience

@python4data.science
4.8K followers 4 following 109 posts

Teaching materials for the cusy training courses on a Python-based data science workflow: cusy.io/seminars

PostsRepliesMedia
Python4DataScience @python4data.science · 29/09/2026
Git 2.56 introduced a number of exciting features that improve the way we work with merges: 'git add --resolved' and 'git merge-base'. www.python4data.science/en/latest/pr... www.python4data.science/en/latest/pr... #Git #DeveloperExperience #DX
python4data.science
Git branches
$ git branch [-a] [-l " GLOB_PATTERN"], shows all local branches in a repository.--a, also shows all removed branches.,,-l, restricts the branches to those that correspond to a specific p...
012
Python4DataScience @python4data.science · 23/09/2026
Having first taken a closer look at CSVW for table dialects and schemas, we have now examined data resource and data package formats in detail, so that we can publish and cite data more effectively: www.python4data.science/en/latest/pr... #Python #DataScience
python4data.science
Cite data
Similar to the way in which researchers routinely provide bibliographical references to other academic sources, data should also be cited. Although data is frequently shared, it is not always cited...
021
Python4DataScience @python4data.science · 21/09/2026
CSVW is a metadata standard for CSV files. If it becomes widely adopted, it will make processing CSV files so much better: www.python4data.science/en/latest/da... #Python #DataScience
python4data.science
CSV
Overview:,,, Data structure support,--, CSV is used to store tabular data, but unlike other serialisation formats reviewed here, it’s not suitable for (nested) objects.,, Standardisation,--, CSV is...
021
Python4DataScience @python4data.science · 18/09/2026
We have significantly revised the section on data validation and cleansing and expanded it to include data quality criteria and processes: www.python4data.science/en/latest/cl... #Python #DataScience
python4data.science
Cleaning and validating data
‘Garbage in, garbage out’ is a stark reminder that it is almost impossible to draw meaningful insights from poor-quality data. Whilst there are highly regulated, safety-critical sectors in which da...
021
Python4DataScience @python4data.science · 03/09/2026
We have updated the section on text comparison – featuring difflib, TheFuzz and textacy: www.python4data.science/en/latest/cl... #Python
python4data.science
String comparisons
difflib: You can compare strings using the get_close_matches option of Python’s built-in difflib library. If you want to specify the minimum match score required for a suggestion to be displayed, y...
011
Python4DataScience @python4data.science · 18/08/2026
At first, it can sometimes be difficult to imagine a practical use for decorators. We will therefore now present a range of different examples: python-basics-tutorial.readthedocs.io/en/latest/fu... #Python #Decorators
python-basics-tutorial.readthedocs.io
Decorators
Functions can also be passed as arguments to other functions, and the results of other functions can be returned. For example, it is possible to write a Python function that takes another function ...
021
Python4DataScience @python4data.science · 11/08/2026
Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: www.python4data.science/en/latest/pe... #Python #pandas #DataFrame
python4data.science
Parallelise pandas
In Enhancing performance, some possibilities are described for improving the performance of pandas. However, there are also special libraries that can parallelise the processing of data frames. cuD...
011
Python4DataScience @python4data.science · 05/08/2026
We have expanded the section on mocks for testing and added spies: python-basics-tutorial.readthedocs.io/en/latest/te... #Python #Testing
python-basics-tutorial.readthedocs.io
Mock
In this chapter, we will test the CLI. To do this, we will use the mock library, which has been included as part of the standard Python library since Python 3.3 under the name unittest.mock. Object...
051
Python4DataScience @python4data.science · 03/08/2026
• How can you make your code more secure? • How can you ensure separation of concerns? • How can you continuously verify the security of your dependencies? • How do you create SBOMs? www.python4data.science/en/latest/pr... #Python #SupplyChain #Vulnerability #SBOM
python4data.science
Security
As of July 2026, PyPI hosts over 750,000 packages, and this number is growing daily. An average Python project includes dozens of additional dependencies – packages that you have never explicitly c...
031
Python4DataScience @python4data.science · 03/08/2026
🧵 1/2 Attacks on the software supply chain are increasing rapidly. We have therefore updated our guide on how to make your projects more secure, adding answers to the following questions:
python4data.science
Security
As of July 2026, PyPI hosts over 750,000 packages, and this number is growing daily. An average Python project includes dozens of additional dependencies – packages that you have never explicitly c...
121
Python4DataScience @python4data.science · 24/07/2026
I agree with you. Recently, a car manufacturer listed a Trojan in its SBOM. Nevertheless, SBOMs can be a first step towards securing your own project. In the ‘Analysis’ section, we described how SBOMs can be used to detect malware: www.python4data.science/en/latest/pr...
python4data.science
Software Bill of Materials (SBOM)
A Software Bill of Materials (SBOM) is a document used to exchange information about software and its composition. This format is primarily used in the security field to check software and its depe...
100
Python4DataScience @python4data.science · 24/07/2026
Did you know that with shot-scraper video you can explain your user interface using a storyboard: python-basics-tutorial.readthedocs.io/en/latest/do... #Python #Docs #Video
python-basics-tutorial.readthedocs.io
shot-scraper
shot-scraper is a tool to automate the process of updating screenshots. Installation: Use: shot-scraper can be used in several ways …for single screenshots on the command line: …or with additional ...
010
Python4DataScience @python4data.science · 22/07/2026
Software Bill of Materials (SBOMs) are commonly used to check software and its dependencies for vulnerabilities using CVE and OSV. We have described how to create these for Python packages here: python4data.science/en/latest/pr... #Python #SBOM #Vulnerability
python4data.science
Software Bill of Materials (SBOM)
A Software Bill of Materials (SBOM) is a document used to exchange information about software and its composition. This format is primarily used in the security field to check software and its depe...
121
Python4DataScience @python4data.science · 20/07/2026
Specification-Driven Development (SDD) was developed to improve the predictability and maintainability of agentic software development. In particular, we are continuing to give OpenSpec a chance: agentic-software-engineering.readthedocs.io/en/latest/pr... #CodingAgent #SDD #DX
agentic-software-engineering.readthedocs.io
Spec-Driven Development
Development teams are increasingly facing challenges regarding predictability and maintainability when requirements and context are stored exclusively in short-lived chat threads. To address this i...
041
Python4DataScience @python4data.science · 15/07/2026
We are switching from pre-commit to prek. Whilst prek is also faster, the decisive factors were the security-related features: pinned SHA revisions and cooldown: www.python4data.science/en/latest/pr... #Python #DX #DevOps #DevSecOps #Security
python4data.science
prek
prek is a variant of pre-commit, written in Rust, for managing and maintaining multilingual commit hooks. An essential task is to make the same scripts available to the entire development team. pre...
111
Python4DataScience @python4data.science · 13/07/2026
We have added the `errors` argument to the instructions for reading and writing files, so that files with encoding issues can be read more effectively. And, fortunately, UTF-8 becomes the default in Python 3.15: python-basics-tutorial.readthedocs.io/en/latest/sa... #Python
python-basics-tutorial.readthedocs.io
Files and Directories
pathlib implements path operations using pathlib.PurePath and pathlib.Path objects. The os and os.path modules, on the other hand, provide functions that work at a low level with str and bytes, whi...
031
Python4DataScience @python4data.science · 09/07/2026
In our experience, data scientists have been using coding agents for several years now. We aim to provide them with a secure development environment featuring sandboxing and appropriate rules.
110
Python4DataScience @python4data.science · 08/07/2026
We have released a new version. As well as the usual software updates, it includes agent-based code generation and securing the software supply chain: www.python4data.science/en/26.1.0/ #Python #DataScience #AgenticCoding #Claude #SoftwareEngineering #SoftwareSupplyChain #SupplyChain
python4data.science
Python for Data Science
This is a tutorial on Data Science with Python. This immediately raises the question: What is Data Science? The term has become ubiquitous, but there is no single definition. Some even consider the...
1102
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 25/06/2026
#TIL that there is also a pre-commit check for uv audit: www.python4data.science/en/latest/pr... #Python #CVE #Vulnerability #Git #ITSecurity #SupplyChain
python4data.science
Security
In previous chapters, we have already provided some tips designed to help ensure safer operation. Here, we would like to summarise and expand on the individual elements once again. We will be using...
021
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 23/06/2026
Guardrails enable you to defend against prompt injection or jailbreak attacks by your coding agents. In our tutorial, we show you the options available to you for this purpose: agentic-software-engineering.readthedocs.io/en/latest/se... #PromptInjection #Jailbreak #AI #Agentic #Coding #ITSecurity
agentic-software-engineering.readthedocs.io
Guardrails
The term AI guardrails first appeared in 2023, when ChatGPT launched its consumer service. The initial versions contained numerous security and privacy vulnerabilities, which ultimately prompted ma...
041
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 22/06/2026
We have added a section on @tiangolo.com’s Library Skills to our agent-based software engineering tutorial: agentic-software-engineering.readthedocs.io/en/latest/sh... #AI #Agentic #Coding #Skills
agentic-software-engineering.readthedocs.io
Skills
As coding agents evolve from simple chat interfaces towards autonomous task execution, context engineering has become a critical challenge. Agent Skills provide an open standard for modularising co...
452
Python4DataScience @python4data.science · 11/06/2026
We have expanded our tutorials to include answers to the question of how coding agents can assist data scientists with programming: • python-basics-tutorial.readthedocs.io/de/latest/li... • github.com/veit/python-... #Python #DataScience #CodingAgents #Claude #Cursor
Note

We also use static code analysis and linting for the generated code in agent-based software development:

AGENTS.md
# Code quality and linting
- Use ruff, ty, prek, wily for code quality and linting.
- Run appropriate tooling after making changes to your code to ensure it meets quality standards.

See also: Code quality and linting
172
Python4DataScience @python4data.science · 10/06/2026
With uv≥0.11.19, you can check dependencies in your project for known vulnerabilities: • python-basics-tutorial.readthedocs.io/en/latest/li... • www.python4data.science/en/latest/pr... #Python #uv #Vulnerability #ITSecurity
python4data.science
Security
In previous chapters, we have already provided some tips designed to help ensure safer operation. Here, we would like to summarise and expand on the individual elements once again. We will be using...
051
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 01/06/2026
If uv is not available in a Python environment, you can still specify the versions using pylock.toml. We have described how to do this here: python-basics-tutorial.readthedocs.io/en/latest/li... #Python #DevSecOps #ITSecurity
python-basics-tutorial.readthedocs.io
Adding more Python libraries
Although Python’ „Batteries included“ philosophy means that you can already do a lot with the default installation of Python, there will inevitably come a situation where you need functionality tha...
031
Python4DataScience @python4data.science · 24/05/2026
Thank you very much for pointing that out. The link has now been fixed.
010
Python4DataScience @python4data.science · 22/05/2026
pygame is suitable for developing multimedia applications, whilst pytest can be used universally to test libraries and applications automatically. It is used by most well-known Python libraries.
010
Python4DataScience @python4data.science · 22/05/2026
Thanks to Ruff, our Python4DataScience tutorial is now much more consistent: www.python4data.science/en/latest/ #Python
python4data.science
Python for Data Science
This is a tutorial on Data Science with Python. This immediately raises the question: What is Data Science? The term has become ubiquitous, but there is no single definition. Some even consider the...
131
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 07/05/2026
Yesterday, the article by Frank Hofmann and me on open licences for documents, images, audio, video, fonts and hardware was published in Linux Magazine: www.linux-magazine.com/Issues/2026/... #OpenSource #OpenHardware
linux-magazine.com
Keep It Free » Linux Magazine
Linux users associate open source licenses with software, but a bevy of licenses are available for documents, images, audio/video, fonts, and hardw...
052
Python4DataScience @python4data.science · 24/04/2026
We have integrated two features from Git 2.54: • Git history: www.python4data.science/en/latest/pr... • Configuration-based Git hooks: www.python4data.science/en/latest/pr... #Git
python4data.science
Rewriting history
There are several commands in Git for rewriting history. git rebase -i is the best known and most flexible: you can reorder, merge, edit and remove commits. However, this flexibility comes with a d...
031
Python4DataScience @python4data.science · 23/04/2026
The testing section has been extensively revised: • Test examples for cusy.tasks • Added Hypothesis extensions • Agile software development using test-driven development and behaviour-driven development python-basics-tutorial.readthedocs.io/en/latest/te... #Python #pytest #Agile #TDD #BDD
python-basics-tutorial.readthedocs.io
Testing
Basically, a distinction is made between static and dynamic test procedures. Static test procedures, are used to check the source code, although this is not executed. They are divided into- reviews...
021
Python4DataScience @python4data.science · 16/04/2026
Today is our last day at PyConDE (@pyconde.bsky.social). We’re looking forward to having lots more interesting conversations with you, whether there are any new developments in research software using Python or on LLM coding agents #PyConDE #RSE #Python #AI #LLM
011
Python4DataScience @python4data.science · 11/04/2026
Final preparations for PyConDE & PyData next week in Darmstadt. We’d love to see you there. Please feel free to arrange a meeting with us: cusy.io/en/about/eve... @pyconde.bsky.social #Python #PyData
cusy.io
021
Python4DataScience @python4data.science · 07/04/2026
We have analysed the attacks on the software #SupplyChain of LiteLLM and Telnyx and now recommend dependency cooldowns alongside immutable references, trusted publishers and digital attestations to secure deployments: • python-basics-tutorial.readthedocs.io/en/latest/pa... #Python #ITSec #DevSecOps
python-basics-tutorial.readthedocs.io
Publish package
Finally, you can deploy the package on the Python Package Index( PyPI) or another index, for example GitLab Package Registry or devpi. For the Python Package Index, you must register with Test PyPI...
011
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 26/03/2026
In future, Open Research Europe (ORE) will no longer be open solely to projects funded under the EU Programmes for Research; instead, all research institutions in the participating countries will be able to use the platform free of charge: research-and-innovation.ec.europa.eu/news/all-res...
research-and-innovation.ec.europa.eu
A new era for Open Research Europe
European Commission to co-fund new phase of operation for open access publishing platform.
054
Python4DataScience @python4data.science · 23/03/2026
We have now significantly expanded our collection of open data and organised it by topic: www.python4data.science/en/latest/da... #Python #DataScience #OpenData #Agriculture #Biology #Chemistry #Climate #Weather #ComputerNetwork #Energy #Finance #GeoScience #Healthcare #ImageProcessing #Medicine
python4data.science
Open data
A topic-based overview of public repositories containing research data. Agricultural sciences: AQUASTAT Dissemination System, Global information system of the Food and Agriculture Organization of t...
031
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 12/03/2026
The German Research Association (@dfg.de) has established a binding framework for the use of AI. The aim is to ensure the legally compliant and transparent use of AI: www.dfg.de/en/news/news... #Research #RSE #Science #DataScience #ArtificialIntelligence #AI #Compliance
dfg.de
Artificial Intelligence in the Review Process
121
Python4DataScience @python4data.science · 11/03/2026
We are a community sponsor and would love to meet you at the conference. #Python #DataScience
020
Python4DataScience @python4data.science · 03/03/2026
… and now we have also moved the pytest configuration to the pyproject.toml file: python-basics-tutorial.readthedocs.io/en/latest/te... #Python #Testing #pytest
python-basics-tutorial.readthedocs.io
Configuration
You can use configuration files to change the way pytest runs. If you repeatedly use certain options in your tests, such as--verbose or--strict-markers, you can store them in a configuration file s...
021
Python4DataScience @python4data.science · 26/02/2026
Since tox 4.44.0, tox.ini has been frozen. Only the toml configuration is supported for further development. We have therefore updated our tutorial accordingly: python-basics-tutorial.readthedocs.io/en/latest/te... #Python #testing #tox
python-basics-tutorial.readthedocs.io
tox
tox is an automation tool that works similarly to a CI tool, but can be run both locally and in conjunction with other CI tools on a server. In the following, we will set up tox for our Items appli...
121
Python4DataScience @python4data.science · 18/02/2026
We are pleased that we were able to develop an idea on how to better deal with LLM agents in the future.
020
Python4DataScience @python4data.science · 11/02/2026
We analyse the memory consumption of our applications with memray and also monitor it continuously with pytest-memray: python4data.science/en/latest/pe... #Python
python4data.science
Memray
Memory usage is difficult to control in Python projects because the language does not explicitly indicate where memory is allocated, module imports can significantly increase consumption, and it is...
010
Python4DataScience @python4data.science · 09/02/2026
Git 2.53 provides faster insights into the repository structure with 'git repo structure'. However, to better understand this, it is helpful to be more familiar with the Git data and storage models: www.python4data.science/en/latest/pr... #Git
python4data.science
Git Internals
So far, we have looked at how you can use Git to manage the different states of your code. Now we want to show you the data and storage models that underlie Git. Data Model: You will be able to use...
021
Python4DataScience @python4data.science · 19/01/2026
The section on performance measurements and finding bottlenecks has been significantly expanded to include cProfile/profiling.tracing, tprof, and profiling.sampling/Tachyon: www.python4data.science/en/latest/pe... #Python #Performance
python4data.science
Performance
Python can be used to write and test code quickly because it is an interpreted language that types dynamically. However, these are also the reasons it is slow when performing simple tasks repeatedl...
042
Python4DataScience @python4data.science · 16/01/2026
We have updated the documentation section with references to README, CONTRIBUTING, CHANGELOG, etc. python-basics-tutorial.readthedocs.io/en/latest/do... #Python #Documentation
python-basics-tutorial.readthedocs.io
Document
In order for your software package to be useful, documentation is required that describes how your software can be installed, operated, used and improved: Those who want to use your package need in...
041
Python4DataScience @python4data.science · 07/01/2026
We have updated the FastAPI extensions. It is very surprising to us that millions of extensions are being downloaded that have not been updated for over a year. www.python4data.science/en/latest/da... @fastapi.tiangolo.com #Python #FastAPI #REST
python4data.science
Extensions
Administration: SQLAlchemy Admin for Starlette/FastAPI, Flexible admin interface for SQLAlchemy models. Downloads Contributors Commit activity Licence,, Piccolo Admin, Simple but powerful admin int...
031
Reposted by Python4DataScience
Veit Schiele @veit-schiele.de · 21/12/2025
I took a look at the changes coming with Python 3.15 – and I can't wait to put them to productive use. I've already updated our tutorials: • Performance measurements: www.python4data.science/en/latest/pe... • Tachyon: www.python4data.science/en/latest/pe... #Python
python-basics-tutorial.readthedocs.io
Unicode and character encodings
Special characters and escape sequences:\n stands for the newline character and\t for the tab character. Character sequences that begin with a backslash and are used to represent other characters a...
152
Python4DataScience @python4data.science · 17/11/2025
We have updated the section on pytest with many exciting use cases * on command line options * on generating markers * and on parameterising exceptions python-basics-tutorial.readthedocs.io/en/latest/te... #Python #Testing #pytest
python-basics-tutorial.readthedocs.io
pytest
pytest is an alternative to Python’s Unittest module that simplifies testing even further. pytest automatically recognises tests based on filenames and functions that start with test_, while unitte...
142
Python4DataScience @python4data.science · 21/10/2025
We have updated our tutorial to data management with DVC. It also allows you to create lightweight data science and data modelling workflows and execute them in a parameterised manner: www.python4data.science/en/latest/pr... #Data #Versioncontrol #Git #DataScience #Modeling #Python
Precision-Recall-Curve comparison between workspace and HEADReceiver operating characteristic (ROC) comparison between workspace and HEADConfusion Matrix comparison between workspace and HEAD
063
Python4DataScience @python4data.science · 25/09/2025
Now we have also described how to use uv reliably for Cursor: www.python4data.science/en/latest/pr... #CursorAI #Python #Packaging #uv
python4data.science
Configuring Claude Code or Cursor for uv
How do we configure Claude Code or Cursor to automatically use uv instead of pip for Python package management? Claude Code Claude Code uses CLAUDE.md files to configure your project’s storage and ...
021
Python4DataScience @python4data.science · 24/09/2025
We have now described how to create a configuration for Claude Code so that it uses uv reliably: python4data.science/en/latest/pr... #ClaudeCode #Python #Packaging #uv
python4data.science
Configuring Claude Code for uv
How do we configure Claude Code to automatically use uv instead of pip for Python package management? Claude Code uses CLAUDE.md files to configure your project’s storage and context, ensuring a co...
130