blog.khinsen.net
Knowledge refinement in the digital era
Knowledge refinement is the ever ongoing process in science (and beyond it) that shepherds knowledge from lab notebooks into journal articles and then on to review articles, monographs, reference handbooks, university textbooks, and finally professional domain expertise and school education for a wider public. It has been going on for a few centuries, but we hardly talk about it. In fact, I made up the term because I couldn't find an established one. Computational knowledge has not yet found its place in the knowledge refinement process. Why not? And what can we do to make it happen?
First of all, what is the role of knowledge refinement?
One answer is that knowledge becomes more reliable in the refinement process. Any single experiment, no matter how carefully done, can turn out to be wrong for lots of reasons. In contrast, the synthesis of hundreds of independent related experiments will remain trustworthy even if one flaky input experiment has to be withdrawn.
Another role of knowledge refinement is making knowledge more accessible. When we learn from others, rather than from our own experience, we start with a highly refined view of a domain, and dive into the details only later and when we see a need for it. Schools teach only refined knowledge, which thus shapes general culture.
Finally, with refinement, knowledge also becomes more long-lived. A useful picture here is pace layers: each refinement step pushes knowledge down into a more slowly evolving layer.
Each knowledge refinement step has several interrelated aspects:
* Summarization. A large number of works from the preceding step are cooked down into a common essence that emerges from them.
* Curation. Domain experts select what they see as most relevant, discarding what they consider technical details, or only of historical importance.
* Integration. Each step draws on broader domain coverage than the preceding one. The relation to already established knowledge is made clearer.
* Simplification and clarification. Each step is written for a wider and less specialized audience than the preceding one. Jargon is replaced by more widely understood vocabulary. New concepts are introduced and the explanation is structured around them.
All these processes require reformulation. The output of a refinement step is not simply an annotated collection of quotes from its inputs. You can't edit a journal article into a chapter for a university textbook.
In the digital era, it has become widely accepted that software and electronic datasets are representations of scientific knowledge. See for example pages 9 to 11 of the UNESCO Recommendation on Open Science, which discuss "open scientific knowledge". For now, however, this is mostly wishful thinking. While code and data certainly are scientific _information_ , it is difficult to consider them to be _knowledge_ as long as they don't participate in the knowledge refinement process.
Let me illustrate this with analogies from pre-digital science. The predecessors of today's digital datasets are tables of observations recorded in lab notebooks. The predecessors of today's scientific software are mathematical equations. Consider the foundational research on electromagnetism that was done in the 19th century. Examples of original sources are the lab notebooks of Michael Faraday, or on the theoretical side James Clerk Maxwell's paper "A Dynamical Theory of the Electromagneticm Field" (already a refined version of his personal notes, which are probably lost). Today's textbooks in physics or electrical engineering mention both Faraday and Maxwell in historical overviews, but they don't ask you to consult their original work, nor do they cite passages from it in their pedagogical presentation. Instead, they show highly refined versions of Maxwell's theory in modern vector notation, which was largely created for this purpose. The textbooks also summarize the kind of experiments done by Faraday and many others, with pictures and diagrams of modern rather than historical instruments and experimental setups. Instead of real data, they show plots of idealized synthetic data for illustration.
Whereas nobody would seriously consider publishing a lab notebook as a journal article after just some light editing, many researchers seem to believe that computational notebooks, such as the popular Jupyter notebooks, could replace journal articles. However, the history and practice of Jupyter shows that it is by design a tool for interactive computational exploration. A Jupyter notebook is an annotated chronological record of a single computation. There is no way to summarize multiple computations. There is no way to focus on the essential aspects of a computation while relegating implementation details to an appendix. There is no way to explain, in more than words, algorithms from a library that is used in the notebook. Moreover, the order of presentation is constrained by the flow of the computation. In a typical Jupyter notebook, one of the very first lines says `import matplotlib`, a technical detail that in a journal article should be relegated to a reference list, because it is irrelevant to the scientific message.
If you look at today's published Jupyter notebooks, most of them are complements to journal articles. The play the role of an appendix, a repository for technical details. Notebooks that are independent publications do exist, for example in the Journal of Digital History, but the computations they contain, basic data analysis and visualization, are simple and not the main topic of the paper. Moreover, the Journal of Digital History has developed its own extensions to Jupyter that add support for hiding technical details from view while reading the article.
While I focused on Jupyter because of its popularity, other notebook technologies such as Quarto or Emacs' Babel extension to org-mode are not fundamentally different. As long as the executable content is a single computation that runs from top to bottom, the above criticisms remain valid.
Various ideas for improving the publishing technology for code are being explored. One of them is removing the linear execution constraint, e.g. by using the dataflow paradigm. A few notebook systems have adopted this approach, most notably Observable. The most interesting system I have seen so far in this space is Lopecode, which adds a hypertext structure to Observable's dataflow model for computation. My own experiment on supporting the publication of refined digital knowledge is HyperDoc, which focuses on (1) integrating code and its execution into hypermedia networks and (2) building bridges to existing hypermedia systems. It has the advantage of being applicable to existing code in any programming language (though this requires significant implementation work that hasn't been done yet), and thus avoids a complete break with the past six decades of computational research.
Much more remains to be done, however, before computational knowledge can become an integral part of the refinement process. Computational knowledge is most of all _formalized_ knowledge, meaning knowledge expressed in terms of a formal system that can be processed mechanically, by a computer. Refinement techniques for formalized knowledge are constrained by the formal system they depend on. Program source code lives in a formal system defined by a programming language. It is processed mechanically by a compiler or an interpreter. The focus of a programming language is, not surprisingly, the construction of executable programs. There is a well-established refinement process for source code: scripts become library modules, which are complemented by tests and documentation and then published to become off-the-shelf ingredients for constructing more complex programs. Like knowledge refinement, code refinement makes code more reliable, more accessible, and long-lived, but only from the "code as a tool" perspective. It tends to reduce rather than augment its value as a knowledge repository. Software engineering priorities such as optimization and reducing redundancy tend to make the code less readable for researchers who are not professional software developers, and impenetrable to anyone who is just starting to explore a new domain of knowledge.
This is why knowledge refinement in computational science remains limited to non-formal language. Textbooks remain printed books (or their digital twins, PDF files), merely referring to and writing about machine-readable artifacts such as code. The textbooks discuss concepts, including algorithms and data structures, but don't provide examples that readers could run and then build on in their own explorations. On the code side, there are often highly refined implementations for use as off-the-shelf tools, but their correspondence to what the textbooks say is unclear and not verifiable by the reader.
There has been work on improving the explanation side, for example Bret Victor's explorable explanations that integrate interactive exploration tools into formerly passive textbook material. However, the strict divide between explanation and code for use in practice remains.
This state of the art is tolerable for low-complexity computational knowledge, e.g. in linear algebra or signal processing. A textbook on Fourier transforms and good FFT library are good enough for learning these methods. But we need something better for complex models and methods.
Consider computational models of complex systems such as force fields for biomolecules, or climate models. On the machine side, they are implemented as software modules for use in simulation, using standard programming languages. On the human side, there are papers describing the principles behind their design, but lacking many details because these details would be overwhelming to a human reader. So if you really want to know how such models work, your only choice is wading through enormous amounts of source code, taking notes carefully of the findings that are relevant to your question. And if you ever want to _compare_ two such complex models... good luck exploring two large software packages in parallel. Unfortunately, that's something you can't avoid doing during knowledge refinement. For example when you write a review paper on protein force fields, or on climate models.
When models become too complex to be represented in traditional scientific notations (equations, diagrams, etc.), they need to be represented in _digital_ scientific notations. They need to be represented in formal systems that are made for documentation and exploration, rather than for plugging into a simulator. And they require tools that treat them as _data_ , not _code_.
An idea I have been pursuing for a while is the use of specification languages as digital scientific notations. See my project Leibniz for the details. Specification languages are more general than programming languages in the same way that mathematical equations are more general than mathematical functions (see here for a discussion). In particular, specification languages do not impose an executability condition on everything you write. This allows you to write incomplete function definitions or data structures. I am working on this idea and I remain optimistic that it will work out, but I won't declare victory before I have _actually_ written that review of biomolecular force fields. Meanwhile, I'd be happy to see other work in this space, ideally pursuing very different ideas.
Digital scientific notations are a necessary first step for enabling the refinement of digital scientific knowledge, but the real works starts once we have them. We must then develop the tools that let researchers explore and examine publications containing formalized knowledge. We must also find refinement strategies that actually work in practice. There remains a lot to be done!
