Reposted by Aaron Sterling
this finding matches my experience: the valuable thing that knowledgeable human devs can bring to a project, that the agents aren't yet good at, is ontology creation.
Aaron Sterling
@aaronsterling.bsky.social
742 followers 2K following 2K posts
CEO, Thistleseeds. Personal account. Current primary project: tech for substance use disorder programs.
![It is worth pausing for a moment to review ontology creation by humans and by LLMs, because there is empirical data that might look contradictory at first glance, but, in fact, paints a unifying
picture. LLMs are not as good as humans at ontology creation (sometimes called “ontology learning”), as shown in [4, 15, 6]. However, at least according to the OntoURL benchmarks,
LLMs are better than humans at reasoning over an ontology that already exists[29].
Despite the previous results, the quality of LLM-generated ontologies can be higher than
the quality of ontologies created by novice human engineers[19]. This is not a contradiction,
because an LLM’s ability to 1-shot ontology creation is directly related to how completely
humans already ontologized the space through documentation. The need for humans to pre-
ontologize the space can be seen in [21], which presented LLMs with well-structured gibberish,
and the LLMs were unable to ontologize the gibberish, showing an inability to reason over
semantic relations between concepts.
One goal of Ontology-First Agent Design is to focus human expert input where it is most
needed: creation of the ontology (at the start), and refinements to the ontology to improve the
system’s functionality (a feedback loop at the end). The LLM does the work in the middle,
where it is most effective.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:qnnbtstx3a4edcwu2jkz4h7d/bafkreib4fapxwu7nvriftegh7j3fxk72skewqi5i65bdqtfo5xdbp3xkiu)
