Les LLMs et les choses
The material turn comes to AI
Whatever your take on it, the technology we call AI is fundamentally shifting how we process information and interact with computers in our day-to-day lives. Om Malik:
“Artificial intelligence doesn’t just search; it synthesizes, contextualizes, and presents information in a user’s preferred format.“
This shift is having a profound impact, particularly where work is text-based: in education and the humanities. At the same time, it’s becoming evident that the current LLM-based AI is running into constraints. Along with many other experts, Yann LeCun has strongly emphasized this. David William Silva summarizes LeCun’s position as follows:
„Large Language Models are a dead end on the path to human-level intelligence. They are, at their core, text-prediction engines, extraordinarily good at retrieving, recombining, and generating language, but fundamentally incapable of understanding the world they talk about. They lack common sense, causal reasoning, and any model of physical reality (...) no amount of scaling, that is, bigger models, more data, more compute, will ever bridge that gap. To get anywhere near genuine intelligence, AI must go far beyond text and learn from high-bandwidth sensory experience: video, spatial data, interaction with the physical world.“
LeCun is now pursuing the ‘World Model’ approach instead, an alternative that has gained the support of Tim Berners-Lee as well. „Real intelligence“, Le Cun writes, „does not start in language. It starts in the world.“ The goal is to capture and handle “real-world sensor data” that is “unpredictable” and “noisy.” A similar statement can be found on the website of the AI company general intuition: „human intelligence far exceeds language (...) Truly intelligent machines must move from words to worlds“. General intuition’s Pim de Witte and Packy McCormick recently published an extensive guide to World Models. These, the two explain, are “action-conditioned,” because actions compress “how humans respond to the countless variables in their environments.” World models can therefore be defined as “systems that learn from watching the world and the actions taken in it.”
The material turn comes to AI
The transition from words to the world comes as no surprise to cultural theorists, as the ‘world-as-text’ approach established by the ‘semiotic turn’ has already been criticized for its one-sidedness. Context isn’t just text. Culture comprises not only symbols but also material objects and social practices. AI solutions won’t achieve greater accuracy until things and practices can be digitized as well.1
The world-as-bottleneck also serves as the focal point for yet another excellent text from Chris Walker this week. He recalls the extent to which scientific discoveries in the industrial modern era were grounded in “contact with reality” and physical practice—specifically in “hands-on experimental work through which scientists build intuitions and encounter surprises that force deeper reflection.” These avoided the “exploitation trap,” where—as is the case with LLMs—only “low-hanging cross-domain connections in published literature” are established: impressively fast, to be sure, and insightful to a certain extent, but ultimately remaining within “well-explored territory.” Irritating insights from the field that prompt new conceptual frameworks are inevitably ignored.
The world is not enough
In contrast to language models, are world models better suited to navigate messy and dynamic contexts? Beyond romantic assurances and ethical concerns, there are compelling reasons to believe that we are ultimately facing a basic impossibility, notwithstanding several probable advances. Why? Because world or contextual knowledge is frequently ‘implicit’ expertise, which brings with it a series of difficulties for digitalization. First, as far as LLMs are concerned:
Tacit knowledge is not reflected upon by actors and therefore isn’t put into words, which is why it cannot be fed into LLMs.
Even with reflection, it is questionable whether every relevant aspect can be articulated and thereby digitized.
Assuming that were possible, we still would encounter a changed situation with every new ‚application‘ as the field evolves, necessitating the constant input of the latest local expertise.
But also when it comes to world models. Chris Walker asks: „Could you build a surveillance apparatus comprehensive enough to capture all of this? Maybe. Brain-computer interfaces might someday access knowledge that even the knower can’t articulate.“ However, he suggests that „total capture can be worse than incomplete capture,“ because taste and judgment matter: „The challenge here is not codification but curation: selecting and assembling the right bundle of already-codified information at inference time.“
That means we have to address two further issues:
If we assume that dynamic knowledge can be captured in real time via action-based world models, we are still left with the challenge of identifying which knowledge is necessary for a specific task.
Finally, Wittgenstein’s rule-following problem presented itself, meaning that
an ‘abductive’ moment of judicial capacity would endure.2
Watching the AI conversation unfold, I’m increasingly curious about who is actually benefiting from whom. The humanities and social sciences, by handing over administrative tasks as well as the evaluation of enormous amounts of quantitative data to AI? Or developers and engineers who need to upskill in qualitative aspects? Mētis, taste, judgment, context, tacit knowledge, the move from language to practices—and more recently, Peirce’s abduction: So many terms from philosophy, science studies, and humanistic inquiry are gaining sharpness and relevance in the wake of AI debates that well-conceived humanities might even have a bright future.
That said, the current craze for words such as „taste“ will pass more quickly than the skill itself can be mastered. For the cultivation of the irreducibly human demands immersion, tenacity, patience, and consequently, time.
This did not, however, imply that AI had then attained consciousness, mind, intention, understanding, discernment, humor, or genuine creativity.
Walker: „This creates a new kind of knowledge worker: one who has done enough of the underlying work to know what good looks like, and whose value lies in framing the right problems, prioritizing what data needs to be captured, curating and assembling it for AI consumption, and evaluating whether the AI output actually makes sense.“