For the official version of record, see here:

Thylstrup, N. B., & Møller Hartley, J. (2025). “Argh! The World Doesn’t Fit the Model!”: Small Acts of Worldmaking in Data Annotation for News Media. Media Theory, 9(2), 105–132. https://doi.org/10.70064/mt.v9i2.1273

“Argh! The World Doesn’t Fit the Model!”: Small Acts of Worldmaking in Data Annotation for News Media

NANNA BONDE THYLSTRUP
University of Copenhagen, DENMARK

JANNIE MØLLER HARTLEY
Roskilde University, DENMARK

Roskilde University, DENMARK

Abstract

This article examines how data annotation in news organizations serves as a site of friction and world-making, shaping how AI systems categorize and interpret journalism. Through an ethnographic study in a Danish news organization, we identify three interrelated sites of friction: taxonomic constructions, organizational infrastructures, and aspirational realities. Rather than a neutral task, annotation is a contested process where journalistic epistemologies and machine learning logics collide, producing small acts of world-making that influence how journalistic knowledge is computationally instantiated. By centering friction as an analytical lens, we extend the infrastructural turn in media theory, showing how annotation practices – caught in between the journalistic and the computational domain – themselves function as infrastructural sites of negotiation and power. We thus show how AI is not seamlessly integrated into news organizations but shaped through ongoing struggles over classification, professional authority, and epistemic legitimacy, highlighting the infrastructural politics that underpin the computational turn in journalism.

Keywords

Digital infrastructure, machine learning, world-making, friction, data annotation, computational journalism

Introduction

The computational logics of machine learning (ML) systems are increasingly penetrating social and political worlds generating multiple moments of friction. These frictions emerge not merely in the direct sense that AI models are being deployed to automate decision-making across domains, but in the generative tensions between computational systems and human expertise that shape what can be known and actioned in the world. Drawing on Tsing’s (2005) concept of productive friction as “the awkward, unequal, unstable and creative qualities of interconnection across difference”, this article offers conceptual and empirical insights into how different but interconnected points of friction between computational and human knowledge practices become sites of world-making.

While research and public discourse have predominantly centered on bias and fairness in AI outputs, an emerging body of scholarship in critical dataset studies (Thylstrup, 2022) is shifting attention to the epistemic dimensions of dataset creation and input. This work examines how knowledge is constructed, contested, and codified during the dataset development and circulation process, particularly through the interplay between technical requirements and domain expertise as organizations increasingly build their own AI models across different sectors (Parrish et al., 2023; Smart et al., 2024). This perspective moves beyond traditional concerns about technical dataset construction and representational bias to investigate how the very process of creating training data shapes what counts as knowledge within these systems. These productive frictions are particularly visible in data annotation practices – the focus of this article – where technical requirements meet professional judgment in ways that both challenge and reshape epistemic frameworks in and across the news media domain.

Contrary to the promise of frictionless universal datasets that could seamlessly translate human knowledge into machine-readable formats, this article addresses data annotation practices as crucial sites where productive frictions generate new forms of world-making. Through examining these practices, we show how datasets are adapted and fitted to specific contexts through what we might call friction-generated knowledge practices. These practices emerge through necessary tensions between universal taxonomies and local knowledge, technical and domain expertise, and descriptive versus aspirational approaches to reality (Wang et al., 2022; Miceli et al., 2022).

Our conceptualization draws on both Amoore et al.’s (2024) concept of world-making and Tsing’s (2005) understanding of friction as generative of new social realities. We emphasize how data annotation serves as a critical site where reality is actively constructed through productive tensions between different forms of knowledge and practices.

Building on the rich tradition of media and cultural studies that has long conceptualized representation itself as constitutive and world-making (Hall, 1997; Laclau & Mouffe, 1985; Butler, 1993), we approach computation not as oppositional to representation but as extending these constructivist dynamics into new technological domains. This perspective acknowledges how journalistic representation has always been constitutive rather than merely reflective of reality (Tuchman, 1978; Gitlin, 1980). What Fazi (2024) argues regarding computation being generative extends this constructivist understanding into computational contexts: annotation practices do not simply describe reality but actively participate in constructing the worlds that ML systems will inhabit and interpret. In this sense, our analysis builds upon rather than rejects the constitutive power of representation, examining how annotation practices reconfigure these dynamics within the specific technical affordances and constraints of ML systems.

Our analysis homes in on three crucial friction-generating junctures in dataset construction: First, in the development and application of classification schemes, where annotators must translate rich domain expertise into discrete, machine-readable categories (Jaton, 2021). Second, in the organizational infrastructures that shape how annotation work is carried out, revealing how institutional arrangements create specific conditions for how knowledge is formalized and encoded (Seaver, 2017), both re-enforcing and mitigating frictions in specific ways. Third, in what we might call epistemic negotiations, where annotators navigate between describing the world “as it is” and actively shaping how it could be represented in ML systems (Passi & Jackson, 2018).

This examination offers several contributions to critical dataset studies and critical ML and media and journalism research more broadly. By illuminating how power operates through the granular practices of data annotation, we reveal dataset construction as a fundamental act of world-making that determines how reality becomes knowable to AI systems. Where classical theory has focused on how organizations construct reality through their selection and framing practices (Tuchman, 1978; Gitlin, 1980), our analysis reveals how this construction now happens within the distributed process of ML – representing a new form of what Bourdieu called “symbolic power” to name and categorize reality. This supports emerging work in critical ML studies on how struggles over representation and meaning-making now extend into the technical infrastructure itself (Ridell, 2019; Miceli et al., 2021; Bennett et al., 2025). Our analysis suggests the need for theoretical frameworks that can account for datasets not as passive repositories but as active sites where knowledge is constructed, contested, and legitimized through the distributed processes of ML.

Dataset annotation and ML in news

The world-making nature of dataset annotation – how it shapes what counts as knowledge and how this knowledge is categorized – takes on distinctly different forms and stakes across institutional contexts. For instance, the epistemic and political implications of annotation work in, for example, content moderation within a US corporate context differ markedly from those within a public service news organization operating under welfare state principles. While critical dataset studies has revealed the complex politics of annotation work, detailed empirical studies of how these dynamics play out within specific institutional contexts can further enrich our understanding of how professional values, organizational contexts, and societal roles shape the construction of ML training data.

Recent work in critical dataset studies examines how data annotation involves fundamental choices about what features of reality matter and how they should be understood (Kapania et al., 2023; Miceli et al., 2020). Rather than neutral representation, annotation practices are shaped by what Passi and Jackson (2018) call “data vision” – the interplay between formal abstraction and discretionary choices that shapes how datasets align with algorithms. This epistemic work is deeply shaped by organizational hierarchies and power dynamics (Wang et al., 2022; Miceli et al., 2022).

Udupa et al.’s (2023) work on “ethical scaling” provides particularly valuable insights into how the discursive-material interplay between human and computational elements shapes AI systems in ways that reflect broader cultural politics. Drawing on Gillespie’s (2020) understanding of scale as “a specific kind of articulation” where different components are bound together to operate as one, Udupa et al. propose ethical scaling as a framework. This framework recognizes scaling as a process that makes “the small…have large effects” while understanding this process as fundamentally political. They define ethical scaling as “a normative framework for platform governance” that “calls for a transparent, reflexive, and replicable process of iteration for content moderation with community participation and global parity.” Crucially, they argue that ethical scaling “involves ethical means for scaling a complex process through reflexive iterations” rather than merely focusing on model accuracy. Theirs is a critique of AI-assisted moderation and the liberal framing of the “human versus machine” conundrum; while different subject matter, their work also resonates strongly with our observations of how taxonomic frictions reveal tensions between ostensibly universal classification systems and local journalistic contexts, as well as how ostensibly small changes can have large effects.

Smart et al. (2024) deepen this understanding through a genealogical analysis comparing contemporary data annotation to 1970s experimental psychology. Their work reveals data annotation as technology-mediated social categorization that may reinforce static social categories. Using Bourdieu’s concept of habitus, they explore the tension between subjective interpretation and objectivity claims, particularly highlighting how non-WEIRD (Western, Educated, Industrialized, Rich, Democratic) workers often categorize data about WEIRD societies, creating a “remarkably complex feedback loop” in global AI systems. Bennett, Catanzariti & Tollon (2025) further develop this discussion of loops by introducing “representation coils” – self-reinforcing feedback loops where assumptions about skill and task complexity shape work structures that validate these initial assumptions. Their analysis shows how ML practitioners’ framing of annotation tasks as simple and universal leads to workflows that constrain annotator agency, thereby appearing to confirm the original assumptions about the work’s simplicity. This framework is particularly relevant for understanding the student annotators in our study who, on the one hand, stand out from the stereotypical outsourced annotator described in critical data, AI and platform studies, but, on the other hand, may still find their knowledge contributions minimized due to their position within organizational hierarchies despite having recognized educational expertise.

The institutional context of journalism offers particularly rich terrain for examining these dynamics. Research has mapped how computational practices have transformed news production and dissemination under various terms: computational journalism (Diakopoulos, 2015), data journalism (Fink & Anderson, 2015), data-driven journalism (Parasie, 2015), robot journalism (Carlson, 2015), and journalism as programming (Parasie & Dagiral, 2013). Diakopoulos (2015) emphasizes how designers make critical editorial decisions about algorithmic parameters, defaults, and system configurations. Studies of audience measurement software’s influence on editorial and gatekeeping processes illuminate questions of platform ownership and big tech’s influence, with tools like Google Analytics advancing news commercialization and audience segmentation in ways that reshape information diversity and newsroom decision-making. As Cubitt (2010) insightfully notes, spreadsheets have become one of the dominant media forms of the early 21^st century, suggesting how computational logics have reshaped information organization and presentation in ways that extend beyond traditional understandings of media formats. This perspective helps us recognize dataset construction as an emergent media practice with its own formal logics and constraints.

Emerging research provides valuable insights into how personalization projects reshape news values (Schjøtt Hansen & Hartley, 2021) and challenge public service principles like universalism (Sørensen, 2019), and examines how recommender systems can support democratic values such as diversity (Helberger, 2019). The transition of news organizations from ML consumers to developers (Hansen & Thylstrup, 2023; Hartley & Thylstrup, 2024) creates new opportunities to examine how professional and institutional values shape the development of AI systems.

Pallauda et al (2021) frame data annotation politics as part of a broader critique of ML’s dataset culture, highlighting how current mainstream practices prioritize rapid scaling and benchmarking over careful consideration of representation, labor conditions, and ethical data management. While Kapania et al. (2023) demonstrate how “representationalist thinking” in annotation work often seeks to eliminate subjectivity in pursuit of neutral labels, Fleisig et al. (2024) identify an emerging paradigm shift toward a “perspectivist turn” that values diverse annotator viewpoints. This shift holds particular promise for understanding how different institutional contexts might foster distinct approaches to managing annotator diversity and expertise.

Our ethnographic study of data annotation practices at a Danish news organization built on public service principles provides insight into how the epistemic work of dataset construction is shaped by and must adapt to specific institutional contexts. By examining how professional knowledge and technical requirements intersect in the development of AI systems for news organizations, we explore how the political and epistemic stakes of annotation work are fundamentally tied to their institutional setting, professional values, and broader societal role. This situated analysis enriches both journalism studies and critical dataset studies by illuminating how different institutional contexts produce distinct forms of knowledge work with varying implications for ML development.

Our ethnographic study of data annotation practices at a Danish news organization provides unique insights into these dynamics. As one of the first news organizations to develop its own in-house ML capabilities for news content recommendation and classification, this organization offers an ideal site to examine how professional knowledges and technical requirements intersect in the development of AI systems. In the following sections, we first describe our methodological approach and provide context about the concrete AI development initiative we observed and which serves as a foundation for developing the notion of ‘small acts of world-making’. We then highlight how these small acts emerge out of three key sites of friction that emerged in the annotation process.

The case: Data annotation in a legacy tabloid news organization

Our ethnographic study examines how friction both enables and constrains AI development at a Danish news organization. Following Tsing’s (2005) conceptualization of friction as “sticky engagements,” we analyze how global AI technologies and local journalistic practices create “heterogeneous and unequal encounters” that lead to “new arrangements of culture and power.”

To provide necessary context: the organization is a major Danish tabloid newspaper engaged in an ambitious AI development initiative jointly funded by a Danish Applied Research Foundation and the news organization itself. The project aims to develop natural language processing models and recommender systems while maintaining democratic values and public service principles. The stakes of this experiment are significant: for the organization, it represents an opportunity to maintain relevance in a digital media ecosystem increasingly dominated by platform giants; for the Foundation, it offers a chance to develop AI applications aligned with Danish cultural and democratic values; for the student-annotators, it provides valuable professional experience in emerging fields.

The team structure itself reflects the complex epistemic politics at play. The development team consists of data scientists and ML engineers who design the overall framework and technical specifications. They work in consultation with (but largely separate from) the editorial team, which consists of professional journalists who continue their regular publishing work. The annotation team consists of two women university students with backgrounds in rhetoric, IT, and communication, who serve as crucial intermediaries between these worlds. This team composition creates a particular dynamic where technical expertise, journalistic judgment, and academic knowledge must be negotiated without clear hierarchical authority over epistemic decisions.

We here take the organization, a national tabloid newspaper, to be what Tsing would call a “zone of awkward engagement” where globalized aspirations for the opportunities offered by ML meet the particularities of the Danish journalistic context. The physical architecture of the workplace embodied these sticky engagements: the development team occupied space one floor below the editorial section, creating what at first appeared to be simple organizational inefficiency. However, as Tsing notes, “roads create pathways that make motion easier and more efficient, but in doing so they limit where we go.” Similarly, we came to understand this separation as both enabling and constraining how different forms of expertise could interact. We observed a Research and Development project jointly funded by a Danish Applied Research Foundation and the news organization. The project’s goal of developing natural language processing (NLP) models and recommender systems while maintaining democratic values exemplifies what Tsing describes as the “sticky materiality of practical encounters.” Just as Tsing’s analysis shows how global capital doesn’t simply flow but must be produced through specific encounters, we found that AI systems aren’t simply implemented but must be negotiated through concrete practices that both enable and resist their implementation. Our data collection focused on these sites of “sticky engagement”: over 350 hours observing how universal technical standards met local journalistic practices; 8 in-depth interviews examining how different forms of expertise created both pathways and roadblocks; analysis of documentation revealing how global standards were transformed through local implementation. Following Tsing’s approach to global connections, we paid particular attention to how ML development operated not as a “well-oiled machine” but through friction. We identified three key sites of sticky engagement: taxonomic frictions where global classification systems met local journalistic knowledge, creating both enabling constraints and constraining enablements; organizational frictions where different forms of expertise both facilitated and impeded each other; aspirational frictions where the desire to implement universal systems encountered local resistance and reimagining.

Theorizing data annotation: Infrastructural friction and world-making

Our theoretical framework examines how data annotation practices serve as infrastructural moments where global technical systems meet local knowledge practices. Following Edwards’ (2010) analysis of knowledge infrastructures, we understand these encounters not as sites of overt power relations, but as complex sociotechnical arrangements where different ways of knowing and organizing information come into productive tension. We bring together three theoretical strands: generative AI as world-making systems (Amoore et al., 2024); infrastructural approaches to data practices (Edwards, 2010; Bowker & Star, 1999); and Tsing’s (2005) concept of friction in global connections.

Drawing on Tsing’s (2005) analysis, we understand data annotation not as smooth implementation of universal standards but as a site of “sticky engagement” where different knowledge practices meet and transform each other. Just as Edwards shows how climate data infrastructures emerge through the complex interaction of local measurement practices and global standardization efforts, we demonstrate how AI systems emerge through concrete practices that both enable and complicate their implementation. This helps us move beyond simple narratives of either technological determinism or local resistance to examine the material and practical dimensions of how AI systems are built and maintained.

Questions about how ML models understand and interact with reality have been central to data science. However, following Edwards’ insight that knowledge infrastructures are simultaneously social, technical, and epistemological achievements, we examine how these abstract models encounter specific journalistic practices. Amoore et al.’s (2024) analysis of world-making in AI systems takes on new significance when viewed through an infrastructural lens. Like Edwards’ (2010) analysis of climate data systems, we can examine how AI systems emerge through the complex interplay of local practices, technical standards, and epistemic frameworks. Rather than focusing on world-making as a top-down process, we examine how different knowledge practices come together to knowledge infrastructures (Edwards et al., 2011); i.e., the networks of people, artifacts, and institutions that generate and maintain specific ways of knowing.

Building on these insights, we understand annotation practices as crucial sites where different forms of knowledge infrastructure intersect. Building on Bowker and Star’s (1999) foundational work, we understand classification systems as both technical and social achievements that structure how reality becomes knowable and actionable. Their concept of “infrastructural inversion” – making visible the typically invisible work of classification – helps us analyze how annotation practices construct rather than merely describe reality. As they argue, classification systems become most visible at moments of breakdown or resistance, which we find particularly relevant for understanding how annotators navigate cases where established categories fail to capture real-world complexity. These perspectives also help us understand how annotation practices involve continuous negotiation between universal aspirations and local needs. As Lampland and Star (2009) demonstrate, standards are never truly universal but always require local adaptation and “tinkering” to work in specific contexts. This theoretical insight helps explain the hybrid taxonomies we observed, where annotators combined elements from different classification systems to meet their specific needs. STS scholarship on infrastructure (Star & Ruhleder, 1996) further enriches our understanding of how annotation practices become embedded in larger socio-technical systems. Infrastructure, in this view, is not just technical but relational – it becomes real infrastructure in relation to organized practices. This helps us analyze how annotation practices create specific conditions for knowledge production through their relationship with existing journalistic practices and organizational structures. Recent anthropological and STS work on algorithmic systems and ML (Seaver, 2017; Jaton, 2021) provides additional theoretical resources for understanding how human judgment becomes encoded in technical systems. These scholars emphasize how ML models don’t simply automate existing practices but create new forms of knowledge production that combine human expertise with computational logics. This perspective helps us understand annotation not as simply translating between human and machine understanding, but as creating new hybridized (Rella et al., 2024) forms of knowledge production. Tsing’s (2005) concept of friction enriches this infrastructural analysis by highlighting how these knowledge practices emerge through specific, material encounters. Like the “sticky engagements” Tsing describes in global commodity chains, annotation practices involve concrete moments where different ways of knowing must be negotiated and transformed. These frictions are not simply obstacles to be overcome but, as Edwards et al. (2011) argue about data friction, necessary features of how knowledge infrastructures develop and operate.

By bringing together perspectives from critical data studies, AI and platform studies, and STS, we can better understand how the abstract promise of AI in journalism becomes concrete through specific choices, struggles, and negotiations in annotation practices.

Small acts of world-making in data annotation

Our ethnographic study reveals three distinct sites where world-making practices unfold in the development of AI systems for news. Each site represents a space where different forms of reality are actively constructed and negotiated through annotation practices: taxonomic constructions, organizational infrastructures, and aspirational realities. These sites are not merely locations where technical work happens, but rather crucial junctures where decisions about what counts as news, how journalism should work, and what values should be encoded into AI systems are made through daily practices of classification and categorization. Through careful observation of these sites, we can see how the abstract promise of AI in journalism becomes concrete and also made durable through specific choices, struggles, and negotiations.

The first site emerges in the development and application of taxonomies, where annotators must translate the rich complexity of journalistic judgment into discrete, machine-readable categories. The second manifests in the organizational infrastructures that shape how annotation work is carried out, revealing how institutional arrangements create specific conditions for world-making through the interpretations annotators make. The third appears in what we call aspirational realities, where annotators navigate between describing the world as it is and resisting these descriptions of reality in turn actively shaping how reality should be represented in AI systems.

These sites are interconnected but distinct, each revealing different aspects of how AI systems come to “know” and process journalistic content. Together, they show how the development of AI for news involves constant negotiation between technical requirements, professional knowledge, and normative ideals about journalism’s role in society. By examining these sites in detail, we can better understand how the future of journalism is being actively shaped through seemingly routine practices of data annotation – what we in this article term small acts of world-making.

Site 1: Taxonomic encounters – World-making through classification

The frictions involved in the worldmaking practices of AI became immediately apparent during the project’s early meetings, where the very foundation of data annotation work was being negotiated during the process of developing a taxonomy for classification of a dataset of news articles consisting of several thousands of articles pulled from the news content that had been published by the news organization. While recognized as crucial for developing transformer models, data annotation emerged as a complex and rather expensive challenge requiring networked knowledge. As one data scientist explained: “There really aren’t any annotation resources. There’s experience with automated systems like Mechanical Turk, but there aren’t any Danish annotators.” While some saw this as an opportunity for outsourcing, others recognized the inherent complexity of annotation work. Another data scientist thus pushed back: “No, I don’t think so. It’s not that simple to annotate. Are we just being super arrogant and thinking we know how to do it, or is there already some best practice when we’re developing our protocol?” The project manager’s response offered a perspective on how these frictions could generate productive adaptations: “80% of the NER (Named Entity Recognition) is based on people who have spent a lot of time thinking about these things. The last 20% we’ve added ourselves. We’re standing on the shoulders of others.” Later, another data scientist explained through metaphor: “It’s a little like taking a suit from the hanger, and tailoring it to us.” This tailoring required numerous discussions and meetings to settle the emerging frictions in this work of building the taxonomy.

The challenges of taxonomic world-making became particularly evident in discussions about how to categorize crime and conflict. During one meeting, the project team grappled with fundamental questions about semantic relationships: “You’ve been very consistent with the taxonomy,” one participant noted, “but if we open Pandora’s box…” This sparked deeper discussion about classification principles. While some participants were interested in categorizing articles by type and genre rather than just content, others insisted on maintaining focus on their specific goals.

The conversation revealed complex decisions about categorical relationships emerging through friction. “Crime is under war and conflict,” one annotator noted. “When I think of war, I think of Ukraine or something like that.” This prompted debates about whether crime should be its own category or remain under broader classifications like “society” or “war and conflict.” The annotation team wrestled with questions of scale and specificity: should they distinguish between different types of crime, such as gang-related versus financial crime?

These broader tensions materialized concretely when the news organization’s managers and developers gathered to discuss competing taxonomies for their AI systems. The discussion centered on friction between the Interactive Advertising Bureau (IAB) taxonomy used by big tech companies and the International Press Telecommunications Council (IPTC) used by news organizations. As one developer noted: “The IAB taxonomy tends to focus on cars and washing machines, which is far from the content we publish here.”

This friction generated a new hybrid approach. The project team developed their own taxonomy called IEB, combining elements from IAB, IPCT, and the newspaper’s own categories fitting their specific profile of content. Through this process of encounter and adaptation, the annotators weren’t simply applying existing standards – they were engaging in world-making through the development of new ways of representing journalistic content. Thus while positioned as a neutral general representative taxonomy once finished, the resulting hybrid taxonomy manifested as an active intervention in how this particular AI system would order and produce future news and in particular what would be recommended to different audiences. Many of these decisions were made by the annotators, which sometimes wanted to add categories, but were also told that they had to limit the number of categories so as not to make taxonomy too complex. The data scientists had the interest of making the taxonomy as simple as possible in order to limit the possible margin of error once the taxonomy was implemented in the ML models. Furthermore, it was argued in meetings that the hybrid taxonomy should not differ too much from the existing ones from the IAB, as it would have to work with recommending advertising in a distant future. Nor should it differ too much for the content to also work with other infrastructures such as Google and META, who would use existing taxonomies to determine how and which content would show up in search engines for example. Similarly to what was observed by Jaton (2021), the annotators in this case had to translate rich domain expertise into discrete, machine-readable categories. While the annotators perceived the IPCT to be ‘closer’ to the journalistic content they had to categorize, their work nevertheless consisted in finding the type of articles and the categories which really did not fit into the existing taxonomies and adding them. As well as removing categories – like the cars and the washing machines – which had very few or no articles in them. The lines of productive frictions emerging in the work of annotators thus both related to issues of simplicity vs. complexity and editorial values vs. more commercial values. Through this the annotators are made aware of the possible consequences of their work, although they were never directly involved in the testing of the ML models once the work with the dataset was done.

Site 2: Organizational encounters – World-making through spatial practice

The second site of world-making emerges through the frictions generated by organizational structures that shape how annotation work is carried out. In our own field notes, one of us remarked:

The general impression of the department where data science is located is that it is separated from the rest of the editorial staff. It is also very quiet. Maybe conversation is going on in Slack or other chat functions. There is occasional scattered conversation in the office space, which most often concerns how a model is performing. Most of the talking happens among those who are doing development. 11.25: Three people come in – they’re from the editorial department. They stand out by having louder voices. They shout hello to everyone and remark: “how hardworking you all are.” When they came in, the two who were otherwise talking fell silent.

Notably, even within in-house annotation teams, this organizational set-up creates distinct conditions for how specific frictions became productive, or even limited the frictions because of institutional constraints on the level of the news organization. The annotators, working part-time alongside their studies in rhetoric, IT and communication, expressed general satisfaction and contentment with their work conditions. However, their position at this friction-point between technical and editorial worlds generated specific forms of knowledge production. As one annotator explained: “We are quite isolated… sometimes you can have the feeling that you are at the bottom of the business in one way or another, and the work we contribute is actually quite important, because if we don’t tag properly, if we haven’t created a proper protocol that allows others to tag properly or annotate properly, then the models won’t work. So we are like the forgotten children haha.” The quote illustrates how the annotators are aware of the consequences and importance of their work, but also how they do not feel involved in the whole process, but instead isolated to one part of it. Interestingly, the annotators were also aware that the spatial organization was isolating the journalists from the process of annotating, and the annotators were wishing they could sometimes ask journalists how they would categorize the content they produce.

The organizational separation generated particular challenges in understanding and encoding journalistic judgment. One annotator described their struggle with basic categorical distinctions: “We had an example in a project in which we had a category called news… We had to talk about how it would actually be nice to know when a journalist or what a journalist would define as news… But we just never did. Maybe because we are on another floor.”

This spatial lay-out shapes how journalistic knowledge gets translated into machine-readable formats. The annotators have to rely on their own embodied gut-feeling (Hartley & Thylstrup, 2024) of what a news story is and how different stories could be categorized, compared to other genres in the dataset, without having journalistic training. The space of separation from editorial staff conditions how “news” as a genre is understood and interpreted. This understanding becomes durable in the systems and models they build when annotators discuss these decisions with data scientists. These encounters revealed a point of friction between the wish and ideal of the annotators to reflect the journalistic domain (without always having the journalistic domain expertise), while the data scientists were more concerned with the functionality of the categories; for instance, that the model would be consistent in the categorization and learn how to recognize similar articles, independent of what the articles were about.

More than simply an organizational inefficiency, however, the mundane separation also pragmatically enabled the annotators to produce categories at a pace that would not have been the same if they would have had to spar more frequently with journalists. Thus, it can be argued that organizational structures can work to minimize unwanted friction to make the process of building ML models, and especially the time-consuming and expensive task of annotating large datasets that are then further used for training the ML models to recognize future content, more efficient. The part of the project dedicated to the annotation could not drag out too long, as the datasets were needed for the training of the models.

Site 3: Aspirational encounters – World-making through resistance and reimagining

The third site of small-scale world-making manifests in how annotators navigate between describing existing realities and actively shaping desired futures. These frictions between representation and transformation generate specific possibilities for how AI systems will understand and reproduce journalistic content. As one annotator reflected: “I’m just trying to make it as objective as possible. And then of course there are some changing values and a changing morality. But it’s not like I think that now I’m the [newspaper] reader. I think I am just starting from the protocol we have written.” As previously noted by Kapania et al. (2023), annotators’ work is often framed within a “representationalist thinking” that seeks to eliminate subjectivity in pursuit of neutral, objective labels. Interestingly, this is echoing how journalists have historically operated through ‘norms and rituals of objectivity’ (Tuchman, 1972) and how the categorizations of different news stories inside the newsroom were also a part of power struggles in the daily negotiations between different beats and news organizations in a relational field (Møller Hartley, 2011). Similarly, in the dataset annotation practices, the apparent neutrality often generates productive tensions in how reality gets encoded, perhaps more even outspoken as the annotation practices have not yet been routinised to the same degree as journalism as it is quite a new profession inside news organizations. However, the point of friction concerning objectivity reflects another underlying point of friction; namely, the active vs. descriptive roles of the annotators. One annotator articulated this friction directly: “It probably depends on whether you are interested in modelling reality or whether you are interested in modelling how one thinks reality should be… It will be a political, or whatever you call it, a very opinionated bias on that model. But that is also sometimes what you have to do to counteract. There are also some things you want to avoid. Things like racism and all that. You are not interested in the models reflecting that.” This tension between describing and transforming reality became particularly visible in how annotators approached issues of representation. One annotator described their handling of an article about clothing for craftspeople: “We have put a lot of work into making it as fair as possible when we tag… We have one example where there was something with a clothing shop for craftspeople. And here we thought automatically that we had a category called ‘men’s fashion’, because we thought that craftspeople are men. In that case, we were a bit biased. But then we also thought that, of course, we should also tag it as women’s fashion.” Again, this reflects how annotation is embedded in larger issues of representation through mediation, in similar ways that the journalistic practices are framing the social reality though the choice of sources, angling of the story and the language used (Entman, 1993). Again, this reflects how annotation is embedded in larger issues of representation through mediation, in similar ways that journalistic practices frame social reality through the choice of sources, angling of the story and the language used (Entman, 1993). The annotators thus function as infrastructure in AI development, i.e. discrete yet essential labour that shapes how these systems understand the world. As infrastructural workers, the annotators are aware of the debates on bias in large datasets, so it’s not surprising that they feel an urge to address this issue in their daily practices.

This friction also reveals a deeper distinction between future-making and past-preserving in ML, which was deemed especially important in news. In one meeting between the developers and the annotators, they discussed how the model would react to “new and unknown phenomena,” using Covid-19 as an example. One data scientist theorized that it would take a long time for the model to recognize such new articles or topics, which was not part of the annotated material, and in news there’s just no time to wait for that, he argued. This illustrates how categorizations become somewhat static representations of reality, and how world-making in this case involves negotiating between preserving existing categories and enabling new ones. Whereas journalists can respond to ‘newness’ quite fast and in fact consider it the main task of their job and professional identity to describe new phenomenon, the categories in datasets need to be more static for the ML model to work and it takes more time to add or remove certain categories as they might not be relevant with the same weight at any given time. Especially since annotation is not considered an ongoing task, but a first phase of an ML project after which the dataset is considered somewhat sealed and finished and ready to move into the scaling and training of the ML models. Yet we also know from decades of research into journalistic practices that these are surprisingly durable and static despite the focus on ‘the unexpected’, because journalism too needs to be routinised to be efficient and fast (Tuchman, 1972). In that way, by referring to the protocol the annotators are to some extent routinising the unexpected and in the same way removing future frictions by making the categorizations more static. The decisions made by annotators represent more than simple classification choices – they are active interventions in how AI systems will understand and reproduce social categories. The annotators find themselves caught between describing the world as it is and transforming it by challenging apparent and obvious categories. When annotators choose to tag “craftsperson” in gender-neutral ways, it can thus be seen as a moment where friction generates new possibilities for how AI systems will understand and represent gender and different stories in future contexts.

Reimagining media power through infrastructural friction and small acts of world-making

This study’s examination of data annotation practices in newsrooms offers critical insights into the shifting loci of media power in computational journalism. While classical media theory has long analyzed editorial selection, framing, and institutional structures as central to the construction of journalistic reality (Tuchman, 1978; Gitlin, 1980), our analysis reveals how these processes now unfold within the more concealed, infrastructural layers of ML development. The annotation of training data – often framed as a routine, technical task – emerges instead as a key site where competing logics of computational abstraction and journalistic judgment collide. The annotators are caught in the middle with no training in either of those domains and yet they experience these frictions and are able to translate them into small acts of world-making in various ways. The sites of categorizing, organizing and re-imagining news content for ML development are producing moments of friction that actively shape how ML systems will come to “know” and process news content.

We argue that friction – conceptualized as the points of resistance, negotiation, and misalignment that emerge when journalistic knowledge is translated into machine-readable categories – must be foregrounded as a central dynamic in the infrastructural politics of AI in journalism. Rather than assuming a seamless integration of ML models into newsrooms, our study shows how annotation work is continuously shaped by epistemic and institutional struggles: friction emerges in the classificatory practices that determine what counts as news, in the organizational structures that separate annotators from journalists, and in the aspirational negotiations over how reality should be represented once that reality is moved from the editorial desk into the ML models. These processes reveal the extent to which computational infrastructures are not merely technical but deeply social and political formations (Star & Ruhleder, 1996; Bowker & Star, 1999; Parks & Starosielski, 2015).

Crucially, friction does not merely operate as an obstacle but as a generative force. As annotators contest, reinterpret, or subtly reshape taxonomies, they engage in small acts of world-making – interventions that, while incremental, have profound consequences for how journalistic content is structured and processed by ML. The hybrid taxonomies that emerge from this work show how the perceived universality of computational classification is, in fact, always contingent upon local adaptations, professional norms, and tacit epistemic judgments (Lampland & Star, 2009). Further we see how the frictions emerge when the local is adapted and connected to a larger global infrastructure, but also that it is in this adaptation there is room for small acts of world-making. This echoes Amoore et al.’s (2024) argument that ML’s world models do not neutrally reflect reality but actively construct it, shaped by embedded political logics and governing rationalities. In the case of news annotation, these adaptations are not simply technical but deeply political: they determine whose perspectives, values, and assumptions about journalism become encoded into the AI-driven infrastructures of news; in similar ways to how the media has constructed reality, as has been highlighted by representation theory (Hall, 1997).

These findings contribute to and extend the ongoing infrastructural turn in media theory (Starosielski, 2015; Plantin & Punathambekar, 2019; Velkova & Plantin, 2023). While media scholars have increasingly attended to the material infrastructures of news production – whether in the form of server farms (Velkova, 2016), recommendation algorithms (Helberger, 2019, Schjøtt Hansen and Hartley, 2021), or platforms (Plantin & Punathambekar, 2019; Kristensen and Møller Hartley, 2023) – our study demonstrates that human practices of classification and annotation are themselves key infrastructural processes that shape journalistic reality. By small acts of worldmaking they shape how datasets are made and how journalistic content is categorized, which is then deployed and used for future training of foundational models for the distribution of news. These practices, though often invisibilized in discussions of ML development, are integral to the production of knowledge and power within computational news environments.

Friction, therefore, is not only a technical challenge but a social question. This calls for an expanded theoretical lens that accounts for how media infrastructures themselves become sites of meaning-making, shaping not only what is considered legitimate journalistic knowledge but also how this knowledge is computationally instantiated. Much focus in media studies and journalism research has been on journalistic practices in the intersection with the computational, overlooking other foundational meaning-making processes in rooms outside the newsrooms, which merits attention in future studies.

Through the lens of friction, we have shown how ethical scaling (Udupa et al., 2023) helps us recognize that scaling makes ‘the small…have large effects,’ treating this amplification as a political process rather than a technical one.

The analytical power of friction lies in its ability to make visible the active processes of knowledge construction that might otherwise appear as neutral technical implementation. Rather than positioning friction as universally positive or negative, we have examined how specific frictions enable certain forms of knowledge while constraining others. This allows us to ask not just where friction occurs, but what kinds of world-making it enables or prevents. In cases where we observed limited friction, this raises important questions about pre-existing alignments between journalistic and computational logics that merit further investigation.

Finally, while much of the discourse on ML in news media oscillates between utopian promises of automation and dystopian fears of displacement (Carlson, 2018; Diakopoulos, 2015), our article indicates a more complex reality: ML integration is neither seamless nor uncontested. Instead, it is shaped by the everyday frictions through which professional expertise, institutional norms, and ML logics are made to fit – or fail to fit – one another. Recognizing these frictions as constitutive rather than incidental allows us to move beyond simplistic narratives of technological disruption and instead interrogate how power, labor, and epistemic authority are reconfigured in the digital infrastructures of journalism.

Concluding thoughts: The politics and ethics of world-making in AI-development

This article has explored data annotation for AI-driven journalism as a site of epistemic negotiation and world-making, where competing knowledge systems, professional values, and computational imperatives converge. Our analysis reveals how the construction of AI-ready datasets involves complex epistemic work that goes far beyond technical translation – it is fundamentally about negotiating how knowledge is constructed, validated, and contested.

The rich work taking place in critical dataset studies on the epistemic work and politics of annotation is crucial for advancing these insights. The tensions we observed between universal taxonomies and domain expertise, between standardized protocols and professional judgment, and between descriptive and normative classification goals, highlight the limitations of viewing datasets as neutral representations (Kapania et al., 2023). Instead, we must recognize datasets as infrastructures that actively shape knowledge production through their ontological and epistemological commitments (Miceli & Posada, 2022; Smart et al., 2024). This suggests that the key challenge is not simply about addressing bias or fairness in annotation, but about examining how dataset construction practices fundamentally structure what kinds of knowledge are possible within AI systems (Fazi, 2024; Zylinska, 2024).

At stake is the need to develop richer theoretical frameworks for understanding datasets as sites of epistemic work. If annotation decisions shape how AI systems interpret and generate knowledge, then those involved in this process must be recognized as epistemic agents engaged in complex acts of knowledge production rather than mere technical facilitators. This reframing carries significant implications for critical dataset studies, suggesting the need to examine how authority and expertise are negotiated through the infrastructural layer of dataset creation, validation, and contestation (Jaton, 2021; Seaver, 2017).

This points to several crucial areas for future research in critical dataset studies: How do different forms of expertise and knowledge claims compete and combine in dataset construction? How are epistemic standards negotiated across different professional and organizational contexts? How do power dynamics shape whose knowledge counts in dataset development? What theoretical tools can help us better understand datasets as sites of epistemic work rather than mere technical artifacts?

Ultimately, as critical dataset studies shows, we must move beyond viewing datasets as static repositories and engage with them as dynamic sites where knowledge is actively constructed, contested, and legitimized (Passi & Jackson, 2018; Wang et al., 2022). By examining dataset practices as epistemic negotiations and small acts of world-making, we can develop richer theoretical frameworks for understanding how AI systems shape knowledge production. This suggests the need for expanded dialogue between critical AI studies, science and technology studies, and epistemology to better theorize the complex knowledge work embedded in dataset construction and contestation.

References

Amoore, L., Campolo, A., Jacobsen, B., & Rella, L. (2024) ‘A world model: On the political logics of generative AI’, Political Geography, 113: 103134.

Bennett, S.J., Catanzariti, B., & Tollon, F. (2025) ‘“Everybody knows what a pothole is”: representations of work and intelligence in AI practice and governance’, AI & SOCIETY, 1-12.

Bojczuk, I., Starosielski, N., & Pasek, A. (2024) ‘Flying the skies to wire the seas: Subsea cables, remote work, and the social fabric of a media industry’, Media, Culture & Society, 46(2): 358-375.

Butler, J. (1993) Bodies that Matter: On the Discursive Limits of “Sex”. London: Routledge.

Carlson, M. (2018) ‘The robotic reporter: Automated journalism and the redefinition of labor, compositional forms, and journalistic authority’, in S. Lewis (ed.) Journalism in an Era of Big Data:  Cases, Concepts, and Critiques. London: Routledge, pp. 108-123.

Cubitt, S. (2010) ‘Making space’, Senses of Cinema, 57.

Demuro, E., & Gurney, L. (2021) ‘Languages/languaging as world-making: the ontological bases of language’, Language Sciences, 83: 101307.

Diakopoulos, N. (2015) ‘Algorithmic accountability: Journalistic investigation of computational power structures’, Digital Journalism, 3(3): 398-415.

Edwards, P.N., Mayernik, M.S., Batcheller, A.L., Bowker, G.C., & Borgman, C.L. (2011) ‘Science friction: Data, metadata, and collaboration’, Social Studies of Science, 41(5): 667-690. https://doi.org/10.1177/0306312711413314

Entman, R.E. (1993) ‘Framing: Toward Clarification of a Fractured Paradigm, Journal of Communication’, 43(4): 51–58: https://doi.org/10.1111/j.1460-2466.1993.‌tb‌013‌04.x

Fazi, M.B. (2024) ‘Machines That Create: Contingent Computation and Generative AI’, Media Theory, 8(2): 01-12.

Fleisig, E., Blodgett, S.L., Klein, D., & Talat, Z. (2024) (The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels’, arXiv arXiv:2405.05860.

Gillespie, T. (2020) ‘Content moderation, AI, and the question of scale’, Big Data & Society, 7(2). https://doi.org/10.1177/2053951720943234

Gitlin, T. (1980) ‘Making democracy safe for America’, Columbia Journalism Review, 18(6): 53.

Hall, S. (1997) ‘Representation: Cultural representations and signifying practices’. London: SAGE Publications.

Hartley, J.M. (2011) ‘Routinizing Breaking News: Categories and Hierarchies in Danish Online Newsrooms’, in D. Domingo & C. Paterson (eds.), Making Online News: Newsroom Ethnography in the Second Decade of Internet Journalism (Vol. 2, pp. 89–102). New York: Peter Lang. http://www.peterlang.de/‌download/datasheet/61890/‌data‌‌sheet‌_311064.pdf

Hansen, K.B., & Thylstrup, N.B. (2023) ‘Stack bricolage and infrastructural impermanence in financial machine-learning modelling’, Journal of Cultural Economy 17(1): 20–38. https://doi.org/10.1080/17530350.2023.2229347

Hartley, J.M. & Thylstrup, N.B. (2024) ‘The Algorithmic Gut Feeling – Articulating Journalistic Doxa and Emerging Epistemic Frictions in AI-Driven Data Work’, Digital Journalism 13(3): 438–456. https://doi.org/10.1080/21670811.‌2024.‌2319641

Helberger, N. (2019) ‘On the democratic role of news recommenders’, Digital Journalism, 7(8): 993–1012.

Jaton, F. (2021) The Constitution of Algorithms: Ground-Truthing, Programming, Formulating. Cambridge, MA: MIT Press.

Kapania, S., Taylor, A.S., & Wang, D. (2023) ‘A hunt for the snark: Annotator diversity in data practices’, in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1-15). New York, NY: Association for Computing Machinery (ACM). https://doi.org/10.1145/3544548.3580681

Kristensen, L.M., & Hartley, J.M. (2023) ‘The Infrastructure of News: Negotiating Infrastructural Capture and Autonomy in Data‐Driven News Distribution’, Media and Communication, 11(2): 307-318. https://doi.org/10.17645/mac.v11i2.6388

Laclau, E., & Mouffe, C. (1985) Hegemony and Socialist Strategy: Towards a Radical Democratic Politics. London: Verso.

Lampland, M., & Star, S.L. (Eds.) (2009) Standards and Their Stories: How Quantifying, Classifying, and Formalizing Practices Shape Everyday Life. Ithaca, NY: Cornell University Press.

Mackenzie, A., Waterton, C., Ellis, R., Frow, E.K., McNally, R., Busch, L., & Wynne, B. (2013) ‘Classifying, constructing, and identifying life: standards as transformations of “the biological”’, Science, Technology, & Human Values, 38(5): 701-722.

Malevé, N. (2024) ‘Lost in Compression: Models of Authorship in Generative AI’, Media Theory, 8(1): 205-228.

Miceli, M., Posada, J., & Yang, T. (2022) ‘Studying up machine learning data: Why talk about bias when we mean power?’, Proceedings of the ACM on Human-Computer Interaction, 6(GROUP): 1-14.

Miceli, M., & Posada, J. (2022) ‘The data-production dispositif’, Proceedings of the ACM on human-computer interaction, 6(CSCW2): 1-37.

Parasie, S., & Dagiral, E. (2013) ‘Data-driven journalism and the public good: “Computer-assisted-reporters” and “programmer-journalists”’, New Media & Society, 15(6): 853-871.

Parks, L., & Starosielski, N. (eds.) (2015) Signal Traffic: Critical Studies of Media Infrastructures. Urbana, IL: University of Illinois Press.

Paullada, A., Raji, I.D., Bender, E.M., Denton, E., & Hanna, A. (2021) ‘Data and its (dis)contents: A survey of dataset development and use in machine learning research’, Patterns, 2(11).

Passi, S., & Jackson, S.J. (2018) ‘Trust in data science: Collaboration, translation, and accountability in corporate data science projects’, Proceedings of the ACM on human-computer interaction, 2(CSCW): 1-28.

Plantin, J.C., & Punathambekar, A. (2019) ‘Digital media infrastructures: pipes, platforms, and politics’, Media, Culture & Society, 41(2): 163-174.

Ridell, S. (2019) ‘Mediated bodily routines as infrastructure in the algorhythmic city’, Media Theory, 3(2): 27-62.

Schjøtt Hansen, A., & Hartley, J.M. (2021) ‘Designing What’s News: An Ethnography of a Personalization Algorithm and the Data-Driven (Re)Assembling of the News’, Digital Journalism 11(6): 924–942. https://doi.org/10.1080/21670811.‌2021.‌1988861

Seaver, N. (2017) ‘Algorithms as culture: Some tactics for the ethnography of algorithmic systems’, Big Data & Society, 4(2): 1-12.

Smart, A., Wang, D., Monk, E., Díaz, M., Kasirzadeh, A., Van Liemt, E., & Schmer-Galunder, S. (2024) ‘Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation’, arXiv preprint arXiv:2402.06811.

Star, S.L., & Ruhleder, K. (1996) ‘Steps toward an ecology of infrastructure: Design and access for large information spaces’, Information Systems Research, 7(1): 111-134.

Starosielski, N. (2021) ‘Afterword: Infrastructural Fantasies’, Canadian Journal of Communication 46(2), 363–370. https://doi.org/10.22230/cjc.2021v46n2a4147

Starosielski, N. (2015) The Undersea Network. Durham, NC: Duke University Press.

Thomassen, L. (2006) ‘Beyond representation?’ Parliamentary Affairs, 60(1): 111–126. https://doi.org/10.1093/pa/gsl051

Thylstrup, N.B. (2022) ‘The ethics and politics of data sets in the age of machine learning: Deleting traces and encountering remains’, Media, Culture & Society 44(4): 655-671.

Tuchman, G. (1978) Making News: A Study in the Construction of Reality. New York, NY: Free Pres.

Udupa, S., Maronikolakis, A., & Wisiorek, A. (2023) ‘Ethical scaling for content moderation: Extreme speech and the (in)significance of artificial intelligence’, Big Data & Society, 10(1). https://doi.org/10.1177/20539517231172424

Yang, T., & Miceli, M. (2024) ‘“Guilds” as Worker Empowerment and Control in a Chinese Data Work Platform’, Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2): 1-27.

Velkova, J. (2016) ‘Data that warms: Waste heat, infrastructural convergence and the computation traffic commodity’, Big Data & Society, 3(2): 2053951716684144.

Wang, D., Prabhat, S., & Sambasivan, N. (2022) ‘Whose AI Dream? In search of the aspiration in data annotation’, in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Association for Computing Machinery. https://doi.org/10.1145/3491102.3502121

Zylinska, J. (2024) ‘Diffused Seeing: The Epistemological Challenge of Generative AI’, Media Theory, 8(1): 229-258.

Nanna Bonde Thylstrup is Associate Professor in Modern and Digital Culture at the University of Copenhagen and PI of the ERC-funded DALOSS project on data loss. She researches the politics and ethics of data, machine learning, and digital infrastructures, with a focus on how loss is socially and politically constituted. She is the author and editor of Uncertain Archives (MIT Press, 2021) and The Politics of Mass Digitization (MIT Press, 2019), and regularly contributes to scholarly and public debates on emerging technologies.

Email: nannab@hum.ku.dk

Jannie Møller Hartley is Professor of Communication and Journalism at Roskilde University, co-leading the Centre for Data and AI and PI of AUTOPUBLICS. Her research spans journalism, audience studies, and data science, focusing on algorithmic governance, datafied everyday life, and journalism’s role in democratic publics. She is the author of News Online (2012), Datapublics – The Construction of Publics in Datafied Societies (2023 Bristol University Press) and is currently writing ‘Just for You – Inside the News Personalization Machine’ (Nordicom).

Email: jath@ruc.dk

Conflicts of interest
None declared

Funding
None declared

Article history
Article submitted: 01/02/2025
Date of original decision: 18/04/2025
Revised article submitted: 02/05/2025
Article accepted: 15/05/2025