Alternative ways of algorithmic seeing-understanding

I want to use a case from Brazil to elaborate on how technology-enabled seeing-understanding is such a complex and layered action. Monteiro and Rajão (2017) studied how remote sensing plays a role in detecting and tracking deforestation in the Amazon rainforest. Although there is plenty of satellite data being gathered, all of it needs to go through the scrutiny of scientists/analysts who outline the deforested areas in the images. The satellite data, although it may seem really complete and detailed, as it comes from expensive and precise instruments of seeing, doesn’t tell the full story in itself. The scientists/analysts need to do a lot of work on it to make it useful: stitch together multiple datasets, collaborate across fields and locations, add in their qualitative fieldsite knowledge, etc. These specialists interpret what they see through the “technological” lens of the satellite to determine, both as scientists and concerned citizens, whether what they are seeing is deforestation or just a cloud passing by. And more, they have to somehow balance their interpretation to make sure it won’t be misunderstood by society and policymakers, who may see the final results as exaggerated or inaccurate 2.

Looking at satellite images, the scientists/analysts are trying to understand and make sense of the world, interpreting through many layers of analysis, situated knowledge, and embodiment. Their decisions of what to see have direct effects in the world, which also informs their particular ways of seeing-understanding. In other words, the information infrastructures for monitoring forest loss operate “at the intersection of mutually interfering political demands, incommensurable goals, and competing organisational logics” (Vurdubakis & Rajão, 2020, p. 18). Although “the Amazonian rainforest remains the most closely observed in the world” (Vurdubakis & Rajão, 2020, p. 18), what is made visible depends on the sociotechnical “unstable network formations” of geospatial technologies and humans.

This discussion is, of course, not really new. Many scholars, particularly historians and theoreticians of vision and Science, Technology, and Society (STS) have discussed what is behind the effort of vision. Berger (2008), for example, talked about “ways of seeing,” saying that the process of seeing paintings, as well as the process of seeing anything, is much less spontaneous and natural than we tend to believe—it depends in large part on habits and conventions. There are many more who discuss this, such as Crary (1992), for whom the body is an intrinsic part of our ways of seeing; Parks (2005), for whom the satellite, much like previously described, can work as a way of seeing that hides/shows in particular ways; and Impett (2020), for whom different computer vision models can see artistic production differently. The fact is: neither in the artistic paintings, nor in the satellite views of the world, or even in the algorithms of computer vision is there a “view from nowhere” (Haraway, 1988), a distant, removed way of seeing that is not also tied to a particular way of understanding. Seeing–understanding is an ideological process, and whenever technology is involved, there are many more layers to be considered.

So, with that established, we can jump to Computer Vision today. The most prevalent and powerful algorithms of seeing–understanding in our society have strong links to consumer surveillance and the military. Or, as described by artist and researcher Adam Harvey, “The most powerful computer vision surveillance tools developed today come from Facebook, Intel, IBM, and Google and are primarily used for tracking consumers” (Harvey, 2020). In the logic of “data capitalism” (West, 2017), companies are using their enormous collected data to create automated personalized predictions of behavior at scale: “increasingly targeted on individual actions, sentiments, and desires rather than stratified population aggregates” (Mackenzie, 2013, p. 392). Alongside the purposeful erosion of privacy in the “invisible visual culture” of machine-seeing, military technology uses are of particular concern. From missile guidance to drone operation, companies and states work together to create dangerous and lethal seeing-machines that “surveil and punish,” often automatically.

The story goes that “If the only tool you have is a hammer, you may be tempted to see everything like a nail.” The way the hegemonic algorithms of seeing-understanding work is directly tied to the surveillance and military gaze, and their economic, racial, social ways of seeing-sensing-detecting and understanding-sensemaking-comprehending. So, just as Katherine Ye in the first chapter of this publication discusses how Silicon Valley creates tools for certain people and by certain people, in the same way these algorithms of seeing-understanding are seeing and understanding for some people, and some people/things are being seen and understood. This, which may seem obvious, is in fact a very active exercise of power that we need to center in our attempt to think about alternatives.

Algorithms of seeing-understanding, as much as any other algorithms, are not black boxes, even if they are undoubtedly clouded by many layers of corporate secrecy and technical languages. Their ways of seeing-understanding, and not only the uses they are being put to, are being directly shaped by particular visions and imaginaries of what seeing-understanding through machines is and can do. This relates to Louise Amoore’s calls for focusing on the opacity of algorithms instead of trying to make them “transparent” or “ethical” (Amoore, 2020). She says: “Where the political demands for transparency reassert the objective vision of a disembodied observer, to stay with opacity is to insist on the situated, embodied, and partial perspective of all forms of scientific knowledge.” Amoore is pointing exactly to the need to foreground how understanding is happening and what it is privileging, rather than trying to somehow believe we will ever reach an imagined algorithmic objectivity. Although I may agree with Fei-Fei Li that what really matters is understanding the images, I argue that there are many different ways of seeing-understanding that can be considered or privileged, and that this is crucial for breaking from problematic hegemonic perspectives.

Thus, we can tentatively conclude that our ways of finding alternatives could/should focus both on: 1) Experimenting with ways of seeing and understanding that are different from the imperatives of military-surveillance Computer Vision; and 2) Breaking away from the power dynamics that mire current Computer Vision algorithms, inverting or subverting power relations.

How could this look in practice? I have three suggestions to help us start imagining and conjecturing alternatives. These are (once again) not comprehensive, but provocations for further discussion.

1) The need for a disobedient gaze. How can algorithms turn around what/who is seen, and focus instead on surveilling the most powerful, as opposed to those marginalized?

One of the projects that I would say advances in this direction is Forensic Architecture, from the U.K. Their objective is using the large amount of data generated by surveillance/data capitalism in order to create different ways of seeing, a “counter-forensics” that surveils states, police, and the military. More related to Computer Vision, their project Triple-Chaser created an algorithm that captures images from the Internet and tries to detect what types of bombs are being used in certain regions of the world. Their goal is to pressure the companies that create these bombs and hold them responsible. In direct collaboration with this project is Harvey’s VFRAME, which more broadly offers open-source computer vision detection tools to help/support the work of activists and journalists that cover wars: “Facilitating visual testimony from conflict zones around the world by using computer vision to navigate large volumes of citizen and journalist collected data” (Harvey, 2020).