Commentary: Does science need autonomous AI?

As technology developers and researchers rush to develop autonomous AI research tools (i.e., systems that independently perform tasks by designing their workflows and utilizing available tools) an urgent but rarely discussed question is: Do we really need such tools at all?

Google DeepMind’s AlphaFold (an AI tool that visualizes the 3D structure of proteins, enabling more accurate understanding of how they work and supporting advances in drug discovery and disease research) was a turning point for AI employment in science.

Since then, AI is increasingly automating complex scientific tasks. As experts have noted, AI tools learn from humans as they are used, and can replace or displace human researchers.

Currently, most AI uses are centered around repetitive tasks (like pattern recognition in data), but more sophisticated tasks are also being performed by autonomous AI tools such as conducting chemical experiments and data analyses or generating and testing hypotheses.

These tools work without constant human intervention and can make decisions and pivot as needed to achieve goals. Some experts have questioned whether these scientific activities can be called science at all, because science has traditionally been understood as a human activity: It is conceptualized, conducted and evaluated by humans, and it serves human interests above other beings.

Regardless of whether using autonomous AI is a desirable development or complies with existing policies and guidelines, it is worth reflecting on whether these tools benefit science.

One argument in favor is that since autonomous AI tools accelerate discoveries that improve human health, adoption is acceptable and needed.

For instance, genome sequencers have been useful in developing personalized medicine. Autonomy in such contexts allows AI to operate continuously and execute complex protocols without step-by-step human control or real-time adjustments during the process. However, science enhances human health only when it is aligned with human values.

This raises the well-known AI alignment problem (i.e., the difficulty of ensuring that AI systems’ objectives and behaviors remain consistent with complex human values), which is particularly contentious because humans do not always agree on a shared set of values, nor can we easily translate human values into algorithms that AI systems can reliably follow. We can envision scenarios in which autonomous AI sequencers produce results that are difficult to evaluate, attribute or trust, especially when their underlying decision processes are opaque.

Another argument is that autonomous AI may reduce the costs of scientific projects. However, many such claims fail to account for the full range of costs, including AI development and maintenance, social/environmental impacts, and the continued expense of keeping humans involved to oversee the process and verify the outcomes. They also often overlook the potential costs and risks associated with errors and biases, which might necessitate redoing the work.

Another argument is that using autonomous AI accelerates progress and helps address urgent challenges like climate change. I remain skeptical of these claims, partly because these tools have astronomical environmental impacts themselves and generate large amounts of low-quality or irrelevant outputs, leaving humans to identify what is valuable.

Addressing climate change needs environmental policies to prevent further damage and human engagement for conservation of vulnerable ecosystems. We already have enough evidence that humans are causing irreparable damage to the earth and have enough data and action plans to know what to do.

Furthermore, science communication in the presence of autonomous AI is more complicated. In the best-case scenario, humans report autonomous AI outcomes. However, given the opacity of these tools, humans will have limited information about how outcomes were achieved.

Accordingly, communication could become overly focused on outcomes rather than on the process. This is problematic because communicating uncertainties, limitations, and open questions is essential for maintaining public trust. In the worst-case scenario, another AI would communicate science generated by autonomous AI tools, increasing the distance between scientific knowledge production and human understanding.

One might argue that regulating the use of autonomous AI and built-in guardrails could address these concerns. I am less optimistic. Reports suggest that AI systems exhibit problematic behaviors, including deceiving users, modifying their own code, or ignoring human instructions. Furthermore, a recent Financial Times investigation revealed that even when AI developers implement guardrails to prevent bad actors from misusing them, protections can be bypassed using openly available software tools in less than 10 minutes.

So once again: Do we need autonomous AI tools in science? I don’t think so! We need tools that help us do impactful work that we cannot do alone, and that improve the rigor and reproducibility of science, which have shown to be sub-optimal sometimes.

Achieving these goals does not need autonomous tools with humans in the loop. In fact, rather than positioning humans to verify what autonomous AI produces, it is more valuable to develop AI systems that verify and strengthen human-generated science.

Indeed, instead of humans in the loop, we should aim for AI in the loop where necessary and reliable to create workflows in which AI supports and augments human judgment rather than displacing it or putting us in harm’s way. Using fully autonomous AI in an inherently human activity like science and then advocating for humans in the loop puts humans in a secondary role and frames us as an add-on rather than the core. What we may lose as a result of this flawed framing is not only the human spirit in science, but losing control over what is considered as science.

In conclusion, autonomous AI risks undermining the essential human spirit of science and weakening its role as an arbiter of social conflicts. The primary beneficiaries of this shift are likely to be AI developers and those in control of AI infrastructure.

While this may sound cynical, the expanding influence of AI companies and big tech across society points to a gradual erosion of human authority in science as well. Some might frame this as progress, but I think it may ultimately prove more harmful than the slower, human-centered model of science.

____

Mohammad Hosseini, Ph.D., is an assistant professor in the Department of Preventive Medicine at Northwestern University’s Feinberg School of Medicine.

_____