Impressions and Links from
On (Super) Intelligence
University of Exeter.
Exeter, UK. 2025.

I had the great pleasure of taking part in On (Super) Intelligence 2025 (Shape the future of SuperIntelligence Conference). September 16 - 19, 2025. Exeter.

Tried to follow as much as possible. But, well, these notes are, of course, in no way, shape or form complete...
Rather, these notes were written on conference nights, as my way of keeping track of the events that I attended. And as a way of storing links and references for future reference.

Below you will find impressions from the conference, and links for further reading.

University of Exeter.

1. Presentations Wednesday. September 17th.

1.1. Introduction.

Cédric Mesnage (General Chair) gave an introduction to the conference, and welcomed those who had travelled from overseas to attend.

Participants from UK, USA, Switzerland, Denmark, Abu Dhabi, India, Poland. Super Intelligence, Exeter.

In order to hear more about AGI, and beyond.

Levels of AGI. Super Intelligence, Exeter.

''Levels of AGI for Operationalizing Progress on the Path to AGI'' [1].

All exciting stuff, indeed [2].

1.2. Technology's New Clothes.

After the introduction, by Cédric Mesnage,
Bertie Müller (chair of aisb) gave a talk about ''Technology's New Clothes''.

SuperIntelligence, Exeter. September 2025.

Asking us the question: ''Are we past the AI hype''.

I.e. Agents should not be just souped-up chatbots, or?
At some stage there should be a shift from tools that simply follow instructions,
to systems that can act on context, ideally within guardrails set by their creators?

Shibani Ahuja, SVP Salesforce, has introduced an ''Agentic Maturity Model'', a structured framework that provides a clear roadmap for organizations to navigate the complexities of AI agent implementation:

[Level 0] Fixed Rules and Repetitive Tasks: Organizations automate repetitive tasks using predefined rules.

[Level 1] Information Retrieval Agents: Agents assist humans by recommending actions and retrieving information.

[Level 2] Simple Orchestration, Single Domain: Here, agents autonomously orchestrate low-complexity tasks in a siloed data environment.

[Level 3] (The real turning point...) Complex Orchestration, Multiple Domain: Agents orchestrate multiple workflows with harmonized data across multiple domains.

[Level 4] (Still, largely aspirational...) Multi-Agent Orchestration: Where any-to-any-agent operability is achieved across disparate stacks with agent supervision [3].

As Kevin Quigley, Director of Process Improvement at Salesforce customer Wiley, puts it, ''This model articulates how these solutions will come together as the technology matures and ensures that the building blocks we create today will set us up for success in the agentic future'' [3].

Indeed, AI should be a tool that enhances human qualities:

Creativity, together with cognitive flexibility, to create valuable innovations.

Communication. Use human communication skills to collaborate and develop new ideas.

Emotional Intelligence. Ability to join intelligence, empathy and emotions to enhance thought and understanding of interpersonel dynamics.

Critical thinking. (pro)Actively and skilfully conceptualising, applying, analysing, synthesising, and/or evaluating information gathered from, or generated by, experience, reflection, reasoning, or communication, as a guide to belief and action.

A great talk, indeed.

1.3. Simulating Ourselves to Death.

Constantine Andoniou, Abu Dhabi University, talked about ''Simulating Ourselves to Death''.

From the paper abstract:

While the field of artificial intelligence advances toward a vision of superintelligence, defined by capacities that exceed human cognition across all domains, another trajectory is unfolding in parallel: The silent erosion of human interpretive agency beneath the surface of machine fluency...
...
As large language models and neural architectures simulate increasingly sophisticated expressions of understanding, humans encounter a paradox: The more convincing the simulation, the less effort is required to think.
...
This paper proposes that the true risk of superintelligence is not machine domination, but the gradual disappearance of human meaning under the weight of automated simulation.

According to the paper:

As AI advances towards SuperIntelligence...

Human interpretive effort is silently eroded, as we lean on machine fluency.

Paper's thesis: The danger is not catastrophic domination, but the quiet disappearance of human meaning under the weight of automated simulation.

According to Constantine Andoniou:

Human cognition thrives on contradiction, imperfection and dissonance. These generate reflection and growth.

Critical fields (law, ehics, medicine, art) depend on paradox and tension; AI threatens to erase these conditions.

AI systems, built on probabilistic optimizations, are structurally designed to smooth over contradictions and resolve ambiguity.

Conclusion:

Superintelligence is less about machine dominance and more about human decline.

Central danger: (human) Cognitive erosion and interpretive passivity.

Well, you wonder, when Irving J., Good [4] defined Superintelligence (1965) as: ''Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever'', was this the expected outcome....?

Exeter University Library.

SuperIntelligence 2025.

1.4. Re-Envisioning Superintelligence using Generative AI and Science Fiction.

Soumya Banerjee talked about ''Re-Envisioning Superintelligence using Generative AI and Science Fiction''.

Banerjee offered a vision where humans coexist with advanced machine intelligence.
Arguing that this could also be a possibility...

According to Banerjee:

This stands in sharp contrast to prevailing narratives of superintelligence, which tend to oscillate between boundless techno-optimism (superintelligence is seen as the solution to all human problems) and apocalyptic dystopia (superintelligence is an existential threat to humanity).

For more, see: [5], [6], [7].

Lucem sequimur, i.e. ''We follow the light''.

1.5. Panel: ''Computing Intelligence''.

Next up was an interesting panel discussion with Miriam Koschate-Reis, Fabrizio Costa and Alex Shaw.

Super interestingly, as part of the session,
Fabrizio Costa talked about things AI still struggles with:

Continuous learning.

Limitation: AI lacks continuous learning.
Current state: Models are trained once, then essentially frozen.
Problem: Adapting (often) requires full retraining. Costly, brittle, catastrophic forgetting.
Human contrast: We learn continuously.
Implication: Until AI can learn continuously, it will remain fragile in a fast-changing real world.

Compositionality:

Limitation: AI struggles to breakdown and recombine concepts.
Current state: Treats concepts as indivisible wholes.
Problem: Poor at imagining novel combinations.
Human contrast: Naturally mix and match. A bus in new colors, a tune on a different instrument.
Implication: Lack of composionality limits AI flexibility and efficiency.

New conceptual spaces, new concepts:

Limitation: AI cannot invent new abstractions.
Current state: Confined to the represential space of its training data.
Problem: Can interpolate inside known patterns, but not step beyond them.
Human contrast: We invent theories, styles, frameworks - expanding the universe of ideas.
Implication: True creativity requires creating new spaces, not just remixing old ones.

Metaphotic reasoning:

Limitation: AI lacks cross-domain reasoning.
Current state: Locked inside learned manifolds, strong at pattern matching.
Problem: Cannot leap across domains to generate fresh ideas.
Human contrast: We use metaphors, ''generals coordinate soldiers'' like ''immune systems orchestrating cell response''. to transfer understanding.
Implication: Without metaphoric leaps, AI misses a core ingredient of human inteligence.

Alex Shaw added, that true AGI should also be able to do active inference.
There must be a ''built-in'' curiosity, where ''exploration emerges intrinsically''.

Great insights, indeed.

1.6. Books. September 2025.

For more, see: Out of the AI Box.

SuperIntelligence 2025.

2. Presentations Thursday. September 18th.

2.1. AI Agents and the Future of AI Regulation.

Ana Beduschi talked about ''AI Agents and the Future of AI Regulation''.

Indeed, will the General Data Protection Regulation (GDPR) and the European Union's AI Act remain effective or need future-proofing as these AI technologies develop?
Where Beduschi argued that ''While the GDPR and the AI Act remain a suitable baseline, the challenges posed by agentic AI require a more comprehensive approach''.

Indeed, important work in this new world of ''Agentic AI autonomy''.

Agentic AI autonomy, 2025.

AI Regulation. Exeter, 2025.

SuperIntelligence 2025.

Student community life. Exeter. 2025.

2.2. Making, Observing and Measuring Reflective Righteous Superintelligent Agents Safely in a Virtual Environment.

Shagofta Shabashkhan talked about ''Making, Observing and Measuring Reflective Righteous Superintelligent Agents Safely in a Virtual Environment''.

Virtual worlds like Minetest provide an unmatched platform for this exploration: They scale to billions of safe interactions, allow precise instrumentation, and isolate possible failure modes from the physical world. Repeated trials let us probe alignment strategies and intervene before unsafe dynamics amplify.

AI in Sandbox.

Indeed, clever.
Yudkowsky would probably not be entirely convinced though.
There are probably still ''Out of the AI Box'' problems out there...
But, so far, ok...

2.3. Embodied AI: A new paradigm for AI Safety.

Rebecca Raper talked about ''Embodied AI: A new paradigm for AI Safety''.

From the paper abstract:

Various approaches have been taken to look at how we might align AI; that is, ensure that the decisions an Artificial Intelligence (AI) system makes are in line with human values.
...
AI embedded into physical engineering solutions, brings about new potential possible threats and challenges, and (tell us) that there is a research gap in looking at safety when it comes to Embodied AI specifically.

Exeter.

About the book ''Raising Robots to be good'':

This book introduces this pursuit, known as Machine Ethics, and aims to outline some of the art and science behind the field.
...
A new theory is proposed, inspired by how we might envisage moral agency developing in a child. The proposal is that we need to cultivate moral development in a robot, in pretty much the same way we might parent a child [8].

As one reviewer says (on Amazon): ''Striving to be good is an important step''.
Indeed!

2.4. Open Conference.

Headlines for the Open Conference discussions:

This open conference provides a collaborative platform for participants to generate and cluster ideas, which are then voted on and discussed in randomly assigned groups.

Ideas. SuperIntelligence 2025.

Clusters of ideas. SuperIntelligence 2025.

In our group we came up with the following ''clusters of ideas'':

How to construct the thing. One or many paradigms? What should it (ASI) do? etc.
Good vs bad robot. How will ASI change our perception of value? What is the role of humans after ASI?
ASI and education. How will ASI change our relation to work? Can we be conscious and still outsource much of our cognition to ASI?
What about religion in the age of ASI? Can humans correctly label things we don't understand? Will ASI want to leave Earth?

Indeed, all in all, super interesting, and certainly thoughts and material to consider for future classes in Deep Learning...
and beyond...

Exeter 2025.

3. Presentations Friday. September 19th.

3.1. Solving current AI alignment problems is surprisingly easy...

Stuart Armstrong talked about ''Solving current AI alignment problems is surprisingly easy''.

From the abstract:

Current LLMs continue to hallucinate, attempts to make them into unbiased or jailbreak-resistant models have failed. So, it seems that they have an unsolvable safety problem. However, much of the safety, ethics, and alignment problems are actually quite solvable - once we stop trying to force LLMs into being things they are not, and take advantage of what they're actually good at.

What is an LLM really? Exeter 2025.

Indeed, in a world of ''jailbreaks'' LLMs can easily come across as monsters...

But after this presentation of defenses against ''dark prompts'' and more (Where the defense could be other LLMs specialized in finding ''jailbreak'' attempts), it was possible to feel somewhat assured that there are actually people, out there, working on ''jailbreak'' defense mechanisms. Just as there are systems out there that can prevent (at least some) parasocial conversations (that might lead to delusions, psychosis etc).

Concluding, Armstrong told us that we shouldnt get distracted by ideas that AI is:

Uncontrollable.
Unteachable.
Impossible to interpret or outsmart.

On the contrary. Learning to solve todays problems, and learning how to generalize todays solutions is useful and needed. Indeed, meaningful human control is always useful and needed...

With much more on the ''Aligned AI'' website [9]
(''Aligned AI'' was co-founded by Rebecca Gorman and Stuart Armstrong).
Including the vision:

We believe AI has the power to radically enhance human potential. We are working to ensure that the development of AI safely and democratically empowers every human being [10].

And including many interesting articles, E.g.

Do we do better with LLMs - or do they delude us into thinking so?
Someone tried to measure how much AI speeds up coding when in the hands of experienced developers...
And it turns out that when experienced developers think AI gave them a speed-up (average: +20%), the cold, hard fact is that it slows them down (average: -20%) [11].

A great presentation indeed.