Small-world Networks of Agents Brainstorm AI Risks to Support Ideation

Problem

The paper addresses a gap in the capability of participatory AI risk assessment, specifically the challenge of surfacing indirect or systemic harms. This is particularly relevant in the context of increasing reliance on AI systems, where traditional methods may overlook complex interdependencies and emergent risks. The work is presented as a preprint and has not undergone peer review.

Method

The authors propose a three-stage ideation support tool designed to facilitate the identification and prioritization of AI risks:

• Stage 1: Dynamic stakeholder discovery is conducted based on the specific use cases of AI, allowing for a tailored approach to identifying relevant stakeholders.
• Stage 2: The tool simulates stakeholder interactions using large language models (LLMs) organized in a small-world network topology. This structure is intended to enhance the ideation process by mimicking real-world social dynamics and information flow.
• Stage 3: Risk prioritization is performed using network centrality measures, specifically betweenness centrality, to identify which risks are most critical based on the simulated stakeholder interactions.

The evaluation method focuses on the effectiveness of the ideation tool in generating a diverse set of risks, leveraging data from 45 AI practitioners for initial risk generation and involving 11 teams of non-western young chatbot users in the ideation process.

Results

The results indicate a significant increase in the novelty of identified risks:

• The ideation tool achieved a novelty increase of 1.1 points over single LLM brainstorming.
• It also showed a 0.5 point increase in novelty compared to agentic LLM brainstorming.
• Treatment teams utilizing the proposed tool identified a broader range of risks, including systemic, human-computer interaction, and environmental risks, when compared to control teams.

Limitations

The authors do not report any limitations in their study. However, the absence of reported limitations may suggest a need for further validation in diverse contexts or with larger participant groups to assess the generalizability of the findings.

Why it matters

This work has significant implications for downstream research and practice in AI risk assessment. By enhancing the ability to surface indirect and systemic harms, the proposed tool could lead to more comprehensive risk evaluations in AI deployment. This could ultimately inform better governance and regulatory frameworks, ensuring that AI systems are developed and implemented with a more holistic understanding of their potential impacts.