Ethical Risks of Framing AI as a Colleague in Business

Summary

This study looks at the ethical effects of framing generative artificial intelligence as an “AI colleague” or assistant in business settings. Using virtue ethics and organizational virtue as a lens, the authors analyze policy documents from China alongside international frameworks such as UNESCO, OECD, the NIST AI RMF, and the EU AI Act. They study how institutional mechanisms like notice, human oversight, risk assessment, and traceable remedies are used to support prudence, responsibility, and fairness. The analysis finds that policies focus on controllability, accountability, and explainability, but give limited attention to how anthropomorphic narratives shift attribution of responsibility. The authors recommend adding anthropomorphic-design risks to risk checklists, strengthening cues that signal human ultimate responsibility, and improving transparency and uncertainty communication in conversational systems.

What the study examined

This work explores the ethical consequences of presenting generative artificial intelligence as an “AI colleague” or assistant in corporate settings. The authors approach these consequences through virtue ethics and the related idea of organizational virtue, which emphasize prudence, responsibility, and fairness in institutions.

The study compares governance texts from China with international frameworks including UNESCO, OECD, the NIST AI Risk Management Framework, and the EU AI Act. It focuses on how institutional mechanisms—such as transparent notice, human oversight, risk assessment, and traceable remedies—are built into policies to support ethical behavior within organizations.

Key findings

Across the policy texts analyzed, regulators and standard-setters commonly stress three themes: controllability, accountability, and noticeability or explainability. These mechanisms are presented as ways to keep systems under human direction and to make system behavior more understandable.

However, the analysis shows that policies pay relatively little attention to the ways anthropomorphic or colleague-like narratives can shift responsibility away from humans. When AI is framed as a peer or teammate, people may be more likely to diffuse responsibility, weaken prudential judgment, or erode organizational integrity.

• The policies studied institutionalize tools like oversight and risk assessment but do not consistently address the specific risks tied to treating systems as social partners.
• Transparency measures are emphasized, but requirements for communicating uncertainty and attribution in conversational systems could be stronger.

Why it matters

How we talk about and design AI affects behavior inside organizations. Framing systems as colleagues can improve workflow, but it can also change who is seen as responsible for decisions and actions. That shift matters for organizational trust, accountability, and ethical practice.

The authors recommend practical policy changes: add anthropomorphic-design risks to risk-assessment lists, strengthen cues that remind users of human ultimate responsibility and internal accountability structures, and refine transparency requirements so conversational systems clearly communicate uncertainty and limits.

FAQ

What main problem did the study identify?
The study found that framing AI as a colleague can shift responsibility away from humans and that policies give limited attention to these attributional effects.

What changes do the authors recommend?
They recommend adding anthropomorphic-design risks to risk assessments, reinforcing cues of human ultimate responsibility, and improving transparency and uncertainty communication in conversational systems.