A Framework for Measuring the Benefits of AI

May 1

5 min read

176

Introduction

Significant research has been invested in studying AI risks, a response to the rapid pace of deployment of highly capable AI models across a wide variety of use cases. Over the last year, governments around the world have established AI Safety institutes tasked with developing methodologies to assess the impact and probability of various AI risks. These range from existential threats, such as the potential 'loss of control' over AI systems, to more immediate concerns, including the perpetuation of historical systemic biases.

While the investment and research into AI risks are crucial (Trustible is proud to contribute to the US AI Safety Institute Consortium) focusing only on the potential harms is omitting the other side of the equation when it comes to AI ethics: benefits. The basis for many ethical decisions is that the benefits of using AI for a certain task outweigh the risks. However, you can only make that assessment if the potential benefits of an AI system are explicitly identified and quantified.

While several organizations, including NIST and MITRE, have started to create taxonomies and frameworks to categorize AI risks, we believe it is equally important to apply a similar structured approach to AI benefits. Developing a framework for AI benefits will help organizations conduct AI impact assessments, make more informed ethical decisions, and ensure that they are investing in AI capabilities that can yield measurable benefits for their organizations, users, and society at large.

In this whitepaper, we outline our framework and taxonomy for measuring the benefits of AI. We include the stakeholder groups considered, the categories of benefits, and an initial list taxonomy labels along with proposed metrics for measuring each one. The goal of this blog post is to enable organizational AI leaders with practical guidance they can use to ethically justify the use of AI and measure their intended benefits.

Stakeholders

Following our ethical AI framework, our benefits taxonomy focuses on 3 key stakeholder groups: People, Organizations, and Society.

People refers to individuals who benefit directly from an AI system. They may be the ‘user’ of the AI system, such as a student using AI to help them learn a new concept, or the ‘target’ of an AI system, such as when AI is used to help veterans fill out confusing insurance claims paperwork.

Organizations refers to businesses, non-profit groups, government agencies, or any other institution that develops or deploys AI. While individuals at these organizations may personally benefit, we typically think of benefits at this level as more abstract concepts such as increased productivity, improved reputation, or direct financial gains.

Society refers to communities, groups of people, and cultural elements. Societal AI benefits could include things like improved public health, reduced community violence, or improved safety of vulnerable populations.

Categories of AI Benefits

In addition to identifying each type of beneficiary, we also identify different categories of benefits for each stakeholder group. We typically think of and measure benefits for each group and category differently.

People	Organizations	Society
Physical Health Impacts to overall physical health of an individual	Financial Impacts to key financial aspects of an organization including business costs and revenue	Justice Impacts to crime, violence, and access to justice
Psychological Emotional or mental impacts to an individual	Customer Satisfaction Direct impacts to customer satisfaction outcomes	Public Health & Safety Impacts to public health
Financial Direct impacts to income or economic standing of an individual	Operational Impacts to organizational efficiency or ability to scale	Environmental Impacts to the environment
Privacy Impacts to protection of personal information	Reputational Direct impacts to the reputation of an organization	Opportunity Impacts on access to economic, education, or other opportunities
Freedom Impacts to an individual’s freedom and autonomy	Legal Direct impacts to an organization’s regulatory and civil liability or exposure	Community Impacts to social interactions and shared experiences
Reputational Impacts to an individual’s social standing, reputation, or dignity	ESG Impacts to environmental, social, and governance factors of an organization	Culture Impacts to key cultural traditions, norms, or values
Opportunity Impacts to access to economic, educational, or other opportunities

Taxonomy

In our effort to help individuals and organizations develop a pragmatic approach to artificial intelligence, we have developed a comprehensive taxonomy of AI benefits. This taxonomy serves as a structured framework to systematically identify and measure the positive impacts that AI can have on various organizational processes and outcomes. By delineating clear categories and metrics, this taxonomy helps stakeholders understand the full spectrum of advantages AI can offer, facilitating informed decision-making and strategic investments in AI technologies.

Name	Description	Stakeholder	Benefit Type
Reduced Task Time	Reducing the time it takes an individual to perform a discrete task.	Personal	Financial
Reduced Task Cost	Reducing the financial costs to an individual to perform a discrete task.	Personal	Financial
Increased Scalability	Enabling a process to efficiently scale to additional users, sectors, or business units.	Organizational	Operational
Improved Task Accuracy	Increased accuracy or performance for a discrete task.	Organizational	Customer Satisfaction
Product Personalization	Customization or personalization of a product/service that increases usability or user value.	Personal	Psychological
Increased Creativity	Increased access to new ideas, or innovative ways of creating products/services.	Personal	Psychological
Improved Knowledge Discovery	Easier and more accurate access to information necessary to accomplish a task.	Organizational	Operational
Improved Customer Retention	Increased probability that a customer continues to use a product/service.	Organizational	Customer Satisfaction
Improved Physical Health	Improved physical health outcomes or information about bodily health.	Personal	Physical Health
Increased Diversity	Improved diversity of communities or groups of people.	Societal	Community
Increased Inclusion	Increased acceptance of people with diverse backgrounds.	Societal	Community
Improved Environmental Sustainability	Improved environment including lowered carbon emissions, reduced waste, or using recycled materials.	Societal	Environmental
Repetitive Task Automation	Automating highly inefficient repetitive tasks, particularly those that contribute to harmful bottlenecks or bloated bureaucracy.	Organizational	Operational
Reduced Risk	Reduction in potential for harm, especially potential legal harm.	Organizational	Legal
Improved Learning	Improved educational outcomes or experiences including faster learning rates or academic achievements.	Personal	Opportunity
Improved Accessibility	Increased accessibility in society for people with disabilities.	Societal	Freedom
Improved Organizational Efficiency	Improvements to overall operational efficiency, including helping facilitate cross team processes, reducing organizational friction, or reducing redundancies.	Organizational	Operational
Increased Product Revenue	Direct increases in product/service revenues or contract growth rates over time.	Organizational	Financial
Improved Public Reputation	Improved social standing and reputation with core stakeholders.	Organizational	Reputational
Increased Product Safety	Improved safety features, detection mechanisms, or failsafes.	Personal	Physical Health
Reduced Carbon Emissions	Reduction in energy usage, water consumption, or direct carbon emission outputs.	Societal	Environmental
Increased Access to Education	Improved access to educational services, institutions, or programs.	Societal	Environmental
Reduced Violence	Reduction in criminal activities or physically harmful incidents.	Societal	Justice