# 259 What Microsoft’s Misunderstood Copilot Study Actually Means for the Language Industry

Working with AI: Measuring the Occupational Implications of Generative AI

Introduction to the Research Paper

• The episode discusses a research paper titled "Working with AI: Measuring the Occupational Implications of Generative AI," published by Microsoft.

• The paper received significant attention and various interpretations, including many misconceptions and critiques on social media platforms like LinkedIn.

Overview of Reactions and Feedback

• There was notable emotional response to the paper, including critical feedback from readers who felt it amplified self-serving views from Microsoft engineers.

• The hosts acknowledge that while some reactions were negative, they find the research interesting and worth unpacking.

Purpose and Data Sources of the Study

• The primary aim of the study is to measure how generative AI impacts occupations, contributing to broader discussions about AI's economic implications.

• Microsoft utilized unique data access to analyze user interactions with their AI tools, specifically focusing on 200,000 conversations involving Microsoft Copilot.

Methodology of Analysis

• The study examines user goals versus tasks performed by AI without distinguishing between work-related or leisure use.

• User satisfaction is measured through feedback mechanisms (thumbs up/down), alongside an analysis of task execution success based on internal standards set by the AI itself.

Key Findings from User Interactions

• Knowledge-related work activities showed the highest overlap in successful task completion when using Copilot, particularly in information gathering, writing, and communication.

AI Applicability in Language Professions

Overview of AI Impact on Various Occupations

• The discussion begins with a list of occupations affected by AI, including interpreters, translators, historians, and customer service representatives.

• The range of jobs spans from information-based roles to physical labor positions like dredge operators and water treatment plant operators.

Understanding the AI Applicability Score

• Researchers clarify that the AI applicability score is not a direct replacement score; high overlap does not equate to job loss.

• Despite interpreters and translators ranking high in overlap with AI capabilities, this does not imply their jobs will be automated or made obsolete.

Misinterpretation of Study Findings

• Headlines suggesting that language professionals will become obsolete are misleading; the study emphasizes assistance rather than full task execution by AI.

• The researchers aimed to convey that while certain tasks can be assisted by AI, it does not mean complete job replacement is imminent.

Caveats and Limitations of the Study

• There is concern about how findings may be interpreted; many assume top rankings indicate vulnerability to automation without considering nuances.

• The researchers likely anticipated misinterpretation but stress the importance of understanding data regarding AI's economic impact.

Insights into Translation and Interpreting Work Activities

• The study highlights that while AI can assist in translation and interpreting tasks, it cannot fully replicate the nuanced work activities performed by professionals.

• A distinction is made between "work activities" and "tasks," emphasizing that decomposing jobs into activities doesn't capture their full complexity.

Value Beyond Task Execution

• An analogy used in the research suggests that simply listing disconnected work activities fails to represent the value added through human interpretation and application.

AI's Impact on Translators and Interpreters

Understanding AI's Capabilities in Translation

• The work activities that AI can perform cover only a moderate amount of the tasks required by translators and interpreters, indicating that AI cannot fully replace these roles.

• Research shows that while AI can handle some activities within occupations, it does not significantly cover all necessary work activities for any single occupation.

• Misunderstandings arise when people focus solely on high coverage scores for specific jobs like interpreters and translators without considering the individual components of those jobs.

• No job has an impact score exceeding 50%, suggesting that while AI may assist in certain areas, it does not dominate any single role completely.

• The hype surrounding AI agents stems from their ability to enhance human capabilities rather than replace them entirely; humans integrate various tasks using their cognitive skills.

Business Implications of AI Integration

• Researchers caution against assuming that high overlap with AI activities will lead to job automation or wage loss; business decisions regarding AI use are complex and multifaceted.

• Occupations where AI assists may see augmentation rather than outright replacement, but this is not guaranteed as downstream impacts of technology adoption are unpredictable.

• Insights from previous studies on machine translation's effects on the translation industry highlight existing knowledge about how technology influences business practices.

Limitations in Current Research

• A notable limitation is the lack of consideration for how embeddedness of AI tools (like MS Copilot in Word/Excel) affects usage among knowledge workers compared to non-knowledge workers.

• The design and user interface of tools like MS Copilot could influence how effectively users engage with these technologies, which was not addressed in the research paper.

Conclusion on Applicability Scores

• For critics of applying an applicability score to translators and interpreters, it's suggested to view it more as an MS Copilot applicability score relevant to their specific tasks.

Passenger Attendance and AI: Analyzing Job Roles

Overview of Passenger Attendance Role

• The discussion begins with the mention of passenger attendance as a significant job role, ranking third in a list that includes interpreters, translators, historians, sales representatives, and writers.

• Passenger attendants are defined as individuals who ensure the safety of passengers aboard various modes of transport such as ships, buses, and trains.

AI's Role in Passenger Safety

• The speaker expresses skepticism about how Microsoft Copilot can assist in ensuring passenger safety during transportation.

• It is noted that while AI may handle tasks like providing information or general assistance (which constitutes about 50% of a passenger attendant's duties), it cannot perform physical responsibilities essential for safety.

Weighting Activities in Job Analysis

• The analysis mentions that activities deemed more critical to the occupation are given higher weight; however, serving meals and beverages might not be prioritized highly.

• A hypothetical scenario is presented where the primary responsibility during an emergency (like a ship sinking) would be ensuring passengers can evacuate safely.

Limitations of AI Assistance

• The speaker humorously questions whether Microsoft Copilot could effectively manage rescue operations during emergencies, highlighting its limitations.

Insights from Research Comparisons

• The speaker shares their experience using AI to summarize research papers and compares findings from Microsoft's paper with those from Anthropic.

• There were discrepancies when identifying authorship between the two papers due to AI misidentification issues.

Conclusion on Research Engagement