# 209 Sourcing Language Data from the Four Corners of the Earth with XRI Global’s Daniel Wilson

Detailed Discussion on Language Tech and Low Resource Languages

In this section, Daniel Wilson discusses his background in linguistics, the importance of language technology for low resource languages, and the journey that led to founding xri Global.

Background in Linguistics and Language Technology

• Daniel's academic background is in linguistics with a focus on endangered languages.

• He conducted research in the Caucasus Mountains, recording dialects with limited previous documentation.

Consulting for Nonprofit Humanitarian Space

• Founded X LLC to provide consultancy for nonprofit organizations exploring language technology for low resource languages.

• Explored tools to bring these communities online and enhance digital equality through language technology.

Data Collection and AI Development

• Emphasized the need for data collection to build tools for languages lacking resources.

• Transitioned into AI development with the launch of xri Global to address the challenge of integrating AI with all languages.

Complexity of Languages in the Caucasus Region

This part delves into the linguistic complexity of languages in the Caucasus region, highlighting morphological richness and unique features.

Linguistic Complexity in Caucasus Languages

• The Caucasus region offers a diverse linguistic landscape with 40 to 80 different languages across five language families.

• Some languages from this region are known for their extreme morphological complexity, challenging theoretical frameworks like distributed morphology.

Features of Caucasian Languages

• Mentioned working on a language with 64 cases, showcasing intricate structural and spatial case systems.

New Section

In this section, the discussion revolves around the factors contributing to the development of elaborate spatial case systems in languages.

Factors Influencing Language Development

• The topography, including mountains and cultural intersections, plays a crucial role in shaping elaborate spatial case systems.

• Languages in certain regions do not have clear origins; some belong to language families unique to specific areas like the Caucasus.

• The geographical features of the Caucasus region are highlighted as influential in language evolution.

Xri Global Company Overview

This part delves into an overview of Xri Global, focusing on its mission and target audience.

Xri Global Mission and Target Audience

• Xri Global aims to address the linguistic gap online by catering to three billion people whose languages are not adequately represented.

• Commercialization efforts focus on industries working towards humanitarian causes and geospatial projects.

Low Resource vs. Medium Resource Languages

The conversation shifts towards defining low resource and medium resource languages based on data availability and tooling support.

Language Resource Classification

• Low resource languages lack sufficient online data and tools for effective communication integration.

• Medium resource languages like Vietnamese or Thai have more data available compared to low resource ones but less than high resource languages like English or Mandarin.

Language Model Limitations

Exploring limitations of language models when faced with unfamiliar or low-resource languages.

Challenges with Language Models

• Some language models struggle with lesser-known languages, providing nonsensical responses due to inadequate training data.

Company Processes and Language Data Collection

In this section, the speaker discusses how their company determines the languages to support based on customer needs and the process of collecting language data efficiently.

Customer-Driven Language Support

• The languages supported by the company are primarily determined by customer demand. For instance, if a group of farmers in Kenya requires additional language support beyond what is currently available, they would approach the company for assistance.

Domain-Specific Model Building

• The company adopts a domain-specific approach when building models for different languages. Models are tailored to meet specific needs within a particular domain rather than being generalized across all areas.

• This strategy ensures that resources are focused on relevant aspects of language processing, optimizing efficiency and effectiveness.

Efficient Model Development and Data Collection

This segment delves into the company's system for rapid model development in specific domains and maximizing intelligence extraction per utterance through efficient data collection methods.

Fast Model Deployment Strategy

• The company has established a system focused on swiftly deploying working models in specific domains. The primary goal is to achieve maximum intelligence extraction per utterance efficiently.

• By prioritizing speed and affordability, the company aims to streamline model development processes while maintaining high-quality outcomes.

Optimal Data Collection for Enhanced Intelligence

• Emphasizing the importance of ideal datasets derived from native speakers, the company strives to collect data that reflects perfect linguistic samples for enhanced model performance.

• Through meticulous data collection practices, including pre-processing source sentences and engaging local vendors for native speaker interactions, the company ensures high-quality language data acquisition.

Language Data Sourcing Process

This section outlines the detailed process employed by the company to source language data effectively through pre-processing steps and collaboration with local vendors.

Pre-processing Source Sentences

• Before initiating language data collection efforts, the company undertakes pre-processing activities on source sentences tailored specifically for targeted domains.

• This preparatory step optimizes subsequent data gathering processes by focusing on domain-specific linguistic content.

Collaboration with Local Vendors

• To facilitate efficient language data sourcing, the company collaborates with local vendors who engage native speakers in various tasks using dedicated platforms such as mobile apps or web interfaces.

• By leveraging local expertise and resources, including project management suites for task distribution, the company ensures timely acquisition of diverse language datasets.

New Section

In this section, the speaker discusses the breaking down of language barriers and the importance of accessing knowledge in one's native language.

Breaking Down Language Barriers

• The speaker highlights the significance of breaking down language barriers, emphasizing the joy and fun in enabling access to information in one's own language.

• There is a realization that barriers to accessing knowledge vary globally, with some regions underestimating the extent of these obstacles.

• An anecdote is shared about interviewing individuals to understand how utilizing their native language online could benefit them, revealing differing priorities within communities.

New Section

This section delves into community priorities and challenges beyond linguistic accessibility.

Community Priorities

• An older gentleman expresses that while online language use would be beneficial, more pressing needs like infrastructure development take precedence in communities.

• The discussion shifts towards generational solutions, where acquiring skills through technology could empower individuals to address community challenges effectively.

New Section

Here, the conversation explores prioritizing community needs over external solutions and considerations regarding specific languages like Aeran.

Addressing Community Needs

• Emphasis is placed on addressing immediate community needs rather than imposing external solutions based on perceived benefits.

• The mention of Aeran as a Turkish-like language raises questions about unique scripts and challenges related to tokenization issues faced by certain languages.

New Section

This segment focuses on tokenization challenges for indic languages and advancements in overcoming these obstacles.

Tokenization Challenges

• Tokenization discrepancies between English and indic scripts result in higher costs for AI implementation among certain linguistic groups.

• Despite initial hurdles, progress has been made by native speakers resolving tokenization issues for indic scripts, indicating advancements in this area.

New Section

The discussion transitions to machine translation integration with llms and synthetic data generation through Mt.

Machine Translation Integration

• Integrating machine translation with llms offers enhanced accessibility to resources across different languages, facilitating efficient communication and interaction.

Detailed Overview

In this section, the speaker discusses the development of conversational AI that can run on a single laptop with a GPU, enabling offline communities to interact with quality information even if resources are only available in English.

ASR TTS nmt Development

• Conversational AI developed to run on a single laptop with a GPU for offline communities.

• Reference to "autop polyglot" experiment aimed at generating high-quality data for training language models.

• Idea of using rule-based systems to generate data for training language models more effectively.

• Experimentation with teaching AI to read grammar and compile machine translation engines.

• Creation of autop polygot experiment focusing on rule-based machine translation system.

AI Agent Development

The speaker explores the limitations of rule-based systems compared to large language models (LLMs), leading to the development of an AI agent capable of reading unstructured data from grammars and producing sentences in low-resource languages.

Rule-Based Systems vs. LLMs

• Comparison between outputs of rule-based systems and LLMs in terms of naturalness and human-like capabilities.

• Creation of an AI agent parsing English sentences, learning rules from grammar, and writing sentences in low-resource languages.

• Success achieved by the AI agent in producing sentences in low-resource languages through reading grammars.

Ethical Considerations

The discussion shifts towards ethical considerations related to leveraging agentic systems for harnessing unstructured data effectively, highlighting the potential for solving new problems ethically within language technology.

Ethical Use of Agentic Systems

• Exploration of how agentic systems can utilize unstructured data ethically to address new challenges.

Data Compliance and Collection Challenges

The speaker discusses the challenges related to data compliance, data sovereignty, data transfer, and data localization laws when collecting data globally.

Legal Considerations in Data Collection

• Emphasizes not collecting sensitive personal identifiable information like names, email addresses, or phone numbers to comply with various countries' laws.

• Highlights the importance of adhering to regional regulations regarding non-sensitive personal identifiable information and obtaining necessary consent for data collection.

Evolving Data Laws Landscape

• Mentions the evolving nature of global data laws due to increased awareness and concerns about fairness in AI development.

• Discusses the ethical considerations around using people from developing contexts for labeling tasks in AI systems.

Future Plans: Language Expansion and On-device AI

The speaker shares upcoming plans for expanding language support and introducing on-device AI capabilities.

Language Expansion Goals

• Indicates a growing demand for more languages and aims to exceed 50 new machine translation systems within the year.

• Discusses enhancing AI capabilities through an on-device tool while waiting for advancements in edge device inference processing.

On-device AI Advantages

• Describes how providing AI on laptops with GPU capabilities can benefit communities without internet access or limited resources.