Son nouveau modèle IA va faire oublier les LLMs

Name: Son nouveau modèle IA va faire oublier les LLMs
Uploaded: 2026-06-18T06:30:04.000Z
Duration: 2 h 8 min 51 s

Unresolved Challenges in AI Research

The Persistent Problem in AI

Despite 20 years of advancements in AI, including LLMs and deep learning, researchers still rely on algorithms from the early 2000s for critical tasks like predicting company revenues or real estate prices.

Emerging Solutions

A new model developed in France is beginning to change this landscape. While not widely discussed, it has attracted significant investment from industry players who recognize its potential.

Behind the Scenes of Tech Development

Reality vs. Perception

The tech world often presents a controlled and predictable image, but the reality includes unexpected challenges and bugs that developers face regularly.

Introduction to Guests

Meet Marine Morvent

Host Mathieu Sanchez introduces guest Marine Morvent, a researcher at Inria with a passion for various scientific fields beyond just computer science.

Focus on AI Beyond LLM

Exploring Different Aspects of AI

The discussion will focus on areas outside of LLM technology, highlighting other significant developments within the field of artificial intelligence.

Journey into Technology

Marine's Background

Marine shares her journey into tech, starting with her education at Polytechnique where she specialized in biology before transitioning to bioinformatics during her PhD.

Transitioning to Machine Learning

From Bioinformatics to Methodology

During her PhD, Marine began applying machine learning methods to genomic problems and gradually shifted towards more methodological aspects of machine learning development.

Research Focus at Inria

Cancer Genomics Study

Her thesis focused on using tumor genomic profiles to identify patient subgroups based on biochemical similarities rather than just tumor location, aiming for personalized treatment approaches.

Current Work at Inria

Foundation Models for Tabular Data

Currently, Marine works on foundation models specifically designed for tabular data—structured data organized in rows and columns commonly used across various sectors including healthcare and business analytics.

Addressing Missing Data

Handling Incomplete Information

One major challenge is dealing with missing data within tables. This affects predictive modeling performance as models require complete datasets for accurate predictions.

Quantifying Uncertainty

Understanding Prediction Confidence

Another area of focus is quantifying uncertainty in predictions made by models. For instance, when predicting cancer probabilities, it's crucial to measure how confident we are about those predictions being accurate.

Comparing Approaches: LLM vs Traditional Models

Limitations of LLM

While large language models (LLMs), such as those generating coherent text responses, can be applied to tabular data tasks, they are not optimized for statistical analysis compared to traditional methods tailored for such data types.

Efficacy of Traditional Methods

Gradient Boosting Dominance

Historically, gradient boosting methods like XGBoost have outperformed neural networks and other architectures on tabular datasets since their introduction around 2001 due to their efficiency and effectiveness in handling structured data tasks.

Understanding Foundation Models in Tabular Data

Introduction to Foundation Models

The discussion begins with the importance of knowing when to stop in model training, leading into the concept of foundation models.

A foundation model is defined as a pre-trained model on large datasets that can be fine-tuned for various downstream tasks without starting from scratch.

Characteristics of Foundation Models

Current open-source language models (LMs) are considered foundation models, trained on extensive data and allowing for fine-tuning.

The conversation highlights a revolutionary shift in tabular data modeling akin to early advancements in language models.

Historical Context and Paradigm Shift

Prior to 2023, machine learning paradigms required retraining from scratch with each new dataset due to the absence of pre-trained models.

A pivotal paper titled TPFN introduced a pre-trained model specifically for tabular data, marking the beginning of this new paradigm.

Pre-training vs. Traditional Training Methods

The key innovation lies in pre-training on diverse tables, enabling predictions without parameter adjustments.

Unlike traditional methods like XGBoost, which rely heavily on supervised training, this approach emphasizes a different training phase focused on context-based learning.

Learning Mechanisms: In-context Learning

In-context learning allows models to understand patterns through examples provided within prompts rather than explicit instructions.

This method mirrors how LLMs learn functions implicitly by recognizing relationships between input and output values.

Differences Between Supervised and In-context Learning

Traditional supervised learning minimizes error based solely on training data; however, in-context learning utilizes multiple tables simultaneously during prediction.

The model processes both training and test inputs together to generate outputs based on learned statistical relationships across various datasets.

Internal Mechanisms of Foundation Models

During pre-training, these models engage in meta-learning by developing effective prediction algorithms from the training data without predefined functions or parameters.

This flexibility allows them to adaptively select appropriate statistical distributions for making predictions based on input features.

Clarifying Terminology: Deduction vs. Learning

The term "learning" may be misleading; instead, it should be viewed as deduction where the model identifies patterns rather than modifying its internal weights permanently.

Application of Transformers in Tabular Data

Despite being rooted in deep learning principles, current tabular models leverage transformer architectures tailored for structured data analysis.

Challenges with Missing Data

Addressing missing data remains an ongoing challenge; various strategies have been implemented but no definitive best practice has emerged yet within foundation models.

Synthetic Data Generation

To overcome limitations related to real-world datasets' availability, synthetic tabular datasets are created with specific statistical properties that allow effective model training despite lacking semantic meaning.

Practical Applications and Use Cases

These foundation models demonstrate significant traction across industries due to their ease of use compared to traditional machine learning methods requiring extensive tuning and expertise.

Examples Across Industries:

Agriculture: Predicting crop yields based on historical data.

Healthcare: Assessing patient outcomes post-trauma or predicting disease progression.

Finance: Evaluating loan repayment likelihood or detecting fraudulent transactions.

By structuring insights around timestamps linked directly back to relevant sections of the transcript, this markdown file serves as an efficient study guide while encapsulating critical discussions surrounding foundation models applied within tabular contexts.

Discussion on Limitations and Advantages of Tabular Models

Current Limitations

The main limitation discussed is the high inference cost associated with large language models (LLMs), which makes them less accessible for academic teams.

Training large LLMs can be prohibitively expensive, often costing millions, while training their model, Tabical, takes about 25 days on a single GPU.

Performance Comparison

Compared to LLMs that typically have around 10 billion parameters, Tabical operates with only tens of millions of parameters, making it less resource-intensive.

Inference can be performed on CPUs but is slower than on GPUs; processing 10,000 samples takes about one second on a GPU.

Collaboration and Competition in Model Development

Collaborative Efforts

The team does not view their relationship with other labs as competitive but rather collaborative; both are working towards advancements in model development.

They aim to improve their models without engaging in a race against others, focusing instead on addressing limitations like integrating external knowledge.

Recent Developments

A startup related to TPFN was recently acquired by SAP, indicating significant industry interest and investment in this area.

Open Source Models and Their Importance

Open Source Benefits

The release of an open-source state-of-the-art model allows for academic research and provides companies with transparency regarding pre-training processes.

True open source includes not just the model weights but also the underlying architecture and data generation methods crucial for effective training.

Data Generation Insights

The "prior" generator used for creating synthetic data is essential for pre-training models; its effectiveness significantly impacts overall performance.

Research Opportunities Enabled by Open Source

Academic Research Applications

Researchers utilize the prior from Tabical to continue pre-training LLMs, exploring how they can achieve performance levels similar to tabular foundation models while still understanding text.

Diverse Use Cases

Various research areas are being explored using Tabical's framework, including privacy concerns and survival estimation modeling.

Future Trends in AI Models

Market Dynamics

Despite the dominance of LLM marketing and funding, there is potential for tabular models to create new use cases that may be more efficient or cost-effective.

Adoption Predictions

There’s an expectation that tabular models will gain significant traction across various sectors such as enterprise applications and healthcare due to their ease of integration.

Reflections on AI Evolution

Changing Perspectives

The speaker reflects on how perceptions of AI have shifted since completing their thesis in 2018; initial skepticism has evolved into recognition of practical intelligence demonstrated by modern chatbots.

Ongoing Debates

Discussions surrounding what constitutes true intelligence remain complex; even successful chatbots challenge traditional definitions yet showcase impressive capabilities.

Recommendations and Closing Thoughts