VIDEO
HIGHLIGHT
Log InSign up
INETUM - Inexus - Self Organizing Data
SummaryTranscriptChat

INETUM - Inexus - Self Organizing Data

Log File Analysis and Lineage Extraction

Uploading Log Files

  • The process begins with selecting various log files from different systems, including Oracle, middleware, front end, SAS, PGA, dashboard, DB2, and Kubal JCL.
  • The first log file uploaded is the front-end log associated with an HTML5 JavaScript application.

Extracting Lineage

  • The AI models Gro Llama Code and GPT-4 are tasked to analyze the uploaded log file to extract data lineage independently.
  • Another AI model (GPT-4 or CL Opus) will verify the extracted lineage for integrity and accuracy before representing it as a graph.

Enriching Lineage Data

  • The verification process includes identifying errors and duplicates in the lineage data while merging any duplicates found.
  • By utilizing multiple AIs for independent checks, the risk of hallucination in outputs is minimized.

Growth of Lineage Graph

  • As more log files are added to the analysis, connections within the data increase leading to a more extensive lineage graph.
  • An exported lineage table summarizes user inputs and transformations applied at each step of processing.

JSON File Download

  • Users can download a JSON file containing detailed lineage information such as source elements, transformation rules, targets, and human-readable descriptions.
  • This JSON data can be converted into SQL statements for further use in database queries or applications.
Video description

This video demonstrates how INexus leverages multiple Large Language Models (LLMs) to analyze organizational software application logs and source code (when available) to automatically extract comprehensive data lineage. This non-intrusive process maps how data flows and transforms across all enterprise applications, generating a complete end-to-end lineage. The resulting lineage can be used to auto-generate SQL commands for modern data platforms such as Databricks and Snowflake. Additionally, it serves as foundational intelligence to train an enterprise-wide LLM—giving it full visibility into business rules and data usage, to ultimately transforming it into the cognitive brain of the organization.