VIDEO
HIGHLIGHT
Log InSign up
Finally, The CORRECT Way to Run Local AI on a Mac
SummaryTranscriptChat

Finally, The CORRECT Way to Run Local AI on a Mac

Local LLMs: Insights and Setup in June 2026

Overview of Local LLMs

  • The speaker discusses their extensive testing with various local LLM setups, including Turbo Quant, GGML, and MLX on different hardware configurations.
  • They aim to clarify their current stance on running local LLMs as of June 2026, focusing on the OMLX framework.

Advantages of OMLX

  • OMLX is identified as the most efficient way to run open-source local LLMs on Mac systems due to its unique features.
  • It enhances MLX by adding functionalities like server management and persistent caching on SSD for improved performance during cold starts.

Technical Features

  • OMLX employs a two-tier architecture where "hot blocks" remain in RAM while "cold blocks" are stored on SSD using a Least Recently Used (LRU) policy.
  • The system allows for faster context building without significant resource sacrifice, improving speed compared to native cache storage from MLX LM.

Comparison with Other Tools

  • The speaker compares OMLX with Ollama and LM Studio; while Ollama is gradually supporting MLX models, LM Studio offers a user-friendly interface for model downloads.
  • Despite its initial appeal, LM Studio has become bloated, prompting the need for a more streamlined solution that conserves RAM resources.

Installation and Configuration

  • Users can download OMLX from GitHub; both DMG files and source code are available for installation.
  • Models can be sourced from Hugging Face or ModelScope; users should search for quantized versions suitable for their needs.

Utilizing Mixture of Expert Models

  • For agentic work, mixture of expert models is recommended; specific examples like Gemma 4's 3.6 model at 8-bit are discussed.
  • The web dashboard provides endpoints necessary for running selected models effectively within the user's environment.

Performance Monitoring

  • Performance metrics such as cache size (1.5 GB noted), RAM usage (80 GB), and token efficiency gains are monitored during operation.

Context Size Management

  • Users can adjust context sizes based on model requirements; however, reloading may be necessary after changes to ensure optimal performance.

Personal Preferences in Model Usage

  • Although demonstrated using Claude code, the speaker prefers Open Code due to its better context preservation capabilities given limited hardware resources.

Integration with Other Applications

  • A new integration called OMLX allows seamless use across multiple providers like Pie; configuration details will be provided in future content.

Future Directions

  • The speaker expresses intent to continue exploring various models and applications while inviting viewer suggestions for future projects.
Video description

█▀█ █▀▀ ▄▀█ █▀▄ █▀▄▀█ █▀█ █▀█ █▀▀ █▀▄ ██▄ █▀█ █▄▀ █ ▀ █ █▄█ █▀▄ ██▄ Download oMLX: https://omlx.ai/ This video explores why OMLX is the definitive choice for founders looking to reclaim their data and run powerful LLMs locally on Mac hardware. Key Takeaways: - Why OMLX is superior to Ollama and LM Studio for professional Mac workflows. - The technical benefits of SSD-backed caching and LRU policies for persistent context. - How to set up agentic models like Qwen 3.6 MoE for real-world coding tasks. - A breakdown of why the M5 Max is the current sweet spot for personal AI infrastructure. - Practical steps to integrate local models into tools like Pie and Open Code. Code examples: https://samuelgregory.co.uk/videos/finally-the-correct-way-to-run-local-ai-on-a-mac Work with me: https://samuelgregory.co.uk --- Support the content: https://www.patreon.com/0x5am5 Twitter: @0x5am5 $ cat tools.txt ──────────────────────────────── Kilo: https://samuelgregory.co.uk/kilo-code Replit (Favourite Vibe Code Tool) : https://samuelgregory.co.uk/replit Perplexity (deep research): https://samuelgregory.co.uk/perplexity Claude Code: https://claude.ai/api/referral/jZ9vnMedyQ&v=p-CzOtUYEyA Warp Terminal: https://samuelgregory.co.uk/warp ⚒️ more at https://samuelgregory.co.uk/tools $ cat services.txt ──────────────────────────────── Domain Names: https://samuelgregory.co.uk/namecheap Hosting: https://www.hostg.xyz/aff_c?offer_id=6&aff_id=130549 Online Storage ($200 credit): https://samuelgregory.co.uk/digital-ocean ⚒️ more at https://samuelgregory.co.uk/tools $ cat gear.txt ──────────────────────────────── Sony A7c II: https://amzn.to/40qaYEJ Lens Sigma 16-28mm: https://amzn.to/3IaDzqx Microphone Samson QU2: https://amzn.to/3TkshCE Macbook Pro M1 Max: https://amzn.to/48736M6 $ cat books.txt ──────────────────────────────── The Full Stack Agency: https://flowst8.dev/store Lingo: Agile: https://thefullstackagency.gumroad.com/l/agile-lingo Lingo: Startup: https://thefullstackagency.gumroad.com/l/startup-lingo $ cat timestamps.txt ──────────────────────────────── 00:00 Finally, the correct way to run AI on a Mac 00:29 oMLX has a special trick up its sleeve 02:40 Where do Ollama and LM Studio land? 03:27 Downloading oMLX 03:48 Rundown of the UI and downloading models 04:49 Serving your local model 06:12 Seeing the cache in action 06:54 Playing around with parameters 07:29 Configuring providers in harnesses #LocalLLM #LocalAI #AI