While majority of the GenAI investment \/ capex is focused on new datacenters, GPUs and hardware, is it possible that the long term future of LLM inference and training is actually on local hardware we already have? Two trends worth tracking:<\/p>\n
Our local desktops, laptops and mobile phones hide a surprising amount of compute capacity which is often not used fully. For example, a recent paper estimated<\/a> that M-series chips on Apple laptops can go as high as 2.9 TFLOPS for an M4 and Google\u2019s Pixel 10 Android phone<\/a> can in theory hit 1.5 TFLOPS (for comparison an NVIDIA GeForce RTX 4090 GPU\u00a0can go as high as 82 TFLOPS and the H100 can go to 67 TFLOPS, at FP32).<\/p>\n Local inference stacks like llama.cpp<\/a>, Ollama<\/a> and LM Studio<\/a> have been getting better and better, with underlying improvements such as Apple\u2019s support for inference via MLX<\/a>, support for AMD GPUs<\/a>, and integration into the overall ecosystem via things like MCPs, tools, local web interfaces for coding assistants, etc. All have been showing better and better performance over the past year \u2013 as an example, compare Cline\u2019s recommendations for local models between May 2025<\/a>:<\/p>\n When you run a \u201clocal version\u201d of a model, you\u2019re actually running a drastically simplified copy of the original. This process, called distillation, is like trying to compress a professional chef\u2019s knowledge into a basic cookbook \u2013 you keep the simple recipes but lose the complex techniques and intuition. \u2026 Think of it like running your development environment on a calculator instead of a computer \u2013 it might handle basic tasks, but complex operations become unreliable or impossible.<\/em><\/p>\n and Nov 2025:<\/a><\/p>\n Local models with Cline are now genuinely practical. While they won\u2019t match top-tier cloud APIs in speed, they offer complete privacy, zero costs, and offline capability. With proper configuration and the right hardware, Qwen3 Coder 30B can handle most coding tasks effectively.The key is proper setup: adequate RAM, correct configuration, and realistic expectations. Follow this guide, and you\u2019ll have a capable coding assistant running entirely on your hardware.<\/em><\/p>\n Even OpenClaw (reluctantly) supports<\/a> local models <\/p>\n Because of pressure to squeeze out better performance from existing hardware, open source inference engines such as VLLM and PyTorch, and the models themselves have been focusing on faster inference speed and throughput. For example, VLLM recently announced 38% performance improvements<\/a> for OpenAI\u2019s gpt-oss-120b model. However, what is more interesting is fundamental changes in models themselves. DeepSeek for example, released a paper recently<\/a> showing how to increase transfomer performance via memory lookups. Several model providers such as Google \/ Gemini and LiquidAI<\/a> have been releasing small models intended to run on limited hardware such as phones<\/a>. <\/p>\n2. Model improvements including inference and training.<\/h3>\n