OpenAI\u2019s former CTO Mira Murati\u2019s new startup recently dropped its first research paper, and it\u2019s about fixing something that\u2019s been bugging AI engineers since ChatGPT launched.<\/p>\n
The paper from Murati\u2019s startup Thinking Machines Lab<\/a> (or, just Thinky, for those in the know) is titled \u201cDefeating Nondeterminism in LLM Inference<\/a>\u201d and tackles the problem of \u201creproducibility\u201d in language models like ChatGPT.<\/p>\n A few definitions before we start:<\/p>\n Nondeterminism: <\/strong>Getting different answers when asking ChatGPT the same question twice with identical settings.<\/p>\n LLM: <\/strong>Large language model like ChatGPT, Claude, or Gemini.<\/p>\n Inference: <\/strong>When you ask a language model a question, and it generates a response.<\/p>\n Even when you set AI models to their most predictable setting (temperature = 0), you still get different answers to the same question. Engineers have been pulling their hair out thinking it was just \u201cone of those computer things\u201d that couldn\u2019t be fixed. Turns out, they were wrong.<\/p>\n The discovery came from Horace He, a PyTorch (code for AI) wizard who recently jumped from Meta to join Murati\u2019s team. He\u2019s the guy behind torch.compile, which is that thing that makes AI models run 2-4x faster with one line of code.<\/p>\n Horace and his team discovered the real culprit isn\u2019t the usual suspect: floating-point math weirdness that engineers typically blame; instead, it\u2019s something called \u201cbatch invariance.\u201d Think of it like this:<\/p>\n Imagine ordering the same coffee at Starbucks, but it tastes different depending on how many other customers are in line. That\u2019s essentially what\u2019s happening with AI models.<\/p>\n When an AI server is busy handling lots of requests, it processes them in batches.<\/p>\n Your request gets bundled with others because that\u2019s more efficient, and somehow this changes your specific answer\u2026 even though it shouldn\u2019t.<\/p>\n Follow the logic, and the busier the server, the more your results vary.<\/p>\n This also happens in real life. Have you ever been to your local Starbucks during coffee rush hour? Unless you have a god-tier level barista, your order might not taste the same!<\/p>\n This matters because of these problems:<\/p>\n AI companies doing research can\u2019t reproduce their own experiments reliably.<\/p>\n Businesses using AI for critical decisions get inconsistent results.<\/p>\n Training new AI models becomes way more expensive when you can\u2019t trust your outputs.<\/p>\n Thinky released its solution as open-source code<\/a>, which is true to Murati\u2019s promise of \u201cscience is better when shared.\u201d The team calls their approach \u201cbatch-invariant kernels,\u201d which basically teaches AI servers to give you the same coffee regardless of the line.<\/p>\n This is just the appetizer from a team\u00a0that recently raised $2 billion\u00a0<\/a>without even having a product (although they are working on one, internally called \u201cRL for businesses<\/a>\u201d that customizes models for a company\u2019s specific business metrics, which sounds very cool).<\/p>\n If fixing decade-old \u201cunfixable\u201d problems is their opening move, the AI giants should probably be nervous (though the code is open, so yum yum yum, as the hungry, hungry, AI labs say).<\/p>\n Editor\u2019s note:<\/em><\/strong> This content originally ran in our sister publication, The Neuron<\/a>. To read more from The Neuron, <\/em>sign up for its newsletter here<\/em><\/a>.<\/em><\/p>\n The post Ex-OpenAI CTO Mira Murati\u2019s Startup Fixed a Major \u2018Unfixable\u2019 AI Bug<\/a> appeared first on eWEEK<\/a>.<\/p>","protected":false},"excerpt":{"rendered":" OpenAI\u2019s former CTO Mira Murati\u2019s new startup recently dropped its first research paper, and it\u2019s about fixing something that\u2019s been bugging AI engineers since ChatGPT launched. The paper from Murati\u2019s startup Thinking Machines Lab (or, just Thinky, for those in the know) is titled \u201cDefeating Nondeterminism in LLM Inference\u201d and tackles the problem of \u201creproducibility\u201d […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-4903","post","type-post","status-publish","format-standard","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/4903"}],"collection":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4903"}],"version-history":[{"count":0,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/4903\/revisions"}],"wp:attachment":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}The problem<\/h2>\n
The solution<\/h2>\n
Why this matters<\/h3>\n
Why the AI giants should be nervous<\/h2>\n