{"id":10046,"date":"2026-10-19T12:34:25","date_gmt":"2026-10-19T12:34:25","guid":{"rendered":"https:\/\/cybersecurityinfocus.com\/?p=10046"},"modified":"2026-10-19T12:34:25","modified_gmt":"2026-10-19T12:34:25","slug":"microsoft-nvidia-push-provenance-checks-as-open-weight-ai-adoption-accelerates","status":"publish","type":"post","link":"https:\/\/cybersecurityinfocus.com\/?p=10046","title":{"rendered":"Microsoft, Nvidia push provenance checks as open-weight AI adoption accelerates"},"content":{"rendered":"<div>\n<div class=\"grid grid--cols-10@md grid--cols-8@lg article-column \">\n<div class=\"col-12 col-10@md col-6@lg col-start-3@lg\">\n<div class=\"article-column__content\">\n<div class=\"container\"><\/div>\n<p class=\"wp-block-paragraph\">Microsoft and Nvidia are urging enterprises to assess the security of open-weight AI models throughout deployment, warning that changes to model weights and configurations can weaken safeguards even when a model retains its ability to perform assigned tasks.<\/p>\n<p class=\"wp-block-paragraph\">In a joint paper published last week, the companies identified 30 risk types across five categories and outlined a process for evaluating, mitigating, and monitoring them. The guidance calls for enterprises to verify model provenance, test the configurations they intend to deploy, and reassess risks after material changes.<\/p>\n<p class=\"wp-block-paragraph\">\u201cWeight openness alone does not determine an AI system\u2019s risk,\u201d the paper <a href=\"https:\/\/github.com\/NVIDIA\/OpenEnterpriseRiskManagementGuidance\/blob\/main\/Papers\/Risk%20Management%20Guidance.pdf\" target=\"_blank\" rel=\"noopener\">said<\/a>. \u201cRisk depends on the model, use case, deployment, and safeguards.\u201d<\/p>\n<p class=\"wp-block-paragraph\">The framework covers cybersecurity, data protection, content, human-AI interaction, and bias. Its cybersecurity category includes malicious code generation, malicious tool calls, extraction of the model\u2019s safety policies, excessive token generation, and behavior that diverges from the intended system goal.<\/p>\n<p class=\"wp-block-paragraph\">The paper draws on existing NIST and ISO risk-management standards and is the first in a planned series on managing risks across open-weight AI deployments, the companies <a href=\"https:\/\/www.microsoft.com\/en-us\/security\/blog\/2026\/10\/16\/managing-open-weight-model-risk-in-enterprise-deployments\/\" target=\"_blank\" rel=\"noopener\">said<\/a> in a blog post.<\/p>\n<h2 class=\"wp-block-heading\">Model changes can weaken safeguards<\/h2>\n<p class=\"wp-block-paragraph\">Open-weight models can be fine-tuned, merged with other models, modified, or repackaged for different uses. But those changes can weaken a model\u2019s safety protections even when it continues to perform its intended task well, Microsoft and Nvidia said.<\/p>\n<p class=\"wp-block-paragraph\">\u201cSafety behaviors are often the more fragile of the two,\u201d the companies wrote. Changes to model weights, fine-tuning, quantization and configuration can \u201cdegrade safeguards while leaving task performance intact.\u201d<\/p>\n<p class=\"wp-block-paragraph\">That creates a challenge for enterprise security teams: a model may continue to deliver accurate results on routine tasks even after changes have affected how it responds to unsafe requests. The companies advise enterprises to assess safety and capability separately, test the specific model configuration they intend to deploy, and repeat evaluations after significant changes.<\/p>\n<p class=\"wp-block-paragraph\">\u201cNo volume of behavioral testing establishes what an artifact is, only how it behaves,\u201d the paper said. It recommends using cryptographic signatures to verify that a model matches the publisher\u2019s original artifact.<\/p>\n<p class=\"wp-block-paragraph\">\u201cWhere signatures are unavailable, record the unverified integrity as an open risk rather than treating evaluation results as closing it,\u201d the paper recommended.<\/p>\n<p class=\"wp-block-paragraph\">For enterprises, Gartner recommends independent adversarial testing, static scanning of model files, and provenance verification before and after deployment. Gartner\u2019s proposed vetting process also calls for verifying model sources, checking file integrity, conducting adversarial testing, and recording approved models in an enterprise registry.<\/p>\n<h2 class=\"wp-block-heading\">Security must extend beyond the model<\/h2>\n<p class=\"wp-block-paragraph\">Microsoft and Nvidia caution that evaluating a model alone does not establish the safety of the application built around it.<\/p>\n<p class=\"wp-block-paragraph\">\u201cModel evaluation alone does not establish the safety of an agent or application; enterprises must also assess permissions, data access, connected tools, runtime behavior, and operational controls,\u201d the paper says.<\/p>\n<p class=\"wp-block-paragraph\">The guidance recommends assessing models in their intended operating context and maintaining controls across the model, runtime, harness, and application layers. It also calls for monitoring model inputs, outputs, and tool use after deployment.<\/p>\n<p class=\"wp-block-paragraph\">According to Gartner, open-weight model traffic must be routed through governed AI gateways to enforce authentication, prompt-injection defenses, sensitive-data protection and audit logging.<\/p>\n<p class=\"wp-block-paragraph\">The research firm also recommends isolating self-hosted runtimes and applying controls suited to other deployment models, including private network paths and customer-managed encryption keys for cloud API consumption.<\/p>\n<p class=\"wp-block-paragraph\">Gartner reported that client inquiries about open-weight models through its research inquiry service rose by more than 30% in the second quarter of 2026 compared with the first quarter. The firm projects that open-weight models will account for 50% of enterprise AI use cases by 2028, up from 20% at the time of its report.<\/p>\n<h2 class=\"wp-block-heading\">Staffing and accountability matter<\/h2>\n<p class=\"wp-block-paragraph\">Sanchit Vir Gogia, chief analyst at Greyhound Research, said self-hosting requires enterprises to account for the people and processes needed to maintain safeguards.<\/p>\n<p class=\"wp-block-paragraph\">\u201cSelf-hosting is a staffing decision as much as an infrastructure one, and the people to carry it must move with the model,\u201d Gogia said.<\/p>\n<p class=\"wp-block-paragraph\">He cautioned against assuming that self-hosting necessarily means replacing all the safeguards available through managed services. In his assessment, a significant gap can be the provider\u2019s misuse-investigation function, while customer responsibilities depend on the service and its configuration.<\/p>\n<p class=\"wp-block-paragraph\">Gogia said the Microsoft-Nvidia paper provides useful guidance but leaves important areas for future work. The authors say subsequent publications will extend risk-management guidance across the runtime, harness, and application layers.<\/p>\n<p class=\"wp-block-paragraph\">\u201cThe sharper question is who is contractually obliged to re-test and patch the model when a new stripping technique lands,\u201d Gogia said. For CISOs, Gogia recommends assessing the model, deployment, and intended use before approving production workloads. He also advises security teams to retain control of their risk registers rather than relying exclusively on a vendor-developed taxonomy.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Microsoft and Nvidia are urging enterprises to assess the security of open-weight AI models throughout deployment, warning that changes to model weights and configurations can weaken safeguards even when a model retains its ability to perform assigned tasks. In a joint paper published last week, the companies identified 30 risk types across five categories and [&hellip;]<\/p>\n","protected":false},"author":0,"featured_media":10047,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-10046","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-education"],"_links":{"self":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/10046"}],"collection":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10046"}],"version-history":[{"count":0,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/10046\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/media\/10047"}],"wp:attachment":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10046"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10046"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10046"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}