A $1 billion law firm last week learned a critical cybersecurity lesson: Even something as innocuous as the ceiling on the number of packages allowed in GitHub can increase an enterprise’s threat profile by undercutting the least privilege principle.
When the problem was initially discovered early this month, it presented the consulting firm handling the application issue for the law firm with a seemingly impossible dilemma.
Scott Bellware, co-CEO of BrightWorks Digital, which was doing the work for its law firm client, said the problem cropped up when the BrightWorks team was trying to handle a file transfer.
“We discovered a 500-package limit for GitHub packages for any user other than an organizational admin. As a result, only people with organizational admin privileges can install all packages,” Bellware wrote in a LinkedIn post. “Those without those privileges can only install the first 498 packages. New packages, of course, represent new work. New work, which a significant share of what the team is doing, is stopped in its tracks. The cost of this is understandably eye-watering.”
After trying various work-arounds, Bellware’s team realized the most practical solution would violate least privilege: “Our only option is to give organizational admin privileges to every single contributor on our team of 25+ people. The security implications of this are shocking,” Bellware wrote.
Making the situation worse was BrightWorks’ initial interactions with support for GitHub, which has been owned by Microsoft since 2018.
“After filing a critical support ticket with GitHub, we received a message days later informing us that the person to whom this matter could be escalated has been out of the office,” Bellware wrote. “Literally one single person to whom a critical support matter could be escalated out of the entirety of the GitHub technical staff.”
But Bellware never had to make that security compromise because he reached out to the Microsoft VP in charge of developer communities, Scott Hanselman, who Bellware had known for 25 years.
Hanselman assigned the matter to Martin Woodward, GitHub’s VP of developer relations, who was hired by Hanselman, Bellware said.
Within a day of the request, GitHub increased the accessible package limit to 1,000 “for team members who are not organizational admins,” Bellware said, adding that Microsoft told him that it is working on a permanent fix to the issue and would be releasing it “soon.”
One Microsoft official confirmed the details, but would not speak on-the-record about what happened.
Still, Microsoft’s unusually fast action helped BrightWorks and its law firm client avert a difficult cybersecurity tradeoff.
Value of vendor relationships
In an interview with CSO, Bellware said that when the package limit was set quite some time ago, it was not considered likely that many would need to exceed the 500-package ceiling. Current development efforts, however, are making that more of an issue.
Discussing why they need so many packages, Bellware said there are many ways enterprises can use packages. “You can have a large number of packages because you are incredibly disorganized or because you are incredibly organized,” Bellware told CSO. “We use packages in the way that we are supposed to use them, to track units of deployment.”
The problem started on May 7 when Bellware’s team released its packages to the GitHub package repository. “They turned around and attempted to install the packages in server environments” when they discovered that only one team member — the sysadmin — could do it.
The law firm project was beyond 500 packages, but not materially more, with Bellware estimating that they needed access for about a dozen more packages.
“We experimented with a temporary way to free up some space” and it involved “unpublishing some packages that we knew were not going to be installed anytime soon,” Bellware said.
According to Bellware, GitHub representatives said they are working on major changes so the doubled-ceiling fix is only temporary. “Instead of trying to make a permanent repair to something that they knew would be replaced, they just did a temporary patch,” Bellware said.
Had he not had a personal connection inside Microsoft, it would have been difficult to get the matter fixed, Bellware said, adding that no one at Microsoft or GitHub knew who the client was until the very end. “To get something done with Microsoft, you need to be willing to do something public about it,” Bellware advised.
The package limit “was there for a purpose and it ended up interfering — accidentally — with our purpose,” he said.
The incident is a good reminder that creating something that seems rational and reasonable, such as setting a limit of the number of packages that anyone other than a sysadmin can handle, can deliver unexpected consequences that can derail cybersecurity protections.
For CISOs, it also shows the value of strong vendor relationships, as going public with security-related problems isn’t always a viable option.
More by Evan Schuman:
The most dangerous time for enterprise security? One month after an acquisition
Nearly 10% of employee genAI prompts include sensitive data
The SolarWinds $4.4 billion acquisition gives CISOs what they least want: Uncertainty
>
No Responses