Cloud security provider Wiz announced yesterday that it found a vulnerability in Microsoft Azure’s managed database service, Cosmos DB, which provided read / write access for each database on the service to any attacker who found and exploited the error.
Although Wiz only found the vulnerability – as it called “Chaos DB” – two weeks ago, the company says the vulnerability has lurked in the system for “at least several months, possibly years.”
A sling around Jupiter
In 2019, Microsoft added the open source Jupyter Notebook functionality to Cosmos DB. Jupyter Notebooks are a particularly user-friendly way to implement machine learning algorithms; Microsoft specifically promoted Notebooks as a useful tool for advanced visualization of data stored in Cosmos DB.
The Jupyter Notebook functionality was activated automatically for all Cosmos DB instances in February 2021, but Wiz believes that the current error is likely to go further back – possibly all the way back to Cosmos DB’s first introduction of the feature in 2019.
Wiz does not give away all the technical details yet, but the short version is that misconfiguration in the Jupyter function allows for an escalation of privileges. This exploitation can be abused to gain access to other Cosmos DB customers’ master keys – according to Wiz, some other Cosmos DB customer’s master key, along with other secrets.
Access to the master key of a Cosmos DB instance is “game over”. It allows full read, write and delete permissions to the entire database belonging to the key. Wiz’s chief technology officer Ami Luttwak describes this as “the worst website you can imagine,” adding: “This is Azure’s central database, and we could access any customer database we wanted.”
Unlike volatile secrets and tokens, Cosmos DB’s master key does not expire – if it has already been leaked and has not been changed, an attacker could still use this key to filter out, manipulate or destroy the database years from now.
According to Wiz, Microsoft only emailed about 30 percent of its Cosmos DB customers about the vulnerability. The email warned users to rotate the master key manually to make sure that any leaked keys are no longer useful to attackers. These Cosmos DB customers are the ones who had Jupyter Notebook functionality enabled during the week that Wiz explored the vulnerability.
Since February 2021, when all new Cosmos DB instances were created with Jupyter Notebook features enabled, the Cosmos DB service automatically disabled Notebook functionality if not used within the first three days. This is the reason why the number of notified Cosmos DB customers was so low – around 70 percent of the customers not notified by Microsoft had either disabled Jupyter manually or disabled it automatically due to lack of use.
Unfortunately, this does not cover the full range of vulnerabilities. Because every Cosmos DB instance with Jupyter enabled was vulnerable, and because the master key is not a volatile secret, it is impossible to know for sure who has the keys to which instances. An attacker with a specific goal could have quietly harvested the goal’s main key, but not done anything disgusting enough to be noticed (yet).
Nor can we rule out a broader consequence scenario, with a hypothetical attacker scraping the master key from each new Cosmos DB instance during the first three-day vulnerability window, and then storing these keys for potential later use. We agree with Wiz here – if your Cosmos DB instance can ever has Jupyter notebook functionality enabled, you should rotate the keys immediately to ensure future security.
Microsoft disabled the Chaos DB vulnerability two weeks ago – less than 48 hours after Wiz privately reported it. Unfortunately, Microsoft itself cannot change customers’ master keys; Cosmos DB customers are responsible for rotating the keys.
According to Microsoft, there is no evidence that malicious actors found and exploited Chaos DB before the Wiz discovery. An e-mail from Microsoft to Bloomberg said, “We are not aware that customer data is available because of this vulnerability.” In addition to warning 3,000+ customers about the vulnerability and giving instructions to reduce it, Microsoft Wiz paid a $ 40,000 premium.