Matrix, the open network for secure, decentralized communication, was hacked yesterday. The attackers gained access to the servers hosting Matrix and were then able gain privilege to the production infrastructure by exploiting a vulnerability in Jenkins CI server and then hijacking credentials(SSH logging).
What got compromised? Matrix believes that unencrypted message data, password hashes and access tokens may have been effected. What was left uncompromised were:
- Source code and packages
- Modular.im servers
- Identity server data
The damage may have been a lot worse had it not been for @jaikeysarraf who tipped Matrix off to the Jenkins vulnerability. It was then that Matrix’s investigators realized the full scale of the attack. They were then able to isolate the problem, remove Jenkins, and save the other machines.
This isn’t the first time that Jenkins has posed a major security threat to servers due to credential hijacking. In 2018, ZDNet reported that thousands of servers were vulnerable because two vulnerabilities allowed hackers to gain admin rights using invalid credentials on victims’ servers. The vulnerabilities were patched, however.
In this case, to be fair to Jenkins, the vulnerabilities exist in the plugins or dependencies used by Jenkins. Here are the 3 plugins affected according to MIST.
- A sandbox bypass vulnerability exists in Script Security Plugin 2.49 and earlier … that allows attackers with the ability to provide sandboxed scripts to execute arbitrary code on the Jenkins master JVM.
- A sandbox bypass vulnerability exists in Pipeline: Declarative Plugin 1.3.3 and earlier … that allows attackers with Overall/Read permission to provide a pipeline script to an HTTP endpoint that can result in arbitrary code execution on the Jenkins master JVM.
- A sandbox bypass vulnerability exists in Pipeline: Groovy Plugin 2.61 and earlier … that allows attackers with Overall/Read permission to provide a pipeline script to an HTTP endpoint that can result in arbitrary code execution on the Jenkins master JVM.
Yesterday, Matrix was cautious about making any definitive statements as to whether or not sensitive data had been stolen or downloaded; but today, they’ve been able to provide an update that details exactly how the attacker compromised their machines.
At around 5am UTC on Apr 12, the attacker used a cloudflare API key to repoint DNS for matrix.org to a defacement website (https://github.com/matrixnotorg/matrixnotorg.github.io). The API key was known compromised in the original attack, and during the rebuild the key was theoretically replaced. However, unfortunately only personal keys were rotated, enabling the defacement. We are currently doublechecking that all compromised secrets have been rotated.
Later on, Matrix confirms that encrypted password hashes were stolen.
The defacement confirms that encrypted password hashes were exfiltrated from the production database, so it is even more important for everyone to change their password. We will shortly be messaging and emailing all users to announce the breach and advise them to change their passwords. We will also look at ways of non-destructively forcing a password reset at next login.
In the aftermath, Matrix promises to beef up the security of their production infrastructure. In the case of tools like Jenkins, that calls for more frequent vulnerability checks(all of the vulnerabilities in the NIST’s database were last modified on January 22, 2019).