If one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository.
I am always uneasy when an application even thrust worthy like sentry asks access to ALL my private repositories even when they dont need it.
I feel that in 2022 GitHub is way too critical to many organizations and they should really work on stricter rules. Even google requires special and costly audits from oauth applications to access users critical data like drive and emails.
Github has (had) some of the worst wording in the permission prompts as well as too many 3rd party apps asking for blanket permission to do anything and everything in all your repos. It might as well be "hey, give us root on your servers and our new service will do cool things for you!"
And even worse is the 1000s and 1000s of repos that have signed up for those services including all kinds of popular libraries. They've given the keys to hack the libraries at will to 10s or 100s of random companies that integrate with github. It's insane.
> If one thing can be gotten from that incident is forcing GitHub to create finer OAuth scopes and per repository.
This. Limiting OAuth app access to specific repositories. I really wish this was a thing. For this exact reason I'm creating a new GitHub account for every new integration and share the repo with this account. It's technically against the TOS (they only allow single bot account per user) but there is no other way if you want security.
Maybe GitHub will implement this feature now. My hopes are high.
Aren't OAuth tokens scoped to org? Wouldn't it be easier to create an org per repo? I guess it might get expensive depending on your usage/collaborators.
Edit: nevermind OAuth tokens are not org-scoped, only apps are
Only "app installations" from the GitHub marketplace can be limited to specific repositories. For OAuth apps this is not possible sadly (everything that uses GitHub to login or connect accounts, e.g. Travis CI, Terraform Registry, Heroku).
How _lucky_ that github itself was the subject of the attack in npm.
Unless I'm missing something, this attack could have gone unnoticed for a long time (it would be hard for someone to connect a random breach in their infrastructure to an oauth intrusion affecting two of their service providers).
Neither Heroku nor GitHub are addressing the key part of this - if Heroku's Dashboard apps were compromised, and thus access was given to connected GitHub repos for download - were the secrets in the Heroku config vars for the app also visible? That is the nightmare scenario for the affected app's owners.
"Salesforce continues to investigate this incident in coordination with GitHub and our retained third-party breach vendor. Once we identify how the threat actor gained access to customers' OAuth tokens, we will immediately take appropriate actions."
Sounds like they simply don't know yet how the actor got access and what else was exposed.
The token isn’t nearly as concerning as what it implies: this stolen token would have been stored in heroku’s DB. However it got out likely has broader implications.
Just going to say that this is the kind of scenario I was discussing in https://mjg59.dreamwidth.org/59353.html . RFC 8705 actually provides a mechanism for dealing with this, so I guess I'm going to be trying to implement that for our infrastructure in the near future.
> Based on subsequent analysis, we believe this API key was obtained by the attacker when they downloaded a set of private npm repositories using a stolen OAuth token from one of the two affected third-party OAuth applications described above.
Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub? Seems like a bad idea, though I’m sure it’s hard to get off that practice if you’ve been on it for a long time.
> Does this imply there was unencrypted AWS credentials stored in an NPM repository and/or possibly GitHub?
Large organisation, hundreds of developers. Kind of likely that something is visible somewhere. Still a bit surprising it’s the prod access keys for npm…
We're a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don't show this, you can only see this information in the traffic graphs (/graphs/traffic). If you're seeing cloning of repos that you haven't touched in a while, you've likely been compromised.
For all the complexity of OAuth, an OAuth token is essentially a free pass for $stuff.
Perhaps we could start validating that a token not only is valid, but also is digitally, freshly signed with a known certificate?
I'd say there's a huge difference between:
* I stole a token
* I stole the token AND am able to keep signing said token with my victim's public certificate
A token can be stolen from many places and is comparatively easy to obtain, while full intrusion into an org's infrastructure is less likely, and even if it happens, eventually will be remediated (and the related certificate will be revoked).
I like this idea makes tons of sense… not only is the token valid but it’s from a trusted source… it is not going to prevent future attacks but it will add one more lock for an attacker to break before gaining access that is a good thing…
I think HTTP request signing will be a non-starter for an interoperable standard like oauth because of the difficulty of canonicalization without an authoritative library. AWS can pull that off because they have an SDK that will canonicalize requests for you, and it's tested in lockstep with the server accepting it. Historically, XML (and JSON to a lesser extent) have been security minefields for this exact reason.
AFAIK according the draft HTTP request signing rfc [1] defines explicit instructions for normalising, serialising and signing HTTP headers (not body). However, you can use an HTTP Digest header for example, that is a hash of the plaintext body to authenticate the body
I was confused because I got an email from Salesforce to my personal email. (We use Salesforce at work.) Email looked phishy (e.g. unsecured (non-https) links). Apparently they acquired Heroku in 2010.
So this means they also got access to all the private npm package code? Compromising all the organisations using travis or heroku is one thing. But all npm packages quite another.
It's pretty plain that source code exfiltration occurred, though. It's not clear to me how exactly to confirm that the exfiltration happened to your account or not.
From the other thread:
> GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
Probably for a lot of us, a source code leak is not the end of the world, but leaked secrets (i.e. config vars) should be addressed immediately. Rotating credentials will take time. E.g. 20 heroku apps with 30 secrets per app could realistically take several hours (even days) to fully rotate creds.
A few days before this attack campaign started, I wrote a guide to help security and engineering teams prioritize and remediate thousands of secrets-in-code incidents.
Hope it can help some of the organizations dealing with this right now!
I still don't totally understand, can anyone read between the lines to explain what's going on here? The attackers stole what sort of OAuth tokens with what sort of access?
I could be wrong but you know when you sign up for services you have option of signing in with github or other oAuth providers? I think they used this to gain access to various private repos, npms, cloud credentials.
So it seems like if your company had some super secret project running on github, heroku, AWS, they had full access to it.
This is my rough understanding just skimming so could be wrong.
I guess here's what I'm wondering: did they somehow get multiple Oauth keys, one per account compromised on github -- from both Heroku and Travis, just coincidentally both heroku and travis exposed their customers keys somehow?
Or did they get one Oauth key each from Heroku and Travis that exposed all of their customers data on github? Are there keys that would do this, set up for what purpose?
Nothing can pull the value of secrets from GitHub actions without either running a compromised/modified runner, or the workflow itself encrypting/printing/shipping that actions secret off to somewhere. If you’re really worried, use oidc GitHub actions where possible: https://docs.github.com/en/actions/deployment/security-harde...
Not just any Friday, but Good Friday, which happens to be a legal holiday in many countries, including here in Germany. And since Easter Monday is one too, a lot of people will go on vacation on this "long" weekend (maybe even taking a couple of days off before or after to make it even longer)... Flights and hotel rooms booked, and all that.
My reading is if you allowed Salesforce/Heroku or Travis CI access to your GitHub account/organization, then there is a chance that that grant was leaked to a malicious third party - and they may have accessed your github account and downloaded private material.
GitHub indicates they are performing an audit and if they find such evidence they will notify each account/org within the next 72 hours.
OAuth properties are like jet plane cockpit buttons. For computer science students there should be one semester OAuth course. And in big companies there should be OAuth Expert. That is how much knowledge OAuth requires. If you don't agree with me try using Keycloak. It has EVERYTHING so you don't need to implement.
I am always uneasy when an application even thrust worthy like sentry asks access to ALL my private repositories even when they dont need it.
I feel that in 2022 GitHub is way too critical to many organizations and they should really work on stricter rules. Even google requires special and costly audits from oauth applications to access users critical data like drive and emails.