BSidesSF 2015 - Federating AWS CLI (Ayman Elsawah, Paul Moreno)

I'm going to go ahead and just get a quick show of hands. How many people are actually an Amazon customer or have been in the past? All right. So you guys have probably had a little bit of experience with IAM roles or IAM in general as well as CLI. And I know everybody's really thirsty. I heard the liquor cabinet was locked up with some crypto ransomware. So hopefully we're going to decrypt it after this. We can all have some beers. All right, introductions. My name is Paul Moreno, and I am the security engineering lead for Pinterest. 0:30 I'm focused on federation of all the things, and I like unicorns and cats.

All right, so here's the agenda. We're going to go over why we're doing this, how we can fix this, how ARIS works, which ARIS is a solution that you guys will all see a live demo of, or sorry, a recorded demo of, problems encountered when we were building it, some potential future problems we imagine we'll see down the line, some pre-release improvements, some future improvements, as well as some time for Q&A. So why are we doing this? 1:20 So here's the problem. Many organizations have IAM accounts with poorly controlled credentials that have potentially destructive access to production. So that means you have a developer that's got some sloppy practice in hygiene with the way they handle key data and it ends up on the internet or out of their control, out of your control, and then it has damaging impact on your environment. So despite 2FA-enabled accounts, accessing the Amazon CLI still only requires an access ID and a secret key to utilize it. 1:50 So a little bit more on why we're doing this. This product will reduce the plain text stored keys in repos and on the hardware. I consider everybody's laptop and every developer that we've got at Pinterest kind of a risk, especially considering it has sensitive key data such as this. This will allow federation and directory services. This automates the onboarding and termination, which is very important. Some IT departments have sloppy practices about terminating all the access. And this also enables the use of IAM roles for users and service accounts. 2:22 I don't know if you've got a lot of familiarity with it, but roles are the way to go because you've got a larger policy size you can have, so you can add cool things like the ability to restrict what source IPs the Amazon command line can be accessed from. And this product will also allow the centralized use of existing 2FA, such as through your VPN, so that you don't have yet another OTP on your phone. I think I have 12 just for my current role. So here's some examples of the problem that we've seen. 2:53 You can see there's a Security Week article here about attackers that were scraping GitHub as well as other locations for plain text access IDs and secret keys out in the wild. And they basically use them to stand up Bitcoin mining operations. And sadly I found there's a lot more open source solutions to actually scraping credentials than are to solve this problem. So another example, I'd say this was also pretty lucky. These people got popped with a $2,300, $2,400 bill because some hackers decided to use their account to launch a Bitcoin mining operation as well. 3:30 It's lucky because it was only $2,400 approximately. The larger example would be these guys. So this is an actual quote by anonymous somewhere. So Codespaces, if you guys don't know, was DDoSed with some ransom and before they realized they actually had somebody that had unauthorized access to their Amazon account. What they did is they ultimately tore their infrastructure all the way down, wiped out all of their backups to the point where they were no longer able to do business. That's really sad for a company that's actually touted their ability to protect their customer data from catastrophic events such as this. 4:10 So, how hard is it to find credentials out on the internet? It's actually really fucking easy. It took me about 60 seconds on Pastebin, and I found just a whole set. I actually did have the full link here, but I felt it would be embarrassing to actually leave it up, so I decided to blur everything out. Another quick example, also found on Pastebin, someone that was also gratuitous enough to actually tell us what region they're running their Amazon instance into. 4:42 Let's see here. So let's say I had the credentials. What could you do? Well, the easy thing I would probably do is just go ahead and get a quick list of the Route 53 and see what hosted zones they have. As you can see, it would be pretty easy to get a quick list, and then all I'd have to do is run that one-liner there to delete their entire hosted zone. So that probably wouldn't completely eradicate them off the net, but it would cause a lot of complications for them for a set amount of time. So, or we could try to do something like this where I actually just want to go ahead and create my own user. 5:12 We're just going to go ahead and create a pony and then add that user to the admins group. So after that, I can pretty much do whatever I want from the web console to be on the CLI. So, how can we fix this? So, we had a couple goals in mind when we decided to start development of ARIS. We wanted to get the IAM keys out of the hands of the developers, and out of the users, and even some service accounts in some cases, and allow them to still do business as usual. 5:42 Some pretty basic goals. From here, I'm going to hand it over to Ayman.

So we need a middleman that can also talk SAML and LDAP that will do some glue in between there. We found Shibboleth, which is an identity provider out there that does SSO and such.

No second, Mike. So Shibboleth, it's an SSO, as I said. It allows for attribute-based exchange framework. It also does role-based, but I'm not going to get into detail. It can act as SAML middleware. It can also act as an authentication and authorization middleware, and has a SOAP and HTTP binder. There's tons of documentation out there on this, and we chose this because of that. Also, there's actually Amazon-specific documentation on how to federate it on the web console, but not actually anything about getting tokens. 6:50 That became a pain in the ass. So I'm going to go ahead and forward this over to Ayman, and he's going to tell you a little bit more.

So, what is Eris?

So I'm basically going to reiterate what he just said, but I've got a beautiful Brad Pitt as a widening developer. So the mechanics of this, again, you've got shibboleth and Amazon Web Services have to have a trust established. So there was a lot of legwork setting that up. There was some documentation. It just wasn't super clear. I gave some feedback on it. Again, this was also done on a Linux system. They have a lot of documentation on how to do it in a Windows environment, which proved to be a lot easier. But we went the hard route. 13:29 Shiblet is configured to go and fetch some attributes based on the user and then do some magic to map it to a role. Shiblet ends up taking input from the user on behalf of Eris. Shiblet has that trust established with Amazon already. It generates a SAML assertion based on the attributes of the user and then it passes it back to Eris. Eris writes it to your operating system environment variables. And then it allows you to use the Amazon STS to perform CLI actions. 14:03 Now, time for a demo. I did tell you I like cats and unicorns. So as you can see, I'm going to say hello B-sides. And then I'm going to launch into ARIS. Errors fetches my shell username, and then I enter my password in, which is just password, by the way. And then, as you see, magic happens. 14:35 And as you see there, we've got an access key, secret key, and session token, and we're now able to run commands if we like. And I'm actually comfortable with having that up on the screen because those are invalid after an hour. So you can take a snapshot of that if you want. They won't work. All right. Back to Ayman.

All right. So I'm going to talk a little bit about what we plan to do next with this. There are some potential future problems that we've seen down the pipe and one that we've actually experienced. I'm worried that maybe the Shibboleth URLs or if we use a different IDP that maybe those might change and that would totally screw up this ARIS and that would just stem from package updates or just general code updates, etc. There could be some other STS related changes. As you heard Ayman reiterate how much pain you need to go through with Amazon. 18:02 I also experience a lot of that on a day-to-day basis and I can tell you that they do a lot of things without telling anybody. So that could be something that would happen that would impact the function of ARIS. Changes in the XML response provided by Amazon, it's not very likely but could also happen. Then something that did happen but I decided to leave it in the potential future problems because it could happen again. Certificate trust configuration between IDP and Amazon. Not a problem, right? Actually, part of the configuration process insists that your IDP go and fetch a publicly available XML file to do some validation on their end. 18:39 So, back last month, my IT department that has been using this product calls me up and is like, your shit is busted. I'm like, okay, so I go look at my logs, start digging around, and then I see all signs point to this XML file. And as you can see as I've highlighted, that actually expired that day. Amazon was down for SAML-integrated products for six hours because of this. There's a big fuck-up on their end, it's kind of basics if you ask me. So what we want to do before we decide to open source Eris, we're going to go ahead and do some Mac and Ubuntu support. 19:15 It does work on both right now. It needs a little bit more polish. We want to daemonize it so that it doesn't have to be a user interactive all the time. Improved logging and of course exception handling. Some other feature improvements. I've got an awesome guy in my IT department that's been willing to step up to the plate and help us build a Mac app, which would be very useful. If we decide to open source it as a user, you would input your Amazon account, username, password, and then the IDP URL, and that should be it. And then that would actually let this write to your OS environments, as well as maybe a potential local file if you absolutely need it for something like a browser extension. 19:55 OAuth, maybe. I've actually talked to one of my coworkers about this. I haven't really explored it too much. It may or may not happen. And then something other than usernames and passwords, because those are bad. I think everything's moving towards a cert world, or if we can, we can move it to a cert or a key world where we don't have to deal with passwords anymore. Open source date, ETA, is July this year. Hold tight. I hope to get it done. We're going to make sure that we draw out all the bugs so that you guys don't see any of them. 20:26 And now I'd like to open up for Q&A if anybody's got any questions. So for those of us that aren't familiar with Shimlet, is Shimlet assuming the authentication of the user in order to generate this credential? Correct, but it can be done for other... Sorry. So the question was, Shibboleth is using, is like the crucial element to this working. That's correct. It's the IDP we chose to use for this. I would imagine that you can do this for other things like an Okta or one login, the other, or ping identity. 20:59 The caveat there is all of those present an MFA or a two-factor login. So we would have to modify the ARIS product to actually ingest that and then pass it over to them as an IDP. So this product is using Shibboleth in a non-2FA mode behind a VPN and firewall. So that would be something that we would have to add on top of that. Again, I would rather rely on having other 2FA than adding another one to the mix. But for now, yes, to answer your question, this is tied to Shibboleth. 21:33 Anybody? You know what? I've heard about this. It's a good question. Oh, sorry. Yes, I played with Hologram. So Hologram is actually something I heard at a DevOps meetup at my office, actually. If you want to put me in touch, I'd actually love to talk to them. I actually did mean to follow up with them.

All right. Oh, one more. So the question was if we had a service account that we needed an access ID and secret key, would we try to move it over to this? The answer is yes, I would like to, but of course this needs a lot of polish and we need to make sure that it's not going to become an availability impacting, you know, a cog in the wheel there. 22:28 But I would like to see it used for all of our service accounts. It's a good question. There's a little back story there too. The question was why we decided to go with SAML to do this instead of any of the other options. To be honest, I didn't really see there was a whole lot of other options. 23:00 Maybe Ayman can answer this one.