Ayman Elsawah - Using Paretos Principle for Securing AWS with SCPs - DEF CON 27 Cloud Village

Hey everyone, thanks for coming. Okay, so let's just get into it. Using Pareto's Principle for Securing Arrest with SCPs. So, let me turn on my remote, that would help. And let me put it over here. All right, cool. So a little about me. I like AWS. I've been doing security for a while, whatever, blah, blah, blah. I took some tests. That's not a big deal. I'm also really into coffee. This is my real coffee set at home. So if you want to talk about single origins and pour overs and recipes, I could talk ad nauseum about that. 0:33 I also run a podcast called Getting Into Infosec, and it's designed to help folks get into security by interviewing folks that are out there and see how they got in, because everybody's path into security is different. And then I also wrote a book as well because, well, you know, it's just kind of helping solve the problem as well. So like if you sat down with me for like an hour over coffee, what would I tell you? This is what I would tell you. And I like Hawaiian shirts. So on Fridays at work, you see me, I'll probably be wearing a Hawaiian shirt. So that's a little about me. 1:05 Cool. So, I have no idea about you guys. This is the first time that the Cloud Village existed, so I just want to get to know everyone out there a little bit. How many folks out there manage AWS accounts, 10 or less? 10 or less AWS accounts? Okay. How many manage 25 or more accounts? All right. All right. And how many manage 75 or more accounts? All right. Very cool. Very cool. Okay. And how many of you guys are using AWS organizations and SCPs? 1:38 Okay, cool, cool. So what's Pareto's principle, right? The idea is that with just a little bit of work, you can accomplish a lot of work, okay, or a lot of results. So 20% of effort and you get 80% of the results. And that's where SCPs come in. They're just beautiful. So with only a little bit of effort, Code right a few lines of configuration you can really apply a lot of security to a lot of your accounts but there's a lot of steps that are required before that and also we'll talk a little about cloud security issues for those that are you know not too familiar about AWS again just trying to level set here 2:18 So, what are some common issues, right? IAM, having keys and source code, having really egregious permissions on boxes like for say, you know, EC2 star, IAM star, storage, leaky buckets, right? We've heard too many stories, logging, right? I mean, really, what AWS is, is you're giving the So, like, infinite amount of resources globally, right, to an engineer, and this is what you have, but this is what you end up with, right? 2:51 It's just, it's, you know, AWS has a shared security model, we all know about it, but still, it makes it very easy for someone to just trip over themselves. And so, you know, we have leaky buckets. I think we all know or we may know about all these leaky buckets that are out there. Someone even created a GitHub page that lists all the S3 bucket leaks out there. Very convenient if you're trying to prove your case or just keep track. And there's just, you know, it goes on and on and on and on. 3:23 And someone said that there is like 9 billion data records of S3 from S3 bucket leaks. Same thing, IAM is the new perimeter. So, you know, no longer is it network security, it's more IAM. Everybody might have heard of a Capital One breach recently. So, you know, a lot of things went wrong there, but, you know, we don't want permissions like EC2 star, IAM star. I have actually seen this out there. I have seen public machines that are running WAFs with EC2 star on them and I'm like having a heart attack. 4:01 And, you know, in this case, you know, I was at a company once where someone emailed their IAM keys to someone else, and I'm not sure how the keys got compromised, but $40,000 later, a lot of EC2 instances were deployed, and it was not fun. AWS did reverse it, but still, we don't want to be in that situation. Yeah. Who here has heard of Codespaces? 4:32 All right. So, Codespaces was like a GitHub-like company that provided, as you can see, rock solid and secure affordable hosting. So, we all know it's secure because it says it's secure, right? So, this was their website on June 14th, 2014. This was their website on June 18th, 2014. Their root keys got compromised and the attacker basically deleted all their backups, all their instances, everything. 5:07 And they literally went out of business overnight. Okay? So this is a target for us. For AWS folks, for cloud folks, this is our target. We need to know this. It's a part of our history. And so it's something that I recommend everyone go ahead and look up. I pulled this from archive.org and you can do that same as well. So, why AWS organizations and SCP? Okay. So basically with a few lines of configuration or quote unquote code, we can enable a lot of change and a lot of protection in our accounts. 5:42 But the reason I did this talk is because, so SCP has been around for a little bit and I'm still not seeing too many organizations taking advantage of it, at least to their full extent. I think people are a little afraid of SCP. But it's a really good way to guard your security and protect yourself if you have things even securely configured properly. So I'll talk a little about multi-account strategy. 6:13 It's kind of a foundation to this. So if you are Using multiple accounts, you want to take advantage of the multi-account strategy. You want to have a logging account for your logs to go to. You want to have a security account for your security ops and your tools to run from. You want to have networking where you can do your transit gateways and things like that. Shared services and on and on and on. So it's really important to take advantage of the multi-account strategy. AWS first started off, I don't think they realized how 6:47 How big this would explode. And so they've incorporated some multi-account stuff. I would look it up. It's part of the well-architected framework. I'm not going to talk anything negative right now. So, once you have this set up, okay, then you can take advantage, full advantage of AWS organizations. So, it kind of works on top of that. You have your master account. If anyone's familiar with any sort of directory, you have different OUs and you could have sub-OUs and you could apply policies at the OU level. 7:23 So, this is really how security control policies work. Okay. And so, this is kind of a quick outline of what I'm going to talk about. You know, if you're coming in starting with AWS organizations and SCPs from scratch, okay? So, we have the master account. Everything starts with the master account. How many, not to embarrass you, but how many here have master accounts with resources in them? 7:59 You don't have to raise your hand, it's okay. Having a master account with resources is bad. You don't want to do that. Why? Because when you're going to apply a policy, you don't want to apply the policy at the master level. You want to apply it at a different OU level. And so you want a master account that's empty and has no resources. So that's really your biggest step in all this is getting, creating a master account, moving your consolidated billing to another, to that account. 8:29 There's a few steps involved. One, call your rep. Create the account, create a S3 bucket for that account where your consolidated billing reports can go to, and then create a support account. And if you're on an enterprise level support plan, then definitely your rep will take care of most of this for you. In one organization, it took a month for us to get this done. We had to sign a new contract, had to go through legal, blah, blah, blah, blah, blah. I'm like, what is this? Why is this? And they had to sign it with ink. It wasn't DocuSign. 9:00 I was like, really? So it was quite interesting. But after you have the new AWS account, then you can create the org and link and unlink accounts, and it's pretty straightforward from that, from there. And here you are. So now once you have your master account, you create the organization, okay? Organizations has two modes. There's billing mode and then the full access, the full features mode, which lets you take advantage of service control policies and everything I'm going to talk about today. 9:33 So if you have an organization already and you have a master account with no resources, then go ahead and make sure your organizations is enabled for full features. It's just a really quick CLI command or you can go into console and do it there as well. You also might need to increase your limits on AWS organizations as well. So I have 10 here because that's what I was doing in my experimental account. But yeah, so you might want to look into that as well. 10:04 Okay, so here I've created an organization. I'm logged in. I created a couple different OUs. It's like super easy. And hopefully everyone has a dev staging and prod environment. And hopefully not everyone is doing So, this is the ideal way. So, I created a prod, staging, and dev account. I also created an other account because there's a few accounts that I just didn't know where they should go. So, you might have a subsidiary that you're, you know, they want to have their own deal with AWS and they don't want you messing with that. 10:43 So, you know, other might be a good place to put it. And you can have sub-OUs. There's no need to create just three top-level OUs. You could have many different sub-OUs. It's really up to you how complicated you want it to go. The only one that really would be outside is your root account, your master account, and that would be in the root OU. Pretty straightforward. And then just go ahead and enable your service control policies. It's okay. Nothing's going to happen here. Just go ahead and enable it. All right. 11:14 So, applying SCPs. Okay. Be really careful about SCPs. I did say it's okay to enable them. Yes. But when you're going to apply them, take the usual methods about applying into dev, applying into staging, and then prod. These things are really powerful. They're instantaneous. And when you apply it at the OU level, it takes effect for all your accounts under your OU. So, you know, 11:45 Take that into consideration. And please for, please don't deploy things on Friday. Just don't do it, okay? Okay, so in AWS organizations, this is not SAP directly, but when you have your master account set up, This is a really nice feature to take advantage of. Just create a CloudTrail, if you don't have one already, create a CloudTrail in your master account and it will automatically apply to all your, your whole organization. So you will only see this configuration when you're doing it from your master account. 12:20 And so of course have that bucket in a separate account for logging, okay? And the reason it's in a separate account for logging is for integrity purposes, limited access, you don't want everyone to have access to that logging account. And so go ahead and set this up and remember the name of your cloud trail. And I'll tell you why. In the next one here we have an SCP Talk on information security covering CISO, vCISO, red team, threat modeling, Enterprise Security Weekly, Security Cafe, Getting Into Infosec, Ayman Elsawah, Adrian Sanabria. 12:53 What I'm referring to is the specific CloudTrail name that you specified. And the reason is that say you have other groups out there that maybe want to have their own CloudTrail or want to do whatever. And you can let them. They can create their own CloudTrails if they want. They can delete it. They can do whatever. But don't touch this CloudTrail. So with this SCP, it'll allow you to stop. And you can see here, I'm logged in as root, as admin and I'm trying to disable it and it says you're not on the necessary permissions. 13:26 So, it works. I have a question here.

So, earlier when you created that trail, you created it from master account for all the member accounts in one go? That's right. Yeah, I'll be happy to take questions at the end. So, again, so, you know, We're trying, so VPC flow logs, right? You want to protect those possibly if you really rely on them from a security purpose. So, you know, this is a service control policy that will protect them and prevent someone from deleting them. 14:05 And, you know, you could tune to your liking, right? Do you want to specify a specific VPC flow log? You could really play with it as much as you like. Okay. A lot of times you have resources only in US West and East, for example, or two or three regions. There's no reason to have all your regions enabled. And so this will kind of limit your attack surface on where regions would be set up. And your attack could come from outside or inside. 14:37 I mean, a lot of times employees I don't want to say a lot of times. Sometimes an employee could set up an instance in, I don't know, London or Asia with GPU instances and just do some crypto mining and no one would know about it until much later. You know, it really depends how good your monitoring is, how mature your environment is for looking at resources outside your environment. So, again, this is not rocket science, but this will limit your attack surface. 15:08 So it's a really good way to just, hey, I don't think we'll ever have a need for US West 1 or anything outside of the US, whatever you can use, you know, you can just play with this to your liking. I did have, oh, so one note is I put like easy to describe for example. So if I want to have a tool or if there's a tool that's going to do easy to describe, then I could run it and it won't break that tool. So if you have monitoring tools, then you want to add a couple not actions here so that your tools won't break. 15:41 Something to take into consideration. And so this is an example of trying to launch something from a region that is not on that list. The next thing is limiting the type of instances that are running in your environment. There's two ways to do it. You could do a deny or you could do a permit list. I'll show you both. In this case, I am denying these particular instance types, right? Is there a need for your organization to use a P3 instance, right, with like four GPUs? 16:13 I don't know. Probably not. So in this case, we're limiting the type of instances so that either accidentally or maliciously, we don't launch instances that are going to cost us thousands and thousands of dollars. And I want to draw attention to the string-like condition here. It lets you use wildcard conditions, so it's pretty versatile. Here's one of allowing it. 16:44 So, you know, I'm limiting to just general purpose instances. So, here we have a bunch of instances listed. We have A, T, M, and M5 and M4. And so, okay, that's all instances we need, right? It depends on your organization. You could actually add more to this. You could limit it to particular AMIs if you want. You could limit, you know, you could have, you can make this as Thank you. Thank you. 17:29 Here's a good one. Limiting root actions, right? You want to deny root actions across your accounts. And so, this is a really good way to do so. Using the principal ARN and literally as root, you can't do anything. I'm logged in as root and I can't even do a describe on instances. I can't launch anything. I can't do a single thing. So, it's really, really powerful. This is one of my favorites. So, If you have that founder that still has access to the root account, which happens a lot, right? 18:04 The founder, because the email was created with their credit card and their email address. So I would recommend you talk to them first before doing this. That might be a good idea. You do want to keep your job. Here's another one. You want to enable encryption. So you don't want to allow any objects to be put into your S3 buckets unless they're meeting this criteria. And in this case, the criteria is that it's encrypted with AES-256, for example. 18:37 Same thing with S3 public access. Here's another one. I'm still experimenting with this a little bit, but it seems to be working. Okay, so troubleshooting. There's a lot of troubleshooting that I went through in this. I definitely recommend you take a deep reading of IAM and IAM syntax, especially when you're looking at the conditional operators and trying to put a list of things. 19:11 So it's quite tricky, especially in this condition map, trying to define it. But, you know, it's all based on IAM, so it's pretty straightforward. And, you know, just some little, like, things. Even I was trying to do a little Easter egg as far as a version number and it turns out that, you know, AWS doesn't like that. Does anyone know what that date is? 19:42 Oh, right. Yeah, the date, it's supposed to be, yes. But that date right there, November 5th, 1955. Does anybody know that date? Yes, back to the future. Pretty good. Yeah, I was watching at the time. Actually, I have a few dates throughout the presentation, little Easter eggs from different things throughout movie history. So yeah, and then same thing here. If you want to deny a role, then you could specify the role. 20:12 But then if you want to specify an actual IAM user, you will need to use the condition string not like. That was quite annoying as well. So... That's something there. This is a really cool website I found after doing a lot of this stuff anyway. It's asecure.cloud. The person that's created this is amazing and they have service control policies, security groups, a lot of different things that will let you even put confirmation templates together and just deploy. 20:45 Of course, you know, deploy cautiously to dev. But it's a good library of a lot of different service control policies. The ones I showed you are the ones that I created and kind of implemented. But that person has a lot of ones like preventing people from deleting KMS keys, for example, right? That's a really good one. So, I mean, imagine if someone deleted your KMS keys, you can no longer decrypt your information. So, it's quite, quite precarious. 21:17 And that's about it. This is my contact info. I blog about stuff on AWS quite some time, you know, frequently. And I'd like to open up for questions, if there's any questions.

During a... Yeah. Yeah. Yeah. That's exactly right. You'd have... So you have to take that into consideration for things. I mean, I think AWS is working solely to limit those root actions. I'm kind of annoyed that there are still things that you need root for, but yeah, that's exactly right. Yes.

So the question is, if I get it right, and the previous question was, do you have to, when you want to do things that require root action, are you going to have to remove the service control policy? So that was the previous question. And this question is, what do you do about over-permissioning?

Oh, yeah. There are a few tools out there that would give you a list of permissions that are recently used by particular users and AWS has some out-of-the-box section as well. Yes.

That's right.

Only CloudTrail. Only CloudTrail. So that screen there was a typical CloudTrail creation screen, but you only, you have that extra dialog box for when it's being done from the master account. And so all the settings that you have there are going to create a CloudTrail in each new account or each account and any new accounts. That's right. Well, depending on how you configure it. So, if you configure for all regions, then yeah, that'll apply. But it'll apply for the same bucket. And the great part is for new accounts that are created later, which is like a lifesaver. 23:40 Yes.

That's right. That's right. Yes.

Yes, yes, procurement is, there are a couple tools out there actually that I found that will help you find shadow IT accounts out there. So they'll hook up into your Expensify and to your expensing tools and look at your credit card expenses and identify, oh, someone created an AWS account. Oh, look at this. And they'll identify this shadow IT stuff for you. So I find that. And it's really interesting because we ran into issues where any of us didn't have permission to do things with that account. 24:37 You probably ran into the same thing. And so you had to get the account owner to give you permission to handle that account. Yes.

Oh yeah, don't give them that permission. So that is only, you can only do that from the master account. So in any case, you want to limit access to your master account to only a limited few people who can apply those SCPs. So, it's an administrative thing. So, you know, a limit, I mean, in general, a general rule of thumb is limit master account access to only a few admins. Does that answer your question? Yeah, that answers my question.

The question was, how do you limit access from people changing the flags on SCPs? And the idea would be to limit access. Yes.

Say again? Oh, Security Hub. Yeah. I've played with it with GuardDuty. I mean, I've used it, so it won't monitor everything, but in this, what I'm talking about, I guess, is more prevention, so kind of setting up guard rails to prevent things from happening. So if you're noticing things, you could use SCPs to prevent things, but you want to get at the root cause of what's happening in your environment. 26:13 Okay.

So the question is, do you have, do you recommend any tools to manage privileged IAM accounts in AWS? There's a couple, I mean, there's a couple tools out of the box from AWS that will have you run permissions and show you what permissions IAM users are using out of the box. And so you can do an audit of that. I recommend, you know, just as a best practice, having different roles. You know, you have a developer, maybe a super developer, a network admin, IAM admin, you know, very few super admins, right? 27:06 And then try to use automation so that things are run and pushed through Terraform and GitHub so that there's an audit log and audit trail of what's done. And so if you can see things done through a GitHub commit, And it's been approved, that PR has been approved, then you have some sort of audit trail and some permissions. Does that answer your question? Kind of. Kind of, right?

That could be a thing. That could be a thing. I haven't used it, but I think that could be a great, that, I mean, the, I think the, you're unlimited in what you could do. Yes?

I don't know.

Right. I can imagine you can. I believe you can.

Yeah. Yeah. So the question is, can you use ICPs to enforce tags? I believe you can. So basically, what I would imagine is, you know, when you have an EC2 run, put a condition to require a tag, for example. But I haven't created one, but yeah, I would imagine so. Yes. So going to his question, what product would you use for ordering? So, there's a few tools out there. 28:49 I guess ask me afterwards, I can answer. There's a few out of the box from CloudTrail, I'm sorry, from AWS that will let you take a look at permissions. Are you talking about permissions? Permissions question? So if I understand the question, what tools are there available to take a look at permissions and audit logs or just permissions? So with audit logs, I mean, so CloudTrail, you have now a lot of the IAM logs are going in CloudTrail. 29:21 You could use any sort of tool, Elks, Blanc, Sumo Logic. There's a few, there's a bunch out there. Yeah, I don't recommend specific vendors necessarily. So, yeah.

There are. So the question is, are there anomaly tools out there that look for anomalies? Well, there's GuardDuty out of the box for AWS. I don't work for AWS, by the way. There are other tools that will ingest your logs and look for anomalies. A lot of times you have to write those out. But yeah, I mean, Security Hub and GuardDuty and the whole ecosystem, that's AWS's recommendation? Yes.

So, yeah, these slides will be posted. Yes.

I have, but I say it with a sigh. So, it's... I don't know if I want to go on record. But yeah, I've used AWS. I've used AWS config. The question is, have you used AWS config and what are your thoughts on that? I have thoughts on that. I think, you know, it works for some organizations. It's something good. I think it's better than nothing. 31:02 But I think with the convergence of a lot of tools that are coming out there, there's a lot of overlap. A talk on information security covering CISO, vCISO, red team, threat modeling, Enterprise Security Weekly, Security Cafe, NIST, FedRAMP, GDPR, HIPAA, SSO, IAM, OAuth, phishing, ransomware, pentest, OSCP, NIST, FedRAMP, GDPR, HIPAA, SSO, IAM, OAuth, phishing, ransomware,