How to Clean up your Security Groups with PiaSoft
We just released a new feature: Cleaning up your Security Groups with VPC Flow Logs. Our solution runs entirely on one instance, the PiaSoft Flow Logs Viewer, and takes less than 10 minutes to set up.
Here is the fastest way to get rid of unused security groups and security group rules.
Start with Our Setup process. Like we said, this takes less than 10 minutes and uses some AWS Magic: CloudFormation.
Once you've finished Setup, log in with the Access Key you created during setup:
This next step works best with plenty of Flow Log Data. See below to generate plenty of data. Navigate to 'Security Groups' and click 'Find Unused Security group rules':
Next, select the VPC you would like to analyze. You can queue jobs for each VPC. The highest level a Flow log can be created at is a VPC, so we scan VPCs. By default, we will pull up to 1,000,000 flow logs per ENI, or as many are available.
Click "Start Search Job" to kick it off, and see the record that it has started:
About Job Duration
Jobs can take as little as two minutes or longer than an hour. It all depends on how much flow log data is available. A t2.micro can retrieve and process 8 log streams of 1 million events each in a little under 10 minutes. Larger instance sizes can process logs faster.
A job will be marked "Finished At" when it has completed.
Atop the page is a list of Security Groups. The rightmost column will tell you whether the group is attached to any ENIs or whether it is not attached. You can delete Security Groups that are not attached.
Click the Security Group name to get the results.
Legend
Rule in Use: Traffic is being accepted through this rule
Unused: Flow Logs are flowing, we had enough data to analyze, but no traffic matched this Security Group Rule. It may be safe to delete.
Not Enough Data: There were not enough Flow Logs from ENIs attached to this Security Group. See 'Generating Plenty of Flow Logs' below.
Generating Plenty of Flow Logs
To get the most value out of this product, you should turn on VPC Flow Logs for as many ENIs as possible. AWS recently made CloudWatch Logs about 90% cheaper for Flow Logs users, so go wild.
Turn Flow Logs on for all important ENIs. Turn it on at the VPC level if possible.
Only ACCEPT data is used for this analysis, so your flow logs can be created with Accept only.
Collect several days worth of ordinary traffic data. If you have a process that runs once a week, let it run.
The job can pull as many or as few logs as you want. The default is 1 million logs per log stream.
We welcome any and all feedback! Please let us know if you have questions or comments.