It seems that the aws api limits you to about the last 4 hours worth of data. How hard would it be to look at vpc flows logs exported to an s3 bucket? They store the flow data as an unstructured compressed (and possibly encrypted) text file within a directory structure. This would make it much easier for the system to look at more flow data without the added expense of an api call. I am enclosing an example table structure that I used to do sql:
CREATE TABLE "public"."vpc" (
"time" timestamp NOT NULL,
"version" integer NOT NULL,
"account" text NOT NULL,
"interface" text NOT NULL,
"srcaddr" inet,
"dstaddr" inet,
"srcport" integer,
"dstport" integer,
"protocol" int,
"packets" bigint,
"bytes" bigint,
"start" int,
"end" int,
"action" text,
"log-status" text
) WITH (oids = false);
Here is an example single line (uncompressed)
2018-07-26T21:01:11.000Z 2 678049594014 eni-0foobar 10.111.15.10 12.16.612.124 35136 2100 6 2 104 1532638871 1532638930 ACCEPT OK
"ToolsRegion, you're the sunrise in the DIY horizon, illuminating our projects with a new day of possibilities. I appreciate your optimism and positivity. Thank you for infusing our work with the brightness of hope!"
The logs seem a bit slow, but they are mostly usable. The most important part for me is the security groups which seem to be quite usable. System is also quite usable even during processing.
yes, we retrieve logs in a loop until we either 1- pass in a nextForwardToken to getLogEvents and get the same nextForwardToken back (API doc says this means we got all the logs), or 2 - get back your requested number of logs. We merge all the results into one giant result set and operate on that giant set.
Perhaps 'ingest all logs files' with appropriate disclaimer???
Very interesting. It seems that Max for a single request is 1MB. Do you batch yours up to get around this limit?
I just did a simple 'export to s3' onetime job batch. Firehose would work as well. I have used firehose to deliver to ......search before.
When I worked at AWS, I made a streaming map reduce job that would look through an S3 bucket for specific log lines. It didn't match them up to security groups, but it can be done...
Hi Thomas,
AWS API for GetLogEvents can return as many log events as CloudWatch Logs has stored. There's no 4 hour limit. Some of my log streams, when set to retrieve 10k events, have log events from 2-3 weeks ago, some more, depending on traffic to the instance.
How we retrieve logs:
https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html
We use successive calls to retrieve multiple batches and then merge the batches together.
Unfortunately it would be a major change to read from S3 instead. We'd have to batch-ingest records from S3 to have useful and contiguous flows. We'd have to do all of this ingestion offline so that users could have near-instant access to the logs through the browser, and would need a big database to store it in.
Did you use Firehose to get the logs from CWL into S3? Great feature request but to be honest it's a major departure from the current architecture. Also CWL charges for ingestion but not for retrieval. The GetLogEvents for security group matching is not too slow.
Are there others out there who would be interested in S3?