What would users like to see?

It seems that the aws api limits you to about the last 4 hours worth of data. How hard would it be to look at vpc flows logs exported to an s3 bucket? They store the flow data as an unstructured compressed (and possibly encrypted) text file within a directory structure. This would make it much easier for the system to look at more flow data without the added expense of an api call. I am enclosing an example table structure that I used to do sql: CREATE TABLE "public"."vpc" ( "time" timestamp NOT NULL, "version" integer NOT NULL, "account" text NOT NULL, "interface" text NOT NULL, "srcaddr" inet, "dstaddr" inet, "srcport" integer, "dstport" integer, "protocol" int, "packets" bigint, "bytes" bigint, "start" int, "end" int, "action" text, "log-status" text ) WITH (oids = false); Here is an example single line (uncompressed) 2018-07-26T21:01:11.000Z 2 678049594014 eni-0foobar 10.111.15.10 12.16.612.124 35136 2100 6 2 104 1532638871 1532638930 ACCEPT OK

Would be a nice function to get a list of associated instance ID's so I can see which instance IDS are affected. New table with ENI/InstanceID/(IP? private/public?)/DNS NAME? Really would enable me to then see what actual instance ENI ID (Nice security groups screen which lists all associated Instance IDS, retrieve and collate.

I know the program is written in ruby, but have you looked at hyperscan? Its a really fast regex parser if you need that kind of functionality. https://www.hyperscan.io/