Performance and Evaluation

security performance testing scope evaluating performance focuses predominantly on throughput, continued bandwidth, resource utilization metrics and response time each use case has its own finer grained testing scope the scope defines individual performance test cases those tests have defined inputs, a defined test environment and agreed upon acceptance criteria refer to each use case below for more information on its scope testing environment the testing environment consists of test machines (2), tools installed on the test machines, and a fw all in a controlled network see diagram below for more details how do we test traffic? for performance, we leverage the following tools to determine performance of the traffic over a network running edgesentinel security ping > ping is a great tool for determining latency on a network, based on response time ping can also be used for testing packet loss iperf3 > a commonly used tool for testing bandwidth on a network the iperf tool is installed on both test machines in order for them to communicate and share metrics about bandwidth in addition, this tool allows for setting criteria like interval and over how long to test various use cases this tool also can measure packet loss and mimic up to 100 sessions prometheus > a tool for collecting performance metrics such as cpu and memory utilization grafana > a tool for visualizing the data collected by prometheus fw utilization tools > some fws offer health endpoints to gather utilization on the device traceroute > a tool used for quickly determining if a session is filtered by the fw or not scapy > an application used on test machines to alter src/destination ips to easily test fw filtering how do we determine results? the metrics defined below all represent essential metrics of testing a fw bandwidth > how much data is allowed over a period of time throughput > how much data has actually traveled through the network over a period of time latency > how quickly traffic gets a response from its destination packets lost > simply, how many packets were dropped cpu utilization > amount of load on cpu memory utilization > amount of load on memory max sessions > number of sessions allowed on the device total sessions > number of sessions over x time what are the input criteria? adjusting these input factors can yield different results vcpu memory disk bandwidth result output all results are output into an excel file with a use case specific table format use cases block list via edgesentinel a block list is defined by threat feeds from a third party this threat feed will include large amounts of ips with metadata given this large dataset of ips configured with edgesentinel security, the focus is on the evaluation of performance in relation to network health health in this use case, is measured by latency, bandwidth, throughput, max sessions and packet loss with varying deployment sizes (cpu, memory, disk) and datasets (ip lists), verify acceptance criteria is met additional input criteria number of ips in block list test cases each test case assumes circuit has been provisioned and edgesentinel is running for this test client (already automated and tested in separate integration tests) add third party feed to service verify outbound tester is blocked access to x% (adjustable) of ips in feed verify inbound tester is blocked access to inside of fw verify an update from the third party feed gets configured on x# (adjustable) of fws remove third party feed from service verify outbound tester is allowed access to x% (adjustable) of ips in feed verify inbound tester has access to inside of fw scale testing of feed size for each deployment type test no third party feed service test 1,000 ips from feed test 10,000 ips from feed test 100,000 ips from feed test 1,000,000 ips from feed test 10,000,000 ips from feed for each feed size verify average latency average bandwidth throughput dropped packets average cpu utilization average memory utilization max number of active sessions result output for the block list use case, there are metrics we can determine results by based on the feed size, there are 6 different metrics (see above) we want to verify the results with the result output will look similar for each of those metrics all results are output into an excel file with a use case specific table format for example measuring success each result metric has a specific failure criteria against which a defined threshold and condition is used to determine if a test run passes or not the criteria is metric specific and will be refined to discover and tune ideal firewall vm deployment parameters to achieve performance goals each service provider will work with sea street to define these expected values for all relevant metrics, this way an agreed upon deployment is set and the metric limits are set this output will look similar to the above table with an added 'result' column that uses the agreed upon value for a metric to determine its result