-
Notifications
You must be signed in to change notification settings - Fork 0
Experiments
We conducted throughput and end-to-end latency tests under three different parameters: inter-cluster delay, transmission buffer size, and lateness tolerance.
-
The Inter-Cluster Delay: This is the delay which is applied before sending data from the first to the second cluster. Delays were based on values from a respective file created during the preparation step mentioned earlier. As both clusters run on
localhost
resulting in an inter-cluster delay less than 1 ms, introducing our own delay mechanism was necessary. -
Transmission Buffer Size: We used a buffer equivalent to the window for aggregating records before sending them from the first cluster to the second cluster. The range of buffer values was from 1 ms to 60 seconds. Values exceeding 60 seconds are not worth taking into account because they caused system instability and required additional memory configurations as all the states must be saved in the meantime, i.e., before the window "fires".
-
Lateness Tolerance: This parameter determined the level of determinism in a distributed setting with parallel streams. To manage out-of-orderness, the lateness tolerance was introduced. If the sum of window size and lateness tolerance was exceeded, a record would be dropped. Measurements for lateness tolerance ranged from 1 ms to 10 seconds, as values beyond 10 seconds were considered unsuitable for low-latency data stream applications.
All these measurements were conducted both locally without using Docker and within the Docker environment. Furthermore, each of the experiments has been conducted 3 times and the application has been running for 300 seconds in each iteration. The total experimentation time is 176,400 seconds or 49 hours.
- Developer guide