Aggregation and ICMP proto

[hentheor]

Hi,
For master-nfdump according to man pages, aggregation with -a and -A proto,srcip,dstip,srcport,dstport(,tos?) should have identical output. For ICMP protocol this is not the case eg

nfdump -r nfcapd.202302210610 'proto ICMP ' -n10  -A proto,srcip,dstip,srcport,dstport -O bytes -o extended
Date first seen             Duration     Proto      Src IP Addr:Port          Dst IP Addr:Port     Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2023-02-21 06:11:25.568     00:00:00.256 ICMP     3.121.159.245:0     ->   194.63.203.150:0.0   ........   0      315   461772     1230   14.4 M   1465     1
2023-02-21 06:12:49.488     00:01:51.856 ICMP       23.193.23.0:0     ->   194.63.239.164:0.0   ........   0      180   247957        1    17734   1377     3
2023-02-21 06:13:03.744     00:00:00.128 ICMP     43.192.29.235:0     ->   194.63.214.162:0.0   ........   0      141   206091     1101   12.9 M   1461     1
2023-02-21 06:13:03.744     00:00:00.128 ICMP    71.136.105.182:0     ->    81.186.197.40:0.0   ........   0      141   206091     1101   12.9 M   1461     1
2023-02-21 06:13:07.824     00:00:00.128 ICMP    194.63.167.239:0     ->      43.192.43.0:0.0   ........   0      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:07.824     00:00:00.128 ICMP     13.244.73.214:0     ->     81.186.38.90:0.0   ........   0      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:07.824     00:00:00.128 ICMP     13.244.73.214:0     ->     81.186.30.54:0.0   ........   0      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:17.160     00:00:00.128 ICMP    185.104.85.110:0     ->    194.63.238.66:0.0   ........   0      107   136746      835    8.5 M   1278     1
2023-02-21 06:13:19.720     00:00:00.128 ICMP    81.186.223.160:0     ->       3.28.60.62:0.0   ........   0      107   136746      835    8.5 M   1278     1
2023-02-21 06:12:25.232     00:00:00.256 ICMP     81.186.186.30:0     ->    63.35.250.200:0.0   ........   0      104   132912      406    4.2 M   1278     1

whereas

nfdump -r nfcapd.202302210610 'proto ICMP ' -n10  -a -O bytes -o extended
Date first seen             Duration     Proto      Src IP Addr:Port          Dst IP Addr:Port     Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2023-02-21 06:11:25.568     00:00:00.256 ICMP     3.121.159.245:0     ->   194.63.203.150:8.0   ........ 164      315   461772     1230   14.4 M   1465     1
2023-02-21 06:12:49.488     00:01:51.856 ICMP       23.193.23.0:0     ->   194.63.239.164:8.0   ........   0      180   247957        1    17734   1377     3
2023-02-21 06:13:03.744     00:00:00.128 ICMP    71.136.105.182:0     ->    81.186.197.40:8.0   ........ 164      141   206091     1101   12.9 M   1461     1
2023-02-21 06:13:03.744     00:00:00.128 ICMP     43.192.29.235:0     ->   194.63.214.162:8.0   ........ 164      141   206091     1101   12.9 M   1461     1
2023-02-21 06:13:07.824     00:00:00.128 ICMP     13.244.73.214:0     ->     81.186.38.90:8.0   ........ 164      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:07.824     00:00:00.128 ICMP     13.244.73.214:0     ->     81.186.30.54:8.0   ........ 164      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:07.824     00:00:00.128 ICMP    194.63.167.239:0     ->      43.192.43.0:0.0   ........ 164      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:19.720     00:00:00.128 ICMP    81.186.223.160:0     ->       3.28.60.62:0.0   ........ 164      107   136746      835    8.5 M   1278     1
2023-02-21 06:13:17.160     00:00:00.128 ICMP    185.104.85.110:0     ->    194.63.238.66:0.0   ........   0      107   136746      835    8.5 M   1278     1
2023-02-21 06:12:25.232     00:00:00.256 ICMP    81.186.217.254:0     ->    18.118.210.52:0.0   ........ 128      104   132912      406    4.2 M   1278     1

and

nfdump -r nfcapd.202302210610 'proto ICMP ' -n10  -A srcip,dstip,srcport,dstport -O bytes -o extended
Date first seen             Duration     Proto      Src IP Addr:Port          Dst IP Addr:Port     Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2023-02-21 06:11:25.568     00:00:00.256 0        3.121.159.245:0     ->   194.63.203.150:2048  ........   0      315   461772     1230   14.4 M   1465     1
2023-02-21 06:12:49.488     00:01:51.856 0          23.193.23.0:0     ->   194.63.239.164:2048  ........   0      180   247957        1    17734   1377     3
2023-02-21 06:13:03.744     00:00:00.128 0        43.192.29.235:0     ->   194.63.214.162:2048  ........   0      141   206091     1101   12.9 M   1461     1
2023-02-21 06:13:03.744     00:00:00.128 0       71.136.105.182:0     ->    81.186.197.40:2048  ........   0      141   206091     1101   12.9 M   1461     1
2023-02-21 06:13:07.824     00:00:00.128 0        13.244.73.214:0     ->     81.186.30.54:2048  ........   0      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:07.824     00:00:00.128 0       194.63.167.239:0     ->      43.192.43.0:0     ........   0      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:07.824     00:00:00.128 0        13.244.73.214:0     ->     81.186.38.90:2048  ........   0      116   147505      906    9.2 M   1271     1
2023-02-21 06:13:19.720     00:00:00.128 0       81.186.223.160:0     ->       3.28.60.62:0     ........   0      107   136746      835    8.5 M   1278     1
2023-02-21 06:13:17.160     00:00:00.128 0       185.104.85.110:0     ->    194.63.238.66:0     ........   0      107   136746      835    8.5 M   1278     1
2023-02-21 06:12:25.232     00:00:00.256 0        81.186.186.30:0     ->    63.35.250.200:0     ........   0      104   132912      406    4.2 M   1278     1

If we aggregate on srcport,dstport we get:

nfdump -r nfcapd.202302210610 'proto ICMP ' -n10  -A srcport,dstport -O bytes -o extended
Date first seen             Duration     Proto      Src IP Addr:Port          Dst IP Addr:Port     Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2023-01-02 13:07:19.016 49d 17:07:34.992 0              0.0.0.0:0     ->          0.0.0.0:2048  ........   0     6652    2.0 M        0        3    297  7875
2023-01-02 13:07:19.016 49d 17:07:38.320 0              0.0.0.0:0     ->          0.0.0.0:0     ........   0     8660    1.3 M        0        2    144  3744
2023-02-21 06:09:41.032     00:05:14.896 0              0.0.0.0:0     ->          0.0.0.0:769   ........   0      831   179921        2     4570    216   996
2023-02-21 06:09:41.904     00:04:52.304 0              0.0.0.0:0     ->          0.0.0.0:771   ........   0       43     2231        0       61     51    84
2023-02-21 06:09:37.088     00:05:05.992 0              0.0.0.0:0     ->          0.0.0.0:2816  ........   0       17     1178        0       30     69    48
2023-02-21 06:10:32.288     00:04:07.896 0              0.0.0.0:0     ->          0.0.0.0:778   ........   0        8      372        0       12     46    14
2023-02-21 06:09:44.392     00:03:48.712 0              0.0.0.0:0     ->          0.0.0.0:1281  ........   0        3      160        0        5     53     4
2023-02-21 06:11:25.080     00:00:03.712 0              0.0.0.0:0     ->          0.0.0.0:781   ........   0        1       42        0       90     42     1

Is this behaviour normal?
Thank you.

[phaag]

Why do you thing it is not normal?

[hentheor]

Ok let me explain

The first command aggregates with -A ...,dstport so icmp type.code may not be :0.0 everywhere.
Second command aggregates with -a, which according to man pages is the same (if I understand correctly),
and shows different icmp type.codes. Also now the Tos field is different than zero.

Omitting the proto field in the third command (which is superfluous due to the command line filter) we get another flavor of display with different representation of type.code (type * 256 + code) and Tos field again 0.

So, it seems to me that -a aggregates also Tos (and displays a more clear output) while -A ...,dstport has some incompatibilities with dstport.
Note that the other counters Packets, Bytes, pps,... are identical for the first three commands.

[phaag]

So let me explain:
The icmp type.code 0.0 is indeed a bug, which will get fixed with the next push to the master repo. Apart from that, which results from a unsuccessful try to handle icmp differently, the output are as expected.

There is a subtile difference between -a and -A proto,srcip,srcport,dstip,dstport
The option -a aggregates flows which have been cut into pieces by an active timeout for example. This means the first flow of a sequence is filled with all elements and subsequent flows with same prot,srcip,srcport,dstip,dstport get aggregated. That's why these flows have tos and other elements still available.

-A fields
That option aggregates flows by custom fields.
-A proto,srcip,srcport,dstip,dstport as an example aggregates flows by the same fields as -a but zeros out all other fields. Therefore tos results as 0. This makes sense, because this way you can aggregate flows in a very flexible way and the flows need not to be related to each other in any way.

As soon as you omit the protocol, that field is zeroed out and therefore the protocol can no longer be recognised as ICMP. Therefore the generic integer value in dst port is printed.

That's the reason for different "flavours" of printing. As of printing, nfdump generates a printing format on the fly depending on the selected aggregation. You may however overwrite this any time. In your case you may specify something like:
nfdump -r ... -n10 -O bytes -o 'fmt:"%ts %td %sa %dp %it %ic %pkt %byt %fl' -A srcip,dstport 'proto icmp' so you may force interpreting ICMP type/code.

btw. your command line nfdump -r nfcapd.202302210610 'proto ICMP ' -n10 ... does not work. Make sure the filter is the last argument with or without quotes.

[hentheor]

Peter,
Thank you very much for these enlightening comments.