Doubts in FuryJava's MemoryBuffer::readBytesAsInt64

[LouisLou2]

/** Read {@code len} bytes into a long using little-endian order. */
  public long readBytesAsInt64(int len) {
    int readerIdx = readerIndex;
    // use subtract to avoid overflow
    int remaining = size - readerIdx;
    if (remaining >= 8) {
      readerIndex = readerIdx + len;
      long v =
          UNSAFE.getLong(heapMemory, address + readerIdx)
              & (0xffffffffffffffffL >>> ((8 - len) * 8));
      return LITTLE_ENDIAN ? v : Long.reverseBytes(v);
    }
    return slowReadBytesAsInt64(remaining, len);
  }

I have some doubts about this method above when applying big endian.

Problem Analysis

1. Incorrect Mask Calculation

   • The issue arises when len < 8. The current mask calculation:

     0xffffffffffffffffL >>> ((8 - len) * 8)

     keeps the lower len * 8 bits, but in the Big Endian scenario, the valid data may actually reside in the higher len * 8 bits (the upper bits).

   Example:
   When len = 3, the mask calculation becomes:

   0xffffffffffffffffL >>> 40

   which results in the mask 0x0000000000ffffff, effectively keeping the lower 24 bits. However, if the data is stored in Big Endian, the valid data would actually reside in the upper 24 bits (the first 3 bytes). As a result, this incorrect mask clears the valid data.

2. Incorrect Byte Order Handling

   • The byte order reversal operation should happen before applying the mask. Currently, the reversal happens after the mask is applied:

     return LITTLE_ENDIAN ? v : Long.reverseBytes(v);

     This is problematic because, after the byte order is reversed, the high-order bits may get erroneously cleared due to the mask being applied in the wrong order. Reversing the byte order after masking results in the potential loss of the valid high-order bits.