refactor: Optimize DataFrame Reconstruction & Update Docs for Linux ARM64 Release #795
Conversation
pangjunrong
added
documentation
| placement=binfo.cids[0], | ||
| ) | ||
| ) | ||
| elif binfo.dt == 2: # BooleanArray | ||
| bool_array = pd.core.arrays.BooleanArray._from_sequence(block_data[0]) |
There was a problem hiding this comment.
From what I understand from the pandas source code (_from_sequence, coerce_to_array), it seems we will have an extra mask array constructed by this _from_sequence step, which will then be discarded and replaced by our mask array like in this example:
And also it seems to directly call the constructor of the BooleanArray anyway. I'm wondering why this _from_sequence approach is still faster than the old BooleanArray(data, mask) approach as it seems to only include the overhead of an additional mask construction. I'm I missing something here?
|
Hi @pangjunrong , thank you so much for the PR! The documentation, |
This change optimizes the reconstruction of Pandas DataFrames using the
_from_sequencemethods for ExtensionArrays, which run on bulk conversion routines using vectorized NumPy operations while avoiding the extra layers of validation in the general constructor.It also uses the
_from_mgrmethod for DataFrame construction using BlockManager, which assumes the BlockManager is already in a valid state and skips the validation and setup overhead that the regular pd.DataFrame(BlockManager) constructor performs.This also resolves the warnings mentioned in #786.
Based on a simple generic benchmark between the typical class initialization and use of
_from_sequence, the performance gain seems to be consistent & profound across BooleanArray, DatetimeArray & IntegerArray. However, I will need help for additional verification on this as I don't fully understand the full picture of how Pandas Internals operate at the lower level.Separately, the installation guides are updated to inform users of
connectorx==0.4.3's general availability for Linux ARM64 distributions running on glibc 2.35 & later and the use ofconnectorx=0.2.3for older distributions not covered by our new build process.