[destination-s3-data-lake] Java out of memory issue exporting

[Bayardalai]

Connector Name

S3 Data Lake

Connector Version

0.3.15

What step the error happened?

During the sync

Relevant information

I'm syncing large table about 28GB on source SQL server to S3 Iceberg table with Glue catalog using S3 Data Lake destination. Whenever I run sync operation it hits java out of memory, while I tried to increase replication pod memory limit from default to 10Gi, 20Gi.
All other settings are as default.

This is query to update memory limit of connection.
update connection set resource_requirements = '{"memory_limit": "20Gi", "memory_request": "2Gi"}' where id = '8a
491a0e-7830-4c26-b964-c6f06cef61fe';

Error Message:

Relevant log output

2025-03-12 15:49:24 destination INFO DefaultDispatcher-worker-5 o.a.h.i.c.CodecPool(getCompressor):153 Got brand-new compressor [.zstd]
2025-03-12 15:49:46 destination INFO DefaultDispatcher-worker-1 o.a.h.i.c.CodecPool(getCompressor):153 Got brand-new compressor [.zstd]
2025-03-12 15:49:52 destination INFO DefaultDispatcher-worker-2 o.a.h.i.c.CodecPool(getCompressor):153 Got brand-new compressor [.zstd]
2025-03-12 15:49:52 destination INFO DefaultDispatcher-worker-1 o.a.h.i.c.CodecPool(getCompressor):153 Got brand-new compressor [.zstd]
2025-03-12 15:49:58 replication-orchestrator INFO Records read: 2260000 (2 GB)
2025-03-12 15:49:58 destination INFO DefaultDispatcher-worker-2 o.a.h.i.c.CodecPool(getCompressor):153 Got brand-new compressor [.zstd]
2025-03-12 15:50:13 destination INFO Malformed non-Airbyte record (connectionId = 8a491a0e-7830-4c26-b964-c6f06cef61fe): Terminating due to java.lang.OutOfMemoryError: Java heap space
2025-03-12 15:50:13 replication-orchestrator WARN Attempted to close a destination which is already closed.

Contribute

• Yes, I want to contribute