Skip to content

Commit 4a2df66

Browse files
sgilmore10kou
sgilmore10
and
kou
authored
GH-41675: [Packaging][MATLAB] Add crossbow job to package MATLAB interface on macos-14 (#41677)
### Rationale for this change In #41592, we added a CI job to build and test the MATLAB interface on `macos-14`, which is ARM-based. We currently only package the interface on `macos-12`, which is Intel-based. We should add a crossbow job to package the MATLAB interface on `macos-14` so that we package the interface for both Intel-based and ARM-based macoS. ### What changes are included in this PR? 1. Parameterized the `macos` job in the MATLAB crossbow workflow file to run on both `macos-12` and `macos-14`. 2. Added the bash script `dev/tasks/matlab/rename_macos_dynamic_libraries.sh`. This script is used to uniquify the shared library names generated for the Intel/AMD-based macOS and ARM-based macOS interface installations. This is required because the crossbow job generates one "monolithic" MLTBX file that contains all shared libraries for all platforms. See the [comment](https://github.com/apache/arrow/pull/41677/files#diff-99731016fc39dd96d2d239cf9cc132a8dacb9569c49bea4a64d69911b4dcc8c4R82) at the beginning of `dev/tasks/matlab/rename_macos_dynamic_libraries.sh` for a more in-depth explanation. ### Are these changes tested? 1. Installed the MATLAB Arrow Interface on both Intel-based and ARM-based macOS machines using the MLTBX file generated by [this crossbow job](https://github.com/ursacomputing/crossbow/actions/runs/9274264286/job/25516244719). All unit tests passed. ### Are there any user-facing changes? Users on ARM-based macOS machines will be able to install the MATLAB Arrow Interface via a "one-click" install workflow using the MLTBX file. * GitHub Issue: #41675 Lead-authored-by: Sarah Gilmore <sgilmore@mathworks.com> Co-authored-by: Sarah Gilmore <74676073+sgilmore10@users.noreply.github.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Sarah Gilmore <sgilmore@mathworks.com>
1 parent 9f58990 commit 4a2df66

File tree

2 files changed

+178
-19
lines changed

2 files changed

+178
-19
lines changed

‎dev/tasks/matlab/github.yml

+16-19
Original file line numberDiff line numberDiff line change
@@ -59,8 +59,13 @@ jobs:
5959
path: matlab-arrow-ubuntu.tar.gz
6060

6161
macos:
62-
name: AMD64 macOS 12 MATLAB
63-
runs-on: macos-12
62+
name: {{ '${{ matrix.platform.architecture}}' }} macOS {{ '${{ matrix.platform.macos-version }}' }} MATLAB
63+
runs-on: macos-{{ '${{ matrix.platform.macos-version }}' }}
64+
strategy:
65+
matrix:
66+
platform:
67+
- { architecture: "AMD64", macos-version: "12", architecture-suffix: "x64"}
68+
- { architecture: "ARM64", macos-version: "14", architecture-suffix: "arm64"}
6469
steps:
6570
{{ macros.github_checkout_arrow()|indent }}
6671
- name: Install ninja-build
@@ -73,27 +78,18 @@ jobs:
7378
env:
7479
{{ macros.github_set_sccache_envvars()|indent(8) }}
7580
run: arrow/ci/scripts/matlab_build.sh $(pwd)/arrow
76-
- name: Change shared library dependency name
77-
# MATLAB's programmatic packaging interface does not properly
78-
# include symbolic link files in the package MLTBX - this is a
79-
# bug. As a temporary workaround, change the expected name of the
80-
# Arrow C++ library which libarrowproxy.dylib depends on.
81-
# For example, change libarrow.1500.dylib to libarrow.1500.0.0.dylib.
81+
- name: Update dynamic library names
8282
run: |
83-
pushd arrow/matlab/install/arrow_matlab/+libmexclass/+proxy
84-
SYMLINK_ARROW_LIB="$(find . -name 'libarrow.*.dylib' -type l | xargs basename)"
85-
REGULAR_ARROW_LIB="$(echo libarrow.*.*.dylib)"
86-
echo "SYMLINK_ARROW_LIB = ${SYMLINK_ARROW_LIB}"
87-
echo "REGULAR_ARROW_LIB = ${REGULAR_ARROW_LIB}"
88-
install_name_tool -change @rpath/$SYMLINK_ARROW_LIB @rpath/$REGULAR_ARROW_LIB libarrowproxy.dylib
89-
popd
83+
arrow/dev/tasks/matlab/rename_macos_dynamic_libraries.sh \
84+
arrow/matlab/install/arrow_matlab/+libmexclass/+proxy \
85+
{{ '${{ matrix.platform.architecture-suffix }}' }}
9086
- name: Compress into single artifact
91-
run: tar -cvzf matlab-arrow-macos.tar.gz arrow/matlab/install/arrow_matlab
87+
run: tar -cvzf matlab-arrow-macos-{{ '${{ matrix.platform.architecture-suffix }}' }}.tar.gz arrow/matlab/install/arrow_matlab
9288
- name: Upload artifacts
9389
uses: actions/upload-artifact@v4
9490
with:
95-
name: matlab-arrow-macos.tar.gz
96-
path: matlab-arrow-macos.tar.gz
91+
name: matlab-arrow-macos-{{ '${{ matrix.platform.architecture-suffix }}' }}.tar.gz
92+
path: matlab-arrow-macos-{{ '${{ matrix.platform.architecture-suffix }}' }}.tar.gz
9793

9894
windows:
9995
name: AMD64 Windows 2022 MATLAB
@@ -140,7 +136,8 @@ jobs:
140136
run: |
141137
mv artifacts-downloaded/*/*.tar.gz .
142138
tar -xzvf matlab-arrow-ubuntu.tar.gz
143-
tar -xzvf matlab-arrow-macos.tar.gz
139+
tar -xzvf matlab-arrow-macos-x64.tar.gz
140+
tar -xzvf matlab-arrow-macos-arm64.tar.gz
144141
tar -xzvf matlab-arrow-windows.tar.gz
145142
- name: Copy LICENSE.txt and NOTICE.txt for packaging
146143
run: |

‎dev/tasks/matlab/rename_macos_dynamic_libraries.sh

+162
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
set -ex
21+
22+
# Issue Number 1:
23+
#
24+
# MATLAB's programmatic packaging interface does not properly package symbolic links. If the
25+
# toolboxFolder argumented provided to the constructor of matlab.addons.toolbox.ToolboxOptions
26+
# contains a symbolic link, the ToolboxOptions will resolve the symbolic link and include
27+
# the symbolic link's target as one of the files to package instead of the link itself.
28+
#
29+
# Example:
30+
#
31+
# Suppose you had this folder structure
32+
#
33+
# $ tree /tmp/example
34+
# /tmp/example
35+
# |-- regular_file.txt
36+
# |-- symbolic_link -> regular_file.txt
37+
#
38+
# Passing "/tmp/example" as the toolboxFolder argument to ToolboxOptions' constructor in MATLAB
39+
# returns the following object:
40+
#
41+
# >> opts = matlab.addons.toolbox.ToolboxOptions("/tmp/example", "dummy-identifier");
42+
# >> opts.ToolboxFiles
43+
#
44+
# ans =
45+
#
46+
# "/private/tmp/example/regular_file.txt"
47+
#
48+
# Note that the ToolboxFiles property - the list of files to package - does not include the
49+
# symbolic link.
50+
#
51+
# This is a known limitation with matlab.addons.toolbox.ToolboxOptions.
52+
#
53+
# Why is this a problem?
54+
#
55+
# On macOS, building the Arrow C++ bindings generates the following files:
56+
#
57+
# $ tree arrow/matlab/install/arrow_matlab/+libmexclass/+proxy/
58+
# .
59+
# |-- libarrow.1700.0.0.dylib
60+
# |-- libarrow.1700.dylib -> libarrow.1700.0.0.dylib
61+
# |-- libarrow.dylib -> libarrow.1700.dylib
62+
#
63+
# When "arrow/matlab/install/arrow_matlab" is suppplied as the toolboxFolder argument
64+
# to the constructor of ToolboxOptions, only the libarrow.1700.0.0.dylib is included as a file
65+
# to package. This is a problem because building the MATLAB interface to Arrow creates a shared
66+
# library called libarrowproxy.dylib, which is linked against libarrow.1700.dylib - not
67+
# libarrow.1700.0.0.dyblib.
68+
#
69+
# This can be seen by calling otool with the -L flag on libarrowproxy.dylib:
70+
#
71+
# $ otool -L libarrowproxy.dylib | grep -E '@rpath/libarrow\.'
72+
# @rpath/libarrow.1700.dylib
73+
#
74+
# To prevent a run-time linker error, we need to update the name of libarrowproxy.dylib's
75+
# dependent shared library from @rpath/libarrow.1700.dylib to @rpath/libarrow.1700.0.0.dylib
76+
# because only libarrow.1700.0.0.dylib is packaged.
77+
#
78+
# ==============================================================================================
79+
#
80+
# Issue Number 2:
81+
#
82+
# The platforms that the MATLAB Interface to Arrow supports Windows, Linux,
83+
# Intel/AMD-based macOS and ARM-based macOS. Currently, we create one "monolithic" MLTBX file
84+
# to package the interface that includes all shared libraries for all supported platforms.
85+
# We do this because the File Exchange <-> GitHub Release integration does not support
86+
# platform-specific MLTBX files.
87+
#
88+
# The problem with creating one MLTBX to package the interface for all platforms is that
89+
# the names of the shared libraries built by the interface for Intel/AMD-based macOS
90+
# and ARM-based macOS are identical.For example, building the interface generates the shared
91+
# library libarrow.1700.0.0.dylib on both platforms. To avoid this duplicate name problem,
92+
# we need to uniquify the names of the shared libraries generated for Intel/AMD-based
93+
# macOS and ARM-based macOS.
94+
95+
if [ "$#" -ne 2 ]; then
96+
echo "Usage: $0 <dylib-dir> <arch>"
97+
exit 1
98+
fi
99+
100+
DYLIB_DIR=${1}
101+
ARCH=${2}
102+
103+
if [ "$ARCH" == "arm64" ]; then
104+
IS_ARM64=1
105+
elif [ "$ARCH" == "x64" ]; then
106+
IS_ARM64=0
107+
else
108+
echo "<arch> must be arm64 or x64"
109+
exit 1
110+
fi
111+
112+
pushd ${DYLIB_DIR}
113+
114+
LIBARROW_DYLIB="$(find . -name 'libarrow.dylib' | xargs basename)"
115+
LIBARROW_MAJOR_DYLIB="$(find . -name 'libarrow.*.dylib' -type l | xargs basename)"
116+
LIBARROW_MAJOR_MINOR_PATCH_DYLIB="$(echo libarrow.*.*.dylib)"
117+
LIBMEXCLASS_DYLIB="$(find . -name 'libmexclass.dylib' | xargs basename)"
118+
LIBARROWPROXY_DYLIB="$(find . -name 'libarrowproxy.dylib' | xargs basename)"
119+
if [ $IS_ARM64 -eq 1 ]; then
120+
MEX_GATEWAY="$(find . -name 'gateway.mexmaca64' | xargs basename)"
121+
else
122+
MEX_GATEWAY="$(find . -name 'gateway.mexmaci64' | xargs basename)"
123+
fi
124+
125+
MAJOR_MINOR_PATCH_VERSION=${LIBARROW_MAJOR_MINOR_PATCH_DYLIB#*.}
126+
MAJOR_MINOR_PATCH_VERSION=${MAJOR_MINOR_PATCH_VERSION%.*}
127+
128+
LIBARROW_ARCH_MAJOR_MINOR_PATCH_DYLIB="libarrow_${ARCH}.${MAJOR_MINOR_PATCH_VERSION}.dylib"
129+
LIBARROWPROXY_ARCH_DYLIB="libarrowproxy_${ARCH}.dylib"
130+
LIBMEXCLASS_ARCH_DYLIB="libmexclass_${ARCH}.dylib"
131+
132+
# Delete the symbolic links. These files are not included in the packaged MLTBX file.
133+
rm ${LIBARROW_MAJOR_DYLIB}
134+
rm ${LIBARROW_DYLIB}
135+
136+
# Rename libarrow.*.*.*.dylib to libarrow_(arm64|x64).*.*.*.dylib (e.g. libarrow.1700.0.0.dylib -> libarrow_(arm64|x64).1700.0.0.dylib)
137+
mv ${LIBARROW_MAJOR_MINOR_PATCH_DYLIB} ${LIBARROW_ARCH_MAJOR_MINOR_PATCH_DYLIB}
138+
139+
# Rename libarrowproxy.dylib to libarrowproxy_(arm64|x64).dylib
140+
mv ${LIBARROWPROXY_DYLIB} ${LIBARROWPROXY_ARCH_DYLIB}
141+
142+
# Rename libmexclass.dylib to libmexclass_(arm64|x64).dylib
143+
mv ${LIBMEXCLASS_DYLIB} ${LIBMEXCLASS_ARCH_DYLIB}
144+
145+
# Update the identificaton names of the renamed dynamic libraries
146+
install_name_tool -id @rpath/${LIBMEXCLASS_ARCH_DYLIB} ${LIBMEXCLASS_ARCH_DYLIB}
147+
install_name_tool -id @rpath/${LIBARROWPROXY_ARCH_DYLIB} ${LIBARROWPROXY_ARCH_DYLIB}
148+
install_name_tool -id @rpath/${LIBARROW_ARCH_MAJOR_MINOR_PATCH_DYLIB} ${LIBARROW_ARCH_MAJOR_MINOR_PATCH_DYLIB}
149+
150+
# Change install name of dependent shared library libarrow.*.*.*.dylib to libarrow_arm64.*.*.*.dylib in libarrowproxy_(arm64|x64).dylib
151+
install_name_tool -change @rpath/${LIBARROW_MAJOR_DYLIB} @rpath/${LIBARROW_ARCH_MAJOR_MINOR_PATCH_DYLIB} ${LIBARROWPROXY_ARCH_DYLIB}
152+
153+
# Change install name of dependent shared library libmexclass.dylib to libmexclass_(arm64|x64).*.*.*.dylib libarrowproxy_(arm64|x64).dylib
154+
install_name_tool -change @rpath/${LIBMEXCLASS_DYLIB} @rpath/${LIBMEXCLASS_ARCH_DYLIB} ${LIBARROWPROXY_ARCH_DYLIB}
155+
156+
# Change install name of dependent shared library libmexclass.dylib to libmexclass_(arm64|x64).dylib in gateway.(mexmaca64|mexmaci64)
157+
install_name_tool -change @rpath/${LIBMEXCLASS_DYLIB} @rpath/${LIBMEXCLASS_ARCH_DYLIB} ${MEX_GATEWAY}
158+
159+
# Change install name of dependent shared library libarrowproxy.dylib to libarrowproxy_(arm64|x64).dylib in gateway.(mexmaca64|mexmaci64)
160+
install_name_tool -change @rpath/${LIBARROWPROXY_DYLIB} @rpath/${LIBARROWPROXY_ARCH_DYLIB} ${MEX_GATEWAY}
161+
162+
popd

0 commit comments

Comments
 (0)