cmd/link: issues with Apple's new linker in Xcode 15

[cherrymui]

NOTE -- please read

If you see lines starting with ld: warning but not other outputs, they are just warnings, not errors. You should still get a working binary, if there is no other output. Please only report if you see a link error, or a broken binary, or a warning that is not mentioned in the "warnings" section below. Thanks.

See #61229 (comment) for more notes about malformed LC_DYSYMTAB warning. Thanks.

---

In Xcode 15 beta it comes with Apple's new linker, ld-prime. As of beta 3 it is enabled by default. I did some experiments and found some issues for Go (cgo) with the new Apple linker. Some of the issues are reported in #61190. I'm filing a new issue to track all related faulures.

warnings

Apparently the new linker is more picky. It issues some warnings that the old linker (ld64) doesn't. This includes

ld: warning: -bind_at_load is deprecated on macOS

ld: warning: search path '/usr/local/lib' not found

ld: warning: ignoring duplicate library '-lm'

ld: warning: '.../go.o' has malformed LC_DYSYMTAB, expected 92 undefined symbols to start at index 15983, found 102 undefined symbol

These are just warnings and don't affect the correctness of the build. I'm preparing CLs for fix/work around them.

offset/addend being dropped

Mostly for c-archive, c-shared, and plugin build mode, for some relocations it seems the offset/addend being dropped and the relocation is resolved to the beginning of the section. This may cause failures like SIGILL. E.g.

% go test cmd/cgo/internal/testcarchive                 
--- FAIL: TestInstall (1.10s)
    carchive_test.go:487: [go tool cgo -objdir /var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/carchive_test3584575482/_obj472038228 -exportheader p.h p/p.go]
    carchive_test.go:489: [go install -buildmode=c-archive ./libgo]
    carchive_test.go:489: [clang -fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/go-build2775077316=/tmp/go-build -gno-record-gcc-switches -fno-common -I /var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/carchive_test3584575482/pkg/darwin_arm64_shared/testcarchive -o ./testp1 main.c main_unix.c /var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/carchive_test3584575482/pkg/darwin_arm64_shared/testcarchive/libgo.a]
    carchive_test.go:489: 
    carchive_test.go:489: signal: illegal instruction
...

#60694 is the same failure mode.

debug info issues

The new linker doesn't generate stab symbols for our functions, causing debug info being lost. Failure look like

--- FAIL: TestDWARF (0.00s)
    --- FAIL: TestDWARF/testprogcgo (3.85s)
        dwarf_test.go:171: ErrUnknownPC
FAIL
FAIL	cmd/link	8.822s

There is also issues with combining DWARF into executable/shared object. Currently it is affecting c-shared build mode. ld-prime rejects the shared object if we add a DWARF segment to it.

non-PIE build doesn't work (Intel Mac only)

The new linker silently ignores the -no_pie flag and generates a PIE binary, which won't work because we assumed some addresses are not relocated. CL https://go.dev/cl/461697 changes the default to PIE and CL https://go.dev/cl/511355 contains a workaround if we are asked to build a non-PIE binary.

fixes/workarounds

CL https://golang.org/cl/505415 and the stack, as well as other CLs linked in this issue, contain tentative fixes and workarounds that address some of the issues. I'll also work with Apple to have them addressed on their side.

I'm hoping o have them addressed before Apple's new linker is finally released.

[ismail]

I also have an ld assertion failure (reported to Apple as FB12550006):

❯ go install -ldflags "-s -w" github.com/rclone/rclone@master
# github.com/rclone/rclone
/opt/homebrew/Cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: warning: -bind_at_load is deprecated on macOS
ld: warning: '/private/var/folders/68/1pjj14514sl3vj7lbt69lst40000gn/T/go-link-876362130/go.o' has malformed LC_DYSYMTAB, expected 152 undefined symbols to start at index 168240, found 170 undefined symbols starting at index 135
0  0x104a50380  __assert_rtn + 72
1  0x104a099bc  ___ZN2ld16LayoutExecutable27writeContentWithoutLinkEditENSt3__14spanIhLm18446744073709551615EEEy_block_invoke + 9552
2  0x184981950  _dispatch_client_callout2 + 20
3  0x184994ba0  _dispatch_apply_invoke + 176
4  0x184981910  _dispatch_client_callout + 20
5  0x1849933cc  _dispatch_root_queue_drain + 864
6  0x184993a04  _dispatch_worker_thread2 + 156
7  0x184b2b0d8  _pthread_wqthread + 228
ld: Assertion failed: (addr + content.size() <= sectionEndAddr), function writeContentWithoutLinkEdit_block_invoke, file Layout.cpp, line 5689.
clang: error: linker command failed with exit code 1 (use -v to see invocation)

❯ go version
go version go1.20.5 darwin/arm64

❯ clang -v
Apple clang version 15.0.0 (clang-1500.0.34.3)
Target: arm64-apple-darwin23.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[cherrymui]

@ismail Thanks for the report. The CL stack mentioned above (https://golang.org/cl/505415 and dependencies) also avoids the assertion failure.

[gopherbot]

Change https://go.dev/cl/503935 mentions this issue: cmd/link: always use symbol-targeted relocations on Mach-O

[gopherbot]

Change https://go.dev/cl/503538 mentions this issue: cmd/link: fix up more zero-sized local symbols on darwin dynamic linking

[gopherbot]

Change https://go.dev/cl/502616 mentions this issue: cmd/link: use symbol-targeted relocation for initializers on Mach-O

[gopherbot]

Change https://go.dev/cl/502617 mentions this issue: cmd/link: don't generate DYSYMTAB when external linking on Mach-O

[gopherbot]

Change https://go.dev/cl/503539 mentions this issue: cmd/link: sort HOSTOBJ and UNDEFEXT symbols with undefined symbols in Mach-O symbol table

[gopherbot]

Change https://go.dev/cl/505415 mentions this issue: cmd/link: work around issues with ld-prime

[gopherbot]

Change https://go.dev/cl/508696 mentions this issue: cmd/link: suppress some warnings for ld-prime

[gopherbot]

Change https://go.dev/cl/508697 mentions this issue: cmd/cgo/internal/test: don't pass -lm on darwin

[cherrymui]

Another failure mode: (AMD64 Mac only, not ARM64) as of Xcode 15 beta 4 (dyld-1009.6 BUILD 20:09:19 Jul 7 2023) non-PIE binary linked with ld-prime crashes at startup.

runtime: pcHeader: magic= 0xfffffff1 pad1= 0 pad2= 0 minLC= 1 ptrSize= 8 pcHeader.textStart= 0x0 text= 0x10b865000 pluginpath= 
fatal error: invalid function symbol table
runtime: panic before malloc heap initialized

runtime stack:
runtime.throw({0x10b8d04de?, 0x7ff7b469e7a0?})
	/Users/cherryyz/src/go/src/runtime/panic.go:1077 +0x5c fp=0x7ff7b469e420 sp=0x7ff7b469e3f0 pc=0x10b88ffdc
runtime.moduledataverify1(0x7ff7b469e560?)
	/Users/cherryyz/src/go/src/runtime/symtab.go:533 +0x816 fp=0x7ff7b469e540 sp=0x7ff7b469e420 pc=0x10b8aa376
runtime.moduledataverify(...)
	/Users/cherryyz/src/go/src/runtime/symtab.go:519
runtime.schedinit()
	/Users/cherryyz/src/go/src/runtime/proc.go:724 +0x34 fp=0x7ff7b469e580 sp=0x7ff7b469e540 pc=0x10b8939b4
runtime.rt0_go()
	/Users/cherryyz/src/go/src/runtime/asm_amd64.s:349 +0x11e fp=0x7ff7b469e588 sp=0x7ff7b469e580 pc=0x10b8b859e

And the CLs above are not enough to fix it. I'm looking into it.
Possibly related to the crash in #61190, not investigated yet.

(PIE binaries are fine. Also fine on ARM64 as it is always PIE.)

[piusalfred]

I am on MacOS Sonoma beta M1,Xcode 15 beta. I use go.120. When I try to install gopls this is what I get

# golang.org/x/tools/cmd/goimports
ld: warning: '/private/var/folders/jc/lq1rvy214tq4643flbh887xr0000gn/T/go-link-1689581912/go.o' has malformed LC_DYSYMTAB, expected 75 undefined symbols to start at index 6656, found 84 undefined symbols starting at index 53

[gopherbot]

Change https://go.dev/cl/461697 mentions this issue: cmd/go: default to PIE linking on darwin/amd64

[gopherbot]

Change https://go.dev/cl/511355 mentions this issue: cmd/link: force ld64 when building non-PIE binary on macOS

[cherrymui]

@piusalfred that is just a warning, not an error. You should still get a binary. Does the binary run? If not, see #61190, and post on that issue if you have any update. Thanks!

[cherrymui]

The non-PIE binary crash (#61229 (comment)) is due to ld-prime silently ignores -no_pie flag and generates a PIE binary instead. But that doesn't work because in exe buildmode we assumed the binary is not relocated and so generated non-relocated addresses, which would not be correct if the binary is actually relocated.

The CLs above handle it: 1. switch to PIE by default, 2. force ld64 in exe buildmode.

(#61190 is not this. Still need to investigate.)

[VarusHsu]

I resolved it by disabling cgo.

go env -w CGO_ENABLED='0'

[tkgalk]

That would explain why GoLand marked those tests as failed.

I think this is a different problem, GoLand shouldn't mark tests as failed due to extra output. Can you please describe how to reproduce this issue? Here or in the comments to GO-16873.

I wouldn't agree it's a different problem. GoLand, or any other app, that sees things on stderr instead of stdout should be reasonably able to assume it's an error and treat it as a failure.

[edaniels]

#61229 (comment)

@cherrymui is it possible the .set directives now being used https://github.com/llvm/llvm-project/blame/main/compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h#L20-L22 fix the issue? If so, would it be possible to update the syso files race uses? They're two years old at this point so it seems like this isn't really a common operation. Forgive me if I'm misunderstanding how the internal mechanics of utilizing TSAN works.

[cherrymui]

I tried that a while ago and it appears that changing to .set doesn't help. The warning is still there. We'll need to think more on what the solution is.

[AlekSi]

We'll need to think more on what the solution is

Then maybe the issue could be re-opened and updated? Started seeing has malformed LC_DYSYMTAB with 1.24 out of the blue, and it is weird that the relevant issue is closed (and also mentions beta in the title, etc)

[cherrymui]

Removed "beta."

The output binary is not malformed and is correct.

[AlekSi]

Yes, it works great for me. It caused some other problems due to 1.24's go build -json being called from go test -json, and my consumer could not handle those build warnings. That's how I found that issue.

But – why are those warnings visible at all? Normally, go build does not produce warnings and hides output from system tools. So why make this one warning visible, especially if it is harmless and the user can't do anything about it anyway? It is just scary-looking and confusing.

[AlekSi]

if you pass -race and you're using Go 1.22.x, it is probably the race detector's C code

Is it still the case for 1.24?

[cherrymui]

Is it still the case for 1.24?

Yes. Edited to "Go 1.22.x or newer"

[cherrymui]

But – why are those warnings visible at all?

These warnings are from the Apple C linker, not the Go linker. In general it is possible that the C linker warns on some other user C object file, not necessarily the race detector one. I guess it is possible to parse the warning and the object file it warns on and see if it is the race detector one and hide that one if so. Not sure if this is really a good idea.

That said, if it doesn't work well with go build -json, we should fix.

[AlekSi]

That said, if it doesn't work well with go build -json, we should fix.

No-no, it works fine. It is just that my go test -json parsing code uses DisallowUnknownFields, and 1.24's go test -json calls go build -json that generates this warning with ImportPath field (as documented).

Thank you so much for your answers!

[sbp-bvanb]

FYI: Still occurs while using 1.24.1.

[ejain]

Seen on go version go1.24.2 darwin/arm64, Xcode 16.3.