Corrupt Mach-O file

[kellycampbell]

Bloaty gives a Corrupt Mach-O file error on a file.

lipo -detailed_info gives this output:

Fat header in: REDACTEDSdk
fat_magic 0xcafebabe
nfat_arch 5
architecture armv7
    cputype CPU_TYPE_ARM
    cpusubtype CPU_SUBTYPE_ARM_V7
    offset 108
    size 8291248
    align 2^2 (4)
architecture armv7s
    cputype CPU_TYPE_ARM
    cpusubtype CPU_SUBTYPE_ARM_V7S
    offset 8291356
    size 8307560
    align 2^2 (4)
architecture i386
    cputype CPU_TYPE_I386
    cpusubtype CPU_SUBTYPE_I386_ALL
    offset 16598916
    size 8294024
    align 2^2 (4)
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    offset 34485416
    size 8454008
    align 2^3 (8)
architecture arm64
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64_ALL
    offset 24892944
    size 9592472
    align 2^3 (8)

The hexdump of the first part of the file is this:

0000000 ca fe ba be 00 00 00 05 00 00 00 0c 00 00 00 09  >................<
0000020 00 00 00 6c 00 7e 83 b0 00 00 00 02 00 00 00 0c  >...l.~..........<
0000040 00 00 00 0b 00 7e 84 1c 00 7e c3 68 00 00 00 02  >.....~...~.h....<
0000060 00 00 00 07 00 00 00 03 00 fd 47 84 00 7e 8e 88  >..........G..~..<
0000100 00 00 00 02 01 00 00 07 00 00 00 03 02 0e 34 a8  >..............4.<
0000120 00 80 ff 78 00 00 00 03 01 00 00 0c 00 00 00 00  >...x............<
0000140 01 7b d6 10 00 92 5e 98 00 00 00 03 21 3c 61 72  >.{....^.....!<ar<
0000160 63 68 3e 0a 23 31 2f 32 30 20 20 20 20 20 20 20  >ch>.#1/20       <
0000200 20 20 20 20 31 35 31 39 32 32 31 38 35 30 20 20  >    1519221850  <
0000220 35 30 31 20 20 20 32 30 20 20 20 20 31 30 30 36  >501   20    1006<

[haberman]

Any way for me to reproduce this? Any chance you could post a binary where this is happening?

[kellycampbell]

Sorry, it's from a proprietary smart lock sdk that I'm not able to post. I could run a debug version of bloaty against it and post output though if that would help?

[alaroldai]

I can reproduce this as well - looks like an issue with fat / multi-architecture libraries.
Worked around by running lipo -thin arm64 to produce a single-architecture library, then running bloaty on the result.

[haberman]

Hmm, Bloaty has code to support fat Mach-O files. I thought I had tested it, but maybe it's broken.

I have my hands full at the moment, but I'd be happy to take contributions from anyone who wants to do a little debugging of the code: https://github.com/google/bloaty/blob/master/src/macho.cc#L201-L236

[EricRahm]

FYI @rjmansfield this might be up your alley.

[rjmansfield]

FYI @rjmansfield this might be up your alley.

Yes, I know what this is. The binary above is a FAT/universal archive. e.g. the !<arch> magic in the hex dump. I can reproduce a similar Corrupt Mach-O file message by creating two archives and then doing:

lipo -create libfoo_arm64a.a libfoo_x86_64.a -output libfoo_universal.a

I have been actually been working on adding Mach-O support for Darwin archives i.e. -d armembers and part of that is adding support for universal archives. However, the problem with universal archives is that the vmsize mapping for multiple architectures is somewhat non-sensical in terms of bloaty handles it. i.e each architecture doesn't have their own private address space so either they need to be only compared between arches using file size, or by the architecture slice. I was going to propose to add a mach-o specific --arch option (similar to how other tools support universal binaries), so that a user can select which slice should be analyzed, or if not specified, warn that it Is a universal binary and the vm size results may be confusing.

[rjmansfield]

For completeness, here are full steps to reproduce what I believe is the issue:

$ cat test_arm64.c
int arm64_func() {
    return 0;
}
$ cat > test_x86_64.c
int x86_64_func() {
    return 0;
}
$ clang -arch arm64 -c test_arm64.c -o test_arm64.o
$ clang -arch x86_64 -c test_x86_64.c -o test_x86_64.o
$ ar cr libtest_arm64.a test_arm64.o
$ ar cr libtest_x86_64.a test_x86_64.o
$ lipo -create libtest_arm64.a libtest_x86_64.a -output libtest_universal.a
$ lipo -detailed_info libtest_universal.a
Fat header in: libtest_universal.a
fat_magic 0xcafebabe
nfat_arch 2
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    offset 68
    size 824
    align 2^2 (4)
architecture arm64
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64_ALL
    offset 892
    size 720
    align 2^2 (4)
$ bloaty libtest_universal.a
bloaty: Corrupt Mach-O file

[haberman]

I was going to propose to add a mach-o specific --arch option (similar to how other tools support universal binaries), so that a user can select which slice should be analyzed, or if not specified, warn that it Is a universal binary and the vm size results may be confusing.

From my perspective, the most satisfying solution would be to add a data source that distinguishes between arches (I think this is what you were proposing with -d armembers), and then to extend the VM address space logic in Bloaty to handle multiple memory segments.

I would imagine each memory segment being an integer, and then on the side we'd keep a mapping, which in the case of Mach-O fat binaries would be something like:

• 0 -> CPU_TYPE_X86_64,CPU_SUBTYPE_X86_64_ALL
• 1 -> CPU_TYPE_ARM64, CPU_SUBTYPE_ARM64_ALL

This is satisfying to me for two reasons:

1. It preserves Bloaty's overall baseline guarantee, which is that the totals shown in the output account for the entire file(s) passed on the command-line.
2. It gives flexibility in how the data is analyzed. If you want to process all architectures together, in a way that aggregates all symbols of the same name across architectures, you can (this would be the default behavior). If you want to filter so that only one architecture is visible, you can use --source-filter to filter out the architecture you don't want. If you want an analysis that will show each symbol broken down by how big it is in each architecture, you can pass something like -d symbols,armembers.

The main downside is just that it will require more surgery inside Bloaty to accommodate a larger address size.