16jan2019

Subject: Unexpected symbol value for unininitalized arrays larger than 64K in
COFF object files

I have made an observation I cannot explain regarding the symbol value of
unininitalized C arrays in COFF object files produced by Visual Studio 2015.

Consider the source file testdata_works.c

  char testb[0x10002];

Consider the source file testdata_fails.c:

  char testa[0x10001];
  char testb[0x10002];

Now compile them both:

cl -c testdata_works.c
cl -c testdata_fails.c

This produces the object files testdata_works.obj and testdata_fails.obj.

Now run "dumpbin /symbols" on both object files and pay attention to how the
value of the symbol "testb" differs between the two object files(!) See below
for the gory details.

=======================================
dumpbin /symbols testdata_works.obj

Microsoft (R) COFF/PE Dumper Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file testdata_works.obj

File Type: COFF OBJECT

COFF SYMBOL TABLE
000 01045E97 ABS    notype       Static       | @comp.id
001 80000190 ABS    notype       Static       | @feat.00
002 00000000 SECT1  notype       Static       | .drectve
    Section length   2F, #relocs    0, #linenums    0, checksum        0
004 00000000 SECT2  notype       Static       | .debug$S
    Section length   6C, #relocs    0, #linenums    0, checksum        0
006 00010002 UNDEF  notype       External     | testb

String Table Size = 0x0 bytes

  Summary

          6C .debug$S
          2F .drectve
=======================================
dumpbin /symbols testdata_fails.obj

C:\work>dumpbin /symbols testdata_fails.obj
Microsoft (R) COFF/PE Dumper Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file testdata_fails.obj

File Type: COFF OBJECT

COFF SYMBOL TABLE
000 01045E97 ABS    notype       Static       | @comp.id
001 80000190 ABS    notype       Static       | @feat.00
002 00000000 SECT1  notype       Static       | .drectve
    Section length   2F, #relocs    0, #linenums    0, checksum        0
004 00000000 SECT2  notype       Static       | .debug$S
    Section length   6C, #relocs    0, #linenums    0, checksum        0
006 00010001 UNDEF  notype       External     | testa
007 00010041 UNDEF  notype       External     | testb

String Table Size = 0x0 bytes

  Summary

          6C .debug$S
          2F .drectve
=======================================

We note that the symbol testb has the expected value 0x10002 in
testdata_works.obj but has the unexpected value 0x10041 in testdata_fails.obj.

Finally - I have saved the really strange part for last (which may trigger the
proverbial "a-ha" in those who are well versed in the COFF object file
format):

Decrease the size of the arrays in testdata_fails.c from above 0x10000 to below
0x10000, for example: 

  char testa[0x0001];
  char testb[0x0002];

and compile

cl -c testdata_fails.c

The dumpbin command now produces the expected result for symbols testa and
testb:

=======================================
dumpbin /symbols testdata_fails.obj
Microsoft (R) COFF/PE Dumper Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file testdata_fails.obj

File Type: COFF OBJECT

COFF SYMBOL TABLE
000 01045E97 ABS    notype       Static       | @comp.id
001 80000190 ABS    notype       Static       | @feat.00
002 00000000 SECT1  notype       Static       | .drectve
    Section length   2F, #relocs    0, #linenums    0, checksum        0
004 00000000 SECT2  notype       Static       | .debug$S
    Section length   6C, #relocs    0, #linenums    0, checksum        0
006 00000001 UNDEF  notype       External     | testa
007 00000002 UNDEF  notype       External     | testb

String Table Size = 0x0 bytes

  Summary

          6C .debug$S
          2F .drectve
=======================================

Final remarks: 

I have reproduced the above using clang/llvm-nm 7.0.1 instead of Visual
Studio 2015/dumpbin.

You may rightfully ask how I stubled across this. I was studying the BSD
kernel genassym.sh script, and decided to write a small C++ STL program that
achieved the same task based on any object file format via the LLVM libraries.
I have this program built, but it does not work properly because the data in
the object files are not as expected.

Any suggestions what is going on here and how to fix it so it works for symbol
sizes larger than 64K?

Posted question on StackOverflow as
https://stackoverflow.com/questions/54217746/unexpected-symbol-value-for-unininitalized-arrays-larger-than-64k-in-coff-object

UPDATE: There was an error made when testing this with clang. This caused us
to wrongfully conclude that it does not work with clang. It turns out to work
as expected with clang 7.0.1.