That One Time It Was Glibc


Four years after the compiler was wrong, almost to the day, it was glibc. And glibc had already fixed it, four years earlier.

Loading Haskell with the system linker

By then Sigma was a C++ server that loaded Haskell code, and the Haskell code called back into the server’s C++ routines. The Haskell code changed hundreds of times a day, so the server loaded each new version while it was running and dropped the old one once nothing referred to it. Jon Coens described how that worked in October 2017. The “shared objects” it loaded were really object files, loaded by GHC’s own linker, and they could only go into the lower 2 GB of memory:

We’ve already made a few improvements to the GHCi linker to reduce the impact of these problems, but we’re running out of options.

Ideally we’d switch to using true shared objects (built with -fPIC) to remove this limitation.

In December 2017 I was working on that, in Facebook’s fork of GHC. I was adding two functions to GHC’s runtime, loadNativeObj and unloadNativeObj. They load a shared object with Haskell code using the system linker, and have GHC call dlclose() once the code is no longer referenced from the heap. With the system linker, the shared object can be loaded outside the low memory region, and its DWARF sections are loaded in a way that perf understands.

The change lived in Facebook’s fork until 2020, when Ray Shih upstreamed it with the line “This change is originally written by niteria”.

Who is being loaded

A Haskell library can export functions to C, which is how a C++ server calls into it. When such a library is loaded, its constructors hand the runtime a stable pointer for each of them, through foreignExportStablePtr. The runtime has to free those stable pointers when the library is unloaded, so it has to know which library they came from. The constructors don’t say. They do run inside dlopen, so the runtime leaves itself a note before it calls dlopen, and takes the note away afterwards:

   ForeignExportStablePtr *fe_sptr = NULL;
   loading_fe_stable_ptr = &fe_sptr;
   // dlopen will run foreignExportStablePtr adding StablePtrs from this DLL
   // to fe_sptr
   hdl = dlopen(path, RTLD_NOW|RTLD_LOCAL);
   loading_fe_stable_ptr = NULL;

The note is a static variable in the same file, and foreignExportStablePtr reads it:

static ForeignExportStablePtr **loading_fe_stable_ptr = NULL;

StgStablePtr foreignExportStablePtr (StgPtr p)
{
    ...
    if (loading_fe_stable_ptr != NULL) {
        fe_sptr = stgMallocBytes(sizeof(ForeignExportStablePtr),
                                 "foreignExportStablePtr");
        fe_sptr->stable_ptr = sptr;
        fe_sptr->next = *loading_fe_stable_ptr;
        *loading_fe_stable_ptr = fe_sptr;
    }
    ...
}

It never unloaded

I think the visible symptom was that the Haskell code never unloaded. The write-up has a name for objects that don’t unload, “sticky shared objects”, and it says where that ends:

Eventually so many shared objects are linked at once that the process runs out of space to load in a new object, stalling all updates until the process is restarted.

I can’t run that setup any more, so I had a clanker rebuild the situation in three small files. The first stands in for the runtime’s table of stable pointers:

/* Stands in for the runtime's stable pointer table. */
static int live = 0;
void table_add(void) { live++; }
void table_free(int n) { live -= n; }
int table_live(void) { return live; }

The second is the runtime’s linker. It leaves the note, loads the library, and closes it again once nothing that the library registered is left:

#include <stdio.h>

#ifdef OLD_GLIBC
/* <dlfcn.h> before glibc 2.19, with the __THROW macro expanded. */
extern void *dlopen(const char *file, int mode) __attribute__((nothrow, leaf));
extern int dlclose(void *handle) __attribute__((nothrow, leaf));
#define RTLD_NOW 2
#else
#include <dlfcn.h>
#endif

#ifndef QUALIFIER
#define QUALIFIER
#endif

void table_add(void);
void table_free(int n);
int table_live(void);

/* While a library is being loaded, its registrations are counted here. */
static int *QUALIFIER loading = NULL;

/* Called from the constructors of the library. */
void report(void) {
  table_add();
  if (loading != NULL) (*loading)++;
}

int main(void) {
  for (int round = 1; round <= 3; round++) {
    int mine = 0;
    loading = &mine;
    void *handle = dlopen("./libplugin.so", RTLD_NOW);
    loading = NULL;

    /* Unload: free what the library registered, and close it once
       nothing refers to it any more. */
    table_free(mine);
    if (table_live() == 0) {
      dlclose(handle);
      printf("round %d: closed\n", round);
    } else {
      printf("round %d: still in use\n", round);
    }
  }
  puts("exiting");
  return 0;
}

The third is a C++ library with one static object:

#include <cstdio>

extern "C" void report(void);

struct Registration {
  Registration() { puts("  constructor"); report(); }
  ~Registration() { puts("  destructor"); }
};

static Registration registration;

OLD_GLIBC selects the declaration of dlopen that glibc had at the time.

$ g++ -shared -fPIC plugin.cpp -o libplugin.so
$ gcc -O2 -DOLD_GLIBC -rdynamic host.c table.c -ldl -o host && ./host
  constructor
round 1: still in use
round 2: still in use
round 3: still in use
exiting
  destructor

Three rounds of loading and unloading, and the library is never closed. The registration from the constructor never shows up in mine, so the host never frees it, and the library always looks as if it were still in use. Its constructor runs once, because dlopen doesn’t run the constructors of a library that is still loaded. Its destructor only runs when the program exits.

The assignment that wasn’t there

It is a miserable kind of bug to chase. It goes away without optimization. It also goes away if you put an fprintf between the assignment and the call to see what is going on.

I read the disassembly. This is the call to dlopen in the optimized main:

	movl	$2, %esi
	leaq	.LC0(%rip), %rdi
	call	dlopen@PLT
	xorl	%edi, %edi
	movq	$0, loading(%rip)
	movq	%rax, %r12
	call	table_free@PLT

The note is never left. The only write to loading is the one after the call, and the count passed to table_free is a constant zero.

A leaf that calls back

dlopen comes from glibc. From version 2.15 to version 2.18, glibc declared it like this:

extern void *dlopen (const char *__file, int __mode) __THROW;

__THROW looks as if it were about exceptions. In those versions it also said leaf:

# define __LEAF , __leaf__
# define __THROW	__attribute__ ((__nothrow__ __LEAF))

This is what leaf promises, according to GCC’s manual:

Calls to external functions with this attribute must return to the current compilation unit only by return or by exception handling. In particular, leaf functions are not allowed to call callback function passed to it from the current compilation unit or directly call functions exported by the unit or longjmp into the unit.

And this is what the compiler does with the promise:

The compiler takes the hint that any data not escaping the current compilation unit can not be used or modified by the leaf function.

dlopen breaks that promise whenever a constructor of the library it loads calls back into the file that called dlopen. Ours did. So GCC took the hint. The note is a static variable, a leaf function can’t read it, and it is overwritten right after the call. That makes the first assignment dead, and GCC removed it. The constructors ran, found no note, and recorded nothing.

The stable pointers were still created, and the exported functions don’t go through them, so the Haskell code ran as usual. The only thing missing was the record of which library they belonged to, and that only matters when the library is unloaded. Then none of them are freed, and a stable pointer that is never freed is a reference into the library that never goes away.

The fprintf brought it back because fprintf is not a leaf function, so the compiler has to assume it might read the note.

With the declaration glibc has today, the note is written before the call and the count is read back after it:

	movl	$0, -36(%rbp)
	movq	%r13, loading(%rip)
	call	dlopen@PLT
	movl	-36(%rbp), %edi
	movq	$0, loading(%rip)
	movq	%rax, %r12
	call	table_free@PLT

And the program does what it should:

$ gcc -O2 -rdynamic host.c table.c -ldl -o host && ./host
  constructor
  destructor
round 1: closed
  constructor
  destructor
round 2: closed
  constructor
  destructor
round 3: closed
exiting

GCC 4.8.5 behaves the same as today’s GCC, with either declaration.

Fixed four years ago

glibc had fixed this in August 2013. Mike Frysinger’s patch says why:

Since the dlopen funcs might invoke a constructor that is in the same compilation unit as the caller, we cannot mark them as leaf funcs.

It went into glibc 2.19, which came out in February 2014. The glibc we were building against was older than that.

So the workaround went into my change, as one word and a comment:

// volatile works around
// https://sourceware.org/ml/libc-alpha/2013-08/msg00465.html
static ForeignExportStablePtr ** volatile loading_fe_stable_ptr = NULL;

GCC’s manual recommends the same thing for the one such case it mentions, a signal handler that uses static variables:

The only compliant way to write such a signal handler is to declare such variables volatile.

In the small program it is -DQUALIFIER=volatile. With it, the old declaration gives the same three constructors and three destructors as today’s.