What Would You Do if You Weren't Afraid?


A service I worked on hit a time bomb, and the whole service went down. I tried to fix it by patching the binary on every machine. I didn’t pull it off in time.

The time bomb

The service was Sigma, the one from The Compiler Is Wrong. We integrated a lot of code that other teams maintained, because our data sources used their clients.

I don’t remember what the time bomb was. We practiced gradual rollout, and everything was fine for a while. A gradual rollout catches code that breaks when it starts running. It doesn’t catch code that works until a moment that comes for every server at once.

Monitoring caught all the servers going down, and they crashed again on restart. The cause was obvious from the core dumps, and there was no lever to stop it.

The slow way out

The way out was to push new code. That took forever to build, and a deploy could take minutes on top of it.

Nopping it out

The time bomb could be nopped out. A check like that compiles down to a comparison and a jump. Overwrite the jump with instructions that do nothing, and the check is gone.

I had a clanker write the smallest program with a time bomb in it, to show what that means:

#include <cstdio>
#include <cstdlib>
#include <ctime>

// A time bomb: this check was fine on the day it was written.
static const time_t kDeadline = 1400000000;  // 2014-05-13

int main() {
  if (time(nullptr) > kDeadline) {
    fprintf(stderr, "deadline passed, refusing to start\n");
    abort();
  }
  puts("serving requests");
  return 0;
}
$ g++ -O2 service.cpp -o service
$ ./service
deadline passed, refusing to start
Aborted

In the compiled program the whole check is three instructions, and the last one is two bytes long:

<main> (File Offset: 0x1080):
  55                    push   %rbp
  31 ff                 xor    %edi,%edi
  48 89 e5              mov    %rsp,%rbp
  e8 b5 ff ff ff        call   <time@plt>
  48 3d 00 4e 72 53     cmp    $0x53724e00,%rax
  7f 10                 jg     <main+0x23>

A patch instead of a binary

Doing that by hand on one machine is easy. Getting a patched binary to every machine is not: it was hard to package, and the deploy would still take minutes.

The idea came from the days when anime releases sometimes came with binary patches. A fix was a small file, and xdelta turned the release you already had into the fixed one.

Our deployment had a pre-deployment command. I believe it was designed for one-time setup before a new version runs. I put the patching script into it, so that every machine would patch the binary it already had.

For the program above, the script is one command:

# Nop out the two-byte jump at 0x1091, but only if those two bytes
# are the jump we expect.
[ "$(od -An -tx1 -j $((0x1091)) -N 2 ./service | tr -d ' \n')" = 7f10 ] &&
  printf '\x90\x90' | dd of=./service bs=1 seek=$((0x1091)) conv=notrunc status=none
$ sh patch.sh
$ ./service
serving requests

The numbers, Mason. What do they mean?[1]

The numbers were offsets. I knew where the jump was in a loaded binary, because I had it in gdb. But the script patches a file, and between a file and a loaded binary sits the layer of linkers and loaders.

This is the jump in the toy program, in gdb:

(gdb) x/i main+17
   0x555555555091 <main+17>: jg     0x5555555550a3 <main+35>

The script above writes to 0x1091. Both numbers name the same two bytes, and three programs had a say in them.

The compiler put the jump 17 bytes into main.

The linker put main at address 0x1080 of the program. It also wrote down which part of the file holds which addresses:

$ readelf -lW service | grep -E 'Type|LOAD'
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x0007b0 0x0007b0 R   0x1000
  LOAD           0x001000 0x0000000000001000 0x0000000000001000 0x0001c9 0x0001c9 R E 0x1000
  LOAD           0x002000 0x0000000000002000 0x0000000000002000 0x000168 0x000168 R   0x1000
  LOAD           0x002d80 0x0000000000003d80 0x0000000000003d80 0x0002b0 0x0002d0 RW  0x1000

The second LOAD line is the code: addresses from 0x1000 come from offset 0x1000 in the file. For the code of this small program the two columns agree. The last line shows that they don’t have to: the data at address 0x3d80 sits at offset 0x2d80.

The loader picked where the whole program goes in memory. Under gdb it starts at 0x555555554000. Outside gdb it starts somewhere else on every run. This is a two-line program that prints the address of its own main:

$ ./where; ./where; ./where
main is at 0x560b9824b050
main is at 0x563169557050
main is at 0x5572a4fcf050

So the address from gdb has to be undone in two steps. Take away where the loader put the program, and what is left is the linker’s address. Look that up in the linker’s table, and you have the offset in the file.

The program's pieces in the file and in the running process: the code is at offset 0x1000 in the file and at address 0x555555555000 in memory, and the jump is 0x91 bytes into it.

gdb has a table with both steps in it:

(gdb) info proc mappings
Start Addr         End Addr           Size               Offset             Perms File
0x0000555555554000 0x0000555555555000 0x1000             0x0                r--p  service
0x0000555555555000 0x0000555555556000 0x1000             0x1000             r-xp  service
...

The jump is in the second mapping, so its offset is 0x555555555091 - 0x555555555000 + 0x1000, which is 0x1091.

Back then executables usually weren’t position independent, so the loader had less to say and the linker more. Built that way, the same program has its jump at address 0x401081 and at offset 0x1081 in the file.

I think I did all of this by hand, sifting through assembly with debug symbols. That was slow.

And whichever way you get the number, it holds for one exact build. The same source compiled with -O1 has its jump at 0x117d, not 0x1091. There the check in the script fails, nothing is written, and the service still refuses to start.

Too slow

I didn’t pull it off in time. I had tested it on one machine, but someone managed to build the package faster than expected. The proper fix fixed it.

What would you do if you weren’t afraid?

The question is a callback to one of Facebook’s “propaganda” posters, the same family as “Move fast and break things”.

A poster with the words: What would you do if you weren't afraid?

The poster at Facebook’s Menlo Park campus in March 2013. Photo by Thomas Hawk, CC BY-NC 2.0.

Later I turned the story into a flash talk for an audience slightly larger than my team: us and the consumers of our service. People generally liked it. Someone even suggested doing it for a larger internal audience.