The Virtual Void

A whole fleet of obfuscation techniques inside a delicious burger

2024-11-21T00:00:00-06:00

A data-centric public web page uses some fun (but ultimately ineffective) obfuscation methods to spoil naive attempts at scraping their data. It’s not completely clear to me why it was worthwhile to obfuscate the data in this way, preventing scraping? Data licensing issues? Or just being paranoid about people trying to rebuild the functionality? Someone having fun?

But I don’t mind, there are some fun puzzles ahead.

As so often, le voyage begins in the Network tab of the browser showing these files with the .hwx extension which look like this:

00000000  a8 3c 14 d6 a7 d1 6e fc  fa fb b8 c1 a5 ae 6e ff  |.<....n.......n.|
00000010  c0 59 63 dd 8d bc b5 19  05 af 30 88 9a e8 db bc  |.Yc.......0.....|
00000020  15 2e 90 45 0d a3 65 41  0b ff 2a 3e dd 10 4b c1  |...E..eA..*>..K.|
00000030  3d f7 91 5a c2 4a dd 13  f9 5d a7 98 f9 e0 3e 41  |=..Z.J...]....>A|
00000040  fa fd ed 95 ef 44 d8 82  a1 2d 20 59 6c 8a fc 16  |.....D...- Yl...|
00000050  e4 93 e3 cb 20 39 bf 95  bf fb 3f 16 3c e7 4e 77  |.... 9....?.<.Nw|
00000060  39 a8 53 60 c8 71 56 18  9a 26 d5 22 e8 81 8d ac  |9.S`.qV..&."....|
00000070  38 8b c8 0e 59 c2 f0 14  d9 b2 0c 7b d8 76 e7 bf  |8...Y......{.v..|
00000080  9c ff 69 58 fd aa e2 b2  31 f1 44 4c a9 26 a0 19  |..iX....1.DL.&..|
00000090  de 4a 48 f7 4e a8 d6 9c  79 b0 12 74 98 d3 d3 c2  |.JH.N...y..t....|
...

with no discernable structure. 4280 bytes with a quite uniform distribution of byte values.

This looks like the output of a compression or encryption algorithm (high entropy). The next step was to see if the decoder code could be found in the Javascript portion of the web site.

The requests were XHR requests initiated by a script of the webpage that was minified but not obfuscated. Using the debugger it was easy to catch the XHR request and the processing of the response. The important code looks like this:

xhr.onload = function (e) {
    if (this.status == 200) {
        var uInt8Array = new UInt8Array(this.response);
        var data = new TextDecoder("utf-8").decode(uInt8Array);
        // (way to go about parsing JSON, but that's a different story)
        callback(datatype == "json" ? eval("(" + data + ")") : datatype == "xml" ? $(data) : data)
    }
};

This does not look suspicious after all. Since the code also redefines TextDecoder that seemed like an obvious suspect but the code turned out to be indeed a simple UTF-8 decoder. Inspecting the values in the debugger while it was running showed that the data was indeed processed here, but unexpectedly during the code to new UInt8Array. The debugger allowed to step into the code which was part of an anonymous source. The code looked like this:

  window._0xedc3 = [
    'length'
  ];
  window.UInt8Array = function (_0x7f4ax2) {
    var _0x7f4ax3 = new Uint8Array(_0x7f4ax2);
    var _0x7f4ax4 = _0Xedc3(_0x7f4ax3[0]);
    var _0x7f4ax5 = (_0x7f4ax4(_0x7f4ax3[1]) << 16) + (_0x7f4ax4(_0x7f4ax3[2]) << 8) + (_0x7f4ax4(_0x7f4ax3[3]));
    var _0x7f4ax6 = 4;
    var _0x7f4ax7 = new Uint8Array(_0x7f4ax5);
    var _0x7f4ax8 = 0;
    while (_0x7f4ax6 < _0x7f4ax3[_0xedc3[0]] && _0x7f4ax8 < _0x7f4ax5) {
      var _0x7f4ax9 = _0x7f4ax3[_0x7f4ax6];
      _0x7f4ax9 = (_0x7f4ax9 ^ (_0x7f4ax6 & 255)) ^ 163;
      ++_0x7f4ax6;
      for (var _0x7f4axa = 7; _0x7f4axa >= 0; _0x7f4axa--) {
        if ((_0x7f4ax9 & (1 << _0x7f4axa)) == 0) {
          _0x7f4ax7[_0x7f4ax8++] = _0x7f4ax4(_0x7f4ax3[_0x7f4ax6++])
        } else {
          var _0x7f4axb = _0x7f4ax4(_0x7f4ax3[_0x7f4ax6]);
          var _0x7f4axc = (_0x7f4axb >> 4) + 3;
          var _0x7f4axd = (((_0x7f4axb & 15) << 8) | _0x7f4ax4(_0x7f4ax3[_0x7f4ax6 + 1])) + 1;
          _0x7f4ax6 += 2;
          for (var _0x7f4axe = 0; _0x7f4axe < _0x7f4axc; _0x7f4axe++) {
            _0x7f4ax7[_0x7f4ax8] = _0x7f4ax7[_0x7f4ax8++ - _0x7f4axd]
          }
        };
        if (
          _0x7f4ax6 >= _0x7f4ax3[_0xedc3[0]] &&
          _0x7f4ax7[_0xedc3[0]] >= _0x7f4ax5
        ) {
          break
        }
      }
    };
    return _0x7f4ax7
  }

Superficially, I was confused (as intended) because it looks like kind of infrastructure code that was somehow decompiled? But wouldn’t UInt8Array be defined natively?

But after seeing the values before and after the processing, it was completely clear that this was the code that was responsible for decoding of the data (it could still have been some benign compression scheme, being minified in a particular way).

Inside the code, one other function with a weird name is called, _0Xedc3, its code looks like this:

window._0Xedc3 = function (_0xc5c0x2) {
    var _0xc5c0x3 = _0x88FE88;
    var _0xc5c0x4 = _0xFE88AA;
    var _0xc5c0x5 = _0xEE8080;
    var _0xc5c0x6 = _0xA0A0F0;
    for (var _0xc5c0x7 = 0; _0xc5c0x7 < _0xc5c0x2; ++_0xc5c0x7) {
      _0xc5c0x8()
    };
    function _0xc5c0x8() {
      var _0xc5c0x9 = _0xc5c0x3;
      _0xc5c0x9 ^= (_0xc5c0x9 << 11) & 16777215;
      _0xc5c0x9 ^= (_0xc5c0x9 >> 8) & 16777215;
      _0xc5c0x3 = _0xc5c0x4;
      _0xc5c0x4 = _0xc5c0x5;
      _0xc5c0x5 = _0xc5c0x6;
      _0xc5c0x6 ^= (_0xc5c0x6 >> 19) & 16777215;
      _0xc5c0x6 ^= _0xc5c0x9
    }
    return function (_0xc5c0xa) {
      var _0xc5c0xb = _0xc5c0xa ^ (_0xc5c0x3 & 255);
      _0xc5c0x8();
      return _0xc5c0xb
    }
  }

Ok… At this point chatgpt could already provide quite good guesses about what these functions do. The latter function is a simple XOR cipher based on a PRNG to generate a key stream to XOR the plain text against. The former function implements a simple LZ compression scheme, somewhat similar to LZ4, a dictionary coder without an entropy stage.

On the cipher side, the PRNG used is a (variant of?) xorshift128. The state is seeded at the beginning using

var _0xc5c0x3 = _0x88FE88;
var _0xc5c0x4 = _0xFE88AA;
var _0xc5c0x5 = _0xEE8080;
var _0xc5c0x6 = _0xA0A0F0;

setting the initial 128 bit state. The argument given to the constructor is used as an additional seed to skip the given number of rounds of the key stream.

The function itself returns a function that can deobfuscate a single byte. It does it in the obvious way by XORing the byte with the current value of the PRNG and then updating the PRNG state.

The grand scheme of the LZ compression was already figured out by chatgpt, so I only had to fill out the details here:

   window._0xedc3 = [ // obfuscated way of calling "length"
    'length'
  ];
  window.UInt8Array = function (_0x7f4ax2) {
    // create a copy of the input
    var _0x7f4ax3 = new Uint8Array(_0x7f4ax2);
    // seed the cipher from the first byte of the input
    var _0x7f4ax4 = _0Xedc3(_0x7f4ax3[0]); 
    // read the next 3 bytes from the input, decrpyt, and interpret them as a BE 24-bit integer representing the length of the output
    var _0x7f4ax5 = (_0x7f4ax4(_0x7f4ax3[1]) << 16) + (_0x7f4ax4(_0x7f4ax3[2]) << 8) + (_0x7f4ax4(_0x7f4ax3[3]));
    // input offset, skip over header
    var _0x7f4ax6 = 4;
    // output array
    var _0x7f4ax7 = new Uint8Array(_0x7f4ax5);
    // output offset
    var _0x7f4ax8 = 0;
    // iterate until either the input or output is exhausted, `xyz[_0xedc3[0]]` is a funny way to call `length`
    while (_0x7f4ax6 < _0x7f4ax3[_0xedc3[0]] && _0x7f4ax8 < _0x7f4ax5) {
      // read a byte from the input
      var _0x7f4ax9 = _0x7f4ax3[_0x7f4ax6];
      // use another cipher to decrypt the byte, XORing with a constant and the current offset
      _0x7f4ax9 = (_0x7f4ax9 ^ (_0x7f4ax6 & 255)) ^ 163;
      // _0x7f4ax9 is a control byte where each bit indicates whether each of the next 8 bytes is a literal or a backreference
      ++_0x7f4ax6;
      // now look at each bit in the control byte
      for (var _0x7f4axa = 7; _0x7f4axa >= 0; _0x7f4axa--) {
        // if the bit is 0, the next byte is a literal
        if ((_0x7f4ax9 & (1 << _0x7f4axa)) == 0) {
          // decrypt the byte and write it to the output
          _0x7f4ax7[_0x7f4ax8++] = _0x7f4ax4(_0x7f4ax3[_0x7f4ax6++])
        } else {
          // if the bit is 1, the next 2 bytes are a backreference
          var _0x7f4axb = _0x7f4ax4(_0x7f4ax3[_0x7f4ax6]);
          // the higher 4 bits of the byte indicate the length of the backreference in the range of 3-18
          var _0x7f4axc = (_0x7f4axb >> 4) + 3;
          // the lower 4 bits of the byte and the next byte form a 12-bit offset to the start of the backreference in the range of 1-4096
          var _0x7f4axd = (((_0x7f4axb & 15) << 8) | _0x7f4ax4(_0x7f4ax3[_0x7f4ax6 + 1])) + 1;
          _0x7f4ax6 += 2;

          // copy the backreference from the output to the output
          for (var _0x7f4axe = 0; _0x7f4axe < _0x7f4axc; _0x7f4axe++) {
            _0x7f4ax7[_0x7f4ax8] = _0x7f4ax7[_0x7f4ax8++ - _0x7f4axd]
          }
        };
        
        if (
          _0x7f4ax6 >= _0x7f4ax3[_0xedc3[0]] &&
          _0x7f4ax7[_0xedc3[0]] >= _0x7f4ax5
        ) {
          break
        }
      }
    };
    return _0x7f4ax7
  }

Let’s convert it into Scala code:

// I split off the ciphering from the PRNG compared to the JS code for more clarity
object XorShiftPRNG {
    def apply(seed: Int): () => Int = {
        var state0 = 0x88FE88
        var state1 = 0xFE88AA
        var state2 = 0xEE8080
        var state3 = 0xA0A0F0

        def next(): Unit = {
            var next = state0
            next ^= (next << 11) & 0xFFFFFF
            next ^= (next >> 8) & 0xFFFFFF
            state0 = state1
            state1 = state2
            state2 = state3
            state3 ^= (state3 >> 19) & 0xFFFFFF
            state3 ^= next
        }

        for (_ <- 0 until seed) {
            next()
        }

        () => {
            val res = state0 & 0xFF
            next()
            res
        }
    }
}

object DeobfuscatingDecoder {
    def apply(data: Array[Byte]): Array[Byte] = {
        val prng = XorShiftPRNG(data(0) & 0xff)
        /* deobfucate by XORing with PRNG generated OTP */
        def d(i: Int): Int = {
            val d = data(i) & 0xff
            val pad = prng()
            val res = (d ^ pad) & 0xff
            //println(s"At $i: ${d.toHexString} ^ ${pad.toHexString} = ${res.toHexString}")
            res
        }

        val l0 = d(1)
        val l1 = d(2)
        val l2 = d(3)
        println(s"Length bytes: $l0 $l1 $l2")

        val length = (l0 << 16) + (l1 << 8) + l2

        println(s"Decoding ${data.length} bytes to $length bytes")
        val out = new Array[Byte](length)
        var i = 4 // input offset, skip past header
        var j = 0 // output offset
        while (i < data.length && j < length) {
            var c = data(i) & 0xff // read control byte, each of the 8 bits determines whether a byte is a literal or a copy
            c = (c ^ (i & 0xFF)) ^ 0xA3 // deobfuscate, yeah...
            i += 1
            for (k <- 7 to 0 by -1) {
                if ((c & (1 << k)) == 0) { // literal byte
                    out(j) = d(i).toByte // deobfuscate and copy
                    j += 1
                    i += 1
                } else { // back reference
                    val b = d(i) // deobfuscate
                    val count = (b >> 4) + 3 // count = 3 - 18
                    val offset = (((b & 0xF) << 8) | d(i + 1)) + 1 // read another byte for a 12-bit offset = 1 - 2^12
                    for (l <- 0 until count) {
                        out(j) = out(j - offset) // copy from window
                        j += 1
                    }
                    i += 2 // skip ahead in input
                }
                if (i >= data.length && j >= length) {
                    return out
                }
            }
        }
        out
    }
}

But unfortunately, it did not work… Already the length decoding was off. Since I knew the input and output from the debugger, I new that the highest byte of the length had to be zero (output length was < 2^16), so the value of the keystream had to be same as byte 1 (the second byte) of the input to XOR to 0. What could be wrong?

some coding error in the PRNG
some off by one error in the seeding
something else

I checked the code again, and then generated the keystream to figure out at which position the required byte to decrypt the first length byte would be. The position should have been the seed value, i.e. the first byte of the input, but the first occurrence of the needed byte in the key stream was not part of the first 256 bytes which ruled out a simple seed initialization error.

There had to be a another issue. I went back to the debugger to compare the PRNG state with my values. And then I saw it: The initialization code was funnily obfuscated itself:

var _0xc5c0x3 = _0x88FE88;
var _0xc5c0x4 = _0xFE88AA;
var _0xc5c0x5 = _0xEE8080;
var _0xc5c0x6 = _0xA0A0F0;

These are not literal values but symbol names… In the debugger I could figure out the values which were these after all:

var state0 = 8111253   // 0x7bc495
var state1 = 4544506   // 0x4557fa
var state2 = 13986479  // 0xd56aaf
var state3 = 16744512  // 0xff8040

This was enough to fix the PRNG and decode/decompress the data.

I called it success for the day and kept the remaining puzzle of how and where the constants are defined for another day.

With fresh eyes I set out to find the constants the next morning. A simple search in the network tab over all received data did not help. However… stepping through the decode method, there were some clues…

How does _initNav come into scope and play here? And what is a burgerImgDef?

Here is the relevant part from initNav:

try {
    var burgerImgDef = $("#nav img").attr("src").split("/C/").slice(1);
    for (var i = 0; i < 3; i++) eval(µ.base64.decode(burgerImgDef[i]))
} catch (e) {};
$("#nav img").remove();

evil eval on board… Here is the relevant part from the viewer HTML:

<div id="nav"><div class="content"></div><div class="buttons"><div class="burger"><div class="line"></div></div><div class="cross"><div class="line"></div></div><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACkAAAApCAIAAAAnApehAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyJpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMy1jMDExIDY2LjE0NTY2MSwgMjAxMi8wMi8wNi0xNDo1NjoyNyAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENTNiAoV2luZG93cykiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6RTQ2NTkyNUQ3OTNEMTFFOTg0Q0FGMTM0N0REMzhBODYiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6RTQ2NTkyNUU3OTNEMTFFOTg0Q0FGMTM0N0REMzhBODYiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDpFNDY1OTI1Qjc5M0QxMUU5ODRDQUYxMzQ3REQzOEE4NiIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDpFNDY1OTI1Qzc5M0QxMUU5ODRDQUYxMzQ3REQzOEE4NiIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/PrV01G8AAATqSURBVHjarFg5M21NFOWUBEVgKCJSBMjMSkyJFLnhFxhSU2z4A4YcIWKzkiFASqKMgeES8da9+3zLervPS9yvq+6t7j6n9969x7VPbiqVys3N/f7+zsnJiaLoOzMw5yYnNsc/luFTm9gL+P/6+nJPjQJJgVee8bNndsC9RK7YOT4+3tvbu7+/f3h4eHx8fH5+LikpKS0tLSsrq6ysbG9vb2tro/R60BGMBf34+CDvRBmxBMutra3Dw0NVUqJiCgoKwL6np6e5uZk07XrKPuat5FSrNr+8vFxdXd3d3TUSoT5MblWy/Xd2dg4ODtbW1rr3/9Il7+1ujzE/P7+xsRFeLtF+ia7Q19c3Pj4eHrediJLaYzv8+vo6Ojq6vr5O4/F+NmzH+Yf6mk0g+tjY2Nvbm4rLpxHOm8ZIHXoeGRmBjW1pqiYz46FcVdV8gS8fHR0NDQ1dXFw4hf/4GgPs5eVleHj45ubGhYpGEa9IBXCpOtcjVVVVS0tLxcXFqvmIMxNzamoKjBlyEMiZlhpyvqaa1zAz9tfX15OTk05JkS7m5uagavVzTTU0laqB7BlOalG9JSiDvvpjxAXMTK8ODemuouZQJZv3hOztnbW1NXChlD9Rv7KyolriPt/Rl40x1atOwH+TQzMmKCBbcCfPZvDG/f19dxj/p6enOVmMhoYGuoLJCi5QvmW9WOfb29uJGSD7obe3JXjZMs+2kKtd5QnT5y8GvY8mw//BwYExyrPqhChXX6WwjY2NFmlmAs7VvcPcHuZ89STwgolbWlrS4WuWdolaXd1EoVhxdP4nhMuUrpxrBrTcgIldPX3+7u7uXxpT6rr8ygzLPOrnzsymal7J5nh0e3ub1jkWQAEueYWKZbCRFiNNA1pZJmrenoIjlukYs5kT2XbOz8+z8TW4i6uqRv/p6SmuoZiFRVozeZau7i6Gf3CMaygAFyOBXN2ZXwc372AEbQmOaZ1jFzO7ukaIHYDSSCKEaQxFmjkxtHgl7gNbxjovLy93GCE8z/BQNSbyc/hEgQYP4rZpafC4oqKCXq2OrRMKZ1d0YMYJqgDLQTlTAAB1Wv/4dXR0MExV7Y6oogZCoigzHHyglM5MFA4oNo7v1tbW/Pz89/f3MBDPzs6yjDGVwHQGXqhj8b2NPd2bfvE/+rnmHzQPf2Hk3t5eB8QcEv3dYIOnYKS7uzuuobbV1NTU1dWF/sO1a/X19SFC0ohyxU2X6lzMbrA0KlhMn9pACxPiyxAuOpBKeO/6Ei10SgdY/ceyZIbeqb+/PyyFrnFx9VhLi4N7LD98NDAwUFNT86NXfXtiYsIcQTOrBp5CYMWH/6r92inCndEfqXyRA0nT09PV1dWaZ9jguGzKhOpSimtZ7BFozszMqJSRFnb7LyoqAnu0MNp3hQpnhWA4uNaEoBhzMJ6dnQVlLaNplyQ53qmurm55eRkqcv6l2CMstbSr7dDRYER0YjBz6ENR2FJjAhkXFxfhek4ybQZc6+vQrW2i/15YWAC15P4GqDGMHF7l6uoKncTOzo4WEidu+MnE4ti+O4TI84d3KpVKzEcK109OTjY3NwFskfPDak0TYlJYWAhj2feWsK/TYuO/t7gvHw5CYX6YGUCZgHhPmWEfmTBQiO07U2IhcF/PYvqfn58Ofrgs4RA433EIx320CUuiW2L+R4ABAA+f0k7RJTqsAAAAAElFTkSuQmCC/C/d2luZG93Ll8weEEwQTBGMD0weEZGODA0MDt3aW5kb3cuXzB4ODhGRTg4PTB4N0JDNDk1O3dpbmRvdy5fMHhGRTg4QUE9MHg0NTU3RkE7d2luZG93Ll8weEVFODA4MD0weEQ1NkFBRjs=/C/d2luZG93Ll8wWGVkYzM9ZnVuY3Rpb24oXzB4YzVjMHgyKXt2YXIgXzB4YzVjMHgzPV8weDg4RkU4ODt2YXIgXzB4YzVjMHg0PV8weEZFODhBQTt2YXIgXzB4YzVjMHg1PV8weEVFODA4MDt2YXIgXzB4YzVjMHg2PV8weEEwQTBGMDtmb3IodmFyIF8weGM1YzB4Nz0wO18weGM1YzB4NzwgXzB4YzVjMHgyOysrXzB4YzVjMHg3KXtfMHhjNWMweDgoKX07ZnVuY3Rpb24gXzB4YzVjMHg4KCl7dmFyIF8weGM1YzB4OT1fMHhjNWMweDM7XzB4YzVjMHg5Xj0gKF8weGM1YzB4OTw8IDExKSYgMHhGRkZGRkY7XzB4YzVjMHg5Xj0gKF8weGM1YzB4OT4+IDgpJiAweEZGRkZGRjtfMHhjNWMweDM9IF8weGM1YzB4NDtfMHhjNWMweDQ9IF8weGM1YzB4NTtfMHhjNWMweDU9IF8weGM1YzB4NjtfMHhjNWMweDZePSAoXzB4YzVjMHg2Pj4gMTkpJiAweEZGRkZGRjtfMHhjNWMweDZePSBfMHhjNWMweDl9cmV0dXJuIGZ1bmN0aW9uKF8weGM1YzB4YSl7dmFyIF8weGM1YzB4Yj1fMHhjNWMweGFeIChfMHhjNWMweDMmIDB4RkYpO18weGM1YzB4OCgpO3JldHVybiBfMHhjNWMweGJ9fQ==/C/d2luZG93Ll8weGVkYzM9WydceDZDXHg2NVx4NkVceDY3XHg3NFx4NjgnXTt3aW5kb3cuVUludDhBcnJheT1mdW5jdGlvbihfMHg3ZjRheDIpe3ZhciBfMHg3ZjRheDM9IG5ldyBVaW50OEFycmF5KF8weDdmNGF4Mik7dmFyIF8weDdmNGF4ND1fMFhlZGMzKF8weDdmNGF4M1swXSk7dmFyIF8weDdmNGF4NT0oXzB4N2Y0YXg0KF8weDdmNGF4M1sxXSk8PCAxNikrIChfMHg3ZjRheDQoXzB4N2Y0YXgzWzJdKTw8IDgpKyAoXzB4N2Y0YXg0KF8weDdmNGF4M1szXSkpO3ZhciBfMHg3ZjRheDY9NDt2YXIgXzB4N2Y0YXg3PSBuZXcgVWludDhBcnJheShfMHg3ZjRheDUpO3ZhciBfMHg3ZjRheDg9MDt3aGlsZShfMHg3ZjRheDY8IF8weDdmNGF4M1tfMHhlZGMzWzBdXSYmIF8weDdmNGF4ODwgXzB4N2Y0YXg1KXt2YXIgXzB4N2Y0YXg5PV8weDdmNGF4M1tfMHg3ZjRheDZdO18weDdmNGF4OT0gKF8weDdmNGF4OV4gKF8weDdmNGF4NiYgMHhGRikpXiAweEEzOysrXzB4N2Y0YXg2O2Zvcih2YXIgXzB4N2Y0YXhhPTc7XzB4N2Y0YXhhPj0gMDtfMHg3ZjRheGEtLSl7aWYoKF8weDdmNGF4OSYgKDE8PCBfMHg3ZjRheGEpKT09IDApe18weDdmNGF4N1tfMHg3ZjRheDgrK109IF8weDdmNGF4NChfMHg3ZjRheDNbXzB4N2Y0YXg2KytdKX1lbHNlIHt2YXIgXzB4N2Y0YXhiPV8weDdmNGF4NChfMHg3ZjRheDNbXzB4N2Y0YXg2XSk7dmFyIF8weDdmNGF4Yz0oXzB4N2Y0YXhiPj4gNCkrIDM7dmFyIF8weDdmNGF4ZD0oKChfMHg3ZjRheGImIDB4Rik8PCA4KXwgXzB4N2Y0YXg0KF8weDdmNGF4M1tfMHg3ZjRheDYrIDFdKSkrIDE7XzB4N2Y0YXg2Kz0gMjtmb3IodmFyIF8weDdmNGF4ZT0wO18weDdmNGF4ZTwgXzB4N2Y0YXhjO18weDdmNGF4ZSsrKXtfMHg3ZjRheDdbXzB4N2Y0YXg4XT0gXzB4N2Y0YXg3W18weDdmNGF4OCsrIC0gXzB4N2Y0YXhkXX19O2lmKF8weDdmNGF4Nj49IF8weDdmNGF4M1tfMHhlZGMzWzBdXSYmIF8weDdmNGF4N1tfMHhlZGMzWzBdXSA+PSBfMHg3ZjRheDUpe2JyZWFrfX19O3JldHVybiBfMHg3ZjRheDd9" /></div></div>

What’s up with the burger anyway?

Oh, an inline base64 image used for the navigation bar? But after the image data there’s some garbage which after a closer look turns out to be three more base64 sections of data deliminated by "/C/". The first of these three sections decodes as the final missing piece of the puzzle:

window._0xA0A0F0=0xFF8040;window._0x88FE88=0x7BC495;window._0xFE88AA=0x4557FA;window._0xEE8080=0xD56AAF;

Tada, and as you might have guessed, the other two sections contain the decoder source code snippets from above.

Obfuscation summary

Here’s a summary of the obfuscation methods used:

use a simple cipher to obfuscate the data and remove clues as how to decode it (but also spike a lot of interest…)
initialize the decode methods inside of a innocuous function to initialize the navigation bar (eval still sticks out its ugly head here)
hide the PNRG constants in variables whose names look like constants (as silly as obfuscation goes)
replace a call to length with indexing syntax, abc[xyz[0]] (an obfuscation classic that never disappoints)
use weird looking symbol names (e.g. there’s both _0Xedc3 and _0xedc3 but whom can you fool with that?)
store the code in a symbol that is called UInt8Array (compare with Uint8Array from JavaScript, I wondered how redefining Uint8Array would work but only noticed the slight difference while writing this, ok, one extra easter egg, you had some fun, I had some fun)
use a second simple cipher scheme for the control values in the LZ compression part for no good reason (maybe the stream cipher was introduced only later?)
hide obfuscation functions in the HTML as random base64 image data (hiding the methods in a burger was my personal delicious main prize of the hunt…)

Anyways, thanks for the quest, unknown author! It was fun to solve it!

Winning a scroll prize

2023-12-11T00:00:00-06:00

Recently, I have been taking part in the Vesuvius Challenge, an effort and a competition to read the Herculaneum Papyri, ancient scrolls that were carbonized and buried in the eruption of Mount Vesuvius in 79 AD, by virtually unrolling them from CT scans and trying to get ML models to read the carbon-based ink on carbonized papyrus…

I have been fortunate to now have been awarded a prize for my open source contributions with the Vesuvius Segment Browser.

From the official blog:

Another new entrant to the community is Johannes. His Segment Browser tool lets you browse all released segments in 2D, showing the inference from Youssef’s First Letters Prize model. A true community service! Besides this Johannes has been making many contributions to Volume Cartographer.

The project was already running for a while when I discovered it when it made headlines that the first word was found. In a weekend of frenzy lots people joined the chat and tried to see what could be read from the scrolls so far (i.e. only a few words so far but still…).

I only joined the later stages of the competition, which mainly are about scaling and refining the existing techniques to find viable segments of texts and refine ML models to make sure they can detect ink on them.

One main challenge is that the data amounts are huge and wrangling data efficiently is key (each scan of a scroll is about ~2TB of raw data).

So far, I’ve built a few components to help with the effort:

a web app, that
- lists all the publicly released segments
- allows browsing the surface volumes and ink detection results in the browser
- a distributed infrastructure that allows to run ML algorithms on GPU nodes
- initial work on stitching together all the segments as if the scroll was unrolled
a volume data server, that remixes the original voxel data from the scans into more readily available blocks at different resolutions
Vesuvius GUI, a Rust GUI app that can be used to browse the original voxel data for all the released scan volumes interactively, pulling data on demand from my volume data server, so that no initial download is necessary
data loading improvements to the main segmentation application (VC)

You can read more about these applications and the infrastructure at my Vesuvius Sub-Blog.

This has been a truly full stack experience spanning everything from building and tweaking javascript in a web application, running a K8s cluster for hosting, optimizing data access patterns, training and running ML models in the cloud, writing a multi-platform GUI app in Rust, lots of geometry maths and algorithms, to creating fun visualizations with Blender. Obviously, the web server component has a Pekko HTTP backend.

Here’s a rough overview of the architecture:

A fly-through of a small part of the ink of the first letter of the first word (exported to MagicaVoxel and illuminated and rendered with Blender):

Your browser does not support the video tag.

A cutout of the first letter of the first word (rendered with MagicaVoxel):

Your browser does not support the video tag.

Finding the top-p elements as used in Nucleus Sampling

2023-08-29T00:00:00-05:00

Here’s a quick report on some experiments on algorithms to figure out the top-p elements of a distribution.

Why Nucleus Sampling?

The final stage of the current LLMs need to figure out how to generate the next token given a distribution over all tokens. Using the maximum element directly (greedy search) is the simplest solution but does not lead to interesting results. On the other end of the spectrum, sampling from the full distribution is risky because a significant share of the probability mass may consist of a large number of low probability tokens and accidentally selecting an unlikely token may lead the further text generation astray.¹

Many potential improvements have been proposed. A simple, and widely-supported one is top-k sampling, where the k most probable tokens are selected and the probability mass is redistributed over those k tokens. It is easy to see when this can go wrong, e.g. in a common scenario where there’s a single token with a very high probability and all other tokens are similarly unlikely. Or, in the opposite case, of a somewhat flat distribution over many tokens where that flat part is cut short seemingly arbitrarily.

A more sophisticated approach is top-p sampling, also known as Nucleus Sampling. Here, the top tokens are selected until the cumulative probability mass exceeds a threshold p.

While working on llama2.scala, the practical problem of how to figure out the top-p elements out of the 32000 potential tokens (of the commonly used vocabulary), can be selected in a performant way. This is only relevant for small models where sampling time might become significant.

I implemented a few algorithms and compared them.

Naive Sorting

The naive approach is to sort the tokens by their probability and then select the top-p elements keeping track of the cumulative probability. This is the baseline to beat. This is simple to implement using a sorting implementations from your runtime of choice.

Since we need to keep track of the original indices of the elements (i.e. the identity of the token in the vocabulary), sorting can be done on the list of indices and then the original elements can be looked up. Alternatively, for better locality, the probabilities and indices could be kept together (leading to better locality but a more expensive swap operation).

Drawbacks of this approach:

In the common case only few elements will be selected, so sorting the whole array is wasteful.
The sorting algorithm does not take some properties of the distribution into account:
- The distribution is likely to be skewed (expecting an inverse power law distribution)
- The distribution sums up to 1

Filter out some elements and then sort

The first improvement is to filter out some elements that are impossible to be selected and then sort the remaining ones going on with the naive approach.

To figure out a conservative bound, consider this: In the edge case, only one element is selected that has exactly the probability p. In that case, all the remaining n-1 elements will have to share the remaining probability of 1-p. Thus, all elements below (1-p) / (n-1) can be filtered out immediately because they will never be selected. Note, how the denominator will only get smaller when more elements are selected. Also, when the single element has a higher probability than p, the numerator will be bigger. Thus, any other scenario will lead to an even higher value, so the given value is a conservative bound.

Filter out some elements and then iteratively select the next element

Instead of using a full-blown sort, the idea is here that after filtering we will be left with a small enough number of elements that using something like a selection sort becomes feasible (especially in the common case). In this algorithm, we keep track of the cumulative probability and select the next biggest element as long as the cumulative probability is still below p.

The runtime depends on the numbers of elements that will ultimately be selected. In the worst case, all elements will be selected and the list is traversed k times. One way to speed up the iterations, is to take into account, that we know the probability mass that is still available in each iteration. If we find a new maximum element that is bigger than half of the remaining probability mass, we can stop this iteration early and go on with selecting the next element.

Build a histogram of elements

Observing that selecting the top-p elements is equivalent to finding the probability of the last element that is still included in the selection, one idea is to build a histogram of the elements to filter in and out whole buckets after a single iteration over the values.

After the first step, we know a more precise bound on the probability of the last element. We can then only operate on the bucket that contains the last element and find the exact result.

A difficult question in this algorithm is how to set up the buckets. Even with top-p, the last element might already be part of a long, flat tail, so that regardless of exact choice of buckets, the last element might be part of a big bucket. Another consideration has to be how to practically calculate the bucket without spending to much computation time on it. A seemingly obvious choice might be to use a logarithmic scale, but calculating logarithm is expensive. Another possibility would be to extract the exponent of the floating point number and use that as the bucket.

Use a quick select algorithm

The quick select algorithm is a well-known algorithm to find the k-th smallest element in an array. It is a variant of the quick sort algorithm that only recurses into the part of the array that contains the k-th smallest element (and avoids sorting as much as possible).

I found it hard to implement, it’s easy to get the indices slightly wrong and choosing the right pivot is difficult. Also, care needs to be taken that the algorithm actually terminates. Depending on the choice of pivot it can happen that the algorithm does not make progress (WHY? Is that even true or a bug in the implementation?) It seems the way this is dealt with is by choosing the pivot randomly which means that eventually the right pivot will be chosen to make progress (there’s always one that works).

Results

Running bench/jmh:run -f1 -wi 2 TopP* at 7f0774b3 on my machine gives the following results:

             sorting  thrpt    5    122.276 ±   1.498  ops/s
       filterAndSort  thrpt    5   2417.850 ± 135.264  ops/s
   filterAndFastSort  thrpt    5  17899.529 ± 600.464  ops/s
       filterAndScan  thrpt    5  23479.921 ± 895.475  ops/s
quick-find-max-top-p  thrpt    5   5294.213 ± 302.202  ops/s
           histogram  thrpt    5  19456.664 ± 210.921  ops/s
        quick select -- has still bugs

where

sorting = Naive idiomatic Scala sorting
filterAndSort = Filter out some elements and then sort, idiomatic Scala
filterAndFastSort = Filter out some elements and then sort, avoid expensive Scala collections
filterAndScan = Filter out some elements and then iteratively select the next element
quick-find-max-top-p = Tries to use filtering and selecting in each step, iterating over all elements all the time
histogram = Build a histogram of elements, then use previous algorithm on the last bucket

The results are an average over 100 different distributions (generated by running the stories15M model).

Observations

Just doing the filtering does most of the work. Due to its simplicity this is what I contributed to llama2.c.
Doing filtering and sorting in Scala requires a bit of work to avoiding the overhead of Scala collections.
Avoiding a full-blown sort and replacing it with a selection sort gives a little bit of extra performance.
Building a histogram before doing the selection sort steps, does not seem to be quite worth it. In theory, it should further reduce the number of elements to consider for the selection sort, but the additional complexity is not well amortized.
In general, a first filtering step is worth it, maybe just because it reduces the amount of data by such a degree that further iterations over all elements are much cheaper than before.
The results are an average over 100 distributions. In many cases, only a handful of elements is selected (for top-p = 0.9) A heuristic could be developed to detect cases with very few elements (e.g. while filtering count elements above a certain threshold), and use a fast-path for those cases.
More work is needed to get the quick select algorithm right.

Code

You can find the implementations and benchmarks in the top-p-benchmarking branch.

Conclusion

This was a fun experiment doing some algorithm engineering and benchmarking. I hope you enjoyed it as well!

This is a consequence of autoregressive LLMs “just predicting the next word/token”. This is often stated as an seemingly intuitive drawback of how LLMs are trained and how they generate text. For me, it is not all that clear, why predicting just the next token is obviously weak. That said, of course, this main architectural property has consequences. As I see it, a (2023 generation) LLM is a very complex state machine that can solve a (maybe surprisingly) wide range of tasks given enough depths/width/parameters. The state is built up by calculating key/value pairs for each token for each layer (and multiple heads) of transformers. The whole machine is instructed and clocked token-by-token of the sequence. In generation mode, the machine generates its own next instruction auto-regressively. So, the choice of the next token is very important as the machine cannot take back its choice. The only thing it might do, is try to weasel itself out of a bad choice by generating more text. Note, how this kind of backtracking by adding apologies, is only applicable to some language generation situations. For example, in a chat situation, the machine can apologize and use more user input as ways to take the conversation further getting meaningful back on track. However, e.g. when generating code or generating under a stricter regime, this might not be feasible and will have to lead to incoherence, hallucinations, or outright garbage. ↩

Experiments with io_uring

2022-10-11T00:00:00-05:00

This is a quick writeup of my experiments with io_uring.

What is io_uring?

io_uring is a generic, consistent API that allows to (batch) submit Linux kernel IO operations in an asynchronous way ultimately requiring few or zero actual syscalls. It uses a pair of buffer rings for submission and completion data shared between user and kernel space (but that’s just an implementation detail).

To deconstruct the above sentence:

generic: io_uring is generic because it supports a growing list of IO-related syscalls
consistent: The submission and completion mechanism is the same for different kinds of APIs. Submission entries must by default fit into the 64-byte submission entry, completions into a 32-byte completion entry. For some operations the size of a single entry must be doubled which can be configured during creation of the ring.
batch submit: Any number of calls (up to the configured size of the submission ring) can be submitted in one go.
asynchronous: The API itself is asynchronous in that submission of an operation will not always lead to immediate completion. Instead, some operations may only be completed later. By default, submission are directly executed inline (in the same thread) during submission if possible. Otherwise, operations may only be completed later.
few or zero syscalls: Submissions and completions are serialized into shared memory and can (in theory) be accessed by the kernel and the user application concurrently. The default behavior requires a single io_uring_enter call to make the kernel aware of the latest submissions. An optional SQPOLL mode allows the kernel to run a dedicated thread to poll submission from the application in which ultimately no syscalls are required any more (though, then kernel and application need to (busy-) poll the queues).
buffer rings: Application and kernel communicate using shared memory and a submission/completion protocol. In theory, this can be faster than syscall based IO submission.

Why io_uring?

syscalls can become a bottleneck in applications that require lots of IO. Batch submission and the potential polling mode means you can trade off different latency and throughput requirements.

Until io_uring came along, Linux did not have a proper asynchronous interface for doing buffered ¹ file IO. In general, io_uring can hide an internally blocking operation inside its own blocking thread pool just providing an asynchronous interface to an actually blocking operation (similar as e.g. Java NIO asynchronous file operations were implemented). Adding more (internal or not) thread pools somewhat defeat the purpose of asynchronous execution (which should be as light-weight as possible). So, later an asynchronous buffered read mode was proposed and added to the kernel (and io_uring) for file-systems that support it.

Registering buffers and file descriptors

Using user-space buffers and file descriptors in the kernel comes with some overhead. Therefore, it is possible to preregister buffers and descriptors using the io_uring_register syscall.

In many server applications that mutiplex handling of multiple connections with a single thread, the connection is first polled to be ready for reading, then a buffer is allocated, and finally the data is read. With io_uring it is possible to avoid the polling step completely and read data asynchronously directly. However, that means that buffers need to be provided upfront for every waiting read call (which in a web server would be all idle connections). To avoid a per-read (per-connection) read buffer for every outstanding read, with io_uring buffers can also be provided more dynamically using a special operation, IORING_OP_PROVIDE_BUFFERS.

You can then use IOSQE_BUFFER_SELECT for submissions to have io_uring select a buffer from the given set when needed.

A downside of this method of providing buffers is that the kernel may choose buffers arbitrarily. After use, the application has to provide them again. While the original IORING_OP_PROVIDE_BUFFERS call can use a contiguous amount of memory to provide multiple buffers at once, spent buffers will fragment that memory over time. The application will either have to

provide them back to io_uring in a one-by-one fashion after usage (spending one SQE per buffer after usage), or
keep track of which buffers have been redeemed, try to coalesce their memory regions and provide them later with fewer SQEs

In later kernels, it is possible to use IORING_REGISTER_PBUF_RING to create yet another ring just for buffer management, so that providing buffers can happen more efficiently without spending SQEs just for providing single small buffers.

Execution Modes

As shown in the Missing Manuals - io_uring worker pool, an IO operation can go through three different paths when submitted:

Be executed directly inline during the execution of io_uring_enter
Is not ready but can be polled for readiness. In that case, it is automatically polled. When the operation is ready, it is retried and completed.
If it only can be executed in a blocking fashion, it is enqueued to a run queue to the blocking kernel worker thread pool and executed and completed when done.

Inline execution

Simple networking applications (like an echo server or a simple HTTP server) will spent a significant portion of CPU time in the kernel doing TCP work. Whether you use regular syscalls or io_uring, by default much of the TCP processing time will be spent on a user thread (especially on the write side).²

The test application is a simple HTTP server which fakes request processing:

Has a simple (single-threaded) event processing loop which blocks on completion events from io_uring (setting the min_complete argument of the io_uring_enter syscall to 1).
Accepts any amount of incoming connections.
Tries to read an incoming HTTP request (but does not validate it at all) and sends out a response (it will never have to wait on network writes).

Here is a flame graph of simple HTTP server processing using io_uring (click for interactive version):

The blueish and green frames on the left side are the application. Since this application does not do any actual processing, it is only a very small share of total processing time.

The big red humps are frames executing in the kernel. All of those frames are on top of the io_uring_enter call, i.e. executing while doing the submission call. The biggest share of that call is the big hump in the middle which is the actual TCP sendmsg implementation. You will see the same frames on a blocking server as well.

The small part on the left seems to be the overhead of actual submitting events using io_uring.

The small hump on the right is part of the TCP recvmsg implementation.

Async submission

There’s an alternative mode that can be selected using the IOSQE_ASYNC flag per SQE where the task is not executed inline at all but queued for io_uring’s internal worker queue.

One problem when used together with IOSQE_BUFFER_SELECT shows up in the flamegraph (click through for interactive flame graph):

io_uring will spawn a massive amount of worker threads and all these threads contend on acquiring a pre-provided buffer which will slow everything down.

I filed an issue about that and it was confirmed that using this usage pattern in particular but more generally IOSQE_ASYNC is not recommended. In any case, you should probably stay away from using IOSQE_BUFFER_SELECT when you expect the requests to end up on the internal worker pool.

Inline vs. Async

It may be surprising that using an asynchronous interface may not automatically lead to asynchronous execution or more parallelism. For application servers that are single-threaded, there is the option to force spreading the internal kernel load to the internal worker queue (if you have cores to spare) by using IOSQE_ASYNC. However, as seen, this comes with extra overhead and potential extra scalability issues.

For CPU-bound workloads the inline execution behavior seems useful as it will provide the fastest execution path. For low-latency applications the time spent in the kernel might be an unwanted interruption but for lowest-latency you will anyway need spare CPU resources to avoid any extra delays. In that case, distributing work over multiple worker threads each owning a separate ring (and maybe polling) should give the best solution and best control over how resources are spent.

Usage from the JVM

To support io_uring from the JVM, only a few native calls need to be implemented:

Access to the io_uring_setup, io_uring_enter, and io_uring_register syscalls (e.g. by supporting all syscalls by linking to libc’s syscall symbol)
Memory mapping the io_uring data structures into memory
A way to manipulate the native (non-JVM-heap) memory of the rings and IO buffers.

The JVM’s standard way of interfacing with the native world is using JNI. We only need to provide simple wrappers for three syscalls with simple interfaces so this is simple to do. Alternatively, using a simple JNA declaration like this also works:

import com.sun.jna.Library;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;

public interface LibC extends Library {
  int SYSCALL_IO_URING_SETUP = 425;
  int SYSCALL_IO_URING_ENTER = 426;
  int SYSCALL_IO_URING_REGISTER = 427;
  int syscall(int number, Object... args);
  Pointer mmap(Pointer addr, long length, int prot, int flags, int fd, long offset);
}

Using JNA will introduce some overhead over JNI, but it seems relatively fine compared to all the other work.

After mapping a few of io_uring’s internal data structures, creating a uring can look like this:

val params = new IoUringParams // not in the fast-path, use JNA mapped data structured
val uringFd = libC.syscall(LibC.SYSCALL_IO_URING_SETUP, sizeOfRing, params)

val length = params.sq_off.array + params.sq_entries * 4

val sqRingPointer = libC.mmap(new Pointer(0), length, LibC.PROT_READ | LibC.PROT_WRITE, LibC.MAP_SHARED | LibC.MAP_POPULATE, uringFd, LibC.IORING_OFF_SQ_RING)
val cqRingPointer = libC.mmap(new Pointer(0), params.cq_off.cqes + params.cq_entries * 16 /* sizeOf(IoUringCqe) */ , LibC.PROT_READ | LibC.PROT_WRITE, LibC.MAP_SHARED | LibC.MAP_POPULATE, uringFd, LibC.IORING_OFF_CQ_RING)
val sqePointer = libC.mmap(new Pointer(0), params.sq_entries * 64 /* sizeOf(IoUringSqe) */ , LibC.PROT_READ | LibC.PROT_WRITE, LibC.MAP_SHARED | LibC.MAP_POPULATE, uringFd, LibC.IORING_OFF_SQES)

// FIXME: check all result values

// convert JNA pointers to DirectByteBuffer
val sqRingPointerByteBuffer = sqRingPointer.getByteBuffer(0, length)
val cqRingPointerByteBuffer = cqRingPointer.getByteBuffer(0, params.cq_off.cqes + params.cq_entries * 16)
val sqes = sqePointer.getByteBuffer(0, 64 * params.sq_entries)

From here on, all datastructures are manipulated using the DirectByteBuffers whose operations (hopefully) are intrinsified and therefore cheap. JNA’s getByteBuffer is expensive, though, so make sure to create buffers spanning the whole interesting region and reuse them later. For example, to access some parameters, you can use code like this:

def sqHead() = sqRingPointerByteBuffer.getInt(params.sq_off.head)
def sqTail() = sqRingPointerByteBuffer.getInt(params.sq_off.tail)
val sqMask = sqRingPointerByteBuffer.getInt(params.sq_off.ring_mask)

def cqHead() = cqRingPointerByteBuffer.getInt(params.cq_off.head)
def cqTail() = cqRingPointerByteBuffer.getInt(params.cq_off.tail)
val cqMask = cqRingPointerByteBuffer.getInt(params.cq_off.ring_mask)

Here is one way of representing an operation with a case class (which is not quite optimal because of the extra allocation):

case class ReadOp(
    flags:             Byte,
    fd:                Int,
    destinationOffset: Long,
    targetBufferAddr:  Long,
    targetBufferSize:  Int,
    userData:          Long,
    bufferGroup:       Short = 0 // FIXME: buffer selection should be abstracted
) extends Op(IORING_OP_READ) {
  override def prepareSQE(buffer: ByteBuffer): Unit = {
    buffer.put(opId)
    buffer.put(flags)
    buffer.putShort(0 /* ioprio */ )
    buffer.putInt(fd)
    buffer.putLong(destinationOffset)
    buffer.putLong(targetBufferAddr)
    buffer.putInt(targetBufferSize)
    buffer.putInt(0)
    buffer.putLong(userData)
    buffer.putShort(bufferGroup)
    buffer.putShort(0)
    buffer.putInt(0)
    buffer.putLong(0)
    buffer.putLong(0)
  }
}

An operation can be submitted like this:

def submitGlobal(op: Op): Unit = {
  // FIXME: check against overflow
  val curTail = sqTail()
  val index = curTail & sqMask
  sqes.clear()
  sqes.position(64 * index)
  op.prepareSQE(sqes)
  sqRingPointerByteBuffer.putInt(params.sq_off.array + 4 * index, index)
  // FIXME: write barrier needed?
  sqRingPointerByteBuffer.putInt(params.sq_off.tail, curTail + 1)
  // FIXME: write barrier needed?
}

That is, we first figure out the tail of the ring to write to, then prepare an operation using the above method, put the index into the SQE array, and finally increase the tail index. (I’m not sure if the barriers are needed in a simple application, but if multiple threads access the ring, you definitely should add the barriers, e.g. using Unsafe).

Finally, submitting all queued SQEs to the kernel is done by invoking io_uring_enter (the only necessary syscall in the hot path):

val toSubmit = sqTail() - sqHead()
libC.syscall(LibC.SYSCALL_IO_URING_ENTER, uringFd, toSubmit, 1, IORING_ENTER_GETEVENTS, 0)

After the call returns, you can check for completions:

var cqIdx = cqHead()
val last = cqTail()

while (cqIdx < last) {
  val idx = cqIdx & cqMask
  // accessing cqe data via JNA is more expensive but also possible:
  // val cqe = new IoUringCqe(cqRingPointer.share(params.cq_off.cqes + idx * 16))
  // debug(cqe.toString)
  val off = params.cq_off.cqes + idx * 16
  val userData = cqRingPointerByteBuffer.getLong(off)
  val result = cqRingPointerByteBuffer.getInt(off + 8)
  val flags = cqRingPointerByteBuffer.getInt(off + 12)
  handleCompletion(userData, result, flags)
  cqIdx += 1
}
// give back all handled entries (FIXME: memory barriers)
cqRingPointerByteBuffer.putInt(params.cq_off.head, last)

Here we loop through all new CQEs, extract their data (again using the previously created DirectByteBuffers) and defer handling to a completion method.

A final problem is creating or accessing file descriptors from the JVM. It is possible to extract file descriptors from Java RandomAccessFile and Socket using Java reflection from their internals.

In newer kernel versions, more syscalls are available and it might be possible to open files and sockets using io_uring.

The full source code can be found at https://github.com/jrudolph/scala-io-uring. It contains more of infrastructure to provide a more idiomatic API, however, everything should be considered experimental (checking return values, anyone?).

Much more

There’s, of course, a lot more to io_uring not covered here:

More flags like IOSQE_IO_LINK or IOSQE_IO_DRAIN
Full set of supported ops
Cancelling requests
…

References

Buffered means going through the page cache. I.e. when calling read the file is mapped into memory and then copied into user buffers. ↩
If you ever wondered where the 5 times difference between the “Json” and the “Plaintext” tests in the Techempower Benchmark comes from, that’s the reason. The json example uses regular HTTP requests where only a single request can be ongoing per TCP connection. In that case you always require at least one TCP/IP packet per request and response. The plaintext benchmark uses HTTP pipelining, which ultimately allows to transmit multiple HTTP requests and responses in a single TCP/IP packet which decreases the in-kernel CPU overhead of TCP massively. ↩

Building a blinky sign for Twäng (part 1)

2022-06-17T00:00:00-05:00

Twäng released its award-winning debut album POP! in September 2021. To celebrate the release I built a small piece of eye-catching hardware.

The sign was shown in a local news item (German):

Ingredients

Background Music

Before continuing, please head over to one of the online places where you can listen to the music and put it on in the background ;)

or watch a recent live performance (without my participation).

Hardware

WS2812B RGB LEDs on a small PCB (e.g. from ebay or aliexpress)
Raspberry Pi Pico
Another Raspberry Pi Pico (using picoprobe for debugging)
0.6mm copper wire
connection wires
picture frame
bubble wrap

Tools

Soldering Iron
Side Cutters
Third-Hand Tool

Building Log

First version assembly

After one of the rehearsals for our small release concerts I had the idea of creating a small blinking sign.

Should I order a LED matrix to program or could I build something simpler myself? Looking around on youtube for inspiration about how to get creative with WS2812B, I found someone handling a string of these LEDs each mounted on a small PCB. The LEDs were hold soldered together with a piece of wire.

That sounded promising, so I ordered a pack of 50 WS2812B LEDs from ebay on a whim. They arrived two days later and came as a single PCB panel with v-cuts to break them off easily.

I got to work and soldered together the first few pixels. At the input side, I soldered a few wires with pin connectors to try them out with the Raspberry Pi Pico.

The pico-examples repository contains examples for driving WS2812B. I compiled it without changes and uploaded it to the Pico using UF2. For the experiments, the data in pin was connected to the GPIO that was configured in the example. The +5V connection was connected to the 3V3 for now.¹

Each WS2812B RGB LED, in fact, contains three LEDs of red, green, and blue color, together with a small IC for driving these LEDs.

They only use a single digital data line that needs to be chained through all the LEDs. Since there’s no physical clock line, the data rate is predefined and needs to be adhered to accurately to ensure proper transmission. Internally, each LED has a shift register which receives the 24 initial bits (the color value ordered as GRB), and propagates all the remaining bits. This happens until a RESET code is received (low voltage level for at least 50µs). Then all LEDs latch the received values into their display register and use PWM to set the perceived brightness per color led.

Connecting to the Pico and running the pico example without changes worked well and showed the predefined animation patterns of the example on the first three pixels.

This looked promising so I came up with a design about how to wire more of those LEDs to form the letters “pop!”.

The copper wires are quite stiff, so you can form arbitrary shapes and the resulting form will be stiff enough for careful use.

Producing the rounded forms like the letter ‘o’ or ‘p’ was challenging. You need to bend three small pieces of copper wire into a curved form making sure that those three wires which are only a few millimeters apart do not touch. After a bit of trial and error, the best technique was to start with the inner of those three wires and solder those first.

In my first attempts I tried to precut the wire into the right size but I could not handle the soldering of these small parts with my fumbly fingers. A better way was to keep the wire on its reel while soldering the first connection, and only cutting it right before soldering the other end. The third hand tool was indispensable, as you can imagine!

The first test on the full first letter revealed that the color scheme looked weird.

It turned out that the pico WS2812B example, expects a different LED type by default that uses a fourth color channel for a white LED. The boolean rgbw parameter passed to ws2812_program_init controls how many bits are sent per LED, so it was an easy fix to run with the RGB variant of the pixels.

I continued with my quest and soldered another letter.

As you can see, some bridges are needed, first, to connect the inner rail of the power supply to the inner rail of the next letter (I chose to run all letters in the same direction, so the inner rail was +5V). Also, the Dout of the last pixel of one letter needs to be connected to the Din of the first letter of the next letter.

After three letters done, I started doing the first little bit of programming just to give each letter its own color:

Finally, I had also finished the exclamation mark and real programming could begin.

Simple colors firmware

First, constants were defined for the colors per letter.

// COLOR_BRG converts R,G,B value to a 24bit BRG value as expected by WS2812B
const uint32_t p_1_color = COLOR_BRG(240, 5, 2);
const uint32_t p_1_o_color = COLOR_BRG(20, 0, 40);
const uint32_t o_color = COLOR_BRG(12, 35, 120);
const uint32_t o_p2_color = COLOR_BRG(28, 70, 6);
const uint32_t p_2_color = COLOR_BRG(200, 40, 0);
const uint32_t p_2_excl_color = COLOR_BRG(90, 13, 1);
const uint32_t excl_color = COLOR_BRG(150, 12, 8);

Determining the color values was not as straight-forward as expected. Turning on all LEDs with the same brightness value, did not gave a neutral white but was heavily shifted to blue. Two potential explanations come to mind:

colors are just not calibrated well
a voltage supply smaller than 5V might affect different colors in different ways (red LEDs have a smaller voltage drop than blue ones and there’s no information about how WS2812B implement the constant current source for the LEDs and how it is affected by input voltage)

In the end, I found that a balanced white had an RGB value of 255, 130, 80.

Pixel are chained one after each other and when sending data to them. You send color values in the same order as the pixels are wired. Here is a schematic of how the pixels are wired (this schema belongs to a latter revision while code and pictures above to the first, sorry for that):

The association between pixel IDs and letters was done using a simple bitmap:

const int num_leds = 32;
const int p_1_leds[] = {1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};
const int o_leds[] =   {0,0,0,0,0,0,0,0,0,1,0,0,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0};
const int o_p_2_leds[]={0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}; // only want that middle one
const int p_2_leds[] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,1,1,1,1,1,0,1,1,1,1,0,0,0};
const int excl_leds[]= {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,0,1,1,1};

By design, some pixels are shared between two letter and get their own colors.

Transmitting the colors works like this:

void show_all(uint t) {
    for (int i = 0; i < num_leds; ++i) {
        if (p_1_leds[i] && o_leds[i])
            put_pixel(p_1_o_color);
        else if (p_1_leds[i])
            put_pixel(p_1_color);
        else if (o_p_2_leds[i])
            put_pixel(o_p2_color);
        else if (o_leds[i])
            put_pixel(o_color);
        else if (p_2_leds[i] && excl_leds[i])
            put_pixel(p_2_excl_color);
        else if (p_2_leds[i])
            put_pixel(p_2_color);
        else if (excl_leds[i])
            put_pixel(excl_color);
    }
}

That is, we cycle over all the LEDs by id, figure out which letter the pixel belongs to, and transmit the desired color.

Here’s the result:

As a finishing touch I put it into a frame and added a sheet of paper and some bubble wrap for diffusion and style.

Improved hardware version

User testing revealed that the letter shaping was not easily readable. This is no surprise as the initial version only had ~ 30 pixels which doesn’t allow for much details. I had tried to mimic the album art by fusing adjacent letters. The result was nicely colorful but still not an obvious representation of the album cover.

Only a few days before the concert, I decided to redo the hardware slightly. Last minute changes to hardware are much more frightening than doing software fixups.

On one hand, there’s only one piece of hardware available which already works, and if it gets broken in the process, there’s no way of going back other than fixing whatever breaks. On the other hand, the software already depends on the topology of the hardware and also needs to be changed accordingly.

After all I overcame my fears and attempted some cosmetic surgery…

I added a few pixels here and there and made the separation between the letters and the exclamation mark clear.

The backside:

Leading to the final result:

Animations

With the hardware done, I could focus on animations. There are three kinds of animations:

Static colors (as seen above) with a slight shimmer by randomly fluctuating pixel colors
Fireworks with rockets and flashing letters
A snake wandering through the letters

Here’s a shot from the fireworks animation:

More about the animations in a follow-up blog post.

In the meantime, you can find the code at https://github.com/jrudolph/twaeng-pop-sign.

For full brightness you can run WS2812B at +5V, but then you need to make sure that the signal level of the data line get to 0.7 times 5V which is 3.5V and therefore slightly too high to be driven by a 3V3 microcontroller. With my pixels both using the VBUS pin (nominally at 5V USB voltage) and VSYS pin (VBUS - the voltage drop of a Zener Diode of ~0.3V, i.e. at ~ 4.7V) of the Pico both also work flawlessly. ↩

Maximum compression ratio with Zip/Gzip/Deflate

2022-01-03T00:00:00-06:00

Quite a while ago an issue was reported for Akka HTTP where incoming data, compressed using Content-Type: gzip, was uncompressed without applying any limits, in the worst case leading to an OutOfMemoryException and a shutdown of the server. This can be used for a classical denial of server attack, a “zip bomb”, where compression is used to amplify the amount of data an attacker has to send to bring down a machine.¹

To reproduce the attack I gzip-compressed files of various sizes that contained a repitition of a single character for maximum compression. It turned out that there seems to be a hard compression ratio limit with gzip. This is somewhat surprising as, in general, with run length encoding you could represent such a repetition with log(n) compressed bytes.

Here’s a table of various compressed file sizes (all bytes 0):

Original Size	Gzip Size	Ratio
1000	29	34.5 : 1
1000000	1003	997 : 1
1000000000	970501	1030 : 1
10000000000	9704731	1030 : 1

So, the ratio seems to converge to approximately 1030 : 1. Why is that the case? We’ll try to find out in the rest of this post.

Let’s have a look at how these 1000000 bytes look in compressed form:

$ dd if=/dev/zero bs=1000 count=1000 status=none|gzip|hexdump -C 
00000000  1f 8b 08 00 19 85 d9 5b  00 03 ed c1 01 01 00 00  |.......[........|
00000010  00 82 20 ff af 6e 48 40  01 00 00 00 00 00 00 00  |.. ..nH@........|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000003e0  00 af 06 9e cb 79 12 40  42 0f 00                 |.....y.@B..|
000003eb

Weird, most of the file is just zero bytes (which hexdump abbreviates with *). Let’s have a look at the specifications for gzip, RFC 1951 (DEFLATE) and RFC 1952 (GZIP) to make sense of it.

When I do these manual decoding exercises, I usually copy the hex data of a file into an editor and then snip off bytes from the beginning and annotate them with their meaning, keeping the tail of undecoded data at the bottom. If you want to follow along you can copy the data from the above hexdump into an editor and follow the steps.

Gzip is just a small wrapper around deflate compressed data. The Gzip header is at least 10 bytes:

         +---+---+---+---+---+---+---+---+---+---+
         |ID1|ID2|CM |FLG|     MTIME     |XFL|OS | (more-->)
         +---+---+---+---+---+---+---+---+---+---+

For our file:
         
ID1 + ID2: 1f 8b 
CM:        08 -> DEFLATE compressed
FLG:       00 -> no extra data
MTIME:     19 85 d9 5b -> time of compression
XFL:       00 -> no extra flags
OS:        03 -> Unix OS

As no flags (FLG) are specified the rest of the file is DEFLATE compressed data (if you use gzip <filename>, instead of piping the input data in, the gzip header also contains the original filename). At the end of a gzip file there’s a trailer containing a CRC32 over the uncompressed data (here: 9e cb 79 12) and 4 bytes of the size of the uncompressed data modulo 2^32 (here: 40 42 0f 00 => 0xf4240 = 1000000).

So now we we’ve have dealt with all the Gzip related data, we can decode the DEFLATE compressed data which starts at offset 0x0a:

ed c1 01 01 00 00 00 82 20 ff af 6e 48 40 01 00
# Lots of 00 bytes
00 af 06

DEFLATE compressed data needs to be interpreted as a bitstream. The order in which to read the bits from the bytes and how to interpret them is the single most confusing thing about the DEFLATE specification. RFC 1951 says the bits of a byte are ordered like this:

         +--------+
         |76543210|
         +--------+

That is, the least-significant bit of each byte needs to be read first. Let’s convert our data byte-by-byte into a bit stream

ed       c1       01       01       00       00       00       82

11101101 11000001 00000001 00000001 00000000 00000000 00000000 10000010

20       ff       af       6e       48       40       01       00
00100000 11111111 10101111 01101110 01001000 01000000 00000001 00000000

Lots of 00 bytes

af       06
10101111 00000110

Now we have to read the bits in each byte from right to left to get the final bitstream:

10110111 10000011 10000000 10000000 00000000 00000000 00000000 01000001

00000100 11111111 11110101 01110110 00010010 00000010 10000000 00000000

Lots of 0 bits

11110101 01100000

We can now just concatenate the bits and start reading DEFLATE blocks.

1011011110000011100000001000000000000000000000000000000001000001
0000010011111111111101010111011000010010000000101000000000000000

Lots of 0 bits

1111010101100000

The first bit of a DEFLATE block defines if it is the last block and the next two bits specify which compression type was used for that block. DEFLATE defines 3 compression types:

00 = uncompressed
01 = predefined Huffman tables
10 = custom Huffman tables
11 = reserved

For our data:

1  - BFINAL => Last block
01 - BTYPE = 10 => custom Huffman tables

As you can see, here’s another oddity: when we read a fixed number of bits (like for the compression type), we need to reverse them again before interpreting them. In our case, the compression uses custom Huffman tables. That makes sense because our original file contains very specific data (only zero bytes) and also it’s long enough to warrant the extra overhead of specifying a set of custom Huffman tables. As you will see most of the following non-zero data will be the definition of the custom Huffman tables.

DEFLATE is a combination of dictionary compression (LZ77) and Huffman coding. LZ77 represents data as a sequence of literal bytes and references to data seen before (as a tuple of offset into previous data and length of data to copy). The LZ77 sequences are then efficiently encoded into a binary stream by using Huffman encoding. DEFLATE uses two Huffman tables. The first table is the LITLEN table. A LITLEN symbol determines either if a literal byte should be decoded (and which) or if this is a reference of a particular length. If it is a reference we will use the second Huffman table for distances to determine the offset of the reference.

The standard defines a set of fixed tables to be used for blocks where including the tables themselves would be too much overhead (BTYPE = 01). In our case, however, we use custom Huffman tables and so the bitstream starts with a definition of the custom tables. These tables themselves are defined using another type of Huffman table, the code length codes.

Section 3.2.7 of RFC 1951 defines how the custom Huffman tables should be constructed.

It starts with some constants to define how big the tables will be. Let’s continue reading:

= HLIT: number of LITLEN codes defined - 257, here 11101 = 29 + 257 = 286
= HDIST: number of distance codes - 1, here 00001 = 1 + 1 = 2
= HCLEN: number of code length codes - 4, here 1110 = 14 + 4 = 18

So, our custom definition contains 286 LITLEN codes and 2 distance codes. They are represented using the 18 code length codes of the code length Huffman table used to transmit the other Huffman tables. This kind of recursive usage of Huffman tables to specify Huffman tables is a bit mind-boggling first but will hopefully clear up later.

The code length symbols are defined like this:

               0 - 15: Represent code lengths of 0 - 15
                   16: Copy the previous code length 3 - 6 times.
                       The next 2 bits indicate repeat length
                             (0 = 3, ... , 3 = 6)
                          Example:  Codes 8, 16 (+2 bits 11),
                                    16 (+2 bits 10) will expand to
                                    12 code lengths of 8 (1 + 6 + 5)
                   17: Repeat a code length of 0 for 3 - 10 times.
                       (3 bits of length)
                   18: Repeat a code length of 0 for 11 - 138 times
                       (7 bits of length)

Next we read the 18 code lengths for the code length table by reading 18 times 3 fixed bits of lengths. The lengths read correspond to the symbols in the order 16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15. Let’s read those:

= 0 (symbol 16)
= 0 (17)
= 2 (18)
= 0 (0)
= 0 (8)
= 0 (7)

= 0 (9)
= 0 (6)
= 0 (10)
= 0 (5)
= 0 (11)
= 0 (4)

= 0 (12)
= 2 (3)
= 0 (13)
= 2 (2)
= 0 (14)
= 2 (1)

We don’t read anything for the last symbol 15 as we should only read lengths of 18 symbols.

So, our code length Huffman table only contains the four symbols 1, 2, 3, and 18 all with bit lengths 2. The specification contains rules how to build a Huffman table from the bit lengths. The result for the code length table is this:

Symbol	Bit length	Huffman code	Extra bits to read	Interpretation
1	2	00	-	Code length “1”
2	2	01	-	Code length “2”
3	2	10	-	Code length “3”
18	2	11	7	Repeat code length “0”, 11 - 138 times

Next we use the code length Huffman table to read 286 LITLEN code lengths and 2 DIST code lengths. Note, that the symbol 18 (and others not in use here) above will read multiple ones so you need to keep track of how many code lengths you have read overall.

         => Symbol 2 => 1 x code length 2 (1 read so far)
+ 1111111 => Symbol 18 + 7 extra bits => 138 x code length 0 (139 read so far)
+ 0101011 => Symbol 18 + 7 extra bits => 11 + 1101010 = 117 x code length 0 (256 read so far)
         => Symbol 3 => 1 x code length 3 (257 codes read so far)
+ 0000100 => Symbol 18 + 7 extra bits => 11 + 0010000 = 27 x code length 0 (284 read so far)
         => Symbol 3 => 1 x code length 3 (285 read so far)
         => Symbol 1 => 1 x code length 1 (286 read so far, i.e. all LITLEN codes)
         => Symbol 1 => 1 x code length 1 (287 read so far)
         => Symbol 1 => 1 x code length 1 (288 read so far)

This translates into these Huffman tables (we read only the bit lengths but I already added the actual codes based on the algorithm from the specification and omitted unused symbols).

LITLEN Huffman table:

Symbol	Bit length	Huffman code	Extra bits to read	Interpretation
0	2	10	-	Literal Character ‘\0’
256	3	110	-	End Of Block
284	3	111	5	Copy 227 + extra bytes
285	1	0	-	Copy 258 bytes

DIST Huffman table:

Symbol	Bit length	Huffman code	Extra bits to read	Interpretation
0	1	0	-	Distance 1
1	1	1	-	Distance 2

If you compare these tables with the predefined ones in the specification you can already see how little codes are needed for our special case file and with how few bits those can be encoded.

Now that we have finally created the Huffman tables for decompression, we can start decoding the actual data.

If you followed along, this is the remaining bit stream, we have already almost reached the long stretch of 0 bits:

101000000000000000

Lots of 0 bits (7736 0 bits to be precise)

1111010101100000

Let’s decode:

=> LITLEN Symbol 0   => Literal byte '\0'
=> LITLEN Symbol 0   => Literal byte '\0'
=> LITLEN Symbol 285 => Copy 258 bytes from ...
=> DIST Symbol 0     => ... distance 1

=> The same, LITLEN Symbol 285, DIST Symbol 0 => Copy 258 bytes from distance 1
=> ...
=> ...
=> ...
=> ...
=> ...

times the same

+ 10101          => LITLEN Symbol 284 => Copy 227 + 10101 = 248 bytes from ...
                  => DIST Symbol 0 => ... distance 1
                => LITLEN Symbol 256 => End Of Block
               => Padding to align to bytes

So the sequence we decoded is this:

2 times the literal byte ‘\0’
3875 times copy 258 bytes from distance 1 (long stretch of 0 bits in compressed data)
copy 248 bytes from distance 1
end of block

In total we have decoded 2 + 3875 * 258 + 248 = 1000000 bytes. Mission accomplished!

Now, back to our original question of what limits Gzip compression. As you can see here (and also when looking into the specification), we can achieve the highest compression when we can copy 258 bytes. In the best case, like with these very specific Huffman tables, we can achieve that with one bit for the LITLEN code and one bit for the distance code. The compression ratio is therefore 258 bytes compressed into 2 bits = 8 * 258 bits / 2 bits = 1032 / 1. In the beginning we measured about 1030 : 1. The difference between that and 1032 : 1 is the overhead of having to include the custom Huffman tables.

A follow-up question is why there is the maximum of copying only 258 bytes in the specification. There would have been room for adding greater amounts. I once read an explanation on StackOverflow (cannot find the link right now), that the length value (range 3 to 258) should be able to fit into 8 bits for performance reasons on 8 bit machines (deflate is really old by now). A great thing is that you can ask question about deflate / gzip on StackOverflow and it’s likely that Mark Adler (who still maintains zlib) will answer. Like for the question “gzip compression ratio for zeros” which answers the same question as I have here.

We thought that we would be safe against such attacks because of our streaming infrastructure, which in theory is correct. However, as so often with bugs like this, unfortunate circumstances led to collecting the stream without limiting the amount of collected data. ↩

Base64 vs. Gzip

2018-10-28T00:00:00-05:00

For personal projects I mostly try to avoid using databases. Instead, I try to use what I call the “JSON repository” pattern. I create a file structure that contains my data in JSON format. I often use the directory structure as an index by placing the JSON data at well-known places for random access.

For example, I wrote a small web crawler a while ago. To make the process incremental between different runs I created this file structure for caching intermediate results until the next run:

data/                          # Repository root
  head/                        # Results of HEAD requests
    http/                      # protocol
      doc.akka.io/             # host
         docs/                 # URI path element 0
           ...                 # ...
             /abc.html.json.gz # result of HEAD request in gzipped JSON format
  response/                    # Results of full GET requests
    # same structure as above
             /abc.html.json.gz # result of GET request in gzipped JSON format
  outlinks/                    # Results of outgoing link analysis
    # same structure as above
             /abc.html.outlinks-v1.json.gz # result of outgoing link analysis in gzipped JSON format

I like this kind of “file system database” because you can browse it with regular tools on the command line. But this shouldn’t be the topic of this post. What I want to talk about in this one is a sub-aspect of storing data in this form.

The good thing about JSON is that it’s human-readable. However, it is less efficient to write, store, and read than a binary format. Also, when you need to embed binary data inside of JSON files you need to encode it in UTF-8 strings which is often done using Base64 encoding. To mitigate at least part of the storage inefficiency of JSON files, I always run gzip on the resulting json. That’s still easy to read on the command line using tools like zless while improving storage space.

Using this scheme, I wondered, how efficient is the encoding of binary data with Base64 first, and then with gzip? More precisely, Base64 wastes space by only using 6 bits out of the 8 bits of a byte. How much of that can be recovered by gzipping Base64-encoded data afterwards? Also, for compressible binary data, how much of the original compression rate can be achieved if data is first Base64-encoded? So, I ran a few experiments.

We have two test bodies:

The UTF8 plain text version of Alice In Wonderland from Project Gutenberg (although, there’s no good reason to base64 encode it in json, it serves as a good example of compressible data)
1,000,000 random bytes

Let’s start with plain Base64 and plain gzip encoding:

Encoding	Random Bytes	of unencoded	Alice Bytes	of unencoded
raw	1000000	x 1	74703	x 1
base64	1333336	x 1.333	99604	x 1.333
gzip	1000173	x 1.0001	27842	x 0.373

The results are unsurprising. Base64 has a fixed overhead of 33%. As Base64 only uses 6 bits per byte, we would expect an overhead of ca. 8 / 6 - 1 = 1 / 3 = 33%.

Gzip worked well on the text file but slightly increased the size of the random data (which is guaranteed to happen in average on random data).

Now, let’s try base64 encoding first and then apply gzip:

Encoding	Random Bytes	of unencoded	Alice Bytes	of unencoded
base64 + gzip	1009020	x 1.009	38064	x 0.510

So, on random, uncompressible bytes, gzip mitigates the effect of base64 encoding, so that only 0.9% overhead are left.

On compressible data, we need less space than the text would have taken in raw, but we lose quite a bit of compression compared to raw compression. The probable reason is that both of gzip’s compression algorithms, LZ77 and Huffman, process input data one byte at a time. Base64’s 8-bit to 6-bit encoding, however, splits up every 3 bytes into 4, so that LZ77 cannot find repetitions as easily because they have to be aligned to those 4-byte base64 blocks.

The Huffman coding is probably also affected for two reasons: First, every character from the Base64 encoding character set will represent 6 bits that were chopped from one or two original bytes. That means that the symbol histogram will not represent the original data bytes symbol histogram perfectly. Second, as the input data takes more space, LZ77 length and distance encoding might require more space (more “extra bits” in the encoding).

To illustrate this effect, I ran a few examples on even simpler input data. Let’s compare gzip compression of raw data versus base64 encoding of the simple string consisting of 1024 times the character e.

If it is compressed raw it can be represented with LZ77 as

Literal character e
At distance 1 copy 1024 characters (somewhat unintuitive that you can reference data that is not yet decoded but it actually works like this)

This can be very efficiently encoded.

Even, if you would disable LZ77 in Gzip, Huffman coding alone could also represent this string efficiently because it would only have to represent the two symbols:

Literal character e
End of block

Even if that would be a quite degenerate case of Huffman encoding, you could still encode it in 1025 bits (e.g. 1024 bits of 1 to represent e followed by a single 0 to represent the end of the block, plus the Huffman table).

Now look at the Base64 encoding of the string of 1024 ‘e’s:

ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVl
ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZQo=

It’s quite obvious that it still has some structure but as you can see it exhibits the predicted 4-byte pattern. It can be encoded with LZ77 as

Literal string "ZWVl" (base64 representation of "eee")
At distance 4 copy 1020/3*4 characters (base64 representation of 1020 'e's)
Literal string "ZQo=" (the trailer which is necessary because 1024 doesn’t divide by 3, so that the base64 encoding has a bit of padding at the end)

This can still be quite efficiently compressed but has more complexity than the plain encoding without base64.

However, this is still just a special case because the repetitions of the character ‘e’ are aligned to 3-byte boundaries which translate into periods of 4 characters with base64.

You can see this when looking at the base64 encoding of "123456712345671234567123456712345671234567", i.e. 6 times the string "1234567":

MTIzNDU2NzEyMzQ1NjcxMjM0NTY3MTIzNDU2NzEyMzQ1NjcxMjM0NTY3

If you look closely you can see that the result is still periodic but only after half of the output. This is because the 7 characters don’t divide by 3 so the next time the string "1234567" aligns at a 3-byte boundary is after 3 periods, that is after 21 input characters or 28 output characters.

It’s pretty clear that this will negatively affect gzip compression afterwards.

Finally, here’s a table showing the difference between compressing the same amount of bytes either of a single character of the string "1234567":

Encoding	4200 * "e"	of unencoded	600 * "1234567"	of unencoded
raw	4200	x 1	4200	x 1
base64	5600	x 1.333	5600	x 1.333
gzip	40	x 0.0095	48	x 0.0114
base64+gzip	47	x 0.0112	81	x 0.0193

In the last line you can see that adding base64 encoding has only little effect on aligned, compressible data but a much bigger effect when the repetitions in the original data don’t align to the base64 3-byte boundaries.

Summary

Putting binary data as base64 into json files and Gzipping afterwards works well because gzip mitigates the overhead of base64 almost completely.
If the binary data is compressible, you’ll lose some potential for compression by using base64 first. (If you want to offset this, you can gzip the binary data first before applying base64.)
Alternatively, you could put binary data not in the json file but next to it on the file system.
Or, use another encoding better suited to json, e.g. see this stackoverflow question

Why isn’t it possible to read gzip files from the middle?

2016-02-04T00:00:00-06:00

For instance, consider the case of a backup gone wrong and the first 100k of your backup.tar.gz went missing for some reason. Why isn’t is possible to start reading a gzip file (or many other compressed formats) from some position in the middle of the file?

I want to give two answers. The first one is a general one. Compressed formats like gzip are often stream formats. The stream encoder (the sender) and the decoder (the receiver) of the data are both stateful. At the beginning of the file they start from a defined initial state and with every byte (or sometimes even bit) processed both advance their state in lockstep.

Why is that a good idea? For one thing, it is efficient because the state doesn’t need to be transferred explicitly between encoder and decoder. Streaming formats usually also limit the amount of state to keep at any time, so you can decode it without having to keep all of the decoded data as it would be for a random access data format.

The downside of a streaming format is that if you start to read from an arbitrary location inside the stream you don’t know with which state to initialize your decoder, so the decoder doesn’t know how to decode the stream. So, this is the first answer.

The second answer is specific for a gzip / deflate compressed stream. gzip is just a thin wrapper around the deflate algorithm which is specified in RFC 1951. Remember, the deflate algorithm is a combination of the LZ77 algorithm and Huffman coding. LZ77 tries to replace a sequence of bytes with a back reference into the past of the stream. I recommend Julia Evan’s cool visualization about how “The Raven” compresses with LZ77.

The output of the LZ77 algorithm is a stream of symbols where each symbol either represents a literal uncompressed byte or a back reference of a certain length into the last 32kb data of the input. These symbols are then efficiently encoded into bits using Huffman coding. The idea of Huffman coding is to compress a byte¹ with less than 8 bits. Sounds magic? Yes, usually you need 8 bits to transfer one of 256 symbols. But only if these symbols are uniformly distributed. If there’s any other distribution, the distribution of symbols already contains some information that you can transfer beforehand (as the Huffman tree in this case) and then transfer the actual bytes with less than 8 bits per byte in average by using short codes for frequent symbols and long codes for infrequent ones.

Here is how gzip/deflate encoders and decoders work in theory²:

Encoder: find back references (LZ77) -> Huffman encode
Decoder: Huffman decode -> resolve back references

A bit we read on a deflate compressed stream is first passed into the Huffman decoder state machine until that reaches a leaf in the Huffman tree in which case an LZ77 symbol is passed on to the LZ77 decoder. The symbol either represents either a literal byte which can be directly emitted or a back reference into the compression window.

Therefore, the decompressor has two stateful components: a state in the Huffman decoder and the contents of the decompression window to resolve LZ77 back references against. Both these states are missing if we start to read in the middle of a deflate stream. We cannot know where we are in Huffman state machine (though, it seems possible to make good guesses). And even if we could Huffman-decode the stream we could only decode literal bytes but wouldn’t be able to resolve back references because we missed the past bytes of the stream.

With the question answered, I wondered if it is inevitable that stream formats cannot be read from the middle. Of course, it is not :)

Many stream formats feature special synchronization byte sequences (which are otherwise unlikely to appear) to mark positions from where the decoder can restart decoding from a defined state. Adding synchronization is a trade-off, however. Streams are often used as an efficient way to represent data. This is in conflict with adding extra data. The synchronization markers add redundant data but, more importantly, each synchronization point also has to reset the internal state. So, every extra knowledge a decoder may have acquired while reading from the beginning must be discarded for the benefit of a decoder that starts from later in the stream.

Popular stream formats that employ synchronization markers are live audio or video streams for which a consumer might want to join the stream later. Here, it makes perfectly sense to add synchronization points, e.g. every second of the data for the following reasons:

a late consumer probably doesn’t care about skipping to the next synchronization point
(video) compression uses correlation between subsequent frames to shrink data, state that is useful for compression will probably only be found in frames of the same scene, so it is no loss to discard any state at a cut between scenes (which offers itself as an synchronization point)

This is different for gzip compressed data for which no general statement about data correlation can be made. Nevertheless, there is some half-official way to synchronize inside of a gzip/deflate stream. This document about “ZLIB flush modes” describes various ways the zlib library can insert synchronization points into a deflate stream. To be actually able to start reading from any synchronization point, aside from emitting a synchronization sequence, you must also reset both LZ77 (i.e. discard the compression window) and the Huffman encoding. A “Sync Flush” emits an empty uncompressed block whose header contains the well-known byte sequence 00 00 FF FF and then starts a new compressed block. This resets the Huffman encoding. However, it doesn’t reset the LZ77 encoding so you may still encounter back references to data that appeared before the synchronization point. For this reason a Sync Flush is not used as an actual synchronization point to allow starting to read from this position but instead it is used to fill the output buffer with padding bits so that full bytes can emitted. To reset the LZ77 compression window, as well, you need to emit a “Full Flush” which looks exactly the same but which guarantees that the encoder won’t emit back references that point to data from before the synchronization point.

Here are some links to read more about gzip/deflate compression.

RFC 1951 “DEFLATE Compressed Data Format Specification”: https://tools.ietf.org/html/rfc1951
Julia Evan’s blog posts about gzip: http://jvns.ca/blog/2013/10/24/day-16-gzip-plus-poetry-equals-awesome/ and http://jvns.ca/blog/2015/02/22/how-gzip-uses-huffman-coding/
A very detailed post about “Dissecting the GZIP format”: http://www.infinitepartitions.com/art001.html
Zlib Flush Modes: http://www.bolet.org/~pornin/deflate-flush.html
Reconstructing corrupt DEFLATEd files: http://www.dfrws.org/2011/proceedings/19-351.pdf
A presentation about synchronization of Huffman codes: http://www.mimuw.edu.pl/~fmurlak/na_katedre/Marek.ppt
Information about “Efficient Huffman Decoding”: http://www.commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art007

In fact, in deflate it’s not bytes (256 symbols) that are Huffman-encoded but either a byte or a back reference. These are represented by one of 285 symbols for which you would usually need 9 bits (with several values unused). See the spec for the details. ↩
In practice, it is a little bit more complex as the complete stream can be divided in separate blocks on the Huffman level to allow different Huffman trees for different parts of the file which makes sense as symbol distribution may change over the course of a stream of data. ↩

Losing track (part 2)

2016-01-26T00:00:00-06:00

In part 1, I introduced the problem of debugging cycling code using stack traces. We have seen that we already lose debugging information if tail call optimization kicks in.

Let’s see what happens if the control flow isn’t linear but there are concurrently running threads of execution.

Let’s run concurrently

Here’s an example that uses reciprocally dependent lazy vals that are evaluated concurrently:

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

object A {
  lazy val a: Int = {
    Thread.sleep(100)
    B.b
  }
}
object B {
  lazy val b: Int = {
    Thread.sleep(100)
    A.a
  }
}

def main(args: Array[String]): Unit = {
  val fa = Future(A.a)
  val fb = Future(B.b)

  val sum: Future[Int] =
    for {
      a ← fa
      b ← fb
    } yield a + b

  println("Waiting for completion....")
  Thread.sleep(10000)
  sum.value match {
    case Some(x) ⇒ println(s"Got result: $x")
    case None    ⇒ println("Future was never completed.")
  }
}

Again, we’ve got the same bug: A.a depends on the value of B.b and vice versa. In the simplest case, we would just access A.a and see what happens. Mildly surprising, it would again fail with a StackOverflowException because the initializers of A.a and B.b would call themselves recursively (huh, recursive evaluation of a val?). But this is not the case I’m interested here.

Instead, we are interested in what happens if A.a and B.b are both evaluated concurrently from different threads. To make that happen we use Future.apply to run the initial evaluation of those lazy vals from different threads. I also added those sleep times to make it more likely that the lazy val initializers are actually executed concurrently.

The future result of those computations are combined in a for-comprehension to a final future value. We sleep in the main thread for 10 seconds and see if we got a result in the meantime.

Here’s what happens:

[info] Running example.CyclicLazyValsComplex 
Waiting for completion....
Future was never completed.

It turns out that we do not get any result (but also no CPU time is spent). What’s going on? Once again, jstack to the rescue, we find that indeed all threads are blocked:

> jps
7480 CyclicLazyValsComplex

> jstack 7480

"ForkJoinPool-1-worker-9"
   java.lang.Thread.State: BLOCKED (on object monitor)
	at example.CyclicLazyValsComplex$A$.a$lzycompute(3complexLazyVals.scala:10)
	- waiting to lock <0x00000007ade6c800> (a example.CyclicLazyValsComplex$A$)
	at example.CyclicLazyValsComplex$A$.a(3complexLazyVals.scala:10)
	at example.CyclicLazyValsComplex$B$.b$lzycompute(3complexLazyVals.scala:18)
	- locked <0x00000007adfb6e80> (a example.CyclicLazyValsComplex$B$)
	at example.CyclicLazyValsComplex$B$.b(3complexLazyVals.scala:16)
	[...]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"ForkJoinPool-1-worker-13"
   java.lang.Thread.State: BLOCKED (on object monitor)
	at example.CyclicLazyValsComplex$B$.b$lzycompute(3complexLazyVals.scala:16)
	- waiting to lock <0x00000007adfb6e80> (a example.CyclicLazyValsComplex$B$)
	at example.CyclicLazyValsComplex$B$.b(3complexLazyVals.scala:16)
	at example.CyclicLazyValsComplex$A$.a$lzycompute(3complexLazyVals.scala:12)
	- locked <0x00000007ade6c800> (a example.CyclicLazyValsComplex$A$)
	at example.CyclicLazyValsComplex$A$.a(3complexLazyVals.scala:10)
	[...]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"main" prio=10 tid=0x00007f6ebc00a000 nid=0x1d3d waiting on condition [0x00007f6ec3e6a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at example.CyclicLazyValsComplex$.main(3complexLazyVals.scala:33)
	at example.CyclicLazyValsComplex.main(3complexLazyVals.scala)

[...]

Found one Java-level deadlock:
=============================
"ForkJoinPool-1-worker-9":
  waiting to lock monitor 0x00007f6e74006218 (object 0x00000007ade6c800, a example.CyclicLazyValsComplex$A$),
  which is held by "ForkJoinPool-1-worker-13"
"ForkJoinPool-1-worker-13":
  waiting to lock monitor 0x00007f6e60003c88 (object 0x00000007adfb6e80, a example.CyclicLazyValsComplex$B$),
  which is held by "ForkJoinPool-1-worker-9"

Java stack information for the threads listed above:
===================================================
"ForkJoinPool-1-worker-9":
	at example.CyclicLazyValsComplex$A$.a$lzycompute(3complexLazyVals.scala:10)
	- waiting to lock <0x00000007ade6c800> (a example.CyclicLazyValsComplex$A$)
	at example.CyclicLazyValsComplex$A$.a(3complexLazyVals.scala:10)
	at example.CyclicLazyValsComplex$B$.b$lzycompute(3complexLazyVals.scala:18)
	- locked <0x00000007adfb6e80> (a example.CyclicLazyValsComplex$B$)
	at example.CyclicLazyValsComplex$B$.b(3complexLazyVals.scala:16)
	[...]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"ForkJoinPool-1-worker-13":
	at example.CyclicLazyValsComplex$B$.b$lzycompute(3complexLazyVals.scala:16)
	- waiting to lock <0x00000007adfb6e80> (a example.CyclicLazyValsComplex$B$)
	at example.CyclicLazyValsComplex$B$.b(3complexLazyVals.scala:16)
	at example.CyclicLazyValsComplex$A$.a$lzycompute(3complexLazyVals.scala:12)
	- locked <0x00000007ade6c800> (a example.CyclicLazyValsComplex$A$)
	at example.CyclicLazyValsComplex$A$.a(3complexLazyVals.scala:10)
    [...]
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Found 1 deadlock.

Very helpfully, in this case, the program ran itself into a stuck state which is again observable from stack traces and for which the JVM even provides a nice deadlock analysis that actually points to the problem: we have two lazy evaluated values that reciprocally need to access each other to initialize themselves. Luckily, access was guarded by locks which again is part of the stack trace so it was easy to figure out what the problem was.

It may not be clear why this is a realistic example. A lazy val is a mechanism often used to implement a cache. A computation is delayed until its result is first needed. A subsequent access waits for the completion of the initial computation, if necessary, and then returns the result.

This is a useful pattern that is used in some form or the other quite often. In more practical cases than this one the cycle may be much less obvious but still you may run into a similar deadlock. Unfortunately, these kinds of situations are more likely to happen under load when threads are preempted to do some other work which introduces delays similar to the artificial delays introduces by the sleep(100) lines.

As seen here, caching based on lazy vals relies on blocking calls to wait for the result of the computation. Nowadays it’s more fashionable (and more reasonable, of course) to use an asynchronous API that allows to subscribe to the result of a lazy computation. This allows to be notified about the completion of a computation without blocking a thread.

A Future-based cache

spray-caching contains a clever caching solution (introduced to spray by Mathias a few years ago) that doesn’t store the results of a computation but which stores the eventual result of the computation, i.e. a Future of the result. This is ingenious because it allows to put computations into the cache that are not yet complete. It sidesteps the so-called thundering herd problem in a very elegant way. The thundering herd problem describes what happens when lots of similar requests arrive at a cold cache (e.g. after a server restart). A simplistic cache may not have noted that it already has started computing the value for a key and may spawn redundant calculations for the same key until finally one of those redundant calculations puts a value in the cache. In the worst case this may lead to another break down of a server because of too much load.

The Future based solution on the other hand can immediately put a result Future into the cache to which subsequent requests for the same key can subscribe to. This prevents redundant computations and also keeps the cache unconcerned about subscribing to an ongoing computation because this will be naturally handled by the Future.

Here’s a super simple implementation of such a cache:

object Cache {
  def apply(f: Int ⇒ Future[Int])(implicit ec: ExecutionContext): Int ⇒ Future[Int] = {
    var cache = Map.empty[Int, Future[Int]]
    val lock = new Object

    key ⇒
      lock.synchronized {
        cache.get(key) match {
          // cache hit
          case Some(result) ⇒ result
          // cache miss
          case None ⇒
            val result = Promise[Int]()
            Future { // run `f` concurrently
              result.completeWith(f(key))
            }
            cache += key -> result.future
            result.future
        }
      }
  }
}

The cache constructor takes a function Int => Future[Int] and returns a function of the same type. The resulting function runs the underlying function only once per key and puts the resulting Future result into a Map. Calls with the same key will instantly return the cached Future.

A more practical implementation would use something more scalable than a simple synchronized block. spray-caching uses ConcurrentLinkedHashMap, for instance. It would also need to enforce some limits on cache size, etc. This simple implementation, though, should serve its demonstrating purpose here.

Let’s put it into use:

object CyclicFutureCache {
  import scala.concurrent.ExecutionContext.Implicits.global

  val f: Int ⇒ Future[Int] = Cache(calculateF)
  // the underlying heavy computation
  def calculateF(i: Int): Future[Int] =
    i match {
      case 42 ⇒ f(i - 19)
      case 23 ⇒ f(i * 2 - 4)
      case x  ⇒ Future.successful(123)
    }

  def main(args: Array[String]): Unit = {
    val sum: Future[Int] =
      for {
        aRes ← f(23)
        bRes ← f(42)
      } yield aRes + bRes

    println("Waiting for completion....")

    sum.onComplete { res ⇒
      println(s"Program was completed with $res")
      sys.exit()
    }

    // sleep while waiting for completion
    while (true) Thread.sleep(1)
  }
}

We have an expensive recursive function calculateF (similar to the one shown in part 1). For efficiency reasons we want to cache results of this function so we put the Cache around it which gives us a new cached function f. This is also what we call for the recursive invocations. Again we introduced the cyclic dependency bug similar to the ones seen in the previous examples.

Here’s the output of the program:

Waiting for completion....

Once again, the futures are never completed and the program is stuck. Let’s see what jstack says:

> jps
7918 CyclicFutures

> jstack 7918

"ForkJoinPool-1-worker-13" #10 daemon prio=5 os_prio=0 tid=0x00007f60642a3000 nid=0x1ee9 waiting on condition [0x00007f601b2d3000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076db30338> (a scala.concurrent.forkjoin.ForkJoinPool)
	at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[...]

"main" #1 prio=5 os_prio=0 tid=0x00007f606400a000 nid=0x1ebb waiting on condition [0x00007f606a856000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at example.CyclicFutureCache$.main(4cyclicFutures.scala:54)
	at example.CyclicFutureCache.main(4cyclicFutures.scala)

[...]

This time we are completely left in the dark. Two threads are just waiting and there’s no sign on the stack that a computation is not yet complete. The program is stuck but we cannot see why.

What’s going on? Remember that a stack trace is a representation of the future of a thread or, in other words, the stack represents the continuation of a thread. The point of an asynchronous API like the Future based one from Scala is, however, to detach the continuation, the “what to do after a computation is complete”, from the thread which starts or runs the original computation.

Therefore, no thread or its stack contain any traces of the continuation. Instead, the continuation is explicitly managed on the heap. When you use onComplete (or any of the other Future combinators) you pass a function value to be invoked when the original computation is complete. This function value is put into a list of callbacks which the Promise keeps that implements the Future.

In our case, the cache created two Promises, one for the result of f(23) and one for the result of f(42). It then called promise.completeWith(...) which makes the promise subscribe to the Future given as an argument. The call sequence was roughly like this:

val f23Promise = Promise[Int]()
val f42Promise = Promise[Int]()
f23Promise.completeWith(f42Promise.future)
f42Promise.completeWith(f23Promise.future)

So, both Promises subscribed to the respective other but neither could make any progress.

In summary, when working with Future, we neither keep track of the history of the computation nor can we easily find out what the future of the computation is because we cannot inspect the list of still-waiting callbacks. Also, a Future consumer usually doesn’t track what it is currently waiting on. Instead, it just registers a callback at the producing Future. So, there’s a reference from the producer to the consumer but not the other way round.

I don’t have any ready made solutions at this point. What is needed is better support to trace Future call chains and execution. At some point I started the better-future-exceptions experiment to see what it would take to improve Future tracing infrastructure. akka-stream faced similar problems and @drewhk implemented a graph output showing who waits for what in the streaming scenario. It would be cool if we had something like this for Futures as well. If you have ideas please tell them in the comments!

You can get the examples from this project at github. They also include an example I have not discussed here which replicates a lazy calculation which actors but there is not much insight gained compared to the future example.

Thanks for reading.

Losing track (part 1)

2016-01-18T00:00:00-06:00

Code has bugs. If you are lucky you spend only half of your life as a coder on debugging code. Most kind of bugs are easy to find and simple to fix. However, there are kinds of bugs that are hidden deeply under a stack of indirection. In that case, you try to track it down using the hopefully helpful tools of your run time environment.

Failure modes may differ wildly depending on the way a program is written and which kind of execution engine is used to run it. There’s a particular kind of bug I want to present here which I’ve encountered in several forms. The thing I want to examine is how “debuggability” can differ quite a lot given the way you wrote the program.

An obvious problem

Here’s the problem in its simplest form:

def a: Int = b
def b: Int = a

a

This is what you get if you run this program:

java.lang.StackOverflowError
  at example.Simple$.a(0simple.scala:4)
  at example.Simple$.b(0simple.scala:5)
  at example.Simple$.a(0simple.scala:4)
  at example.Simple$.b(0simple.scala:5)
  at example.Simple$.a(0simple.scala:4)
  at example.Simple$.b(0simple.scala:5)
  at example.Simple$.a(0simple.scala:4)
  at example.Simple$.b(0simple.scala:5)
  at example.Simple$.a(0simple.scala:4)
  at example.Simple$.b(0simple.scala:5)
  at example.Simple$.a(0simple.scala:4)

The problem is clear. What we have here is a simple calculation where two methods call each other leading to a cycle. I could have shown the even simpler form of def a: Int = a but I guess it’s easier to extrapolate from two methods to the realistic case where the cycle will span lots of methods so the problem will not be as obvious any more.

Tail calls

In the simple case above, the program terminates with a StackOverflowException. But let’s talk about tail-calls. A tail-call is a method call which is the last thing done before a method returns itself to its caller. This means it is a method call whose return value will also be the return value of the method that made the method call.

For example, given

def f(x: Int): Int = x + 12
def g(): Int = f(5)

def main(): Unit = {
  g()
}

the call to f will be a tail-call in g because nothing is left to be done in g when f returns.

def f(x: Int): Int = x + 12
def h(): Int = 12 * f(5)

It is not a tail-call in h, however, because the result of h still needs to be multiplied by 12 after f returns.

If a call is a tail-call some languages and run times can do tail-call elimination to remove the intermediate stack frame when tail-calling a method. This means that you call the new method but leave no trace of the method that was executing before.

Without tail-call elimination, while executing f the stack would look like this:

main -> g -> f

With tail-call elimination, the stack goes directly from

main -> g

main -> f

so that it will afterwards directly return from f to main.

What’s the advantage of doing tail-call optimization? Each stack frame needs memory on the stack so removing a frame will save stack space. This is especially important to prevent that recursive calls will eat up all stack space for deeply nested data structures. Its main reason is to avoid a StackOverflowException.

But think back to our original bug. How did it manifest itself? By throwing a StackOverflowException. So, tail-call elimination and bug-finding may work against each other.

In the original case above we got a helpful StackOverflowException because the JVM doesn’t do tail-calls (yet?). However, the Scala compiler is clever enough to optimize recursive tail-calls. Let’s see what happens if we rewrite the program so that the Scala compiler can apply its tail-call optimization.

A first complication

We now make the example slightly more complex by 1) bringing it in a form that allows tail-call elimination to happen and 2) which shows that the problem gets harder if the cycle is dynamic in nature.

def f(argument: Int): Int =
  argument match {
    case 42 ⇒ f(argument - 19)
    case 23 ⇒ f(argument * 2 - 4)
    case x  ⇒ x + 5
  }

def main(args: Array[String]): Unit = {
  f(0) // everything's fine
  f(42)
}

What’s the result of this program?

In fact, this program runs only for its side-effect which is … heating your home indefinitely. It will not terminate but will spin one of your CPU cores with 100%. To find out what’s going on, let’s try my number one tool of choice for debugging and profiling : jstack, the command line tool bundled with the JDK which allows getting a stack trace for another Java process.

I usually run jstack at least twice as a kind of poor man’s sampling profiler. This way I get 2 samples of what the program is doing (which is already kind of a statistic) and it often is already enough to see what the program is (probably) spending most of its time with.

Let’s do this:

> jps
17452 TailCalls

> jstack 17452
[...]
"main"
   java.lang.Thread.State: RUNNABLE
	at example.TailCalls$.f(1tailcalls.scala:7)
	at example.TailCalls$.main(1tailcalls.scala:13)
	at example.TailCalls.main(1tailcalls.scala)
[...]

> jstack 17452
[...]
"main"
   java.lang.Thread.State: RUNNABLE
	at example.TailCalls$.f(1tailcalls.scala:6)
	at example.TailCalls$.main(1tailcalls.scala:13)
	at example.TailCalls.main(1tailcalls.scala)
[...]

Here, we got two almost equal stacktraces. In both cases, the main thread runs function f. Can you spot the difference?

We were lucky here, because when taking the stack traces the program was executing two different lines (7 vs. 6) in the two cases. The lines directly pinpoint the issue because they correspond to the lines containing case 42 => and case 23 => respectively.

Again, we have a cycle that is only triggered if the method is called with the right argument. Also, we don’t get a StackOverflowException anymore because in this case scalac’s tail-call elimination kicked in.

Fortunately, finding the mistake here was still simple because the cycle could be observed in the stack trace by looking at the differing line numbers. In more practical cases it may be much harder because the stack trace may not directly point to different branches of a conditional expression (if there is one at all) or the hot lines (which are most likely to turn up in the stack trace) are in greater distance to the actual problem.

To summarize, the obstacles for debugging this issue are these:

the program does not fail but it never terminates
the stack trace does not show the argument to the f call
the stack trace does not show the complete “trace” from the beginning because tail-call elimination removed the stack frames when doing the recursive call

This shows that stack “traces” may not always be helpful. In this example, we have already seen that tail-call elimination removed parts of the trace.

I put “trace” in quotes because “trace” seems to imply that the program leaves a track from the program start to the current program position.

However, it is not the purpose of the stack to keep the history of a thread. Actually, it’s quite the opposite: the primary purpose of the stack is to keep all the necessary state to continue the execution of the thread. That is, it contains return addresses and local variable values which are needed when execution returns from currently open calls. The stack contains the future of a thread.

In simple cases, the history and future of a thread luckily agree, so we get useful debugging information. In other cases, however, information that is not needed anymore for continuing the program, can and will be thrown away and that is exactly what tail-call elimination is doing here.

One reason, that the JVM still has no tail-calls is the related concern that meaningful stack traces are lost when tail calls are eliminated naively. As we have seen this is certainly a valid practical argument.

Tail call elimination is not the only reason stack traces may be not useful. You can always convert recursive algorithms into iterative ones and sometimes it is required to do that for algorithms to complete in bounded space. In those cases, a stack trace does not contain the necessary information to debug a code cycle issue because important state like iteration variable values is either already lost (i.e. overwritten in memory) or not easily accessible because the stack trace contains only static information and not dynamic information like local variable values.

But this is only the start. In part 2 of this post, I’ll try to make things more interesting by introducing more complicated control flow (a popular complication these days is running things concurrently…).

I’ll leave you spinning here for a while and will return shortly with the second part of this story.

Thanks for reading!

(If you dare so you may try to register yourself in my private discourse instance and leave a comment. Everything is quite fresh so please expect road bumps and occasional downtime.)

Continue reading with part 2.