Matt Coneybeare


Fixing Rflickr and Xmlrpc to Handle Bignums

| Comments

Flickr has become one of the more popular online photo storage sites and consequently has billions of picture on there. I use my Flickr account, in conjunction with a modded rflickr gem, to display all my photos on my website. It uses the Flickr API to get the pictures by ID, the downloads and caches the pics. Lets talk about these Flickr photos and their identifying ID’s. Every time a new pic is added, it is assigned a numerical ID number, and that ID number has been counting up since day one. In fact, there are so many pics, and the IDs are so big that the ID number of the pics can no longer fit into an Integer container, but now need a Bignum or Bigint to hold the value. Let me explain… for example, one of my more recent pics:
Me and Di - Spinaker Way
… has an id number of 2153550206. This number in hex is 0x805C917E but more importantly, in binary it is:
Decimal: 2153550206
Hex:     0x805C917E
Binary:  1000 0000 0101 1100 1001 0001 0111 1110
“So what” you ask? Let me talk a little bit about signed integers. All modern computers use a binary counting method called two’s compliment to create a signed integer, as in, the integer has a negative sign or a positive sign. In this system, the first bit determines the sign, 0 for positive and 1 for negative. This limits the value of the most positive int to be a 0, followed by 31 1’s, A.K.A. 2^31-1. This maximum value is 2147483647. Uh-oh… The number id of my picture is bigger than that… how can I represent this value using an int? The answer (in ruby) is using a Bignum which is simply twice as long (64 bits) so that there is plenty of room to spare. For example, my number above would become:
Decimal: 2153550206
Hex:     0x00000000 0x805C917E
Binary:  0000 0000 0000 0000 0000 0000 0000 0000 1000 0000 0101 1100 1001 0001 0111 1110
… Giving us 2^63-1 positive numbers instead of 2^31-1. Luckily for us, most of ruby detects and switches to the Bignum seamlessly. That cannot be said for all of the ruby libraries though. I recently came across a problem in which this large picture ID was throwing up an exception because it could not be contained in a 32-bit integer container:
Bignum is too big! Must be signed 32-bit integer!
Tracing the error stack, I was able to look into where the error was generated… /opt/local/lib/ruby/1.8/xmlrpc/create.rb
when Bignum
  if Config::ENABLE_BIGINT
    @writer.tag("i4", param.to_s)
    if param >= -(2**31) and param <= (2**31-1)
      @writer.tag("i4", param.to_s)
      raise "Bignum is too big! Must be signed 32-bit integer!"
It seems that for some reason, the xmlrpc library doesn’t switch over by itself… I can either 1) ask Flickr to change how they represent a photo, 2) rewrite a bunch of code to handle the photos in a different way, or 3) I must set this “Config::ENABLE_BIGINT” value to be true. Lucky for us, option 3 is a one-line fix over in /opt/local/lib/ruby/1.8/xmlrpc/config.rb, line 26:
Change this:
… To this:
Now save and restart your server and all should work. This should work fine if you are on your own server or running locally, but if you are on Dreamhost or another shared server, you must first install your own copy of ruby to make the change to as you do not have permission to change the servers copy of ruby. If you use this method, your file to change will be located somewhere different than /opt/local/lib/ and you will have to do a:
find . -name config.rb
… from your home directory to find the file and make the change. After following all the instructions, my file was located at ~/run/lib/ruby/1.8/xmlrpc/config.rb Good Luck!
EDIT: Just for fun… the last pic whose ID fits into an integer (ID = 2147483647) is:
And the first Bigint pic (ID = 2147483648) is strangely not found…

The requested Flickr photo was not found.
Photo ID: 2147483648


My name is Matt Coneybeare, I design and develop for iOS (iPhone, iPad and iPod Touch), Mac OS X and the Web out of New York. In 2008 I started a software company called Urban Apps that has made some pretty popular apps such as Ambiance and Hourly News. My current Stack Overflow reputation is about 27k.

I was a Rockstar a decade ago, but then went back to school and collected a Bachelor's Degree in Computer Science from U.C. Berkeley. Now I am settled down with my beautiful wife Di and our two doggies Hamachi and Foxy. While coding, I walk several miles/day on my Treadmill Desk. When not at my desk, I love exploring New York City as a Yelp Elite, or training for marathons.

Contact information

Matt Coneybeare