Category: ruby

IP to ASN lookup, using Redis

Hello all, Just recently, I’ve had the need to map an IP address to an AS number in a very fast manner for use in another project. I started by looking for methods of obtaining the data neccessary for creating the IP to ASN map. I do not personally own a router than is running full BGP tables and I didn’t want to abuse one of the looking glass routers for obvious reasons. On my search, I came across two well formatted indexes. One mapping masks to ASNs and the other mapping ASNs to owner names. [Netmask to AS number][1] [AS number to AS owner][2] This is great, because I could create to arrays or hashes, including the keys/values. Now since I am new to Ruby, I started out by searching for the suitable libraries to scrape the data from the two above maps. I found a nice [ruby-curl][3] library on GitHub, which had some nice features, that would allow me to Curl directly into an array using regex with it’s scan() method… how nice! After creating my initial script and pulling the data, for some reason the Curl wouldn’t put all the data into the array… it would stop between 80.x.x.x and 90.x.x.x every time. After messing around with it a bit, I couldn’t get it to work how I wanted, so I switched to using [curb][4], which didn’t have the nice scan() method, but I was able to play with the data just how I wanted in no time. I was able to get the script working by putting all the key/values into a Ruby hash and it was easy to pull the values out how I wanted but this would mean scraping the data from the above tables every time I ran the script. I started searching and was thinking about MongoDB but after some further testing, I decided to go with Redis due to the simple model of data I was playing with (just a bunch of key/values). Now the fun part begins; storing the data for persistence. The [Redis Ruby client][5] is awesome and very easy to use. You import the values with redis.set and call them with redis.get… nice and simple.
irb(main):007:0> redis.set("1.1.1.0/24", "15169")
=> "OK"
irb(main):008:0> redis.set("15169", "Google Inc.")
=> "OK"
irb(main):009:0> puts "Hey! Don't touch ASN #{redis.get("1.1.1.0/24")}! That's owned by #{redis.get("15169")}"
Hey! Don't touch ASN 15169! That's owned by Google Inc.
Since this script needs to support ANY IPv4 address but the table consists of only the netmask used by the specific AS,  which may not be in the same /24 or another mask easily identified by simple string matching, makes this a little bit more difficult. What we need to do, is convert the IP address into a value that can easily be compared to any other and if there is no match, check all masks until there is a match (Example: for a /24… test /24, /23, /22, /21…….until there is a match, which eventually you will run into). This is accomplished by converting the IP to long. Ruby does not have this built in function, so after researching and testing, I was able to come up with:
ip = 1.2.3.4
ipAry = ip.split(/\./)
long = ipAry[3].to_i | ipAry[2].to_i << 8 | ipAry[1].to_i << 16 | ipAry[0].to_i << 24
For 1.2.3.4, this should effectively return 16909060. The script would then take that long IP and search redis. If no match is found, another common network boundry is tested. It will continue searching until redis.get(?) isn’t nil. For example: 16909060 = no match = redis.get(16909060) = nil 16909056 = MATCH = redis.get(16909056) = 15169 Now that the logic part is finished, now time to make the script usable and make some final housekeeping changes. The ip2long code was put into a method, along with the search for a matching long netmask. I then added argument support, so you can pass the IP address to the script. Since the Redis import would occur every time, I had to find a nice way to stop this. I first tried to count the redis keys and if they were less than 50k, the Redis import would be initiated but the “redis.keys(‘*’).count” command would take sometimes up to 10 seconds to complete, which wouldn’t give me the fast lookup I want to achieve. I then decided to just use optparse and choose an option to initiate the Redis import, otherwise, it would just skip it and run the script as if the necessary Redis keys were already there. **The final code I ended up with is:**
#!/usr/bin/env ruby


require 'curb'
require 'redis'
require 'optparse'

def ip2long(ip)
  ipAry = ip.split(/\./)
  long = ipAry[3].to_i | ipAry[2].to_i << 8 | ipAry[1].to_i << 16 | ipAry[0].to_i << 24
  long
end

def getMask(ipLong)
  redis = Redis.new
  (0..31).each {|msk|
    ipRef = (ipLong >> msk) << msk
    if redis.get(ipRef).nil?
    else
      return ipRef
    end
  }
end

def redisImport
  redis = Redis.new
  redis.flushall
  http = Curl.get("http://thyme.apnic.net/current/data-raw-table")
  http.body_str.each_line {|s|
    s.scan(/([0-9.]+)\/[0-9]+\s+([0-9]+)/) {|x,y|
      z = ip2long(x)
        redis.set(z, y)
    }
  }
  http = Curl.get("http://thyme.apnic.net/current/data-used-autnums")
  http.body_str.each_line {|s|
    s.scan(/([0-9]+)\s+(.*)/) {|x,y|
      redis.set(x, y)
    }
  }
end

userIp = ARGV[0]
redis = Redis.new

ARGV.options do |opts|
  opts.on("--charge")    { puts "Charging Redis..." ; redisImport ; exit}
  opts.parse!
end

ipLong = ip2long(userIp)
zMask = getMask(ipLong)
zAsn = redis.get(zMask)

puts "(" + userIp + ") belongs to ASN " + zAsn + " - " + redis.get(zAsn)
The initial Redis import is initiated with the –charge option when running the script. The import usually takes around 1 minute to complete but the IP to ASN queries only take around 200ms afterwards, which is pretty fast. When running the script after the initial “charge”, the script is getting the values directly from Redis and isn’t using Curl to receive the values anymore. This is the nice thing about using a persistent cache, is that you can run the script over and over without losing the data as you would if you stored the data in a Ruby array or hash. If you want to use this script, make sure you have redis-server running and make sure you have the Ruby libraries installed, which you can install with **gem install**. Here is how the script functions:
[[email protected] ~]$ ruby ip2asn.rb --charge
Charging Redis...
[[email protected] ~]$ ruby ip2asn.rb 4.2.2.2
(4.2.2.2) belongs to ASN 3356 - Level 3 Communications, Inc.
[[email protected] ~]$
Leave any comments if you have any suggestions or tips to enhance this script or make it more efficient. Or just for general discussion.. Here is the GitHub repo: [https://github.com/nckrse/ruby-redis-ip2asn][6] [1]: http://thyme.apnic.net/current/data-raw-table "Netmask to AS number" [2]: http://thyme.apnic.net/current/data-used-autnums "AS number to AS owner" [3]: https://github.com/mksm/ruby-curl "ruby-curl" [4]: https://github.com/taf2/curb "curb" [5]: https://github.com/redis/redis-rb "Redis Ruby client" [6]: https://github.com/nckrse/ruby-redis-ip2asn "https://github.com/nckrse/ruby-redis-ip2asn"