Building an OpenBGPD bgplg Looking Glass from Scratch

Anybody who’s spent time troubleshooting BGP advertisements and/or Internet reachability issues has probably made use of one or many looking glass servers. These are servers, usually managed and maintained by ISPs, which provide a set of useful troubleshooting tools (ping, traceroute) and outputs (show ip bgp) that allow you to view the Internet from the perspective of their network. Large, geographically diverse carriers often allow you to select a router you would like to troubleshoot from, which is useful if the issue you are troubleshooting seems to be localized to a specific area.

As a network guy, I’ve spent A LOT of time on various looking glasses, and I’ve even set a few up — albeit many years ago. Internet routing tables can be chaotic at times and, in my opinion, there can never be too many looking glasses. So when the time came to set up a new looking glass at my day job, I figured it would be a good opportunity to take some notes to make it easier for the next guy or gal to get who is willing to show the world what the Internet looks like from their perspective. After a bit of research, I decided that I would install bgplg, a clean, robust OpenBGPD add-on.

Get the OS

I installed OpenBSD 5.2, which can be freely downloaded here

Install the OS

OpenBSD is a very lightweight OS, so I probably went overkill when I configured my VMWare guest with 1GB RAM, 2 CPUs and 15GB HDD space.

The installer is also very lightweight — just select (I)nstall when it asks and start walking through the prompts. I followed the defaults until it asked if I’d like to install X Windows. There’s no need for it on this server so, unless extenuating circumstances warrant it, I’d select "no". A few more questions and then it will ask to install sets. I used the CD as my source and, in keeping with the ideal of a minimal install, removed packages I felt were unnecessary: type "-x*", which will remove all X-related packages and then "-game*" to remove the game set. Simply press Enter when done selecting sets and the install will begin.

When installation is complete, eject your install media and type "reboot".

(Optional) Add an IPv6 Default Gateway

If you’ve configured IPv6, you will notice that there is no default IPv6 route installed. To fix this, edit your network config file (mine is /etc/hostname.em0) and add the following line to the end:

!route -n add -inet6 default <IPv6 default gateway>

When added & saved, you can activate your changes by running the netstart script:

# sh /etc/netstart

Configure bgplg & Apache

You’ll now want to enable bgplg, which is installed by default. The next steps are straight out of the man page:

# chmod 0555 /var/www/cgi-bin/bgplg
          # chmod 0555 /var/www/bin/bgpctl
          # mkdir /var/www/etc
          # cp /etc/resolv.conf /var/www/etc
          # chmod 4555 /var/www/bin/ping
          # chmod 4555 /var/www/bin/ping6
          # chmod 4555 /var/www/bin/traceroute
          # chmod 4555 /var/www/bin/traceroute6

Change the following values in /etc/rc.conf to start OpenBGPD & Apache at boot time:


Note: The default bgplg install runs in at /cgi-bin/bgplg in your Apache document root. Since my server only serves up the bgplg page, I tweaked the DirectoryIndex parameter in my Apache config (`/var/www/conf/httpd.conf1) to the following:

DirectoryIndex /cgi-bin/bgplg

After the change is made, you can reload Apache by running:

apachectl reload

If this setting is not changed, you can still access your looking glass at http://[Your IP]/cgi-bin/bgplg

OpenBSD runs Apache in a chroot. We tweaked a lot of permissions above to give the chroot environment access to basic network tools, but by default things like DNS lookups will still not work. To test DNS resolution and the ping tool from your chroot environment, you can run the following command:

# chroot -g www -u www /var/www /bin/ping

which will likely result in the following error:

ping: socket: Permission denied

This is a result of the nosuid flag being set on the /var partition by default. You can verify this by looking at how /var is currently mounted:

# mount | grep var
          /dev/sd0e on /var type ffs (local, nodev, nosuid)

To fix this problem, remove the nosuid flag from the /var partition in <code>/etc/fstab</code> and reboot your machine. Your should now see something similar to:

# mount | grep var
          /dev/sd0e on /var type ffs (local, nodev)

Verify that your chrooted user is now working:

# chroot -g www -u www /var/www /bin/ping
          PING ( 56 data bytes
          64 bytes from icmp_seq=0 ttl=49 time=57.249 ms
          64 bytes from icmp_seq=1 ttl=49 time=56.861 ms
          --- ping statistics ---
          2 packets transmitted, 2 packets received, 0.0% packet loss
          round-trip min/avg/max/std-dev = 56.861/57.055/57.249/0.194 ms

Also, Apache should be up and running after the reboot. Even though we haven’t configured OpenBGPD yet, you should still be able to test ping & traceroute functionality.

Configure OpenBGPD

Finally, we’ll configure OpenBGPD. To do this, edit the `/etc/bgpd.conf`file and enter the relevant bgpd config information (see the man page for help). My config looks like:

          # global configuration
          AS 65000
          # restricted socket for bgplg(8)
          socket "/var/www/logs/bgpd.rsock" restricted
          # neighbors and peers
          group "peering AS65000" {
                  remote-as 65000
                  neighbor $peer1 {
                          descr   "IBGP-V4"
                  neighbor $peer2 {
                          descr "IBGP-V6"
                          local-address 2001:DB8::2

Once you’re done configuring, you can restart OpenBGPD:

/etc/rc.d/bgpd restart

And that’s it. Next step: create/update your ASN’s PeeringDB record with your sexy new looking glass URL.

Ken Mix - December 13, 2012

Wireshark IP in IP Capture Filter

As anybody working on the back end of VoIP knows, sometimes a packet capture is the quickest way to get to the root of a problem. If traffic volumes are high, this can be a painful exercise for you, the network and the PC or server hosting your analysis program (we prefer Wireshark). Fortunately, our AcmePacket SBCs provide a handy "packet-trace" feature that allows you to specify traffic by interface and source and/or destination IP and port, encapsulates the specified packets in an IP-IP tunnel and sends them to the capture server of my choice.

If we are only running a single capture, we can then set up a capture filter of ip proto 4 to ensure that our file only contains the encapsulated traffic. Wireshark, being the unbelievably useful tool that it is, then allows us to use our standard display filters, such as sip or ip.addr ==, regardless of the fact that there is an extra IP header on each packet. All’s well, right?

Not quite. Unfortunately, there comes a time in every tech’s life when they’re required to troubleshoot the intermittent issue. And sometimes these types of issues require a rolling file capture of, say, 50 files at 50MB each ("ring buffer" capture, in Wireshark terminology). When this happens, a simple capture filter of ip proto 4 means that your already annoyingly-large files are going to contain a bunch of packets from unrelated captures. When dealing with IP-IP packets, using the simple "host" capture filter isn’t an option, as it only looks at the outside IP header. So, what’s the solution?

Byte Offset Notation

Byte offset notation is exactly what the name says — basically you specify a protocol, the offset in bytes from the beginning of the header and the number of bytes to check. Although a bit complicated at first glance, they are one of the most powerful tools I’ve yet found with regards to Wireshark; a filter designed using byte offset notation could filter traffic based upon ANY type of ANY packet. An example is worth a thousand words:

To capture all IP packets with a TTL of 2:

ip[8,1] == 2

*Note:* All headers start with byte zero. The example above captures all packets where the TTL (starting at byte 8 of the IP header, and containing 1 byte) equals 2. If you’re not too familiar with IP header structure, Wikipedia has a usable breakdown here.

To capture based upon the source and destination address of the inner IP header of an IP in IP encapsulated packet, you simply add 20 bytes to each of your values. So, if the source address of an IP packet starts at byte 12 and the destination address starts at byte 16, you would use 32 and 36 for the starting byte values. You’ll also want to convert your IP addresses to hex:

IP converted to hexadecimal = 0xC0A80001

So to capture both directions of a conversation, you’ll want to capture where the source OR destination address equals 0xC0A80001:

ip[32:4]==0xC0A80001 || ip[36:4]==0xC0A80001

Now that we’ve got these down, we find ourselves running more captures, more often. I’m not sure if that’s a good thing, but YMMV.

Enter An IP Here to Convert to Hex:   
Ken Mix - August 01, 2012

Expanding the Scope of the Blog

For the folks following this blog (both of you) it’s probably become readily apparent that:

  1. IPv6 is a pipe dream, or

  2. After all of the initial testing and configuration, it turns out the demand just isn’t there yet, or

  3. We’re pretty lazy (or busy — take your pick)

In reality, it’s a bit of each. While we know IPv6 isn’t really a pipe dream, and that IPv6 traffic increases every day, the portion of our workdays involving IPv6 equals exactly 0.00%. If we want slightly more involvement, we could always work at Google; according to their IPv6 statistics, IPv6-enabled Google users account for 0.68% of their workload at the time of this writing…​

While we’re certain there will eventually be more IPv6 content, we felt it would be handy to have a repository for the rest of the sweet nuggets of info we dig up or develop in the course of our jobs. Hence, Enjoy!

tags: IPv6 Blog
Ken Mix - July 31, 2012

2010 Rocky Mountain IPv6 Summit Scheduled

The Rocky Mountain IPv6 Task Force (RMv6TF) has announced details of their 3rd annual Rocky Mountain IPv6 Summit. The summit, scheduled for May 26-27 in Denver, promises to be an excellent opportunity to learn about IPv6 technologies and trends, and also to meet other people in various stages of IPv6 deployment and/or planning.

Details and registration info can be found here:

I look forward to seeing you there!

tags: IPv6
Ken Mix - February 01, 2010

Cisco ASA IPv6 Failover in 8.2(2): So Far, So Good...

After several months of empty promises, missed dates and missing features, we’d pretty much resigned ourselves to waiting until ASA version 8.3(x) (ETA TBD) for IPv6 failover support. So imagine our surprise when the 8.2(2) release notes showed the following new feature:

IPv6 Support in Failover Configurations — IPv6 is now supported in Failover configurations. You can assign active and standby IPv6 addresses to interfaces and use IPv6 addresses for the failover and Stateful failover interfaces.

The following commands were modified: failover interface ip, ipv6 address.

Keeping in mind past disappointments, we were cautiously optimistic that we might finally be able to swing our IPv6 traffic from a spare Juniper SSG (100Mbps interfaces) to our ASA 5520 failover pair (Gigabit Ethernet interfaces). The upgrade was flawless and, sure enough, there is now a spot to specify a standby IPv6 address. After a week of light testing, I can report that so far things are running as one would expect. Time will tell, but it seems that another major step has been taken towards production deployment of IPv6 in the enterprise.

Ken Mix - January 29, 2010
About Knowledge Bombs
Random bits of knowledge that we don't want to forget and that might help you!
Cody Lerum
Ken Mix