Weblog entries 2012


Here's a one-liner for you today:

  $ sudo ngrep tcp and port 80 -W byline

This prints HTTP traffic to the console, including full body.

2012-12-04 Objective-C with a Makefile

Here's a little example on how to compile Objective-C on the commandline, with a makefile. All code can be downloaded here: test cars.zip or copy/pasted from below.

Create a file called Makefile:

 LIBS=-framework Foundation
 all: test_cars
 test_cars: test_cars.m car.m
 	$(CC) $(LIBS) test_cars.m car.m -o test_cars
 	rm -f test_cars *.o
 Now create a header file:
 #import <Cocoa/Cocoa.h>
 @interface Car : NSObject {
         float fillLevel;
 -(float) gasLevel;
 -(void) addGas;

Plus the implementation:

 #import "Car.h"
 @implementation Car
 -(id) init
     self = [super init];
     if (self) {
         fillLevel = 0.0;
     return self;
 -(void) addGas
     NSLog(@"Added one gallon!");
     fillLevel += 1.0;
 -(float) gasLevel
     return fillLevel;

And the file containing main:

 #import <Foundation/Foundation.h>
 #import "car.h"
 int main(int argc, const char * argv[])
     @autoreleasepool {
         // insert code here...
         Car *c = [[Car alloc] init];
         NSLog(@"Level = %f", [c gasLevel]);
         [c addGas];
         NSLog(@"Level = %f", [c gasLevel]);
     return 0;

Type make in the directory containing the above files, and run the example with ./test_cars. Enjoy the commandline goodness.

The code in car.m and car.h is mostly by your friendly neighborhood Java/Grails developer.

2012-11-11 Starting with storyboard

I've never seriously worked with the storyboard before in Xcode, so here's a summary of what I've gathered so far.

  • The storyboard defines the flow of screens throughout your app, it's basically a collection of view controllers and their NIBs.
  • A storyboard is made up of scenes that are connected with segues (pronounced "SEG-way").
  • A scene here means: a view controller plus view.
  • A segue specifies how you go from one view controller to the next.

Container view controllers (such as a navigation controller, or a tab bar controller) have a special place in a storyboard, since these container view controllers define how the navigation takes place.

Thinking about the implementation details, I'd expect the storyboard to actually show a graphical representation of an XML file. In this file, the list of screens is saved, including the way how the screens are shown (with animations or what have you). Also present would be the type of container view controller.

In other words, in the MVC-pattern, the storyboard is the input for an over-arcing controller. A big data file, a specification, according to which all container view controllers are instantiated with the correct parameters. Instead of coding, you drag and drop this specification using Interface Builder.

You still have to put in some navigation code, though. Basically you code which segue to perform:

 [self performSegueWithIdentifier:@"OfferDetails" sender:self];

If I'd had to put it in one sentence: storyboard replaces the code for navigating between views.

2012-11-09 Full-text searching

In a previous project, I solved the full text search problem as follows.

Management overview: it's very easy to do full text searching but very hard to assign a weight to the results. MySQL supports full-text search but only in a degraded type of database (MyISAM instead of the full-featured InnoDB). The problem can be fixed by creating an "archive" table in this degraded MyISAM format, where you store a business ID, plus a summary of the data you're looking for (i.e. a concatenation of business name, address, reviews, locations etc). The MySQL full-text search facility then becomes available, and you get results as such:
(score, businessID, summary)
(0.99, 12, "Clear Cafe A beautiful restaurant that serves raw, vegan and seafood cuisine www.clearcafeubud.com Restaurants Jl. Hanoman, Ubud, Bali 80571, Indonesia 03618894437 Ubud Cheap and good food Bart I've been there with a 10+ people group and they were happy to accomodate apparently they also blah blah")

As you can see, the summary is just a concatenation of name, address, category, reviews, phone number, everything. There is lots of duplicate information in there, but that doesn't matter for the results.

Tech details: we were doing scientific measurements and these were stored in a bunch of InnoDB tables. You can mix table types InnoDB and MyISAM in a database schema. I created a separate table called 'fulltexts' in the database, with basically two fields: "measurement ID", and "summary". Any human-readable fields from any of the tables were concatenated into a summary. I created a database row-level trigger for a couple of tables, so that each insert or update would concatenate a buch of fields and do an insert into 'fulltexts' (or a replace, if the summary needed updating). Then, this table can be searched by the MySQL facility "full text search", and give you back a result with a relevancy. This is quite nifty and fast.

2012-08-12 Xcode 4.4 won't quit

I've had a problem with Xcode 4.3 and 4.4 where it will not quit after stopping the simulator. The problem is that Xcode thinks that tasks are running (even with the simulator not running). The only solution is to force-quit Xcode, which is a pretty major interruption in your workflow.

Here are some threads that appear to talk about the same problem:
Failure to stop executing task in Xcode 4.3
Why does Xcode 4.3.1 (4E1019) / 4.3.2 (4E2002) hang regularly with iOS simulator?
My 2 XCode 4.4 problems

I haven't found a workaround so far. The last thread on cocoabuilder.com indicates that you should add some extra logging to find the problem:

 defaults write com.apple.dt.Xcode IDEDebuggerLogToFile ~/IDEDebugger.log

Update: the problem seems to occur less, when I quit the task from Xcode. Normally I'd just punch Alt-Q to quit the simulator altogether; now I alt-Tab to Xcode first, then punch Alt-. (Alt-period) to stop the running task.

Update 2: I found a suggestion to clear the project from the organizer window: Xcode 4.3.2 process hangs.

Update 3: It still happens, even with Xcode 4.6... There's another suggestion that says to click the Simulator icon in the dock, which seems to work for now.
XCode 4.3.2, issue with running on simulator

This is my pet issue, and since this is the internet, every time Apple releases a new version of Xcode, I will loudly point to this bug and ask Why haven't they fixed this??!?!?!!111!! When they do fix it, I'll still keep using it as an example why Xcode is "broken".

2012-07-15 strace on Mac OS X

If you're looking for the Linux strace command on OS X, stop looking. OS X doesn't have this command, but what it does have, is dtruss.

I made the mistake of assuming dtruss was a straight replacement, so I ran it with the process name as its argument; i.e:

 $ sudo dtruss /Applications/SomeRandomApp/Contents/MacOS/SomeRandomApp
 dtrace: failed to execute /Applications/SomeRandomApp: file is set-id or
 unreadable [Note: the '-c' option requires a full pathname to the file]

The error message had me stumped, but after some reading up, dtruss is just a different command altogether. Rather you run it as follows:

 $ sudo dtruss -n SomeRandomApp

And in another terminal, or just via the GUI, run the command.

This came in handy when using the excellent JollysFastVNC which was failing to connect over an SSH tunnel without telling me the error. I found out the error as follows:

 $ sudo dtruss -n JollysFastVNC\ Home -t read
  6110/0x3af1f:  read(0x17, "Bad stdio forwarding specification '\\'\r\n\0", 0x1000)		 = 40 0

That gave me a clue that I made a typo in the SSH Options field in JollysFastVNC.

See also Brendan Gregg's blog.

2012-07-12 Deflux SQUID

Another update on the routines that I'm writing for the SAFARI project. I've already explained that we work with a sensor type called TES, and that to read out this sensor, we use a device called a SQUID. Our SQUID is basically a 3x3mm chip, Basically, the TESes and the SQUID are put together in an electronic circuit, which is then immersed in liquid helium.

Since at this temperature (at or below 4 Kelvin), the circuit is superconducting, the SQUID can trap flux because a superconducted current is flowing around in the SQUID.

Since the SQUID is basically a magnetometer, we need trapped flux just as much as we need a punch in the face. The flux can be removed by heating the SQUID a little but, but only just so it no longer superconducts. This is a very common thing for superconducting sensors, so our readout electronics have firmware which contains a "remove flux" procedure.

However, you can't just put the complete procedure in firmware, because we have only one version of electronics and we have multiple different setups in the labs. Each of these setups has a different cryostat with different temperatures (anything between 4 Kelvin and 50 milliKelvin). So the firmware deflux routine has a bunch of parameters which should be controlled through a software routine, which the user can then configure and kick off.

You might wonder why we don't just do it all in software. The reason is, that for the best result, the heating current must be stopped abruptly and the timing aspect is much more tightly controlled by using firmware.

2012-07-09 Mac OS X command line utilities

Recently I upgraded my macbook its SSD drive for a bigger version. Since I wanted to start with a fresh OS X copy, I decided not to use the Migration Assistant but rather just manually copy over the home folders (like Documents, Pictures etc).

I want to re-use the old, smaller SSD, but there might be some files which I forgot to copy. So what I was looking for, is a way to create a disk image like I'm used to when running Linux.

Below, the procedure is documented. It's probably wise to stick nice -n 20 before the commands dd and cat, so normal desktop usage is not bothered by the I/O, but for clarity, I've removed it.

First, connect your USB drive and type mount to show which drive got connected:

 $ mount
 /dev/disk0s2 on / (hfs, local, journaled)
 devfs on /dev (devfs, local, nobrowse)
 /dev/disk1s2 on /Volumes/storage (hfs, local, journaled)
 map -hosts on /net (autofs, nosuid, automounted, nobrowse)
 map auto_home on /home (autofs, automounted, nobrowse)
 /dev/disk2s2 on /Volumes/MacSSD (hfs, local, nodev, nosuid, journaled, noowners)

In this case, the last line shows that /dev/disk2 is the device we need, and partition 2 contains an HFS-formatted partition (or volume, in OS X speak). You can zoom in on this disk with the command diskutil list and show all partitions of all disks.

Since OS X automatically mounts volumes that a user attaches, we can now change to the directory where it's mounted, and zero the empty space:

 $ cd /Volumes/MacSSD
 $ cat /dev/zero > zero.file;rm zero.file

We'll now unmount the volume so we can make a copy of the whole partition (instead of the contents of the volume):

 $ cd
 $ hdiutil detach /dev/disk2s2

Now, we can make a literal copy of this partition:

 $ dd if=/dev/disk2s2 of=MacSSD.hfs bs=1m 

Over USB, this could take a loooong time, though. After that, compress the image with:

 $ gzip MacSSD.hfs

Now it's safe to move this file somewhere else. If you want, you can encrypt the file before doing so.

2012-07-02 Minimizing amplifier chain offset

For the SAFARI project, I'm refactoring some code, and one part of that is a routine to minimize the offset of the front-end electronics or FEE. For the place of the FEE in the setup, see also 2012-03-29 Extensions to Safari library.

The overall setup

Here is a picture of the lab setup.

lab setup with cryostat.jpg

From right to left: the PC with our lab software, the DEMUX board and some other measurement equipment in the rack, and the cyostat with on top the FEE board.

Zooming in on the cryostat; we're looking at a closed cryostat now. It's filled with helium because the scientists are doing a measurement. Before the test, the cryostat was opened:

opened safari cryostat.jpg

The sensor

The type of sensors used by the SAFARI instrument are TESes: Wikipedia article on transition edge sensors. As you can read with in this article, this type of sensor uses a SQUID in its readout electronics.

The TES basically consists of a tiny surface ("pixel") which is biased by a current source from the DEMUX board. When a photon hits the pixel, its resistance increases and the voltage drops. You detect this via a SQUID, whose output is amplified outside of the cryostat.

The TES its setpoint is 20 mOhm and if it's hit, the resistance increases to 100 mOhm. It's superconducting, so it's hard to get it out of its setpoint with heat, so flux is used.

The SQUID is controlled with a DC voltage, meaning we set and characterize it via a DC voltage. It's supplied by our DEMUX electronics board. Reading out happens with an AC voltage. It's supplied by our FEE electronics board, which also amplifies the output so it can be digitized by the DEMUX board. See also this schema:


The signal

We currently use a single SQUID for all the TESes. The SQUID combines these in one signal, which the DEMUX board then splits up again so we can read out each individual pixel. To see what I mean, here is the sensor:

safari sensor.jpg

Description: at the top left, we have the TESes. On the bottom left, we have the LC filters (see below). On the right, we prominently see the black connectors that bring the signal out of the sensor. On the bottom right, we see the single SQUID. Note the connections in the top and bottom middle. If we increase the number of pixels, these connections will stay the same because the TES signals are multiplexed (put together). To get an idea of the physical scale, this thing has a width of 12-13 cm (I haven't measured it exactly, just an estimate).

There is one LC filter for each pixel. Basically, with the LC filter you pick out a certain frequency. Thus the LC filter controls which information the pixel gives us; it makes sure the spectrum we're measuring is nicely divided over the pixels. Wikipedia has an article on LC circuits which is hard to read for me, but do notice the applications header which I've linked to. First entry says that the most common application is tuning, which is exactly what's being done here.

As an aside: two or more SQUIDs

SRON bought the SQUID from elsewhere, namely Magnicon, and then bonded to the silicon substrate. As for the SQUID, there are other possibilities. Per pixel row, you could use one SQUID. Or you could still use the above setup, but with an extra SQUID, which we call the dual-SQUID configuration. This configuration has been designed, but for this to work, we need to characterize both SQUIDs and since they influence each other, we'd need to use a flux-locked loop to lock the first SQUID in its set point, then characterize the other. I might have to expand on this sometime later.

Looking ahead

What I've described above, is just the lab setup. With this setup, we can only detect power intensity, but the SAFARI instrument is a spectrometer. Thus, the instrument consists of an FPU (focal plane unit), which is a box containing mirrors, shutter, filter wheels, a cooler, an FTS and three FPAs (focal plane assemblies). These FPAs are basically little cubes with the above described sensor in there. The focal plane unit is cooled to 1.7 Kelvin, and the focal plane assemblies are cooled to 50 milli-Kelvin via an intermediate 300 milli-Kelvin stage. Outside of the focal plane unit, we also have two "warm" units. Basically these have the function of the current DEMUX and FEE boards, but our current boards are meant for a lab situation, so new electronics will be made.

So the sensor measures power intensity, and the FTS will give us intensity as a function of wavelength. In other words: the instrument will send us power plus mirror position (of the FTS), and these can be transformed mathematically to create a spectrum. Since SAFARI is an infrared instrument, we are looking in the 34 to 210 micrometer wavelength.

Current task

So far I've described the setup. Now on to my current task.

To accurately set the DC bias on the pre-amplifier chain that consists of a single SQUID, an amplifier, and an integrator (??), we need to measure the offset that's introduced by the FEE.

The basic schematic of the SQUID, the amplifier and the integrator look as follows:
schema single squid.png

On the left, we have the single SQUID. It's biased in a range between 22 and 44 uA. On the right, we have two functions. The top right shows a function of the FEE board, and the botton right (with the DC voltage part) shows a function on the DEMUX board; the DAC which sets a voltage, and the integrator that keeps the voltage over the setpoint of the SQUID. Below the SQUID is

This offset can be adjusted via the integrator-offset bias generator (noted as DAC in the above schema (??)). It's also adjusted via:

  • the LF_Gain, which has a range of 100..10k
  • the LF_Input Impedance
  • the LF Amplifier configuration

Note: the ?? indicates I have to check this statement


SPICA - SAFARI Link to the SRON homepage concerning the project

2012-06-01 OS X Security

After the recent FlashBack debacle, it seems prudent to look at the way you work with OS X. A good way to start:

2012-05-19 Telfort ADSL modem Zyxel P-2812HNU-F1 and VoIP Planet

(English readers: this is an explanation on how to configure the modem of a Dutch ISP for VOIP usage).

Onlangs heb ik mijn Telfort ADSL abonnement geupgrade en kreeg een Zyxel modem, type P-2812HNU-F1. Vorig jaar heb ik reeds voor type Zyxel 2601HN-F1 instructies gegeven, hierbij een update.

Om via een SIP account bij VoIP Planet te bellen met een telefoontoestel dat je aansluit op het modem, volg de volgende instructies op.

Om het modem te configureren, log in als admin (standaard wachtwoord: 1234) op het modem via adres en kies in het onderste menu genaamd VoIP voor de optie SIP.

zyxel sip menu.png

Het scherm dat tevoorschijn komt, heeft drie tabbladen. Het eerste tabblad is genaamd "SIP Service Provider". Laat het drop-down menu "Service Provider Selection" staan op "Provider-1" en vul deze als hieronder in. Klik daarna op Apply, en het drop-down menu toont nu netjes "VoipPlanet".

zyxel service provider.png

Klik daarna op het tweede tabblad, en klik op het pen-en-papier icoon in de Modify kolom, eerste rij. Er komt een popup-scherm waarin je slechts een viertal dingen hoeft te veranderen. Eerst moet je de "Active SIP Account" aanvinken, dan vul je de SIP Account Number in, en daarna Username en Password.

Waarbij je natuurlijk de vermeldde SIP Account Number en Username verandert in diegene die je van VoIP Planet hebt gekregen:

zyxel sip account.png

Als het goed is, hoef je niet naar beneden te scrollen. Klik op apply om op te slaan. Ga onderaan het scherm naar het menu System Monitor, en kies voor Log.

zyxel system monitor.png

Activeer de tweede tab, "Phone Log". Hier moet nu staan:

  SIP Registration: SIP:12345: Register Success


zyxel sip success.png

Je kunt nu bellen met het aangesloten (analoge) telefoontoestel via VoIP Planet!

2012-03-29 Extensions to Safari library

I've received the specs for a modification to our software library for the SAFARI project, and after a two-month hiatus, I'm looking again at what's currently present.

What's existing, is basically a script that displays a menu with a bunch of testing options. It's called demuxtest, because it tests the electronics board which we call "the demux board".


The Demux board plays the intermediary role here; all commands are routed through this board to the FEE (front-end electronics) board. If the FEE board isn't present, then it can generate a dummy response for testing purposes.

Zooming in on the electronics, it would look as follows:


You could roughly see the demux board as the digital part, and the rest of this picture as the analog part.

Some visuals

This is our test setup:

demux fee test setup.jpg

The FEE board

This is how the FEE (front end electronics) board looks like:

fee board.jpg

The demux board

This is the demux board:

demux board.jpg

2012-03-20 graphics glitches on OS X Lion

Graphics glitches

For the past days, I've seen a number of glitches on OS X Lion (update 10.7.3) with my mid-2010 Macbook Pro, and apparently, I'm not the only one.

It seems to happen especially on external monitors:

However, iMacs have a lot more trouble:

A possible solution for the graphics glitches is mentioned at the end of the thread:

White screen

UPDATE: this issue seems fixed with the 10.7.4 update

I've encountered a bug where the screen would blank, reproducible as follows:

  • With a closed laptop, hook up an external monitor (mini-DisplayPort) and keyboard
  • Press a key to wake screen, then I start working
  • At the end of day, I unplug keyboard and laptop
  • When coming home, open lid and click "switch user" (I have security settings such that it locks the display when closing the lid)
  • Look at white screen, where the mouse cursor is visible. Sometimes, after up to a minute, the screen will redraw correctly and show the login screen. Sometimes it'll just stay this way, and a hard reboot is required.

Black screen

I've also encountered a bug where the laptop screen would just stay black, not reproducible. I took the following steps:

  • With a closed laptop, hook up an external monitor and keyboard
  • Press a key to wake screen, then disconnect screen with the mini-DisplayPort
  • Open laptop lid, stare at black screen

Remotely I could log into the laptop, though.

Another set of circumstances happens as follows:

  • With a closed laptop, hook up an external monitor and keyboard
  • Go away until display sleeps
  • Come back and discover laptop cannot be woken up, but is still accessible via SSH

The following lines are then visible in the kernel.log:

 --- last message repeated 1 time ---
 kernel[0]: NVDA(OpenGL): Channel timeout!                                                
 --- last message repeated 1 time ---
 kernel[0]: NVDA(OpenGL): Channel timeout!
 --- last message repeated 1 time ---
 kernel[0]: NVDA(OpenGL): Channel exception!  exception type = 0xd = GR: SW Notify Error
 kernel[0]: NVDA(OpenGL): Channel timeout!
 kernel[0]: NVDA(OpenGL): Channel exception! exception type = 0xd = GR: SW Notify Error
 kernel[0]: NVDA(DisplayBase): Channel timeout!

UPDATE: this is still occurring with 10.7.4 update. Workaround is to keep the lid open

2012-03-08 Launchctl on OS X

Comparison of SysV initialization and launchd

If you're an old-school Unix or Linux person, you're probably also used to the SysV system initialization routine. Nowadays, it's slowly being replaced by upstart and the likes, but I'm going to skip that for now. This is meant as a guide for ye olde Unix neckbeards.

Basically, SysV init means:

  • Daemons are controlled with scripts in /etc/init.d
  • These scripts take a couple of standard arguments, such as start/stop/restart/reload
  • There are three or four so-called runlevels, and the higher the runlevel, the more capable the system is (in other words, more daemons are started up)
  • The sequence of starting the daemons (i.e. calling the init.d scripts with "start") is determined by specially named symlinks in directories named for the runlevel, for instance for runlevel 3, the directory /etc/rc3.d contains a bunch of symlinks to start the daemons appropriate for runlevel 3
  • If you want to prevent a daemon from running, then you remove symlinks in the /etc/rcX.d directory. This is usually done through a command which manages this for you. Redhat uses chkconfig, Debian uses update-rc.d.
  • To control specific daemons, you call the appropriate script in /etc/init.d with the start or stop parameter

OS X does this differently, and superficially, much simpler.

When the system boots, the launchd process is responsible for starting all daemons. To sum up the characteristics of this method:

  • Daemons are controlled with the launchd command, there aren't any start/stop scripts
  • Description of how to start a daemon is specified in a configuration file in the folder /System/Library/LaunchDaemons, and these files have the binary plist format, editable via the defaults command
  • Launchd does not support dependencies, thus it's not possible to configure (for instance) starting the database daemon before the webserver.
  • You can prevent daemons from running by using the launchctl command
  • To control specific daemons, you use the launchctl command

There is a further distinction which I haven't talked about. OS X has the concept of user-specific daemons. These are called "launch agents" instead of daemons and they have their configuration files in /System/Library/LaunchAgents, and other locations specified in the launchd man page. Examples of these launch agents: the dock and software for your Wacom tablet.

launchctl examples

Listing present daemons

The stuff below is equivalent to listing the contents of the /etc/init.d directory on Linux.

To see what system daemons are present on the system;

 $ cd /System/Library/LaunchDaemons
 $ ls

Note that this does not tell you whether they've actually been configured to start, or not!

Overview of running daemons

The stuff below is equivalent to listing the contents of the /etc/rcX.d directories on Linux, or using the "chkconfig --list" command on RedHat.

Since launchd is more than just managing daemons, but also a replacement for init, it can show us some bookkeeping on previously run daemons. Use launchctl its 'list' parameter to list the currently running daemons, including the daemons that have already exited (i.e. configured for a one-time run).

To list all daemons:

 $ launchctl list

To show daemons, but filter out those that aren't currently running:

 $ launchctl list | grep -v "^-"

Controlling daemons

The stuff below is equivalent to using the scripts in /etc/init.d on Linux.

To temporarily stop a daemon, you use the unload parameter plus the config filename. For instance to stop the daemon that controls swapping memory to disk:

 $ sudo launchctl unload /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

If it says "nothing to unload", then the daemon wasn't configured to be started in the first place.

To temporarily start a daemon, use the load parameter:

 $ sudo launchctl unload /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

Adjusting the boot process

The stuff below is equivalent to using the chkconfig or update-rc.d commands in Linux.

To start or stop a daemon, and make sure it'll be started the next boot, use the -w flag. For example: to start the dynamic pager now, and after rebooting:

 $ sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

If the daemon wasn't configured to start before, you might want to pass the -F flag, which forces starting.

Configuring a daemon

The stuff below is equivalent to editing a script in /etc/init.d, or a configuration file in /etc under linux.

To configure the start/stop flags of a daemon, you edit the plist file with the defaults command. To read the current configuration:

 $ defaults read /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

To change a config file, simply use the defaults 'write' parameter. As an example, to disable Bonjour advertising on the network:

 $ sudo defaults write /System/Library/LaunchDaemons/com.apple.mDNSResponder \
     ProgramArguments -array-add "-NoMulticastAdvertisements"

Note that whenever software is updated, these changes get written over! OS X does not have a package system to my knowledge, which will save any adjusted configuration files.


Some notes:

  • Launchd doesn't support dependencies, but it's possible to create special configuration files in /System/Library/LaunchDaemons which will start a daemon whenever a client tries accessing a particular network port.
  • Both launchd and launchctl have a man page, and they're okay (not super, though).
  • For more information on how to write the plist files, see also the man page of "launchd.plist"
  • Should you want to periodically run a script, check out the tips here: http://blog.mattbrock.co.uk/2010/02/25/moving-from-cron-to-launchd-on-mac-os-x-server/

2012-02-09 Writing ethernet packets on OS X and BSD

If you want to write raw ethernet frames, you can use so-called raw sockets. Unfortunately, there's a difference between raw sockets under Linux and raw sockets under BSD and derivatives such as Apple's OS X. Linux allows you to create custom ethernet frames with whatever content you want, but BSD assumes you will create IP (internet protocol) packets.

In my case, I wanted to write raw ethernet frames, as in: have complete control of the contents. The content didn't have anything to do with internet; at work we use ethernet to talk to custom electronics. The electronics doesn't contain enough processing power to have a full TCP/IP stack, when ethernet is perfectly fine.

Note: I've tested the code below on OS X, version 10.7 (Lion). I've checked the output using tcpdump on the other end (a Linux box). Most code is from Bastian Rieck's excellent piece "Using FreeBSD's BPF device with C/C++". The only real addition is the write_frames routine that chops up your data before sending it.

    #include <sys/types.h>
    #include <sys/uio.h>
    #include <unistd.h>
    #include <stdlib.h>
    #include <stdio.h>
    #include <fcntl.h>
    #include <string.h>
    #include <sys/ioctl.h>
    #include <sys/socket.h>
    #include <arpa/inet.h>
    #include <net/if.h>
    #include <net/ethernet.h>
    #include <net/bpf.h>
    // Fill in your source and destination MAC address
    unsigned char dest_mac[ETHER_ADDR_LEN]  = {0x00, 0x1b, 0x21, 0x53, 0x83, 0x4f};
    unsigned char src_mac[ETHER_ADDR_LEN] = {0xc8, 0xbc, 0xc8, 0x91, 0x79, 0x2d};
    // My struct for an ethernet frame. There are many like it, but this one is
    // mine.
    struct frame_t {
        struct ether_header header;
        unsigned char payload[ETHER_MAX_LEN - ETHER_HDR_LEN];
        ssize_t len;
        ssize_t payload_len;
    // Some convenience constants
    // Try to open the bpf device
    int open_dev(void)
        char buf[ 11 ] = { 0 };
        int bpf = 0;
        int i = 0;
        for(i = 0; i < 99; i++ )
            sprintf( buf, "/dev/bpf%i", i );
            bpf = open( buf, O_RDWR );
            if( bpf != -1 ) {
                printf("Opened device /dev/bpf%i\n", i);
        if(bpf == -1) {
            printf("Cannot open any /dev/bpf* device, exiting\n");
        return bpf;
    // Associate bpf device with a physical ethernet interface
    void assoc_dev(int bpf, char* interface)
        struct ifreq bound_if;
        strcpy(bound_if.ifr_name, interface);
        if(ioctl( bpf, BIOCSETIF, &bound_if ) > 0) {
            printf("Cannot bind bpf device to physical device %s, exiting\n", interface);
        printf("Bound bpf device to physical device %s\n", interface);
    // Set some options on the bpf device, then get the length of the kernel buffer
    int get_buf_len(int bpf)
        int buf_len = 1;
        // activate immediate mode (therefore, buf_len is initially set to "1")
        if( ioctl( bpf, BIOCIMMEDIATE, &buf_len ) == -1 ) {
            printf("Cannot set IMMEDIATE mode of bpf device\n");
        // request buffer length
        if( ioctl( bpf, BIOCGBLEN, &buf_len ) == -1 ) {
            printf("Cannot get bufferlength of bpf device\n");
        printf("Buffer length of bpf device: %d\n", buf_len);
        return buf_len;
    // Read one or more frames
    void read_frames(int bpf, int buf_len)
        int read_bytes = 0;
        struct frame_t *frame;
        struct bpf_hdr *bpf_buf = (struct bpf_hdr*) malloc(buf_len);
        struct bpf_hdr *bpf_packet;
        int run_loop = 1;
        int i = 0;
        printf("Start reading frames\n");
        while(run_loop) {
            memset(bpf_buf, 0, buf_len);
            if((read_bytes = read(bpf, bpf_buf, buf_len)) > 0) {
                printf("Read %d\n", i);
                // read all packets that are included in bpf_buf. BPF_WORDALIGN is used
                // to proceed to the next BPF packet that is available in the buffer.
                char* ptr = (char*)bpf_buf;
                while(ptr < ((char*)(bpf_buf) + read_bytes)) {
                    bpf_packet = (struct bpf_hdr*)ptr;
                    frame = (struct frame_t*)((char*) bpf_packet + bpf_packet->bh_hdrlen);
                    frame->len = bpf_packet->bh_caplen;
                    frame->payload_len = frame->len - (2*ETHER_ADDR_LEN) - ETHER_TYPE_LEN;
                    // Do something with the frame
                    printf("Got packet, length of frame: %ld, length of data: %ld\n",
                        frame->len, frame->payload_len);
                    ptr += BPF_WORDALIGN(bpf_packet->bh_hdrlen + bpf_packet->bh_caplen);
            } else {
                perror("Meh, couldn't read from bpf device");
    // Read a single frame
    void read_single_frame(int bpf, int buf_len)
        int read_bytes = 0;
        int i;
        struct bpf_hdr* bpf_buf = malloc(buf_len);
        memset(bpf_buf, 0, buf_len);
        char *ptr;
        printf("Headerlength: %ld\n", sizeof(bpf_buf));
        read_bytes = read(bpf, bpf_buf, buf_len);
        if(read_bytes > 0) {
            printf("Got %d bytes\n", read_bytes);
        } else {
            printf("Got 0 bytes\n");
        ptr = (char*)bpf_buf;
        for(i = 0; i < read_bytes; i++) {
            unsigned char byte = (unsigned char) *(ptr + i);
            printf("0x%02X ", byte);
    // Write a single ethernet frame with test data
    void write_single_frame(int bpf)
        ssize_t data_length = 0x4F;
        struct frame_t frame;
        memcpy(frame.header.ether_dhost, dest_mac, ETHER_HDR_LEN);
        memcpy(frame.header.ether_shost, src_mac, ETHER_HDR_LEN);
        frame.header.ether_type = 0x00;
        frame.len = (2*ETHER_ADDR_LEN) + ETHER_TYPE_LEN + data_length;
        // Fill frame with ramp
        unsigned char j;
        for (j = 0; j < data_length; j++) {
            frame.payload[j] = j;
        ssize_t bytes_sent;
        bytes_sent = write(bpf, &frame, frame.len);
        if(bytes_sent > 0) {
            printf("Bytes sent: %ld\n", bytes_sent);
        } else {
            perror("Whoops! Does the device actually have an IP address?");
    // Divide data across ethernet frames
    void write_frames (int bpf, const unsigned char *databuf, size_t datalen)
        size_t start = 0;
        struct frame_t *frame = malloc(ETHER_MAX_LEN);
        size_t bytes_to_send;
        ssize_t bytes_sent;
        memcpy(frame->header.ether_dhost, dest_mac, ETHER_HDR_LEN);
        memcpy(frame->header.ether_shost, src_mac, ETHER_HDR_LEN);
        frame->header.ether_type = 0x0000;
        do {
            // Clear frame
            bzero((void*)(frame+ETHER_PAYLOAD_START), ETHER_PAYLOAD_LEN);
            // Calculate remainder
            if((datalen - start) < ETHER_PAYLOAD_LEN) {
                bytes_to_send = datalen - start;
            } else {
                bytes_to_send = ETHER_PAYLOAD_LEN;
            // Fill frame payload
            printf("Copying payload from %lu, length %lu\n", start, bytes_to_send);
            memcpy(frame->payload, (void*)(databuf + start), bytes_to_send);
            frame->len = ETHER_HDR_LEN + bytes_to_send;
            // Note we don't add the four-byte CRC, the OS does this for us.
            // Neither do we fill packets with zeroes when the frame length is
            // below the minimum Ethernet frame length, the OS will do the
            // padding.
            printf("Total frame length: %lu of maximum ethernet frame length %d\n",
                frame->len, ETHER_MAX_LEN - ETHER_CRC_LEN);
            bytes_sent = write(bpf, frame, frame->len);
            // Check results
            if(bytes_sent < 0 ) {
                perror("Error, perhaps device doesn't have IP address assigned?");
            } else if(bytes_sent != frame->len) {
                printf("Error, only sent %ld bytes of %lu\n", bytes_sent, bytes_to_send);
            } else {
                printf("Sending frame OK\n");
            start += bytes_to_send;
        } while (start < datalen);
    // Create a simple ramp so we can check the splitting of data across frames on
    // the other side (using tcpdump or somesuch)
    unsigned char* make_testdata(int len)
        unsigned char *testdata = (unsigned char*)malloc(len);
        int i;
        unsigned char j = 0;
        for(i = 0; i < len; i++) {
            testdata[i] = j;
            if(j < sizeof(char)) {
                j = 0;
        return testdata;
    int main(void)
        char* interface = "en0";
        unsigned char* testdata;
        size_t testdata_len = 4530;
        int bpf;
        int buf_len;
        bpf = open_dev();
        assoc_dev(bpf, interface);
        buf_len = get_buf_len(bpf);
        //read_single_frame(bpf, buf_len);
        //read_frames(bpf, buf_len);
        testdata = make_testdata(testdata_len);
        write_frames(bpf, testdata, testdata_len);

2012-01-26 TransIP VPS

Since last year, TransIP offers virtual private servers (VPSes). I have been playing with the M version for the past week. This particular VPS has 512 MB of RAM, 10 GB of harddisk space and one CPU core, and I ran UnixBench on it. Here are the results:

 System Benchmarks Index Values               BASELINE       RESULT    INDEX
 Dhrystone 2 using register variables         116700.0   24917048.3   2135.1 
 Double-Precision Whetstone                       55.0       3413.4    620.6
 Execl Throughput                                 43.0       4520.4   1051.3
 File Copy 1024 bufsize 2000 maxblocks          3960.0     882288.4   2228.0
 File Copy 256 bufsize 500 maxblocks            1655.0     260223.5   1572.3
 File Copy 4096 bufsize 8000 maxblocks          5800.0    1378435.4   2376.6
 Pipe Throughput                               12440.0    2369954.3   1905.1 
 Pipe-based Context Switching                   4000.0     387771.4    969.4
 Process Creation                                126.0      14882.1   1181.1
 Shell Scripts (1 concurrent)                     42.4       6816.5   1607.7
 Shell Scripts (8 concurrent)                      6.0        898.3   1497.2
 System Call Overhead                          15000.0    4303337.7   2868.9
 System Benchmarks Index Score                                        1537.7

Edit 29/03/2012: their offer has gotten better, with more disk space.

2012-01-23 Logrotate error for MySQL on Debian 6 Squeeze

When installing a new machine with Debian 6 (Squeeze), I also imported a database from a CentOS 5 box. The next day, the following error appeared in my inbox:

 error: error running shared postrotate script for '/var/log/mysql.log
      /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log '
 run-parts: /etc/cron.daily/logrotate exited with return code 1

Turns out that Debian needs a user called 'debian-sys-maint' and that this user needs a number of rights on the database. The username and password are configured in /etc/defaults/debian.cnf so write these down, then create the user with the following commands:

 $ mysql -u root -p
 mysql> GRANT SHUTDOWN ON *.* TO 'debian-sys-maint'@'localhost';
 mysql> GRANT RELOAD ON *.* TO 'debian-sys-maint'@'localhost';
 mysql> GRANT SELECT ON 'mysql'.'user' TO 'debian-sys-maint'@'localhost';

and then set the correct password for the debian user:

 mysql> SET PASSWORD FOR 'debian-sys-maint'@'localhost' = PASSWORD('secret');

Test whether logrotate won't give any problems:

 $ sudo logrotate -f /etc/logrotate.d/mysql-server