Weblog entries 2011

2011-12-12 Protocol stuff

We have an interesting situation here at work, concerning our internally developed binary protocol.

There are two libraries to parse the protocol. These have different ways to calculate the length of a block of data. And now I want to know which way is the correct. Of course, that's debatable, with a history of hardware that uses the protocol for different purposes.

In the current firmware/software combination, the firmware generates a large number of packets that each fit into the Ethernet maximum packet length (i.e. 1500 bytes). Considering the packet length is 1500 bytes, and our data consists of 2-byte samples, we get 730 samples in a packet. The rest is overhead, namely: 6 bytes primary header, 9 bytes secondary header and 24 bytes data header, plus an end-of-packet byte containing the value 0xA5.

A number of these packets are received by an intermediate piece of software, called the Ethernet Daemon, and then concatenated into a larger packet, where the Data Header/Data part are repeated.

CCSDSpacketconcatenation.png

The discussion piece currently, is how to calculate the length of the block of data. You need to know this in order to find the next Data Header. The following fields in the data header can help:

  • samples64; the number of samples per 64 bit; must be a power of 2
  • sample_count: the number of samples
  • data_bits: number of bits that's used in the data block
  • encoding: the encoding of the data block; two complement, unsigned integer, double or zero-terminated string

Basically samples64 says something about how much samples are contained in 64 bits (eight bytes), and this can be represented graphically in the way below:

CCSDSdata bits.png

The above fields are pretty much explained now, except that the data_bits explanation is rather terse. The thing is, we software engineers round off everything to one byte. But in the real world, that's nonsense. For example, measuring a voltage will result in, say, 12 bits. For a 12-bits sample, we set the data_bits field to 12. Rounding off to bytes, this means that each sample will occupy two bytes. Thus, the samples64 field will be set to "4", meaning: four samples will fit in 64 bits.

The developer of our Python library said that the samples64 field implies, that the length of the Data field is always divisible by 8. Or in other words, that the Data field is created using 8-byte pieces. After discussion, it turns out that's not what the firmware does, in our projects.

His calculation of the length of the Data field was:

  Data field length (in bytes) = 8 * ( (sample_count + samples64 - 1) // samples64 )

(The double slash means: throw away the remainder)

The firmware would send a packet with a number of Data Header/Data fields. The first Data Header says contains 31390 samples, each 16-bits. The second Data Header contains 8760 samples. The above calculation would yield for the first Data Header:

  Data field length (in bytes) = 8 * ( (31390 + 4 - 1) // 4 )
  Data field length (in bytes) = 8 * 7848
  Data field length (in bytes) = 62784

And for the second Data Header:

  Data field length (in bytes) = 8 * ( (8760 + 4 - 1) // 4 )
  Data field length (in bytes) = 8 * 2190
  Data field length (in bytes) = 17520

The Perl library does things differently; its way of calculating the length of the Data field is:

  Data field length (in bytes) = sample_count * (data_bits / 8)

Thus for the first and second Data Headers, this looks as follows:

  Data field length (in bytes) = 31390 * (16 / 8)
  Data field length (in bytes) = 62780
  Data field length (in bytes) = 8760 * (16 / 8)
  Data field length (in bytes) = 17520

For this to work, the data_bits has to be rounded off to the nearest power of two. We don't have firmware that does otherwise, but in theory it's possible. This implies that the following combinations should be expected:

samples64 field data_bits field
1Between 64 and 33
2Between 32 and 17
4Between 16 and 9
8Between 8 and 5
16Either 4 or 3
322
641

We assume here, that the firmware packs the data as efficient as possible, of course. But that's not a guarantee. So the way to calculate the byte length of the Data field, should not include data_bits. It should be as simple as sample count times sample size (in bytes).

The sample size basically comes from the samples64 field:

samples/64 field sample size in bytes
18
24
42
81
160.5
320.25
640.125

Of course, now we're pushing the boundaries of other libraries and hardware. Computer and networking equipment dictates that the minimum transfer unit is the byte. Thus when the samples64 field is higher than 8, software must start ignoring the remainder of the Data field.

  Data field length (in bytes) = sample_count * sample_size_in_bytes
  Data field length (in bytes) = sample_count * ( 8 / samples64 )

Because we want to make sure that we're getting whole bytes, the result should be rounded up.

2011-10-27 Compact Google Calendar header

UPDATE

I have created an official extension, available in the Chrome Web Store:

Compact Calendar extension

Original post

The recent changes to Google Calendar are looking nice, but unfortunately the header and navigation take up far too much space when using my 13" MacBook. For a compact Google Calendar header in your Chrome browser, take the following steps:

  • Install the Live CSS Editor extension for Chrome
  • Go to the options, set them all to "Yes" and click Save
  • Go to your Google Calendar
  • Click the new CSS button that the extension provides
  • Add the following code
 #vr-nav { display:none; }
 #onegoogbar { display:none;}
 #vr-header { display:none; }

Done!

2011-10-11 OS X and Vim

Vim comes with OS X by default, but it doesn't come with X11 support. You can always install MacVim of course, but some of us prefer to use vim in the terminal. Unfortunately, because of the lack of X11 support, copying/pasting doesn't work correctly in OS X.

As an example, copy a paragraph of text from Safari, then paste it into your vim instance in the terminal. Chances are, if options such as AutoIndent are on, the paragraph will be rearranged.

On Linux, we have the + and * buffers for that. That way, a simple "+p command will paste the most recently copied text into the terminal version of vim.

To enable this on OS X, I added a hint to my Vim page.

2011-10-05 From iPhone to Android

This is part 8 of my series where I document going from an iPhone to Android 2.3, running on the Samsung Galaxy Gio.

Syncing, it turns out, is a royal pain in the ass with the combination Android and OS X. There's a whole bunch of sub-optimal and incomplete solutions out there. And that's assuming that your mail, calendar, contacts and notes/tasks are synced via your Google account.

Product Pros Cons
doubleTwistFree version available. Syncs music, video's and photos, wireless in the paid version Does not transcode music and resize photos to appropriate size for device. Does not sync (only download) photos.
iTuneMyWalkmanOpen source. Syncs music via iTunes, via a specially-named playlist. iTunes does transcoding if you care for that. Moves pictures to your harddisk, but doesn't sync them.Doesn't sync anything else than music (photos are copied). Thinks thumbnails are also pictures. Doesn't transfer cover art. Transferring takes a long time, a couple of Gb of music causes a twenty-minute "collecting data from iTunes" message.
PicasaSyncs photos wirelessly Forces you to use Picasa web albums

As far as I know, there is no one-stop solution on OS X, unless you use Samsung Kies on OS X, which only supports a limited number of Samsung phones.

2011-09-21 From iPhone to Android

This is part 7 of my series where I document going from an iPhone to Android 2.3, running on the Samsung Galaxy Gio.

From version 2.2, Android offers the HotSpot functionality. There's another way, namely using the phone as a 3G modem via USB. This is called tethering, and OS X supports this out-of-the-box, just follow the instructions on TodayOnTech.

Another thing I bumped into, is the lack of audiobook support. doubleTwist doesn't really sync this properly. So, you copy them manually, which is not really a big deal for me. However the default media player will list these as music, which it most definitely isn't. The Android blog has a solution for this, creating a zero-length .nomedia file in your SD-card its /Audiobooks directory. Clumsy, in my opinion. The mediaplayer could look at the ID3-tags of the MP3 files, and see that they're marked as 'spoken word' or 'audio book'.

Another missing thing of the stock music player is the lack of control from the lock screen. From a locked iPhone, you can doubleclick the home button and the player buttons will pop up if music is playing. There are a number of Android apps that simulate this, for instance bTunes, mixZing and Lithiumplayer.

Perhaps a solution is to use iTuneMyWalkman, if doubleTwist continues to bug me, then I'll try this solution.

Rooting

On the forums, there are powerusers who manage to fill up their phone with apps. Apparently, the Gio isn't meant for those people. If you're one of those, you want to be rooting your Samsung Gio. That way, you can delete some of Samsung's stock apps. It's a process similar to jailbreaking your iPhone, but since the Android phone manufacturers aren't as tight-assed, it's not really as invasive and/or complicated as the typical iPhone jailbreaking. See also the xda-developers forum for instructions on jailbreaking the Samsung Galaxy Gio.

2011-09-19 From iPhone to Android

This is part 6 of my series where I document going from an iPhone to Android 2.3, running on the Samsung Galaxy Gio.


Android and iOS both have home screens, but they really have a different purpose. In iOS, the home screen contains all apps. In Android, your apps are stored in the app drawer (usually the icon in the lower right corner). The Android home screen is much more flexible. It can contain a number of items, of which the most important are: widgets and shortcuts. Widgets are little apps that run on your home screen, showing the date/time, the weather or what have you. A shortcut is just a way of starting up an app via the home screen.

Thus on Android, you could easily have an empty home screen, if you don't mind starting apps with two taps (first go to the app drawer, then tap the app itself).

At first, the Android home screen is a bit confusing since there are so many possibilities compared to iOS. But there are many cool things here, such as the display of upcoming events in your calendar, right on the home screen:
CalWidget.

2011-09-15 From iPhone to Android

This is part 5 of my series where I document going from an iPhone to Android 2.3, running on the Samsung Galaxy Gio.

I have some problems with the stock music player. First, it'll show all my audiobooks right in between my music. This makes shuffle play worthless, and clutters the list of albums.

Another thing is skipping/pausing. Sometimes it happens that you want to skip a currently playing song when you're on the home screen or in some other app. The iPhone offers a convenient popup that offers basic player controls when you double-click the home button. I haven't found the equivalent under Android. It's certainly not built in, at least, if you don't count Android multitasking: hold down the home button and a popup appears with all running programs. You can now tap the music player and control it.

2011-09-13 From iPhone to Android

This is part 4 of my series where I document going from an iPhone to Android 2.3, running on the Samsung Galaxy Gio.

Another thing I found out; making screenshots is not something that's universal on all Android phones. It totally depends on the specific phone maker and model. My Gio supports it; holding the back button while pressing the home button will do the trick. For lots of other Android phones, there's no such thing.

Captive bugs

A particularly interesting thing about Android is, that there's an alternative in the Android Market for lots of standard apps. For texting and the calendar app, there are replacements.

That's necessary sometimes, because my Samsung Gio has the Captive "sauce" over the stock Android. There's something weird with the calendar app, where if you add an event, it's added by default to the local calendar. In other words, if you set up the phone to sync to your Google calendar, and you add an event, it doesn't show up on Google. How weird is that? Turns out this is an age-old issue, so after creating the event, you'll have to go back and change the calendar.

Accessories

A couple of days ago, I got a silicone case for the Gio, but it's not really to my liking so I ordered a hard case on eBay via the Totomagirl seller. This was surprisingly difficult to find. When iPhone cases are all over the place, some Android phones like the Gio clearly aren't that attractive to makers of accessories.

As opposed to the iPhone, it's really easy to swap out the battery of the Samsung Gio. So I also ordered a battery on DealExtreme (the Galaxy Ace and the Gio use the same batteries). The iPhone with its external batteries (or expensive add-on packs) is less than ideal in this regard.

2011-09-08 From iPhone to Android

This is part 3 of my series where I document going from an iPhone to Android 2.3, running on the Samsung Galaxy Gio.

Previous time, I charged the phone with a cheap charger in my car and at home, which resulted in touchpad unresponsiveness, only to be fixed with a reboot. I suspect the chargers and started using only the Samsung-supplied charger and my PC for juice. We'll see.

I've also started using doubleTwist (yeah, lower "d") for syncing Music from my MacBook to my Samsung Gio. It works fine, but I'm missing some iTunes features. A really useful feature for iPhones and iPods with less than 16 Gb is the option "Convert higher bitrate songs to 128 kbps AAC". doubleTwist currently doesn't support downconverting (also called transcoding) music to a lower bitrate.

As for apps, I haven't really missed anything between my iPhone and my Samsung Gio. All the biggies are here; Facebook, Kindle, Skype, an RSS reader which syncs with Google Reader, a number of SimpleNote clients; all there. There's a couple of extra apps that aren't available on iPhone such as the official Google Reader App.

What's nice as well is that Android apps seem to be able to control the phone, more so than on the iPhone platform. For instance, there are apps that can control the screen brightness with a very wide range. Useful stuff.

The Android Market for apps seems to attract developers who aren't in it for the money. A decent SSH client (tool for IT system administrators) for example is ConnectBot, open source and completely free. The equivalent in the iOS AppStore is TouchTerm, which costs $4. Other app markets for Android seem to be popping up as well; for instance there's the Amazon app store (only available for US residents, unfortunately).

2011-09-06 From iPhone to Android

This is part 2 of my series where I document going from an iPhone to Android 2.3, running on the Samsung Galaxy Gio.

Damnit. When I wake up this morning, something is wrong with the touchscreen. It seems to register touches, but the phone thinks you touched the screen somewhere else. I can't even turn off the phone, because although the power button is a real button at the side of the phone, it asks for confirmation on the screen. After a number of tries, I get it to power off.

That's not good, Samsung. The second day, I have to restart the Gio.

When I cycle to work, I start the media player and realize that there is no album art when you manually sync music to your phone. Which you are forced to do, because the OS X version of Kies doesn't support the Gio. Lifehacker tells me to take a look at DoubleTwist, which I may do later.

One of the weird things about Android is, that you have the home screen as well as the application screen. The home screen contains selected shortcuts to apps, if I can say so, while the application screen contains all apps. Why not just limit it to one screen? There must be some reason, but I haven't found it yet.

The Gio is a real nice, light phone. My iPhone 3G weighs 133 g (4.7 oz), and the Samsung Gio only weighs 102 g (3.6 oz). It's also 1.5mm thinner. Since I remembered dropping my iPhone in the first week after purchase, I ordered a cheap silicone casing on eBay store Cellapod.

Just before dinner, the phone warned me to recharge it, although I hadn't heavily used it. That's okay though, my iPhone 3G is very power hungry as well.

I did a little testdrive to see how the navigation is holding up (quite nicely) and charged the phone using my cheap USB car charger that I used for my iPhone as well. When I came home, the phone wouldn't react to screen touches, and when clicking the home button, a screenshot would be made.

Because the unresponsiveness this morning as well as this evening happened right after charging, I've come to the conclusion that this phone can't handle cheap chargers. I'm going to do some more testing, though.

2011-09-05 From iPhone to Android

Today, my iPhone 3G lost its reception in the middle of the city, and that wasn't the first time. Since this was a good time as any to experiment with Android, I figured I'd see what Google's ecosystem is doing, and I'm going to relate my experiences from my point of view: staunch GMail user, a heavy smartphone user, and one who is used to the Apple way of doing things. That means: a MacBook, iTunes and an iPhone.

Since I was limited on budget but still wanted the equivalent in specifications of an iPhone 3G, I went for the Samsung Galaxy Gio S5660:

http://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Samsung_Galaxy_Gio.JPG/450px-Samsung_Galaxy_Gio.JPG

Wikipedia has all the specs of this phone, but the gist is: an Android phone with a reasonable CPU, a nice 320×480 pixel display, a 3 megapixel cam and enough onboard memory as not to cause any trouble.

The Gio actually has a pretty bad history, as Koen Delvaux diligently blogs. Crashes related to WiFi usage, battery draining, et cetera. But since Samsung rolled out the new Android 2.3 version (nicknamed Gingerbread), these bugs are supposedly fixed.

The phone was advertised as having Android version 2.3, but when I asked the sales guy, it turns out it's still at 2.2 and you have to update it yourself.

Coming home, I discovered that updating the Gio is done through Samsung Kies, a program which runs on both OS X and Windows. Except the OS X version only supports four or five of Samsung's phones. Luckily, I have Parallels running on my Mac, so I can run Kies on a virtualized Windows. For the sake of Google, I repeat: if you have a Mac, you need Parallels in order to run Samsung Kies, because the version for OS X does not support the Galaxy Gio. Samsung Kies has a number of documented problems, for which MobileCowboys thoroughly provides workarounds. But perhaps I was lucky, because installing Kies and updating the phone to Android 2.3 worked fine.

Of course, this will mean a bit of a hassle in the future. Putting music on my iPhone is basically connecting the iPhone, starting iTunes on my Mac, then hit sync. Now I'll have to copy music manually to the phone.

After updating, I configured the phone with my GMail account and installed a bunch of apps (Kindle, Facebook, Skype, etc) that I already used on my iPhone. The configuration of GMail is flawless, even though I use a Google Apps account (I have had my own domain for years now, and mail is handled by GMail).

The iPod application of the iPhone is nice, and supports a sleep timer. I often fall asleep while listening to audiobooks. You need a separate app for this, and I was advised to take a look at MortPlayer. I haven't tried it yet.

2011-08-30 Creating specific password lists with John the Ripper

For security purposes, you sometimes want a dictionary with possible passwords. Perhaps you want to assess password strength in an in-house built application, or you want to test the security of the WPA wireless network.

There are plenty of dictionaries around to use to warn (or exclude) users from creating an account with such a password, or for the latter purpose, to brute force the WPA key.

After the standard dictionaries, a good next step is to create a password list for specific circumstances. For example; you are building an app for a company and want to make sure that easy passwords are excluded. You then need to build a list of passwords that any cracker would try.

There are good commercial products that can do this kind of stuff automatically, but for the poor man, it's easy enough to just get the stuff from the company website, or the Wikipedia article on them. With standard Unix/Linux tools, you're good to go.

I assume we create a directory in your home directory called "crack". First, use wget to get the source of words that shouldn't be a password:

 $ mkdir ~/crack
 $ cd ~/crack
 $ wget "http://nl.wikipedia.org/wiki/Rotterdam" -O wordlist.html

Then use sed to strip out the HTML:

 $ cat wordlist.html | sed -e :a -e 's/<[^>]*>//g;/</N;//ba' > wordlist.txt

Then edit the list with vi, and manually remove some JavaScript cruft at the bottom if necessary, then clean it up a bit.

First put each word on a separate line. The ^M indicates the Enter key. Create it by pressing CTRL+V, then pressing enter.

 :%s/ */^M/g

Remove extra whitespace:

 :%s/ *//g

Remove empty lines:

 :g/^$/d

Done. Now we're going to expand the wordlist with all sorts of word play that people might think of, like using years, or numbers, or using leet speak. John the Ripper by default has a nice set of mangling rules, but it's very limited.

Matt Weir has greatly enhanced the default set of mangling rules, and that's the one we'll be using.

Make sure you have a recent version of John the Ripper, because Matt's rules need it. Your Linux distribution will probably have an older version, so download and compile version 1.7.8 or something later. I'm assuming you'll put John in your home directory in opt, then add it to your path:

 $ mkdir ~/opt
 $ cd ~/opt
 $ wget http://www.openwall.com/john/g/john-1.7.8.tar.gz
 $ tar xfz john-1.7.8.tar.gz
 $ cd john-1.7.8/src
 $ make

(make now lists a number of targets, choose the best for your architecture and re-run make)

 $ make linux-x86-64

When finished, the executable and configuration file reside in $HOME/opt/john-1.7.8/run. Download Matt Weir's configuration file to this directory, but back up the original configuration first:

 $ cd ~/opt/john-1.7.8/run
 $ mv john.conf john.conf.bak
 $ wget http://sites.google.com/site/reusablesec/Home/john-the-ripper-files/john-the-ripper-sample-configs-1/john.conf?attredirects=0&d=1 -O john.conf

Since John the Ripper is used by us in WordList mode, we need to edit the configuration file so it uses the correct rules. Edit john.conf, and search for the line

 [List.rules:Wordlist]

(around line 335) and disable it by adding a few characters, for example

 [List.rules:WordlistXYZ]

Then find the following line:

 [List.rules:Modified_Single]

(around line 21), and change it to:

 [List.rules:Wordlist]

Now add John the Ripper to your path, by adding the following line to your ~/.bashrc:

 export PATH=$HOME/opt/john-1.7.8/run

Logout, and login again. Typing 'john' should print the version. Now mangle your newly tested word list. To make sure the new mangling rules are used, create a file called test.txt with only one line, for example 'rotterdam'. Then run john. It should look something like this:

 $ cd ~/crack
 $ vi test.txt
 $ john -w=test.txt --stdout --rules
 rotterdam
 rotter
 Rotterdam
 rotterdamrotterdam 
 ROTTERDAM
 rotte
 rotterdam1
 rotterdam2
 rotterdam3
 rotterdam4

(skipping a whole lot of lines)

 ROTtERDAm
 ROTTerdAm
 ROTTerDAm
 ROTTeRdAm
 ROTTeRDAm
 ROTTErdAm
 ROTTErDAm
 ROTTERdAm
 ROTTERDAm
 RoTTeRDaM
 Tpyyrtfs,
 Eirrwesan
 words: 4745  time: 0:00:00:00 100%  w/s: 39541  current: Eirrwesan
 $

As you can see, Matt's rules generate all sorts of permutations which you can then use in your software or brute forcing efforts. From one word, 4745 mutations have been created! If you see less (for example between 20 and 40), then the default rules have been used and there is something wrong with the configuration file. Perhaps john couldn't find it?

If this went okay, then re-run john on your custom wordlist:

 $ john -w=wordlist.txt --stdout --rules > wordlist_mangled.txt

Voilà, this resulting word list can now be used in your password strength assessment, or brute forcing efforts.

2011-08-18 Google Chrome on Debian Lenny

The latest packages of Google Chrome won't install anymore on Debian 5.0, nickname Lenny. This is the old stable release of Debian.

For the time being, I'm still using Lenny, so for everybody else, here is a link to the most recent release that still installed on Lenny:

Google Chrome

Note that this is somewhat of a security risk!

2011-06-20 Running yum on low end VPSes

Yum is a dog to run on older systems with small amounts of memory, or when you run a VPS (virtual private server) that does not have much memory. When running a yum update, you get errors such as memory alloc (8 bytes) returned NULL. Recommendations are:

  • Running yum clean all
  • Disabling plugins by using the --noplugins parameter

But ofthen that's not enough. A simple yum update can find a huge number of packages to update and nothing will help you getting the required amount of free memory. You can work around this problem by updating them one-by-one.

First generate a list of packages-to-be-updated, then dump them in a file.

 $ yum -q check-update | cut -f1 -d" " > list

Now use a loop that runs yum for each package listed in the file:

 $ for i in $(cat list); do sudo yum -y update "$i"; done

2011-06-17 Encrypting and decrypting files part 2

Some time ago, I gave a duo of one-liners that encrypts/decrypts files.

This wasn't good enough for normal usage, so here's a better version that takes care of directories as well by running tar on them.

  #!/bin/sh
  source="$1"
  if [ -d "$source" ]; then
      newfilename="$source".tar.encrypted
  else
      newfilename="$source".encrypted
  fi
  if [ -e "$newfilename" ]; then
      echo "ERROR: A filename with the name $newfilename already exists"
      exit 1
  fi
  if [ -d "$source" ]; then
      # Source is a directory, first tar it
      if [ -e "$source".tar ]; then
          echo "ERROR: A filename with the name $source.tar already exists"
          exit 1
      fi
      tar cf "$source".tar "$source"
      openssl des3 -salt -in "$source".tar -out "$newfilename"
      rm -f "$source".tar
  else
      openssl des3 -salt -in "$source" -out "$newfilename"
  fi

And its evil twin, decrypt:

 #!/bin/sh
  origfilename=$(basename "$1" .encrypted)
  if [ "$origfilename" == "$1" ]; then
      echo "ERROR: Encrypted files must have the .encrypted extension"
      exit 1
  fi
  if [ -e "$origfilename" ]; then
      echo "ERROR: A filename with the name $origfilename already exists"
      exit 1
  fi
  openssl des3 -d -salt -in "$1" -out "$origfilename"
  # Check if the output is tarred
  tarfilename="$origfilename"
  origfilename=$(basename "$origfilename" .tar)
  if [ "$origfilename" != "$tarfilename" ]; then
      # It was tarred
      if [ -e "$origfilename" ]; then
          echo "ERROR: A filename with the name $origfilename already exists"
          exit 1
      fi
      tar xf "$tarfilename"
      rm "$tarfilename"
  fi

Save them in your path somewhere and make them executable.

2011-05-19 App Store on OS X a disappointment so far

So far, the OS X AppStore couldn't be called 'wildly popular' since its inception on January this year. Regularly, I checked my installed apps for availability in the App Store, because it allows for such easy updating. Lo and behold, only fairly trivial apps are there, the following list is not available in the App Store:

  • Google software (Chrome, Sketchup)
  • Mozilla software (Firefox, Thunderbird)
  • Adobe software (Flash, Flash Builder CS5, Photoshop etc)
  • Microsoft software (Office, Messenger, Silverlight etc.)
  • OpenOffice
  • Microsoft Messenger
  • Microsoft Silverlight
  • Seashore (painting program)
  • Parallels
  • VLC
  • Skype
  • Calibre (an eBook converter)
  • XBench (a benchmark for OS X)
  • Vuze
  • KisMAC

Now I agree that stuff like a bittorrent client (Vuze) and a network sniffing tool (KisMAC) would probably be refused in the App Store. But all in all, the OS X App Store could be called a disappointment so far.

Note that the Opera browser (which contains a bittorrent client) is in the App Store.

2011-04-21 Not enough CPU time

The datarate of the Demux board of the Safari project is quite high for a desktop PC to work with, even considering the fact that the raw data is being thoroughly filtered before it reaches the PC running the Generic EGSE (the central part of the software, which allows the user to control electronics and plot output).

We've made some tweaks in the past to allow more breathing space, such as good optical GBit network adapters (we dumped cheap DLinks and bought some expensive Intel ones). A separate PC was installed to run a daemon, which concatenates the large amount of small ethernet packets into a couple of big CCSDS packets. We tweaked the kernel TCP buffers and the driver settings on both PCs.

The running problem with the EGSE was unresponsiveness. When retrieving packets and plotting them, the whole software stopped reacting to user input. The cause was that the plotting (and its computations) took a lot of time because computations run in the same thread as the plotting, and although the GUI gets time to accept user interaction at regular interval but this interval is too long. A solution is to allow the GUI to accept user interactions more often. We are testing this solution.

(What didn't help, was that several bugs popped up in the meantime. For instance, it's possible to create two plots for one packet type. What would happen, is that packets would get divided between plots, or packets would interrupt the plotting right after the first plot was drawn, but before the second plot was started. A colleague also found and fixed these.)

We brainstormed with the users, and one had experience with this type of situation. After talking, we had a proposal for a situation where the daemon would maintain a very large buffer in memory or on disk, and the EGSE would then lazily retrieve the buffer. Since the daemon runs on a separate PC, memory contention wouldn't be a problem.

safaridaemoncaching.png

We thought about this solution, but although it's a solution for other (older) programming models, we use the Qt library. The big idea of Qt is that it generates signals when an event happens, and you just write code (a "slot") that handles the signal. The Signal/Slot system of Qt provides an elegant and robust solution to manage a buffer which is filled in with packet coming from a daemon and is read lazily by the EGSE. We're testing this solution.

2011-04-20 Telfort ADSL and VoIP Planet

(English readers: this is an explanation on how to configure the modem of a Dutch ISP for VOIP usage).

Tegenwoordig wordt Telfort ADSL uitgeleverd met een Zyxel modem, type P-2601HN-F1. Het is vrij eenvoudig om via een SIP account bij VoIP Planet te bellen met een telefoontoestel dat je aansluit op het modem. Waarom niet gewoon via Telfort Vast Bellen? Omdat je misschien reeds klant van VoIP Planet bent, omdat het goedkoper is, en omdat de service vele malen beter is.

Om het modem te configureren, log in als admin op het modem via adres http://192.168.1.254/ en klik aan de linkerkant het mapje VoIP open, en klik op SIP. Vul het eerste tabblad als volgt in:

sip voipplanet1.png

Vul het tweede tabblad als volgt in, waarbij je natuurlijk de vermeldde SIP Account Number en Username verandert in diegene die je van VoIP Planet hebt gekregen:

sip voipplanet2.png

Klik op de Apply button en klik aan de linkerkant het mapje Maintenance open, en klik op Logs. Op de eerste regel moet nu staan:

  SIP Registration: SIP:12345: Register Success

Je kunt nu bellen met het aangesloten (analoge) telefoontoestel via VoIP Planet!

2011-04-08 Using the decorator pattern in Python

For my current project, we are trying to look into the future, where our Python scripts might be used on different electronics.

In order for this to work, our Python library for communication with our Generic EGSE software must easily be rewritten. These changes will not be big: communication with an electronics board will take place over a cable of some sorts. That cable and protocol aspect is abstracted away by a daemon, as part of the Generic EGSE.

http://upload.wikimedia.org/wikipedia/commons/f/fa/Altera_StratixIVGX_FPGA.jpg

However, what is not abstracted away, is the contents of the protocol. Our electronics board incorporates an FPGA which takes a certain address (say, address 42) to mean something such as a bias voltage. The electronics board(s) of our project partner might have entirely different addresses, or might require multiple addresses to be set.

Thus, we need the Python scripts to work with a different set of addresses in the future, and we need the flexibility to make exceptions for specific addresses. On the other hand, for now the library needs to work as-is.

The way to typically reach this goal, is to use a decorator pattern. I'm looking into the best way to do this in Python. Since I'm not (yet :-) a guru in the matter of Python, this'll be an interesting experiment.

(Photo by Altera Corporation)

2011-03-03 SAFARI software and hardware setup

Below is a basic setup of our software and electronics.

SAFARI setup software and electronics

We have two PCs, one with our Generic EGSE software and the other with a daemon (which collects data). To give you an idea, the Generic EGSE looks as follows:

EGSE Server.png

The daemon just receives data on an optical ethernet port in raw ethernet frames, then concatenates the packages and transfers them over TCP to the EGSE server.

The reason we use a separate PC for the daemon, is that the EGSE server cannot natively receive data over raw ethernet. Thus we use a daemon for that and the data rate is too high to run both daemon and server on one PC.

The demux is short for 'demultiplexer board', an electronics board:

demux.jpg

This board retrieves the science data from the sensor and then decimates it. There are two filters available for that. Still, the amount of data is considerable. An ethernet frame can contain 1500 bytes of data. Since one sample of the sensor is two bytes, we put 730 samples in one CCSDS packet so it fits into one ethernet frame. A good sampling should take 100,000 samples. This is rather a lot of packets so you'd say this could easily be fitted into jumbo frames. However, the demux board firmware doesn't support jumbo frames. So we concatenate the CCSDS packets on the PC with the ethernet daemon, then transfer them over TCP to the EGSE Server.

2011-03-03 From which host am I logged in

A previous entry documented on how to change your settings in bashrc, according to from which host you logged in over SSH.

The solution used the hostname entry from the "who"> command. Thing is, it didn't work well because the "who" command also outputs any X sessions.

Here is a script snippet that uses the "host" command. Be sure you install this (under Debian and derivatives available with package dns-utils).

  FROM_IP=$(echo $SSH_CLIENT | cut -f1 -d" " | grep -v ":0")
  FROM=$(host -t ptr $FROM_IP | cut -d" " -f5)
  case $FROM in
    myhostname*)
      # Enter your settings here
      set -o vi
      ;;
    otherhostname*)
      # Enter your settings here
      set -o vi
      ;;
    mylaptop*|worklaptop*)
      # Enter your settings here
      set -o vi
      ;;
  esac

The stars behind the hostnames are there to ignore the domain name. This is because FROM=... line will give you the hostname including domain name, for instance, "mylaptop.company.com."

If you want to strip off everything but the hostname, use something like

  FROM=$(host -t ptr $FROM_IP | cut -d" " -f5 | cut -d"." -f1)