The LiveScribe pulse pen has built in handwriting recognition software.  Unfortunately its not very good unless your pen application (penlet) gives it a lexicon.  With a lexicon, it works pretty well – however when my lexicon reached 50,000+ lines I found out about a currently undocumented feature, Ludef files.  Ludef files are a grammar file (instead of a flat list of terms), which suites chords perfectly.  My 50,000 lines dropped to under 50 lines.

Background on Chords

First, I am not a music expert.  I used to play keyboard a bit, and frequently cheated by playing the guitar chords instead of the real music.  But here is a bit of background for those with even less knowledge than myself.  The following may make anyone who knows about music cringe – I am trying to keep things simple for any non-music people, not describe it 100% correctly.

On a piano, there are 7 white keys “A” to “G”.  These make up an octave (the “oct” for 8 comes from including the starting note at the end as well).  There are sharps (“#”) and flats (something that looks like a “b”) which raise or lower the sound by a semitone.  A semitone is one note up or down on the piano, including the black keys.  “C#” for example is the black key to the right of the white key for note “C”.  “Cb” is actually just “B” as there is no black key on the piano between “B” and “C”.  People just write “B” instead of “Cb”.  “Db” is the same black key as “C#” – both names are used in practice.

Chords are three or more notes played together that sound good together.  The chord “C” involves playing “C”, “E”, and “G” together.  Chords can have more notes, and chords can have variations that make them sound different.  For example, the chord C-minor (“Cm” or “Cmin”) is “C”, “Eb”, “G”.  To make a normal chord into a minor chord you lower the second note of the chord by a semitone (“E” became “Eb”).  Minor chords sound more sorrowful.  There are lots of other types of chords – major, suspended, augmented, diminished, power, added, 7ths/9ths/etc.

On top of that, it is common to specify the inversion of a chord to use.  The chord “C” can actually be played as “C”, “E”, “G” or “E”, “G”, “C” (next octave up), or “G”, “C”, “E” (where C and E are next octave up).  To specify which inversion, the notation “C/E” is used.  This means the “C” chord where “E” is the lowest note played.  However, as a lazy keyboard player, I would play the easiest version of the “C” chord in the right hand and just play the lowest note in the left hand.  It sounds good.

Why all of the above?  Well, hopefully you can start to see there are rules to how chords are written.  You have a root note (“A” to “G”), you can have an optional accent (“#” or “b”), you can have various modifiers, and you can have an optional bass note (after a “/”).

Using a Lexicon

Back to writing a penlet.  The handwriting recognition software in the pen does its best, but is not ideal. For example, for my writing it often gets “G” and “9” mixed up. It also gets “A” and “#” mixed up at times as well.  A lexicon is a list of terms that are legal inputs, in my case chords (like “A”, “G#”, “Cm”, “Gsus4” etc).  If you supply a lexicon it gives the handwriting subsystem a hint as it knows what letters can appear where.  For example, chords always start with “A” to “G” (never a “#” or “9”).  After using a lexicon the recognition was pretty accurate.

If you want to develop an application with handwriting recognition, use the new project wizard in the LiveScribe Eclipse plugin.  It has an option to add handwriting recognition to the new project.  I picked the option to type in a subset of characters I wanted recognized.  I added “ABCDEFGabdgijmnosu#/123456789+-”.  As a zero never occurred, I left it out (avoiding confusion with “o”).  I also picked to specify a lexicon.

Well, there were a lot of variations – for example every chord could specify “/<note>” so I wrote a little program to generate all the combinations I wanted to support.  It might not have been perfect, but my first generation created a 16,000+ line file.  Next version with some more fixes was over 50,000 lines.  Egad!  It worked, but was a bit scary.

Gotcha!

When I started out trying the handwriting code with a lexicon, I was bitten by a known issue.  The pen internally converts the lexicon into some internal representation that it then caches.  Redeploying the modified lexicon has no effect unless you turn off and on the pen (which flushes the cache).  Thoroughly frustrating times may eventuate if you don’t know this minor detail – they sure did for me!  I would make changes that had no effect, then come back my next programming session (after pen auto-shutoff) and they would magically work!  Things went much more smoothly once I knew to power cycle the pen after each lexicon change.

Ludef Files

All this work with handwriting recognition was in part triggered by someone else on the forums having some troubles as well.  Maybe due to critical mass of questions, LiveScribe published a sample application using a Ludef file.  A Ludef file specifies a grammar rather than a list of legal values.  This is perfect for chords.  My Ludef file is below.  Note this is not a perfect representation of how chords can be written.  In the forums the sample code contains a different (more complex) grammar file.  I wrote my own because I wanted the grammar to match my code which parsed the writing afterwards.  (Warning, some line wrapping may have occurred.)

The grammar of a Ludef file is fairly clear.  Literal text is in parenthesis, rule names are referenced by putting them in braces, question mark means optional, vertical bar means alternatives, no operator means a sequence.  I am assuming readers are familiar with the concept of a grammar here so I am not going to go into details.

Enabling the Ludef file

To use the Ludef file, I created a project using the Eclipse wizard with a lexicon, then made the following changes.

• Comment out the line icrContext.createAppResource(“/icr/LEX_MyApp.res”).  This assumes your application was called MyApp of course.
• Add a line icrContext.createAppResource(“/icr/LUDEF_MyApp.res”).  This references the compiled MyApp Ludef file.
• In the src/icr directory rename the MyApp.lex file to MyApp.lex.IGNORE.  (You can delete it – I am just cautious and like being able to go back easily.)
• In the src/icr directory create a file MyApp.ludef and put your grammar into it.  The file extension is important.  Its how the build phase knows the file is a Ludef file.

And you are done!

Personally I have not noticed any difference in my application behaviour using a Ludef file or lexicon.  But this is to be expected.  There may be differences in file sizes or performance (I did not check).  What was important to me however was the length of the file (much easier to manage and inspect) and that I could now compare my source code parsing a chord as a string to the Ludef grammar.  Its much easier to make sure the two are in sync.

Having a go with a Ludef file is easy.  The wizard creates enough of a handwriting application to display recognized text on the pen display as it goes.  So its easy to give it a go yourself if you want to.  Of course, you have to get the LiveScribe Penlet SDK (registering as a developer).  But its free.  Oh, having a pen would be useful as well!!!

Conclusion

I don’t think the handwriting software built into the pen is that useful without a lexicon or Ludef file.  It just gets too many errors.  A lexicon file is great if you have a simple list of terms you want to match.  A list of English words for example works fine.  But if the input text matches a grammar, a Ludef file is probably more suitable.  The recognition rates goes up dramatically.  You do still get some errors, but not many.

By the time you see this article, its possible Ludef files are documented in more detail by LiveScribe.  Check out the LiveScribe forums at livescribe.com for more details.