A common thing for a LiveScribe pen penlet (pen application) to do is require the user to draw a button that they can later tap on.  Its sort of funky in a way – making the printed page “active”.  The standard “piano” application is one example of this (tap notes to play a tune) and the translator demo (write word in one language, then say word back in another language).  Oh, before you get to excited about the translator demo, it only knows ten or so words.

This post summarizes the main things you need to do when creating such “buttons” on a page.  Note that the LiveScribe provided “Getting Started Platform SDK” and “Developing Penlets” manual goes into technical details in some detail as well, and the “User Experience Guidelines” manual describes recommended user interaction models for consistency across applications.  The following is my overview.  I purposely have left out details to keep this post length down a bit.

Letting the user draw a button

First thing to do is to allow a user to draw a button.  An application may do this in different ways.

• It may require a user to select an item from a menu (navigated using the cross (up/down/left/right) navigation controls.  The user can then create buttons at any time and in any order.
• It may prompt the user upon start-up and ask for buttons to be drawn.  This will result in the application asking for buttons every time it is started (from a menu).
• It may use handwriting recognition to work out what the user has written and turn it into a button if recognized. 

It depends on your application which is most appropriate.  Personally I find the menu approach painful, although it appears the best choice if you have lots of buttons a user may want to write.  Using handwriting recognition is cute, but does not (easily) allow you to draw a box around the button afterwards.

I was playing with a little application for playing music chords.  You could write down the chord name, then tap it later.  E.g. “G”, “A#” (for A-sharp), “Bb” (for B-flat), etc.  More funky chords are possible such as minors, suspended fourths, major 7ths and so on.  The handwriting engine in the pen works *much* better when you tell it what input is legal.  Specifying a list of words is an easy approach to do this (a lexicon).  A separate blog post talks about using a grammar file instead, which is more appropriate for chords (you can say “a note followed by a modifier” without having to list all the possible combinations).  The point is however in this case it made good sense to use handwriting recognition, allowing the user to write a chord on the paper, which then became a button they could tap on later.

If the user decides when to enter a label (via a menu selection or when the application starts), then it is common to just capture the strokes written but not try to interpret them at all.  When the user pauses (or double taps on the page) that marks the end of the stroke input for the button.  You just trust the user to write down an appropriate label.

Capturing a button in code

To capture a button, a common approach is for the penlet’s main class to implement the StrokeListener interface, then register itself using context.addStrokeListener(this) in the activateApp() method and deregister itself using context.removeStrokeListener(this) in the deactiveApp() method.  The strokeCreated()  method (required by the StokeListener interface) is then called whenever a stroke is created on the page.

If you use handwriting recognition, then it is normal for the strokeCreated() method to pass the stroke to the handwriting engine via the addStroke() method of ICRContext.  If you penlet implements the HWRListener, then you implement hwrResult(), hwrUserPause(), hwrError(), and hwrCrossingOut().  Yes, ICR and HWR are really both used for the same concept.  hwrResult() is the important one – it reports the final result of the recognition engine of what you wrote.

So when strokeCreated() is called, how do you know which button it is for?  Well, its up to your application to know.  E.g. if a user selects a menu item, then you save that selection away in a class variable.  The strokeCreated() method then looks at the class variable to work out what it should do with the stroke.  See this blog post for some suggestions here.

To create a button you can tap on later, you have to create a “dynamic region” and register it with the pen.  It is common to use a bounding box of what was written on the page.  If you used handwriting recognition, you can use the ICRContext.getTextBoundingBox() method to get a bounding box.  Sometimes I create a new Rectangle object with a bit of extra padding around this bounding box to make it easier to tap.  If you just capture the strokes yourself, then you can ask the bounding box of a Stroke using Stroke.getBoundingBox().  (To get the stroke, you use StrokeStorage.getStroke(time) where the time is a parameter of the strokeCreated() method.)  If you want to capture multiple strokes (common), then you can build up a bounding box around all the strokes using Shape.getUnion(rect1, rect2) to merge the rectangles.

So how to work out when a button is finished?  Double taping is one approach (but beware of strokes if the user moves the pen fractionally when they tap on the page – you might not want them as part of your button!), and pausing is another approach.  If using the handwriting engine, it does the pausing code for you.  Otherwise you need to use the Timer class to trigger an alarm after the timeout has expired.  (You reset the alarm timer when strokeCreated() is called.)  I have another blog post talking about timers briefly.

Area IDs

Once you have your rectangle representing the button, you need to tell the pen about it.  You do this by calling context.getCurrentRegionCollection() to get a RegionCollection instance, on which you then call addRegion().  You pass this call the rectangle (or any other shape) for your button.  You also have to pass a Region object.  I find this class name a little confusing – it is not a shape but rather metadata about the shape of the region on the page to be made active.  In particular, you can specify an “area ID” which is a 16-bit unsigned integer.  When the user taps the button on the page, you get back the Region data structure.  (To be precise, the strokeCreated() method of StrokeListener and the penDown() and penUp() methods of PenTipListener are given the Region as an argument.)

So what do you use and Area ID for?  Well, it tells you what button was touched.  If you application has 5 different types of buttons, it could be set to 1 to 5 where each number represents a different button type.  (Note Area ID zero is reserved to mean “no region was touched”.  That is, its an unused area on the page so far.)  To chose what action to take, you can just do a switch on the area ID from the region.  (See Region.getAreaId().)

For my chords application, I had many possible combinations (due to the complexity of the grammar).  In the end I tweaked my grammar so there were around 50,000 legal combinations.  That way each one fitted into a 16-bit integer (64K).  Given a number I knew what chord (and modifier) was selected.  Note it was not perfect however.  I had to take short cuts when there were multiple ways to write the same chord.  So sometimes a user might have written “Asus4” but when the user tapped on the chord I displayed “Asus” (which is the same).  A trifle confusing when the user sees “Asus4” on the page.  (Providing feedback on the display I find useful to confirm what is being played, in case the handwriting recognition software got it wrong for example.)

Another approach I could have used was to use PenletStorage which provides a file system within the pen.  I could have allocated areaId’s one by one and create a new file for each chord written (for example).  Each file might contain the chord name and any other details about the chord I wanted to keep.  The filename might be the area ID.  (A single file could of course also be used with more sophisticated parsing code.  One file per areaId is just easier to explain.)  Another file might keep track of the next areaId to use.

There is nothing wrong with this last approach, but I personally try to avoid it as it opens up a whole new area of complexity (messing around with files) and associated bugs that can sneak in.  I like keeping things simpler.

It may be obvious, but you can of course split up your range of area IDs to be a mix of the above.  For example, IDs 1 to 99 might be reserved for commands, 100 to 999 for files, and 1000 onwards using some form of static allocation.  Whatever makes most sense for your application.

Summary

Creating a button (dynamic region) on a page is pretty easy.  Bounding boxes are a common way to make the region sensitive.  You need to put a little thought into how you are going to allocate area IDs for your buttons if you use them.  Hopefully the above is useful to someone.  All of the information is in the SDK documentation, but sometimes hearing the same information in different forms can help.