This is the first of a series of posts related to a new application in JavaFX I'm developing with the initial intention of interacting with a Lego® Mindstorms® NXT.
The first purpose of this app is, as a preliminary proof of concept, to show how easy is to integrate different software and hardware technologies. For that I'll start by introducing you briefly to the following topics:
- Lego Mindstorms NXT: computational module, with graphical programming environment, initially intended for kids to play with robotics.
- leJOS, firmware to replace the NXT one, includes a tiny Java Virtual Machine, and allows you to programm the robot with Java.
- Glassfish, application server to run Java apps and RESTful web services, implemented using HTTP and the principles of REST.
- JavaFX, the new Java platform for creating and delivering rich Internet applications.
Please notice I don't consider myself in any way an expert in any of the topics I'm covering here. Quite on the contrary, this is an exercise of traversal and shallow engineering.
Before I begin, it's fair I mention the guy whose idea I've taken to start with all of this: Sébastien Stormacq @Sebtso. His entry blog Put a Java Brain in your Lego Robot, from 2010, opened my eyes, with his code I didn't need to start from the scratch. Also I want to thank the team members of leJOS community. They keep vey much alive this fantastic project.
1. The schematics
For starters, let's see a simple overview of the intended project and the technologies involved.
As I said before, there are plenty of blogs out there describing in full detail what it is or what you can do with it, so just for the very beginners here it's in short the main specs.
Lego Mindstorms NXT is a programmable robotics kit released by Lego in 2006. The NXT 2.0 was released in 2009. The kit includes 3 servo motors and 4 sensors (ultrasonic, 2 touch and color), 7 connection cables, a USB interface cable, and the NXT Intelligent Brick, which is the brain of a Mindstorms machine.
 |  |
The brick is the computational module:
- Based on a 32-bit Atmel AT91SAM7S256 ARM7TDMI microcontroller (ARMv4) with 64 kB RAM, 256 kB flash and 48 MHz.
- Attached to it a 8-bit Atmel AVR ATmega48 controller via I2C interface, 512 Bytes RAM, used to collect data from sensors and control motors.
- With one USB 2.0 port, Bluetooth support and a 100x64 pixels black and white display.
Let me say here that, relatively speaking, the brick architecture could be compared to a quite underrated Raspberry Pi (ARMv6 ARM1176JZF-S 32-bit, 512 MB RAM, 700 MHz) with and Arduino (ATMega328, 8-bit, 2 kB RAM) attached.
On the software side, bundled with the kit comes NXT-G, a graphical programming environment that enables the creation and downloading of programs to the NXT. It is powered by LabVIEW from National Instruments. These programs make use of the standard firmware provided by Lego.
This is intended for kids to start programming, very much like MIT's Scratch. It's open to new blocks related to sensors from third-party providers like HiTechnic or Dexter Industries.
While NXT-G provides a very user friendly approach to programming NXT robots, for more advanced programming with a text-based language, there is plenty of unofficial languages for the NXT out there. Three of the most popular of them are NXC/NBC, RobotC and LeJOS NXJ.
3. LeJOS NXJ
Well, why I've choosen LeJOS NXJ in the first place, a Java based language, instead of the others C based languages it should be clear to a regular reader of this blog.
You can find a complete description of leJOS, the firmware replacement for the official one, and leJOS NXT, the language, here. You can find a full tutorial here.
Download the latest version (at the time of this post, 0.9.1beta-3) here. Choose Windows (*_win32_setup.exe) or Linux and Mac OS X version (*.tar.gz).
In short, the firmware includes a tiny Java virtual machine, which allows Lego Mindstorms robots to be programmed with Java. So, first of all, you have to flash this firmware.
Note you'll lose every file stored in the NXT. This process is reversible, and you can restore Lego official firmware afterwards.
To flash the NXT, please follow the detailed Getting Started instructions here according to your operative system. At the end, you'll have your NXT with a brand new leJOS logo and a main menu, quite similar to the official one. Here you'll find detailed instructions of how to use it.
If you want to use bluetooth for communication, enter in the submenu, select Power On, Visibility On, and enter a 4 digit pin (1234 by default). In your PC, search the device and connect to it. In Windows (my case) this is quite simple and it works smoothly.
On the other side, on your PC you'll find in the installation path a bin folder with a bunch of Swing based PC GUI tools, which are batch files like nxjbrowse, to list the files in your NXT, the nxjflashg, to flash the NXT or nxjcontrol, that gives you full control of your NXT:
You can either write Java programs that run on the NXT or run on the PC and control the NXT remotely. You can use any Java IDE or just command line tools.
In the installation path you'll find the samples.zip file, which contains samples to download and execute on the NXT (samples folder) or to run them on your PC (pcsamples folder). Ant build scripts are provided.
Running samples on the NXT
The samples require classes.jar in the classpath. When you run the project from the IDE, a dialog ask you to enter the name of the sample you want to download and run. The view sample, for instance, shows a menu in the NXT LCD and you can read sensors or control motors with the buttons.
package org.lejos.sample.view;
import lejos.nxt.Battery;
import lejos.nxt.Button;
import lejos.nxt.LCD;
import lejos.util.TextMenu;
public class View {
public static void main (String[] aArg) throws Exception
{
String[] viewItems = {"System", "Sensors", "Motors", "Exit"};
TextMenu main = new TextMenu(viewItems, 1, "View Example");
...
int selection;
for(;;)
{
LCD.clear();
selection = main.select();
if (selection == 0) // System Info
{
LCD.clear();
LCD.drawString(sys, 0, 0);
LCD.drawString(batt, 0, 2);
LCD.drawInt(Battery.getVoltageMilliVolt(), 4, 10, 2);
LCD.drawString(tot, 0, 3);
LCD.drawInt((int)(Runtime.getRuntime().totalMemory()), 5, 10, 3);
LCD.drawString(free, 0, 4);
LCD.drawInt((int)(Runtime.getRuntime().freeMemory()), 5, 10,4);
LCD.refresh();
Button.ESCAPE.waitForPressAndRelease();
}
...
}
}
}
As you can see, it's pretty straightforward.Running on your PC with remote control of the NXT
In this case, you must add the jar files from lib/pc, pccomm.jar and pctools.jar, and from lib/pc/3rdparty bluecove.jar. But you must not add classes.jar.
Here it's one of the samples:
package org.lejos.pcsample.tachocount;
import lejos.nxt.Motor;
import lejos.nxt.Sound;
import lejos.nxt.remote.NXTCommand;
import lejos.pc.comm.NXTComm;
import lejos.pc.comm.NXTCommLogListener;
import lejos.pc.comm.NXTCommandConnector;
import lejos.pc.comm.NXTConnector;
/**
* @author Lawrie Griffiths and Brian Bagnall
*/
public class TachoCount {
public static void main(String [] args) throws Exception {
NXTConnector conn = new NXTConnector();
conn.addLogListener(new NXTCommLogListener() {
public void logEvent(String message) {
System.out.println(message);
}
public void logEvent(Throwable throwable) {
System.err.println(throwable.getMessage());
}
});
if(!conn.connectTo("btspp://NXT", NXTComm.LCP)) {
System.err.println("Failed to connect");
System.exit(1);
}
NXTCommandConnector.setNXTCommand(new NXTCommand(conn.getNXTComm()));
System.out.println("Tachometer A: " + Motor.A.getTachoCount());
System.out.println("Tachometer C: " + Motor.C.getTachoCount());
Motor.A.rotate(5000);
Motor.C.rotate(-5000);
System.out.println("Tachometer A: " + Motor.A.getTachoCount());
System.out.println("Tachometer C: " + Motor.C.getTachoCount());
conn.close();
}
}
Notice that the NXTConnector object uses Lego Mindstorms NXT Communications Protocol LCP to establish communication between PC and the NXT either via bluetooth or via USB. As valid URL for your specific NXT, you can set "btspp://<NXT name>", "btspp://<NXT address>", "usb://<NXT name>" or "usb://<NXT address>". To coonect to any device just use "btspp://" or "usb://".
Look at the other samples, to find out the rest of the possibilities.
4. The Glassfish server
With the ability to communicate to the NXT from a PC, a further step is the installation of a server on the PC to allow the connection of several devices simultaneously to the same NXT(s).
So I'll try to develop an application, deployed in a Glassfish application server, that will respond to REST requests from these devices, by establishing a synchronized connection with the NXT.
The LegoNXT class
Continuing the previous work of Sébastien Stormacq, based on a singleton object to guarantee one only access to the NXT, regardless the number of users trying to access to it, I'll add a blocking key, so only one request is processed at a time. We must keep in mind that we are targeting a very slow processor. I also add another key to know if the NXT is connected or not.
Here you can see part of the code:
public class LegoNXT {
private static LegoNXT singleton = null;
private static boolean legoBlocked=false;
private static boolean legoConnected=false;
public static LegoNXT getSingleton() {
final Object lock = LegoNXT.class;
if (singleton == null) {
synchronized(lock) {
singleton = new LegoNXT();
}
}
return singleton;
}
private LegoNXT() {}
public void connect(String NXTName, String Protocol) {
final String connectString = Protocol.concat("://").concat(NXTName);
legoBlocked=true;
legoConnected=false;
try {
Logger.getLogger(getClass().getName()).log(Level.INFO, "Initializing communication with NXT");
NXTConnector cmd = new NXTConnector();
cmd.addLogListener(new NXTCommLogListener() {
@Override public void logEvent(String message) {
Logger.getLogger(getClass().getName()).log(Level.INFO, message);
}
@Override public void logEvent(Throwable throwable) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, "Exception while talking to NXT brick", throwable);
}
});
if(!cmd.connectTo(connectString, NXTComm.LCP)) {
Logger.getLogger(getClass().getName()).log(Level.WARNING, "Failed to connect");
legoBlocked=false;
return;
}
NXTCommandConnector.setNXTCommand(new NXTCommand(cmd.getNXTComm()));
Logger.getLogger(getClass().getName()).log(Level.FINE, "Done");
legoConnected=true;
} catch (Exception e) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, "Exception while initializing connection", e);
legoBlocked=false;
}
}
public void disconnect() {
try {
NXTCommandConnector.close();
} catch (IOException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
legoConnected=false;
legoBlocked=false;
}
public boolean isLegoBlocked() { return legoBlocked; }
public boolean isLegoConnected(){ return legoConnected; }
...
/* NXT methods */
The RESTful web services
The first RESTful web service will search for any NXT device connected to the server.
@Path("search")
public class SearchResource {
@GET
@Produces("application/json")
public String getJson() {
NXTConnectionManager man= new NXTConnectionManager();
NXTInfo[] search = man.search();
JSONObject json = null;
try{
JSONArray arr=new JSONArray();
for(NXTInfo info:search){
JSONObject element = new JSONObject();
element.put(CData.JSON_RESPONSE_DEVICENAME, info.name);
element.put(CData.JSON_RESPONSE_DEVICEADDRESS, info.deviceAddress);
if(info.protocol==NXTCommFactory.USB){
element.put(CData.JSON_RESPONSE_PROTOCOL, "USB");
} else if(info.protocol==NXTCommFactory.BLUETOOTH){
element.put(CData.JSON_RESPONSE_PROTOCOL, "BLUETOOTH");
}
arr.put(element);
}
json = new JSONObject();
json.put(CData.JSON_RESPONSE_DEVICES, arr);
} catch(JSONException jse){
return "{}";
}
return json.toString();
}
}
So when you start the Glassfish server, from any browser you can search for NXT devices connected, and it will answer with a JSON array. In my case, with just one device:
@Path("device")
public class DeviceResource {
private LegoNXT nxt = null;
@GET
@Produces("application/json")
public String getJson(@QueryParam("NXTName") @DefaultValue("NXT") String NXTName,
@QueryParam("Protocol") @DefaultValue("usb") String Protocol) {
String result = null;
if (nxt == null) {
nxt = LegoNXT.getSingleton();
}
int cont=0;
while(nxt.isLegoBlocked()){
try{
Thread.sleep(100);
cont+=1;
} catch(InterruptedException ie) {}
if(cont>CData.TIME_OUT){
String msg = "Device was blocked";
Logger.getLogger(getClass().getName()).log(Level.WARNING, msg);
result = "{ \"status\" : \"" + msg + "\" }";
nxt = null;
return result;
}
}
try {
nxt.connect(NXTName,Protocol);
result = new JSONStringer().object()
.key(CData.JSON_RESPONSE_DEVICENAME)
.value(nxt.getDeviceName())
.key(CData.JSON_RESPONSE_BATTERYLEVEL)
.value((double)nxt.getDeviceBateryLevel())
.key(CData.JSON_RESPONSE_BLUETOOTHADDRESS)
.value(nxt.getDeviceBluetoothAddress())
.key(CData.JSON_RESPONSE_FIRMWAREVERSION)
.value(nxt.getDeviceFirmwareVersion())
.key(CData.JSON_RESPONSE_VOLUME)
.value(nxt.getVolume())
.key(CData.JSON_RESPONSE_STATUS)
.value("OK")
.key(CData.JSON_RESPONSE_TIME)
.value(((double)cont)/10d)
.endObject().toString();
nxt.disconnect();
} catch (Exception ex) {
String msg = "Can not get device information";
Logger.getLogger(getClass().getName()).log(Level.WARNING, msg, ex);
result = "{ \"status\" : \"" + msg + "\" }";
nxt = null;
}
return result;
}
}
Several other RESTful web services are also added to the server in this way.
EDITED on the 1st of February of 2013: By request I've uploaded the server code to my GitHub repo, so if you're interested, grab the code here and give it a try!
EDITED on the 1st of February of 2013: By request I've uploaded the server code to my GitHub repo, so if you're interested, grab the code here and give it a try!
5. The client requests
Those RESTful web services must be called from the client side, and the returned json string must be processed to extract the contained information. For that I'll use the GSON open source Java library, which allows deserialization of collections or nested classes. So I just need one POJO class with the properties expected to be read from the json string for one device, EntSearch, and one POJO class with an arraylist of the first one, EntDeviceSearch, to collect all NXT devices found and their properties:
public class EntSearch {
private String devicename;
private String deviceaddress;
private String deviceprotocol;
private int batterylevel;
private String firmwareversion;
private String status;
private int volume;
private double time;
/* getters and setters */
}
public class EntDeviceSearch {
private ArrayList<entsearch> devices = null;
/* getters and setters */
}
So the request to the server to search for NXT devices, and the deserialization of the response, can be done very easily:
public void search() {
URL theJsonUrl=null;
try {
theJsonUrl = new URL("http://localhost:8080/WebLego/NXT/search");
} catch (MalformedURLException ex) {
System.out.println("URL Error: " + ex.getMessage());
return;
}
String jSonTxt="";
try {
InputStream in=theJsonUrl.openStream();
jSonTxt=IOUtils.toString(in);
} catch (IOException ex) {
System.out.println("URL Error: " +theJsonUrl+" "+ ex.getMessage());
return;
}
EntDeviceSearch devices=null;
try{
devices=new Gson().fromJson(jSonTxt,EntDeviceSearch.class);
} catch(JsonSyntaxException jse){
System.out.println("GSON Error: "+jse.getMessage());
return;
}
if(devices!=null){
System.out.println("Found Devices: "+devices.getDevices().size());
for(EntSearch ent: devices.getDevices()){
... // process each entry accordingly
}
}
else{
System.out.println("Search Error");
}
}
6. The JavaFX UI
Finally we reach the User Interface part. Now we only need to desing a fancy scene and connect it to the server via the RESTful web services. As this is a proof of concept, well be adding just a few controls to the scene, in this first version.
I'll use a Tab Pane with two tabs. The first one, labeled Communication, will have a TableView to list all NXT devices found, just by calling the search RESTful web service. Once a row is selected, the other service (device) is called to fetch the data from the picked device.
You should know by now that there're quite fancy custom JavaFX controls in jfxtras.org. Just drop the last available jar (at the time of this writting, jfxtras-labs-2.2-r5-SNAPSHOT.jar) to your project. To show the data values, a Battery, several Leds and an Indicator from JFXtras project will be used.
This is how it looks like the Communication tab, for now:
I should explain here the model behind this application, as it connects the UI part (FXML, the controller, the controls) with the REST requests from the client to the server part. There're a few tricky things here, as the controls values (JavaFX thread) are bound to the readings of the REST responses (standard Java thread).
But due to the clearly exceded length of this post, I must stop here, and leave it to a second part... So let me just add a short video showing what we've got of the NXTLegoFX application so far:
See you in the (hopefully coming soon) next part of this post! Love to hear any comments from you in the meantime.
Bluetooth works on Mac, you have to restart the NXT right before pairing, though. And it succeeds only when it originates from the Mac. Once paired you get two devices in /dev: A TTY and another one which you should definitely avoid as it is able to crash your Mac. The TTY is okay, though and works as a serial console to your NXT. The moment you access this TTY (e.g. using cat) the Mac connects to the NXT automatically. I was not able to upload something to the brick via Bluetooth, though. I always had to use USB for that.
ReplyDelete