a drop in the digital ocean

Updating the Unity_SerialPort Script

2022-02-21T00:00:00+00:00

I recently made a few updates to the old Unity_SerialPort project I initially put together in 2014. The updates include: compatibility with Unity 2021, reintroduction of the previously removed threading based update loop, inclusion of additional port configuration and finally some minor changes to the arduino code so that it also complies and runs on an ESP32.

This post is a TLDR look at the project and how it’s components can be used in your own projects. As always the source-code is available here.

The Example Unity Project

Figure 1: The Opened Unity Project

The unity project demonstrates the use of serial communication between a Unity application and a micro-controller such as an Arduino. Communication is achieved via use of a configurable com port.

Running the Project

Once you have downloaded and opened the project you should be greeted with a project window similar to that shown in Figure 1. If not please ensure that you have opened “SampleScene” and have also selected the “UnitySerialPort” gameObject. The latter can be found within the hierarchy window.

By selecting the gameObject you will be able to see all the configuration properties and options within the inspector window (Figure: 2). The properties can be modified to match the requirements of the port you wish to open.

Figure 2: The Inspector Window

Assuming that you have already deployed the accompanying Arduino script to a compatible micro-controller; modify the COM port property to match that of your attached micro-controller.

You can then hit play! :)

Click the Com Button (Open-Port) to initialise the connection and open the port. A successful connection to the micro-controller will be indicated by the message: “The serialport: COM(N) is now open!” appearing in the Port Status field (bottom left). If you wish for the open port call to be attempted upon application start; rather than by having to click the button. This can be achieved via toggling the “Open Port On Start” option within the inspector.

A Few Example Calls

The following are a few sample calls that demonstrate the operation/functionality of the UnitySerialPort. In order to trigger the examples via key-press, you must first ensure that the custom inputs are defined via:

Edit > Project Settings > Input.

For reference, I have mapped “Key1” to keyboard key 7, “Key2” to keyboard key 8 and “Key3” to keyboard key 9. The command “SendData” is also mapped to the SPACE keyboard key.

Code 1, shows the code used to trigger the commands. It can be located within the Update() method of GUIManager.cs. An equivalent code block can also be found within the Update() method of UnitySerialPort.cs.

The later is commented out and has solely been included to demonstrate that calls can also be made from that location.

if (Input.GetButtonDown("SendData"))
{ unitySerialPort.SendSerialDataAsLine(OutputString.text); }

// Example of sending key 1 press event to arduino.
// The "A,1" string will call functionA and pass a
// char value of 1
if (Input.GetButtonDown("Key1"))
{ unitySerialPort.SendSerialDataAsLine("A,1"); }

// Example of sending key 1 press event to arduino.
// The "A,2" string will call functionA and pass a
// char value of 2
if (Input.GetButtonDown("Key2"))
{ unitySerialPort.SendSerialDataAsLine("B,1"); }

// Example of sending space press event to arduino
if (Input.GetButtonDown("Key3"))
{ unitySerialPort.SendSerialDataAsLine(""); }

Code 1: Example Event Initialization

The highlighted commands can also be sent via use of the example GUI. To do this just enter a command (e.g. “B,1”) into the “InputField” and then click the “Snd Button” (Send Data). This results in a call to the method GUIManager.SendSerialDataAsLine(data). In this instance the data argument will equate to “B,1”. This method in-turn calls the SendSerialDataAsLine(data) command within UnitySerialPort.cs and the data is sent.

Visual indication of the send is provided by an updated message withing Port Status field, bottom left.

Incoming data is visible via both the “Raw Data” and “Evt Data” fields. The former shows the raw string received by the port. The later, the same data parsed into chunks and stored in an array separated via the “Separator” property. In this example it is a “,” character.

More information on the Arduino side of things can be found via both the original unity3d-serialport-script post and also my simple-serial-string-parsing post.

Using the SerialPortScript in Your Own Projects

The SerialPort Script has been implemented so that all incoming data is made available to other scripts via the registration of five events. The sending of data is handled via the use of static public methods. This means that you do not need to modify UnitySerialPort.cs to be able to use it.

To use the script drop the prefab (or even the script onto a gameobject) into your scene. You then need to create an accompanying script both receive the event notifications; and to send data via calls. The included GUIManager.cs is an example of such a script. It is important to note that input data can be obtained solely via the registration of just the “SerialDataParseEvent”.

The following list details all of the events which can be registered to:

SerialPortOpenEvent
SerialPortCloseEvent
SerialPortSentDataEvent
SerialPortSentLineDataEvent
SerialDataParseEvent

The other events have been included in order to further aid you in developing front end applications and GUI’s more easily and efficiently. The included GUIManager.cs script demonstrates this capability. In addition to the events UnitySerialPort.cs also contains several methods which are publicly accessible:

OpenSerialPort()
CloseSerialPort()
PopulateComPorts()
UpdateComPort()
SendSerialData()
SendSerialDataAsLine()

These are pretty self explanatory but the following section details their use in conjunction with the aforementioned events. For further clarification take a look at the GUIManager.cs scripts and follow each call.

How to register to an event

Code: 2 shows an example of the code required to register to an event for notifications. In this case it is for the “SerialDataParseEvent” event.

void Start()
{
    UnitySerialPort.SerialDataParseEvent += 
        UnitySerialPort_SerialDataParseEvent;
}

private void UnitySerialPort_SerialDataParseEvent(string[] data, string rawData)
{

}

Code 2: Example Event Initialization

In this example UnitySerialPort_SerialDataParseEvent is the method which will be called each time the event is triggered.

if (SerialDataParseEvent != null)
    SerialDataParseEvent -= UnitySerialPort_SerialDataParseEvent;

Code 3: Example Event Destruction :)

You will also need to perform some cleanup when you destroy the object the event is registered to. this can easily be achieved via placing the following code (3) within the scripts OnDestroy() method.

SerialPortOpenEvent & SerialPortCloseEvent

The “SerialPortOpenEvent” event is triggered each time that a serialport is opened via the use of the UnitySerialPort.cs OpenSerialPort() method. In the case of the example project this call is made either by clicking on the GUI’s “Com Button” button when it is in the “Open-Port” state. Once opened, the “Com Button” is then toggled to the “Close-Port” state. It can then be clicked again to close the port.

Closure of the port will call the UnitySerialPort.cs CloseSerialPort() method. This in turn fires the “SerialPortCloseEvent”. Each event can be used by scripts to indicate when the port has been opened or closed respectively. The code used to toggle the button can be found via the GUIManager.OpenClosePort() method.

In addition to the aforementioned button call(s); the UnitySerialPort.cs OpenSerialPort() method can also be automatically called via the UnitySerialPort.cs Start() method. This is achieved by setting the “OpenPortOnStart” property to true. This should only be used when you have finalised your settings for the port; and dependant upon user-case.

The SerialDataParseEvent

The “SerialDataParseEvent” event is triggered each time that data is read by the UnitySerialPort.cs GenericSerialLoop() method. Once the data has been read, the loop performs some rudimentary parsing by splitting the data into chunks; each defined via the user defined “Separator” property. If a separator is not found within the string of raw data then it is left as a single chunk.

The data is then stored into two public properties (RawData and ChunkData) which can be accessed cross thread[1] via other scripts. In addition the data is also made available as the arguments of the SerialDataParseEvent. The event is fired following the property assignment.

SerialPortSentDataEvent

The “SerialPortSentDataEvent” event is triggered each time that data is sent via the UnitySerialPort.cs “SendSerialData” method. This method sends data exactly as provided with no line ending via the SerialPort.Write(data) method

SerialPortSentLineDataEvent

The “SerialPortSentLineDataEvent” event is triggered each time that data is sent via the UnitySerialPort.cs “SendSerialDataAsLine” method. This method sends data exactly as provided with the addition of line ending via the SerialPort.WriteLine(data) method.

Reading Data

I have also recently added the ability for data to either be read in by line; that is separated by a new line character. Or as a chunk separated by a custom delimiter which can be set by the user via the inspector. The reads utilise the SerialPort.ReadLine() and SerialPort.ReadTo() methods of the SerialPort class.

switch (ReadDataMethod)
{
    case ReadMethod.ReadLine:
        rData = SerialPort.ReadLine();
        break;
    case ReadMethod.ReadToChar:
        rData = SerialPort.ReadTo(Delimiter);
        break;
}

Code 4: Data Read Methods

This means that data can now also be sent as a continuous stream which can be parsed into chunks of your own design; rather than solely as many individual lines. Ultimately it gives a little bit more flexibility for your own differing use case scenarios.

Example Data Output

Finally lets have a look at how we can use an event call to output/use the data. Code 5 shows how we can set the value of GUI Text from within the update method.

if (RawDataGUI != null)
    RawDataGUI.text = "Raw: " + unitySerialPort.RawData;

Code 5: Show Data Via a GUI Text

In this instance RawDataGUI is a text field and unitySerialPort is a reference to the UnitySerialPort.cs class

The first thing to be aware of is that if we are using the threading method we need to use the event data to set the value of a local variable so that be accessed across threads. If you were to use it directly then you would cause an exception; this is because the unity API is only available from the main thread.

UnitySerialPort.cs and GUIManager.cs both contain commented examples to further demonstrate this. Have a go with uncommenting and commenting the example calls and you will see what is and isn’t possible.

// Create the array

ParsedEvtData = new int[data.Length];

// Create a string for GUI display

string values = string.Empty;

// Populate both the array and string using the event data

for(int i=0; i<data.Length; i++)
{
    // Convert the data to ints. These can be viewed
    // via the unity editor!

    ParsedEvtData[i] = int.Parse(data[i]);

    // add to the string
    values += i + ": " + data[i];

    // check if we are at the last value and if not add a new line
    if (i != data.Length - 1)
        values += "\n";

}

// Update the variable so the gui can call it up the update method
EvtDataString = values; // e.g.2

Code 6: Convert Data to Int Array

Finally code 6 shows an example on how to use the UnitySerialPort_SerialDataParseEvent event data by converting it to an array of ints which can then be applied. It also outputs each value to the GUI on a separate line so that they can be easily viewed. Again the display call is made within the Update() method of GUIManager.cs.

The Data Read Loop

In order to change method used to run read loop, all you need to do is select either threading or coroutine via the inspector panel. This can be found under options. Selecting threading will result in a port that reads in data on a separate thread; whilst selecting coroutine utilises the same thread spreading the task across several frames.

public enum LoopMethods
{ Threading, Coroutine }

Code 8: The Loop Methods

Due to the nature of coroutines being frame dependant; in some instances you may have a noticeable lag depending upon your application.

Port Configuration

The new port configuration options are applied either within the OpenSerialPort() method as part of the ports Initialisation; or just before the SerialPort.Open() command is called (see Code: 7).

// Initialise the serial port
SerialPort = new SerialPort(ComPort, BaudRate, Parity, DataBits, StopBits);

SerialPort.ReadTimeout = ReadTimeout;
SerialPort.WriteTimeout = WriteTimeout;

SerialPort.DtrEnable = DtrEnable;
SerialPort.RtsEnable = RtsEnable;

// Open the serial port
SerialPort.Open();

Code 7: OpenSerialPort() - Port Configuration

All of the properties are made available to be modified via the inspector. In theory you should generally only need to change the ComPort name and BaudRate values to get up and running. However the other options are now also available for completeness and to cater for those outlier use cases.

The Thread Loop

In both instances the data read is performed via a call to the GenericSerialLoop() method. The only difference is the loop method used to call it. For more information on the GenericSerialLoop() please see the Reading Data section.

Compatibility with Unity 2021

The last time I updated this project was way back in 2016. At that time the update was a recompile and a GUI update to make it compatible with then new GUI system. There have really been no changes to the original code since 2014 when I added events and removed the capability to loop serial reads via a separate thread. However, since 2016 unity has changed massively and the knock on result was need for the project to once again be upgraded. I hope you like the changes and find it/them useful! If you have any questions or feedback please get in touch via social media or through github issues (whilst im working on applying utterances to the blog!)

P.s. Well done for reading this far!

Just in case the related posts don’t pick them up; here are a few links to my previous posts on the topic that you may find useful:

Future plans for this project include implementing a job system approach in addition to the current Threading and Coroutine methods for running the read loop. However time will tell… hopefully it wont be another 8 years :)

A Circuit-python Library for the PAA5100JE

2022-02-15T00:00:00+00:00

Recently I have been playing around with several RP2040 based micro-controllers in order to investigate the RP2040 as a platform for an upcoming project. During my studies I have found that because the RP2040 itself is still fairly new; there is not that large an amount of compatible libraries for existing sensors and breakouts yet. The libraries do exist; just for other platforms such as Arduino and Raspberry PI.

Figure 1: The PAA5100EJ connected to a Feather RP2040

The PAA5100JE optical flow sensor is one such sensor; more importantly, its a sensor I also want to use for the aforementioned (top secret) project. With all this in mind; the only thing for it, was to have a go a porting an existing library over to circuit-python. I chose circuit-python because I want to utilise its HID capabilities further on in the projects development. This decision also had the knock on effect of highlighting the python library for the PAA5100EJ developed by Pimoroni as the best starting point for the project.

Luckily the port was successful and this post outlines how you can use the code in your own projects. It also highlights some of the discoveries and decisions that I made along the way so that I can refer to them in the future.

Prerequisites

Before you can use the code you will need to ensure you have several prerequisites. Assuming you already have a Circuit-python installation, you will also need to install the Adafruit_Circuit_BusDevice library. This library can be located via the previous link or installed via the following pip command:

pip3 install adafruit-circuitpython-busdevice

Command 1: Installation via pip3

Installing the code

The projects github repository contains two circuit-python files: code.py and pa55100ej.py. For those familiar with Arduino; the code.py file is the equivalent to an arduino sketch file in that it is where you put your main code that you want to run. Similarly pa55100ej.py is like an Arduino library called by the main code.py file. These are all you will need to get things up and running.

Figure 2: Installation setup and file structure

Once you have obtained the files you then need to place the file paa5100ej.py in your circuit-python drives designated library location; and also the file code.py in its required location. For the latter this is usually the root. For the former its usually a bespoke library directory also created at the root of the circuit-python drive. Figure: 2 shows my installation setup and file structure within the Thonny IDE. For reference: the lib directory contains the Adafruit_Circuit_BusDevice library library as detailed in prerequisites.

For my setup, I have also two neopixel libs included in the directory as I am using a Feather RP2040 board which has an on-board neopixel. This I annoyingly need to turn off at every boot. If you don’t have the same board, i.e. you are using a pi pico then you shouldn’t need these libraries.

Using the code

With the installation/setup complete; lets move on to putting together the circuit and also having a look at the code itself.

A look at the circuit

First up we connect the sensor to the micro-controller. As I am using the Feather RP2040 I use the connections shown in Code: 0. These will be similar for whichever RP2040 board you use. Just make sure to check your boards corresponding pin diagram.

3-5V - 3.3V
CS - D4
SCK - SCK
MOSI - MO
MISO - MI
INT - 
GND - GND

Code 0: The sensor to Feather RP2040 connections

With the connections made lets move onto having a look at the code.

A breakdown of code.py

The only code you really need to worry about in order to get things working in your own application is that found within code.py. To this end this section takes a look at the file in detail. So it can be adapted for your own needs.

First up we import all the libraries/modules that we will need. These are:

The board module; which contains constants for the pins on the specific board we are using. The busio module which contains classes to support a variety of serial protocols such as SPI and I2C. The digitalio module to provide access to the boards pins. The time module to allow for use of time and timing related functions. Finally we include the pa55100ej library to allow us to interface with the pa55100ej sensor using SPI.

import board
import busio
import digitalio
import time
import paa5100ej

Code 1: code.py module/library imports

Once we have imported all the required modules/libraries we then create a reference to the boards SPI hardware bus. This is then passed to the paa5100ej library where it is used to create a new SPIDevice using the SPIDevice Library.

spi = board.SPI()
cs = digitalio.DigitalInOut(board.D4)
cs.direction = digitalio.Direction.OUTPUT

Code 2: the SPI and CS definitions

In addition to the boards SPI configuration a custom chip select pin is also passed to the class. The CS pin is one which can be toggled to tell the chip that it should listen and/or respond to requests on the SPI bus. This toggle can be controlled manually by code; however by using the SPIDevice Library this is handled for us automatically. More Information on the library can be found here and here

Next we initialise and make a reference to the paa5100ej class. As we do so we pass to it both the aforementioned references to the SPI and CS pins. The class then utilises this information to create a new SPIDevice that handles the SPI interface and allows us to communicate with the sensor.

oflow = paa5100ej.PAA5100EJ(spi, cs)
oflow.set_led_state(True)
oflow.set_rotation(0)

Code 3: Initialising the PAA5100EJ

Finally we create two properties to store any movement outputted from sensor in both the x and y direction. These properties are updated each loop of the main class as a means of storing the cumulative movement detected by the sensor.

tx = 0
ty = 0

Code 4: The tx, ty properties

With our setup complete lets move onto the main loop. If we take a look at our main program (Code: 5) we can see that it consists of a loop which is kept alive via the use of a while true statement. On each iteration of the loop a call is made to the pa55100ej class via the get_motion() function. Any iteration where data is not available is accounted for via wrapping the call in a try block.

A caught exception causes the loop to continue. If data is received this is assigned to the x and y variables which are in turn used to update the values of tx and ty.

while True:
    
    try:
        x, y = oflow.get_motion()
    except RuntimeError:
        continue
        
    tx += x
    ty += y
    
    print("Motion: {:03d} {:03d} x: {:03d} y {:03d}".format(x, y, tx, ty))
    time.sleep(0.01)

Code 5: The update loop

A print command is then used to output the data to the console and/or serial. The command applies simple formatting so that each value is shown to 3 decimal places. Finally we then use a time.sleep() command to provide a pause and to slow things down for brevity.

A look pa55100ej.py

The pa55100ej.py is almost a direct port of the Pimoroni PMW3901/PAA5100EJ sensor library. The main changes are around the structure of both the data read and write commands. Within each of these I had to convert the spi_dev.xfer2() method to one that is available within circuit-python.

def _write(self, register, value):
    GPIO.output(self.spi_cs_gpio, 0)
    self.spi_dev.xfer2([register | 0x80, value])
    GPIO.output(self.spi_cs_gpio, 1)

Code 6: The original write method

For the write this was easily achieved by wrapping the register and value to be called correctly within a bytearray and passing it via spi.write(). The other key thing to note here when comparing the two code blocks (6,7) is the lack of commands required to toggle the CS pin within block 7. As indicated this is thanks to the SPIDevice library which is handling this for us.

def _write_to_reg(self, reg, val):
    with self.spi_device as spi:
        spi.write(bytearray([reg | 0x80, val]))

Code 6: The circuit-python write method

The read command was a little more difficult; my investigations resulted in two methods based around spi.readinto() and spi.write_readinto().

 def _read(self, register, length=1):
    result = []
    for x in range(length):
        GPIO.output(self.spi_cs_gpio, 0)
        value = self.spi_dev.xfer2([register + x, 0])
        GPIO.output(self.spi_cs_gpio, 1)
        result.append(value[1])

Code 7: The original read method

The latter is the one needed within the script to replace the original spi_dev.xfer2() call within the _read() method of Pimoroni code (7).

def _read(self, register, length=1):
    result = []
    with self.spi_device as spi:
        for x in range(length):
            val = bytearray(2)
            cmd = bytearray(2)
            cmd[0] = register+x
            cmd[1] = 0
            spi.write_readinto(cmd, val)
            result.append(val[1])

        if length == 1:
            return result[0]
        else:
            return result

Code 7: circuit-python read method

In order to get the code to function properly I had to break it down into parts. There is probably a more efficient means of doing this; but hey I’m new to the language. I must note that my inspiration/motivation for this came from this post over on the micropython.org forums. This little nuget of information was the glue that brought everything together.

def _register_read(self, register, length=1):
    with self.spi_device as spi:
        spi.write(bytearray([register]))
        result = bytearray(length)
        spi.readinto(result)
        return result

Code 8: circuit-python readinto() method

For completeness code: 8 shows the spi.readinto() method. This may be useful in the future when I once again attempt a similar undertaking. The method first performs a write to a specific register (register) and then reads into a bytearry (result) of a set length (length).

A quick look at the documentation suggests there is no need for a separate write command and this could be amended as shown in Code 9; however I haven’t tested this yet! TBH I missed it when I wrote the code (hey im a python and circuit-python noob remember!).

def _register_read(self, register, length=1):
    with self.spi_device as spi:
        result = bytearray(length)
        spi.readinto(result, register)

Code 9: circuit-python readinto() amended

The final differences between my code and its Pimoroni python counterpart are around the register initialisation code. Firstly I removed the register code required to use the library with a PMW3901 sensor. This however can easily be added back in. Thanks to my having separated each of the initialisation calls into chunks:

self._init_registers_secret()
self._init_registers()
self._init_registers_led()

Code 10: Initialisation of registers

All you would need to do is swap out the register values defined within the _init_registers() method with those that can be found within the Pimoroni pmw3901 class.

Secondarily is the inclusion of a method to toggle the state of the sensors onboard led’s (Code: 11). All this method does is to write a differing value to register 0x6f in order to toggle the lights on or off. I also broke out the led initialisation into its own method (see Code: 10) so that functionality can be controlled upon initialisation. The register and values and code for the led toggle call were located via this issues post.

def set_led_state(self, state=False):
    if state == True:
        self._bulk_write([
        "WAIT", 0xF0,
        0x7f, 0x14,
        0x6f, 0x1c,
        0x7f, 0x00
        ])
    else:
        self._bulk_write([
        "WAIT", 0xF0,
        0x7f, 0x14,
        0x6f, 0x00,
        0x7f, 0x00
        ])  

Code 11: The set_led_state() method

And thats it. Hopefully there is enough there to enable you to use the library with your own PA55100JE and Circuit-python implementations. If all goes well when you play the code you should end up with some glaring lights like those shown in Figure: 3. Additionally a stream of data will be present via serial!

Figure 3: The working circuit

As always if you need any more info I can be contacted by the socials. Your best option would be to open an issue here. As I progress with the building of this Github Pages site I aim to implement comments via use of utterances so eventually your comments/questions may also show up here.

As a final note; i don’t think i will ever get used to not having to terminate statements :)

Updates

NB: Since completing the library I discoverd an official Pico one for Micropython by Pimoroni so that may be a better option for you? This can be found here Although it looks like there may be a few more hoops that will need to be jumped through to get things up and going etc (see their readme for more details).

I have also updated the github repository to include the capability for the initialisation of the registers for a PWM3901 sensor. To swap between the PMW3901 and the PAA5100EJ; just un-comment/comment the corresponding init_registers method. See Code: 12 for more details.

"""Initialize the device registers. You don't seem to need
the secret ones for the PAA5100EJ!?"""

self._init_registers_secret()

""" You can now chose between either the the PAA5100EJ or the PMW3901.
    Just un-comment/comment the corresponding init_registers method"""

self._init_registers_PAA5100EJ()        
# self._init_registers_PMW3901()

Code 12: Initialize either device registers

Finally I have also added the code as outlined in Code: 9 to the script in a method called _register_readinto(). I’ve not tested this yet but it compiles etc.

Photon Control of a MOSFET

2016-11-26T22:40:33+00:00

This tutorial builds upon both the code base and circuitry as developed with the Particle.publish() and Particle.subscribe() example. This time around instead of registering a subscription that can be used to turn on or off an led we set one up to control a MOSFET.

Figure 1: The completed system

The result is a circuit which can be used to control high power devices such as motors and solenoids via an independent power source.

What is a MOSFET

A MOSFET or metal–oxide–semiconductor field-effect transistor; is as the name suggests a special type of transistor that requires very little current to activate. This feature means that they are perfect for use with micro-controllers. A transistor itself is a 3 lead component that generally has 2 basic functions, to switch or to amplify. In this example we will be using one as a switch.

How a MOSFET Works

As stated a MOSFET has three leads. The Source lead is basically an input whilst the Drain lead is effectively an output. The Gate is a control pin that when activated switches the transistor and allows current to flow from the Source (in) to the Drain (out).

An Overview of the Circuit

The MOSFET is connected so that by default the high powered device is connected to V+ but not to ground (V-). The ground however is connected to the transistor’s drain. When the micro-controller sends a HIGH signal to the transistor’s gate; the transistor switches, (connecting the drain and source) and thus completes the circuit for the high powered device.

Figure 2: The MOSFET circuit

In addition to the aforementioned connections to the MOSFET the circuit also has a resistor (10K) that acts to hold the Gate low when the micro-controller does not send a HIGH signal. This is to prevent false readings caused by floating values on the pin.

The Code

The code for this example is almost exactly the same as that developed for the Particle.publish() and Particle.subscribe() example. The only real difference is the name of the event which has been changed to “mosToggle” rather than that of “ledToggle”.

Additionally all the property names such as those used for the pin references have also been updated. The actual functionality of the code however, remains the same.

The Event Subscription Code

The following code block outlines the setup of the program and the registration of the “mosToggle” event subscription.

Code 1: The event subscription code

The next block of code shows the subscription handlers construction. The handler is instantiated as part of the registration of the “mosToggle” event subscription.

Code 2: The event subscription handler

All this code does is set the mosfet pin (D3) to either HIGH or LOW depending upon the data received via the subscription.

The Event Publish Code

The next code block shows the main loop for the button input program. On each iteration of the loop; the value of the button input pin is read and the flag pressed is updated accordingly.

Code 3: The event publish code

The flag pressed is used so that only a single activation of either press or release is used to trigger the event call. What this means is that if a reading of LOW or HIGH is made and the value of pressed has already been set to reflect this state; nothing happens as the code just returns.

The full source code for this example can be found here. A copy of this post published via the IoST Project can be found here.

Photon Publish and Subscription Events

2016-11-26T22:00:47+00:00

This post introduces the use of the Particle.publish() and Particle.subscribe() events to enable photon devices to communicate via the Particle Cloud. This post acts to lay the groundwork for a series of postings that will enable the use of the particle.io platform to formulate a variety of IOT devices.

Particle.publish()

The Particle.publish() command allows you to trigger an event through the Particle Cloud. This means that any hook, application or device that is registered to listen to the event will be notified when it is called.

Calling Particle.publish() when the device is not connected to the cloud will not result in an event being published. This is indicated by the return success code of false. This in turn enables some form of notification to be made if the publish event fails.

Code 1: The Particle.publish() Method

Currently, a Particle device can publish at rate of about 1 event/sec, with bursts of up to 4 allowed in 1 second. A back to back burst of 4 messages will take the device 4 seconds to recover. Code 1 above demonstrates (descriptive) the use of the Particle.publish() command.

Particle.subscribe()

The Particle.subscribe() command allows you to register or subscribe for a notification from a specific Particle.publish() event. This allows devices to talk to each other very easily.

Events received via the subscription are passed to a handler function; itself registered when the subscription is made.

A subscription handler (code 2) must return void and take two arguments, both of which are C strings (or const char *).The first argument is the full name of the published event. The second argument (which may be NULL) is any data that came along with the event.

Code 2: The Particle.subscribe() Method

As with Particle.publish(); the Particle.subscribe() command returns a bool indicating its success. A subscription can be registered when the device is not connected to the cloud. In this instance the subscription will be automatically registered with the cloud next time the device connects.

A device can register up to 4 event handlers. This means you can call Particle.subscribe() a maximum of 4 times; after that it will always return false.

A Worked Example

The following example demonstrates the use of both the publish and subscribe commands by allowing for a button press and release on one device to trigger the turning on and off of a LED on situated on a separate device.

The Push-Button Circuit

Figure 1 shows a simple push-button circuit. The Photons D6 pin is connected to one leg of the push-button. That same leg of the button connects through a pull-down resistor (here 10K ohm) to ground. The other leg of the button connects to the Photons 3v3 supply.

Figure 1: A simple push-button circuit

When the push-button is open (unpressed) there is no connection between the two legs of the push-button, so the pin is connected to ground (through the pull-down resistor) and we read a LOW. When the button is closed (pressed), it makes a connection between its two legs, connecting the pin to 5 volts, so that we read a HIGH. This means that we can use the change from HIGH to LOW to trigger our Publish event.

The LED Output Circuit

Figure 2 shows a simple LED Circuit. The Photons D3 pin is connected to the positive leg of the LED via a 22K resistor. The negative leg is then simply connected to ground. Use of the resistor is to prevent the LED from burning out.

Figure 2: A simple LED output circuit

This setup enables us to control the LED by sending either a HIGH or LOW value to the pin. This in turn means that we can then use the values received via the subscription to determine whether we set the pin LOW or HIGH and thus effectively turning the LED ON or OFF.

The (Push-Button) Publish Code

The following code block provides the full source code for the push button event publisher. All the code does is read the pin connected to the push button (D6) each loop. If the button is down then a down event (“on”) is published. If up, then an up (“off”) event is published.

The flag “pressed” is used to ensure that only the first trigger and release of the button is counted.

Code 3: The (Push-Button) Publish Code

The (LED Output) Subscribe Code

The subscription code is much simpler than that of the publish code. All we are doing here is initialising the led pin (D3) as an output, and then setting it to LOW. We then subscribe to the event ledToggle and pass the handler myHandler at the same time.

The following code block provides the full source code for the led subscription.

Code 4: The (LED Output) Subscribe Code

The handler is triggered each time the event is Published and depending on if the data sent equates to “on” or “off” the status of the LED is updated accordingly.

Figure 3: Both circuits working together

The above image (Figure 3) shows the system in action. As the button is pressed; the other circuits LED lights. A copy of the full source code can be found here. A copy of this post published via the IoST Project can be found here.

A Simple Voltage Divider Circuit

2016-11-11T21:19:31+00:00

Here we have a quick repost of a tutorial which I originally developed for the IoST project that I worked on back in 2015/2016. The tutorial details how to make a simple voltage divider circuit that can be used to measure input from physical variable resistance sensors such as flex sensors and pressure sensors.

A variable resistor is one which changes its resistance when interacted with. Using the flex sensor as an example; as we bend the sensor its resistance increases. We can then measure that change using an a micro-controller; such as an Arduino via one of its analog inputs.

In order to do that however; we first need a fixed resistor (not changing) that we can use for providing a comparison. This is called a voltage divider circuit as the supplied voltage is divided between the sensor and the resistor.

How it Works

The analog read on your micro-controller is basically a voltage meter. Arduinos run at 5v , so at 5V (its max) the analog pin would read 1023, and at 0v it would read 0. The amount of the 5V that both the resistor and sensor gets is proportional to their resistance. So if the the sensor and the resistor have the same resistance, the 5V is split evenly (2.5V) to each part. (analog reading of 512)

A Practical Example

The following is an example of a voltage divider circuit used for a pressure sensor. As you can see the sensor is connected to the Arduino via the pin A0. It is also connected to the 5v supply. The comparison resistor is simply connected to the signal line of the sensor and then to GND. An easy way of looking at it is that the two resistors (one the sensor) are connected in series; whilst the data output is connected between them:

Here is an image of the same circuit however this time physically put together on the breadboard. A few extra wires have been added just to allow for ease of wiring both the 5v and GND lines.

These can be singular wires if you have them long enough.

The Code

The code for this circuit could not be simpler. All we are really doing is reading the value from the A0 pin via an analogRead call. In order to display the data we then make use of the Serial.println command.

A copy of this post published via the IoST Project can be found here.

Controlling virtual experiences using biometrics

2016-03-07T20:11:20+00:00

Over the last year or so I have developed several plugins that allow use of the Microsoft Band, within Android based Unity3D applications. Finally, the time has come to release at least one of them (the most functional if not simple) to the masses.

In this post I will predominately focus on the Unity side of things so that you can get up and running with the plugin asap; however never fear; I will focus on the Java side of things in a future post.

About the plugin

The plugin is structured so that it can and will run in conjunction with other Android Unity plugins. This means that a developer can also leverage other features offered by the Android platform, including: speech recognition, gps, step counting and generic phone sensors such as accelerometers and gyroscopes etc.

Most importantly however; well for me anyway 🙂 is that the plugin can also run on the Samsung Gear VR. By way of result; the MSBand can be used to provide biometric data and multi-modal input that can not only shape and control a players virtual experiences; but also used to evaluate them too…

The use of biometrics in this manner has been a personal research interest of mine for well over a year now.

Figure 1: The Band 2 Sensors

During this period I have found that sensors such as those made available via the Band and (Figure 1) Band 2; show great potential for both applied health and well-being based research, and also targeted games based applications.

Plugin Features

The Key features for this initial release include:

Multiple bands – yes at the same time; this can be quite useful in a VR application
Support for all the sensors made available via the current (Feb) Microsoft Band SDK
Support for both the Band 1 and Band 2 (even at the same time)
Gear VR integration (there is a bit of work to get heart rate to function but it works)

Features that didn’t make it into this release are:

Static access to properties: this is because I opted for the multi band support. However I aim to make it possible in a future release and just need some advice from a Unity plugin or Android guru.
Support for tiles: I have got this working as you will have seen in this post, however I am currently making the process simpler for bespoke implementation. Hopefully this will make it into the next release.

Using the plugin

In order to use the plugin you will need to put a copy of both it and the official Microsoft SDK jar (Currently the Feb 2016 release) within the Plugins > Android folder of your Android Unity project. You will also need to include a copy of the three scripts MsBand.cs, Sensors.cs and MsBandAndroidBridge.cs as located within the MsBandScripts folder and as demonstrated in each of the provided examples.

Simply create a GameObject called MsBandManager and add the MsBandAndroidBridge script as a component. You can the communicate with the band via use of method calls and event registration.

Please note that this script affords communication with the plugin via the UnitySendMessage method. For now I have hard-coded the name of the object to which messages are sent and so the GameObject must be called MsBandManager in order to work. I might update the code for the next release so that the naming is set dynamically as with my UnityEV3 plugin but for now you will have to make do 🙂

Registering for Events and Direct Polling

The unity code allows you to use events in order to register for and retrieve data from the plugin. Each time a message is sent from the plugin to the MsBandAndroidBridge class it is parsed via the receiving function and an event fired. At the same time a corresponding instance of the MsBand class is also updated which enables for direct polling.

Each sensor has its own update event and can find all of the events within the MsBandAndroidBridge class.

A Simple Example

The following code example is for the Accelerometer Sensor. First in your scripts start function we will need to both register for Accelerometer events and initialise the connection to any paired bands. If you wanted; initialisation could be moved to a separate method to allow for scenarios such as an activation on button press etc.

void Start()
{
    // Register for raw Accelerometer events.

    MsBandAndroidBridge.RawAccUpdateEvent +=
        MsBandAndroidBridge_RawAccUpdateEvent;

    // Initialise the connection to all bands 
    // paired with the mobile device.

    MsBandAndroidBridge.Instance.ConnectToPairedBands();
}

Code 1: The Start Method

Depending upon your IDE; when you create the event registration, a corresponding method will also automatically be created that will be called when the event is fired. In this instance the corresponding method MsBandAndroidBridge_RawAccUpdateEvent looks like the following code block. If your IDE does not do this then you can just add the method yourself. Just ensure that the names match etc.

private void MsBandAndroidBridge_RawAccUpdateEvent(float x, float y, float z, MsBand band)
{
   // Do something with data
}

Code 2: The Event Method

As you can see the raw Accelerometer data is passed to the method as three float values. In addition, an instance of the updated band class is also received. This is useful as it provides immediate access to the whole of the current band state.

private void MsBandAndroidBridge_RawAccUpdateEvent(float x, float y, float z, MsBand band)
{
    if (band.BandId == 0)
    {
        // Do something for band 0
    }

    if (band.BandId == 1)
    {
        // Do something for band 1
    }
}

Code 3: Multiple Band Filtering

More useful however is that it can also be used to identify the band in multiple band applications (code 3).

Getting Heart Rate to work on the Gear VR

Currently (I’m working on it now) in order to get heart rate to function on the Gear VR you will need to deploy a a non VR version of your application to the phone first so that you can grant it permission to access the heart rate sensor data.

To do this all you need do is use a standard (non VR) manifest. Once deployed and running you then need to make a connection to a band and then enable the sensor. This will trigger the required intent request. Once permission is granted you can then replace your manifest with one required for a VR application and deploy as normal.

Don’t forget the Manifest

Please make sure that in both (all) instances that you include the required permissions in your manifest as detailed below:

<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="com.microsoft.band.service.access.BIND_BAND_SERVICE" />

Code 4: Android Permissions

In addition to the plugin .jar and Unity3D examples. I have also placed the full source on GitHub. As stated at the beginning of this post I will be publising another post on the Java side of things soon. In the mean time, please fork away and get in touch with any suggested amendments and or fixes etc.

Editor Options

There are also a few options made available via the MsBandAndroidBridge script component that can be set via the inspector (fig 2). Foremost is the ability to auto enable sensors. If set to true this will automatically make the relevant calls to enable the bands sensors during connection and initialisation. This can also be done manually via the Enable & Disable Sensors collection of methods.

Secondly you can select which sensors your application uses via adding their string name to the SensorsToEnable list. If left blank this will default to all sensors. Finally the PreventSleep property can be used to prevent your application from sleeping via the Unity SleepTimeout.NeverSleep command.

Its a wrap(per)

In reality all this plugin is; is a wrapper for the official and excellent Microsoft SDK which uses the Unityplayer.UnitySendMessage method to relay data to the unity application. So thanks and kudos must really go to Microsoft not only for producing a great product, but also for providing great developer support.

With this in mind, hey Microsoft please can I have a free Hololens 🙂 to play with; I promise you wouldn’t regret it 🙂

The official band SDK can be found here.
The plugin can be found via GitHub here.

That all there is to it. I hope that you find the plugin useful and please do send me links to your uses. As a final note, if there is anyone interested in me collaborating with them in a biometric based research project, please get in touch via my facebook page as I am always looking to expand my research network etc.

Capacitive Sensor Development Board

2015-07-20T22:36:45+00:00

This tutorial details how to make a simple breakout for the Arduino CapacitiveSensor library. The library turns two or more pins of an Arduino pins into a capacitive sensor which can then be used to sense the electrical capacitance of the human body.

The above image shows the final product; a sensor breakout capable of handling up-to six soft textile contacts.

Bill of Materials

In order to build the above breakout you will need the following components:

Six 1000 kohm resistors
Some wire
Male and Female header strips each 11 pins in length
A single header pin
Adafruit Perma-Proto (Quater size)

Putting it Together

First; remove every other header from each of the header strips using a pair of pliers. This will result in a strip with six pins remaining as shown in figure 2. Make sure you start removing at pin 2. Once complete, repeat the process with the other strip. Again make sure you start at pin 2.

Next, layout each of the resistors on the breadboard so that there is a gap of 1 row between each and so that all of them connect to one of the boards negative lines. Use some blue-tack to hold them in place and then solder to the board.

At the same time also add a strip of wire to the final row instead of a resistor as shown in the above image (figure 3).

Once the resistors and wires are soldered in place; next add both the header rows so that each pin lines up with one of the resistors. Make sure that the male row is situated between both the resistors and the female row. Once soldered into place you should have something that looks like the following image (figure 4).

Finally add the single header pin to the column with the wire attached as shown above (figure 4). Once soldered into place this pin will become the shared connection line for all of our attached sensor contacts. This is possible because we are using the negative line of the proto-board to share the connection, not supply negative power!

That’s it for the build process; next lets have a look at the Arduino code needed to utilise the sensor breakout.

The Capacitive Sensing Library

The Arduino environment can be extended through the use of libraries, just like most programming platforms. Libraries provide extra functionality for use in sketches, e.g. working with hardware or manipulating data. A number of libraries come installed with the IDE, but you can also download or create your own.

To make use of this breakout we need to download a library from the Arduino web site called the Capacitive Sensing Library. Once downloaded the library can be installed by following the standard library installation instructions which can be found here.

For this tutorial I am just going to skip directly to the code, however Information on how the library works and its features can also be found via the download page. So if you want some more information I highly recommend that you check it out.

The Arduino Code

Thanks to the Capacitive Sensing Library the code for the breakout is dead simple. Therefore, rather than giving an in depth description on how everything works I have commented the code so that you can follow along. All you need to do to get the sensor up and running is to upload the following sketch to your Arduino and then connect the following pins (one to each female header slot) 2,3,4,5,6,7 and finally pin 12 to the single header.

Once uploaded the sketch will output six values to the serial monitor. When you touch any of the male header pins the value of the pin will change to show the capacitance of the touch. You can change the value of the threshold to make the sensor more sensitive. Finally; each time the threshold is exceeded the LED on board the Arduino will light up to show that a touch has been detected. A copy of this post published via the IoST Project can be found here.

Mixed Reality & Sphero Robots

2015-06-06T00:39:08+00:00

The following text is an extended abstract for a another paper I hope to be presenting at this years ITAG. This time round the paper features the research which I am conducting into the use of mixed reality in conjunction with robotics; in this instance, the robots in question are the most excellent Sphero platform.

Applying Mixed Reality & Sphero Robots to Teach STEM Concepts

In this paper we present the application of Mixed Reality Environments (MRE) in conjunction with Sphero; a commercial off the shelf robotics platform. The result is a framework for educational robotics that is not only affordable for use in both a personal and mainstream educational context, but one which is also robust and safe enough to facilitate this task.

The framework builds upon the existing SPRK (Schools Parents Robotics Kids) program; a curriculum developed by the makers of Sphero to teach STEM (Science, Technology, Engineering and Mathematics) based concepts to children (8-13). At the heart of SPRK is the ethos that play is a powerful teacher. We expand this thinking further by allowing for learning to take place within fun games based MRE that not only foster learning, but also creative problem-solving, teamwork and social interaction.

Sphero is a robotic ball that can be controlled by Bluetooth enabled mobile technologies. It can be instructed to roll at a given speed and direction for a given amount of time. Sphero also provides feedback in the form of variable colour illumination. The robot is unique in that not only can it perform as a controllable robot, it can also be used as a controller. These features make Sphero a perfect fit for STEM education as it allows for the direct mapping of core curriculum concepts such as compound measures; through to more advanced topics like applied geometry and computer programing.

The described framework expands upon Sphero’s existing feature set by facilitating use of the sensors and technologies afforded by the mobile device used to control the robot. The result of which is a self-contained system with advanced methods for interaction and feedback that rival, and in some instances surpass those of more expensive robots. This in turn also allows for a more comprehensive level of curriculum coverage. Our enhancements include: audio, object, symbol, text and speech recognition, inertial sensor based input and navigation and the real-time display of performance metrics.

In a typical lesson, students work in small groups to play out an educational game/scenario. In order to complete each lesson they are required, but not exclusively to:

Use Sphero to input commands to manipulate the MRE.
Write computer programs that control how the Sphero rolls and appears.
Input commands that control how the Sphero rolls and appears.

Each learning game/scenario takes place in a specially designed table top gaming space, the arena. The arena facilitates two primary roles. Foremost it allows for physical objects to be scanned and placed so that they can become elements of the MRE. This allows for real-world objects to become both physical and augmented features with which the Sphero can interact, in both physical and virtual space. Secondly the arena also provides visible metrics that a student can use to solve a task.

We present the design and application of two such learning game/scenarios each formulated to demonstrate the full capability of the described framework.

DOI: 10.13140/RG.2.1.2522.3841

Update, the paper was accepted and here is a link to the presentation.

Walking in Place with Wearable Technology

2015-06-06T00:33:47+00:00

The following text is an extended abstract for a paper which I hope to present at this years ITAG. The paper features the continued research and development of the Virtual Cane system I originally developed as part of my PhD.

Walking in Place with Wearable Technology: the development of a system for travel training and applied travel for people with a visual impairment.

In this paper we present the application of commercial off the shelf mobile Virtual Reality and Wearable Health technologies to formulate a novel system for travel training and applied travel for people with a visual impairment (VI).

At the heart of the system is a walking distance estimation algorithm derived from the design and analysis of a custom cane-use walking classification for applied Orientation Mobility Cane Technique (OMCT). The algorithm can be used to track position through dead reckoning, with integral drift corrected for each time the operator completes a specific phase of the walk cycle. Not only does this allow for correctly performed OMCT to be used as a means of navigation within Virtual Environments for those with a VI; but also provides basis for a self-contained indoor and outdoor positioning system.

Virtual representations of unknown Physical Environments (PE) have been demonstrated to allow those with a VI the opportunity to develop both mental-models and navigational strategies in advance, which can then can then be employed in the PE. Typically, such systems are found to disrupt the match between both the mental body-model formed from proprioceptive data and the sensory data supplied by the VE. This is because the methods applied for navigation often require a mental association between peripheral input and the act of virtual translation.

In order to evaluate and ultimately combat this effect, this research investigates alternative methods of navigation based upon an amalgamation of OMCT and the Walking in Place (WIP) technique.

WIP is an established effective method to navigate VE. It enables an individual to traverse a VE without the physical limitations imposed by tracking devices; and it has been shown to improve both an individual’s level of immersion and their spatial orientation. Research is now needed on how WIP might affect these outcomes for those with a VI; and ultimately how it might affect the acquisition of spatial knowledge which can then be applied to a PE.

To demonstrate the effectiveness of the current system we show its application utilising a standard assistive white cane. Finally we discuss how the system can be improved via the design of a custom tip with odometry sensing capability.

DOI: 10.13140/RG.2.1.1252.6245

Update, the paper was accepted and here is a link to the presentation.

Microsoft Band & Unity3D

2015-06-05T18:43:51+00:00

In this post I present a video of a plugin which I have been developing recently to enable communication between the excellent Microsoft Band and the Unity3D games development environment.Currently the plugin only works on Android but I hope to have a Windows Phone equivalent finished soon too 🙂

You need to accept the use of third party cookies to view Youtube content embedded in this site!

To do so click here or to find out more here

As you will see in the video; the plugin makes available all the functionality provided by the official Band SDK (Android) including the ability to create a custom tile on the band that can be used to send instructions to the Unity3D app running on the phone.

I am aiming to make the plugin available in the next week or so; so check back soon and/or contact me for more details. Finally, for a great video detailing an overview of the official SDK’s functionality be sure to check out the Band SDK development team Build 2015 presentation here.