Gesture detection using Arduino MKR1000, MPU6050 and TouchDesigner
The following blog is about how I combined the Arduino and TouchDesigner using OSC to create the gesture detection system.
This article covers the main steps I took to build the system, if you want to have a look at the whole code, you can check the Github repo. The idea for this blog was based on Charlie Gerard’s awesome Play Street Fighter with body movements using Arduino and Tensorflow.js. In Charlie’s example, she used machine learning to train in Tensorflow.js. I prefer to use node in TouchDesigner to trigger the action which makes it easier to analyze the simple data.
References:
https://docs.arduino.cc/hardware/mkr-1000-wifi
https://www.arduino.cc/reference/en/libraries/mpu6050/
https://blog.tensorflow.org/2019/11/how-to-get-started-with-machine.html
https://medium.com/@devdevcharlie/getting-started-with-the-arduino-mkr1000-johnny-five-3e8d1da82a6c
Material:
Arduino MKR1000(another model can work too)
MPU6050(Or other accelerometer/gyroscope)
Battery
Wires
Micro-USB Cable
I used a mini breadboard to put everything together, and also used Apple Watch band to make it wearable.
Step 1: Assembling components
The reason why I used Arduino MKR1000 is the built-in wifi can help me do not have to be tethered to my computer when I run the program, you can also use Arduino Uno for testing. As for the accelerometer/gyroscope, I bought both MPU6050 and MPU9250, the only difference is MPU6050 give you 6 points of data but MPU9250 gives you 9.(6 are enough for this project)
The circuit and final look like this:
Step 2: Setting up the Arduino IDE
Download and install the Arduino IDE from https://arduino.cc/downloads
Open the application
Select Tools > Board > Boards Manager, search for “MKR1000” and install Arduino SAMD
Select Sketch > Include Library > Manage Libraries, search for “WiFi101”, “OSC”, “MPU6050” and install these three
Plug the Micro USB cable into the board and your computer
Choose the board Tools > Board > Arduino SAMD > Arduino MKR1000
Choose the port Tools > Port > /dev/cu.usb…
(If you have troubles about l2Cdev, try to delete the same file of these libraries. There are more detailed Getting Started and Troubleshooting guides on the Arduino site if you need help.)
Step 3: Gathering data
See the code below:
Save, Verify and Upload to your Arduino
You can get the data like that:
Step 4: Getting IP address
Now you should get your computer’s IP address, the only thing your need to change is the SSID and Password at the beginning of the file and upload the sketch. Once it’s uploaded, open the Serial Monitor and make sure it is set to 9600 BAUD. After a few seconds, it should print out the Arduino’s IP address. Copy it and keep it somewhere because we’re going to need it soon.
Step 5: Sending OSC
See the code below and change SSID, Password and IP address. Once it’s uploaded, open the Serial Monitor and make sure it is set to 57120 BAUD.
You should see like that:
Then you can connect the battery and disconnect the cable, open TouchDesigner and add OSC in node, set up IP address and port(57120), you will get the data from your Arduino!
Step 6: Triggering in TouchDesigner(CHOP)
Set the appropriate Trigger Threshold(I set acc1 to 3.5, acc3 to 3)
Clear the attack length and peak length to 0
Logic node change Channel Pre OP to Radio Button
Use the math node to scale the data
And congratulations, you have created your own gesture detection system!
Thanks for reading!
