final project: working keyboard-mouse-guitar

final project: final processing code

i’ve finally figured out how to manipulate the java class Robot in Processing for both the mouse and keyboard inputs. all i needed to do was figure out the try/catch function when creating an instance of a robot.

here’s the final processing code ive been using:

/**
* oscP5message by andreas schlegel
* example shows how to create osc messages.
* oscP5 website at http://www.sojamo.de/oscP5
*/

import java.awt.Robot;
import oscP5.*;
import netP5.*;

OscP5 oscP5;
NetAddress myRemoteLocation;

int value=0;  //incoming
int j=10;  //
int MouseX=500;
int MouseY=500;
int MouseWait=0;
int MouseAxis=1;

void setup() {

/* start oscP5, listening for incoming messages at port 12000 */
oscP5 = new OscP5(this,12000);

/* myRemoteLocation is a NetAddress. a NetAddress takes 2 parameters,
* an ip address and a port number. myRemoteLocation is used as parameter in
* oscP5.send() when sending osc packets to another computer, device,
* application. usage see below. for testing purposes the listening port
* and the port of the remote location address are the same, hence you will
* send messages back to this sketch.
*/
myRemoteLocation = new NetAddress(“127.0.0.1”,12000);
}

void draw() {
}

void mousePressed() {
/* in the following different ways of creating osc messages are shown by example */
OscMessage myMessage = new OscMessage(“/test”);

myMessage.add(123); /* add an int to the osc message */
myMessage.add(12.34); /* add a float to the osc message */
myMessage.add(“some text”); /* add a string to the osc message */
myMessage.add(new byte[] {0x00, 0x01, 0x10, 0x20}); /* add a byte blob to the osc message */
myMessage.add(new int[] {1,2,3,4}); /* add an int array to the osc message */

/* send the message */
oscP5.send(myMessage, myRemoteLocation);
}

/* incoming osc message are forwarded to the oscEvent method. */
void oscEvent(OscMessage theOscMessage) {
/* print the address pattern and the typetag of the received OscMessage */
int receivedValue = theOscMessage.get(0).intValue();

if(receivedValue>400) //axis-two value from accelerometer
{
MouseWait++;
if(MouseWait>5)
{
try
{
Robot robot = new Robot();
if(receivedValue>520)
{
if(MouseAxis==1)
MouseX=MouseX+10;
else
MouseY=MouseY+10;
}
if(receivedValue<510)
{
if(MouseAxis==1)
MouseX=MouseX-10;
else
MouseY=MouseY-10;
}
robot.mouseMove(MouseX, MouseY);
}
catch (AWTException ex)
{
String x=ex.getMessage();
println(x);
MouseWait=0;
}
}
}

else //values under 100, pitch values from guitar
{
if(value==receivedValue && value>5) //the pitch-value has come twice in sequence, this number is not noise. print the letter.
{
int letternumber=0;
//println(value);
fill(0, 102, 153);
value=value+1;
switch(value)    //take the incoming pitch and findand print the corresponding letter
{
case 45:
letternumber=68;
break;
case 46:
letternumber=67;
break;
case 47:
letternumber=85;
break;
case 48:
letternumber=77;
break;
case 49:
letternumber=70;
break;
case 50:
letternumber=69;
break;
case 51:
letternumber=84;
break;
case 52:
letternumber=65;
break;
case 53:
letternumber=79;
break;
case 54:
letternumber=73;
break;
case 55:
letternumber=78;
break;
case 56:
letternumber=83;
break;
case 57:
letternumber=82;
break;
case 58:
letternumber=72;
break;
case 59:
letternumber=76;
break;
case 60:
letternumber=80;
break;
case 61:
letternumber=71;
break;
case 62:
letternumber=87;
break;
case 63:
letternumber=89;
break;
case 64:
letternumber=66;
break;
case 65:
letternumber=86;
break;
case 66:
letternumber=75;
break;
case 67:
letternumber=88;
break;
case 68:
letternumber=74;
break;
case 69:
letternumber=81;
break;
case 70:
letternumber=90;
break;
case 71:
letternumber=200;
break;
case 72:
letternumber=201;
break;
default:
letternumber=32;
break;
}
try
{
Robot robot = new Robot();
if(letternumber==200)
{
if(MouseAxis==1)
MouseAxis=2;
else
MouseAxis=1;
}
else if(letternumber==201)
{
int mousebutton=1;
robot.mousePress(InputEvent.BUTTON1_MASK);
robot.mouseRelease(InputEvent.BUTTON1_MASK);
}
else
{
robot.keyPress(letternumber);
println(letternumber);
}
}
catch (AWTException ex)
{
String x=ex.getMessage();
println(x);
}
value=500;
}
else {
value=receivedValue; //the last two pitches from the guitar are not even. replace the holding variable with the new value.
}
}
}

final project: arduino issues

i recntly rebuilt my computer, and the arduino 0010 build that i downloaded doesn’t seem to be able to connect to my arduino. the error i’m having is:
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_recv(): programmer is not responding

not sure what to do with this. the serial port selector doesn’t show up with my USB connections. all i have is:

/dev/tty.Bluetooth-Modem

/dev/cu.Bluetooth-Modem/dev/tty.Bluetooth-PDA-sync

/dev/cu.Bluetooth-PDA-sync

final project: processing code that works

so i figured out how to manipulate the java AWT robot class in processing. after i did i was pretty darn happy. for now i have the guitar actually acting like a keyboard. here is the code im using:

/**
* oscP5message by andreas schlegel
* example shows how to create osc messages.
* oscP5 website at http://www.sojamo.de/oscP5
*/

import java.awt.Robot;
import oscP5.*;
import netP5.*;

OscP5 oscP5;
NetAddress myRemoteLocation;

int value=0;  //incoming
int j=10;  //
String letter=””;  //complete string of all text that has been printed
void setup() {

/* start oscP5, listening for incoming messages at port 12000 */
oscP5 = new OscP5(this,12000);

/* myRemoteLocation is a NetAddress. a NetAddress takes 2 parameters,
* an ip address and a port number. myRemoteLocation is used as parameter in
* oscP5.send() when sending osc packets to another computer, device,
* application. usage see below. for testing purposes the listening port
* and the port of the remote location address are the same, hence you will
* send messages back to this sketch.
*/
myRemoteLocation = new NetAddress(“127.0.0.1”,12000);
}

void draw() {
}

void mousePressed() {
/* in the following different ways of creating osc messages are shown by example */
OscMessage myMessage = new OscMessage(“/test”);

myMessage.add(123); /* add an int to the osc message */
myMessage.add(12.34); /* add a float to the osc message */
myMessage.add(“some text”); /* add a string to the osc message */
myMessage.add(new byte[] {0x00, 0x01, 0x10, 0x20}); /* add a byte blob to the osc message */
myMessage.add(new int[] {1,2,3,4}); /* add an int array to the osc message */

/* send the message */
oscP5.send(myMessage, myRemoteLocation);
}

/* incoming osc message are forwarded to the oscEvent method. */
void oscEvent(OscMessage theOscMessage) {
/* print the address pattern and the typetag of the received OscMessage */
int receivedValue = theOscMessage.get(0).intValue();

if(receivedValue>200) //axis-one value from accelerometer
{
//Robot robot = new Robot();
println(receivedValue);
}

if(receivedValue>100&&receivedValue<200) //axis-two value from accelerometer
{
try
{
Robot robot = new Robot();
int x=0;
int y=0;
robot.mouseMove(x, y);
}
catch (AWTException ex)
{
String x=ex.getMessage();
println(x);
}
}

else //values under 100, pitch values from guitar
{
if(value==receivedValue) //the pitch-value has come twice in sequence, this number is not noise. print the letter.
{
int letternumber=0;
//println(value);
fill(0, 102, 153);
value=value+1;
switch(value)    //take the incoming pitch and findand print the corresponding letter
{
case 45:
letternumber=68;
break;
case 46:
letternumber=67;
break;
case 47:
letternumber=85;
break;
case 48:
letternumber=77;
break;
case 49:
letternumber=70;
break;
case 50:
letternumber=69;
break;
case 51:
letternumber=84;
break;
case 52:
letternumber=65;
break;
case 53:
letternumber=79;
break;
case 54:
letternumber=73;
break;
case 55:
letternumber=78;
break;
case 56:
letternumber=83;
break;
case 57:
letternumber=82;
break;
case 58:
letternumber=72;
break;
case 59:
letternumber=76;
break;
case 60:
letternumber=80;
break;
case 61:
letternumber=71;
break;
case 62:
letternumber=87;
break;
case 63:
letternumber=89;
break;
case 64:
letternumber=66;
break;
case 65:
letternumber=86;
break;
case 66:
letternumber=75;
break;
case 67:
letternumber=88;
break;
case 68:
letternumber=74;
break;
case 69:
letternumber=81;
break;
case 70:
letternumber=90;
break;

default:
letternumber=32;
break;
}
try
{
Robot robot = new Robot();
robot.keyPress(letternumber);
}
catch (AWTException ex)
{
String x=ex.getMessage();
println(x);
}
value=500;
}
else {
value=receivedValue; //the last two pitches from the guitar are not even. replace the holding variable with the new value.
}
}
}

data readout of accelerometer in processing

after getting the accelerometer values hooked up in processing, i end up with values like this:

the numbers definitely have a sway of noticeable and useable output from the accelerometer.

i just copy and pasted from the serial monitor and copied it into ms excel.

except, as i was saying before, i don’t know enough java to control the mouse input with these values, so this is where my project hits its brick wall.

i’m not sure where i would go from here – maybe talking to some itp kids and continuing surfing forums about how to hack the java on my computer to let me use the robot class.

acceleromter>pd>processing

to get the accelerometer input into processing, i decided to use hans’ firmware designed for pd. this requires a program to be running on the arduino that is controlled by pd, and which i edited to send messages to processing.

 

it definitely gets the job done. 

i modified the values coming in like this, so i can use the same port in OSC to transfer all the data into processing. and once inside processing, i can do three easy if statements to determine which values go with which input. 

0-100 – pitch input from the guitar.
100-200 – x-axis input from the accelerometer.
200-300 – y-axis input from the accelerometer. 

accelerometer input into processing

to operate the mouse with the guitar, i’ve decided to use an accelerometer. when you tilt the guitar down-up, the mouse will go up. when you tilt the guitar sideways, the mouse will go left-right.  i got this accelerometer on sparkfun, and i’m using the x and y axis to control the guitar.  

final project: issues with controlling low-level system inputs

after i got the guitar pitches inputting into processing and being able to control them, i have issues controlling the java library that  would let me make a system character input rather than just in processing. the library is called “robot”, it’s extension being java.awt.robot. when i try to create a robot object in processing, the program throws a java exception.

although i dont fully understand what a java exception is, and after laboring over processing discussion boards and java documentation, i came to a couple possible conclusions: 

1. the robot class is not accessible in mac os x. like this man has found.
2. i don’t know enough about java/processing to figure out how to override this “exception” and manipulate the robot class.  

either way, i don’t seem to be able to take advantage of the robot class in processing.  this is a big brick wall to accomplish the real goal of this project – manipulating the low-level input devices (mouse, keyboard) with a guitar.

final project: the keyboard guitar in action!

the guitar layout

i checked up on some letter use frequencies, and based the keyboard layout on the guitar off of that.   i put the most commonly used letters in the middle, all on the same string, and then the more arbitrary ones on the other strings. this is what it looks like in fretboard imagery: