Embedded Eye

Give your gizmo the gift of sight

Due to my need to develop some form of eyegaze mouse we have been (wife & I) using ardueye as an embedded first stage.  Here is where we have got so far.

1) Hooked up camera at close to correct focal lenght to eye

2) Established that the eye will track left and right and display in some form that we can use.

3) Worked out the drift in the system focusing on the eyeball (similar to a gyro drift).

We have success in the above three.

By just tracking one axis and not using accumulation we could work out the drift so the next step is to get a mega and repeat for two axis and either use the servo out, in arduino or send serial commands to a teensy board which has usb mouse routines on it.

Views: 2134

Comment by Clinton McKinnon on December 14, 2012 at 7:28pm

Mega board has been hooked up as well as the teensy2 for the scroll up scroll down routinues which are as simple as if the ardueye triggers a limit negative or positive during tracking there fires a command for scroll up or scroll down.  Lots of tweaking plus build a permenant holder for the camera but apart from it tracking one way due to noise it all looks positive.  Below is a picture of the gui restrained at 40 & -40 pluse resets back to 0 after every 50 cycles to counter noise and drift.

Comment by Clinton McKinnon on December 15, 2012 at 7:53pm

Here's the code simple as it is to send out to the teensy2 the scroll up down commands.  numbers still need adjusting but you can put this out the bottom of the 1d optical flow code in first app and have some success at tracking the eyeball.  Don't forget to add the appropriate intergers in the global section for the ODOcount etc 


ODOlinearacc = constrain(ODOlinearacc,-41,41);
    odocount = ++odocount;
    if (odocount == 50)
      odocount = 0;
      ODOlinearacc = 0;
    // Dump accumulation i've changed this so that it just prints the raw filtered linear movements
   // Serial.println(Filtered_OFlinear);
     if (ODOlinearacc > 40)
 Serial1.print("t"); // do something here
if (ODOlinearacc < -40)
 Serial1.print("b"); // do something here

Comment by Clinton McKinnon on December 18, 2012 at 7:36pm

New layout with camera mounted on top of actual reading glasses, seems to focus okay.  Changed from the serial port to using the teensy analog inputs as an easy way for the routinue to page up and down.  Seems to be quite successful.

//extract from stonyman first app as modified so far 


ODOlinearacc = constrain(ODOlinearacc,-45,45);
    odocount = ++odocount;
    if (odocount == 1000)
      odocount = 0;
      ODOlinearacc = 0;
    // Dump accumulation i've changed this so that it just prints the raw filtered linear movements
   // Serial.println(Filtered_OFlinear);
     if (ODOlinearacc > 44)
// Serial1.print("t"); // do something here
// digitalWrite(53,LOW);
if (ODOlinearacc < -44)
// Serial1.print("b"); // do something here
// digitalWrite(51,LOW);


teensy code below as modified from teensy examples


// This line defines a "Uart" object to access the serial port HardwareSerial Uart = HardwareSerial(); int incomingByte; int xdata; int ydata; const int analogInPin1 = A0;  // Analog input pin that the potentiometer is attached to const int analogInPin2 = A1; // Analog output pin that the LED is attached to

int sensorValue = 0;        // value int sensorValue1 = 0;        // value

void setup() {  // Serial.begin(9600);         Uart.begin(9600); }

void loop() {         Uart.println("ready"); sensorValue = analogRead(analogInPin1); sensorValue1 = analogRead(analogInPin2);  if (sensorValue > 1000) { Mouse.scroll(2); Uart.println("donet"); delay(200); } if (sensorValue1 > 1000) { Mouse.scroll(-2); Uart.println("doneb"); delay(200); } }

Comment by Clinton McKinnon on December 20, 2012 at 6:19pm

Some comments from me to date on the experiment

We have the eye tracking working on a single axis using just about standard code as supplied with a few mods.  I can now page up or page down as of yesterday, and the eyeball settled down after a few minutes and it was quite controllable.  As a note we used the horizontal axis to page up and page down (scroll up/ scroll down = d2)

You may have noticed that we are using analog reads on the teensy2 this is just practical so not to confuse my scribe as the second part of this is also interfacing some emg kits (muscle detection so that we can detect blink or face scrunch or jaw clamp for the mouse such as single click, double, right etc)

We have only scratched the surface of what can be done with eye tracking and ardueye but an offshoot for the uav community would be eye tracking for camera rotation - just a thought. 

Comment by Clinton McKinnon on January 1, 2013 at 6:52pm

quick update and photos.  Hooked tracking of single axis into our UAV depth sounding boat project on the rudder output for the mega goes serailly to out existing PPM interface board.  A bit jittery but we're scaling up the output from the tracker a huge amount in the PPM interface code.

Sorry no video . 

Comment by Clinton McKinnon on January 1, 2013 at 6:54pm

need to practice keeping eyes steady with smaller movements but it shows promise for any two axis RC project as well.

Comment by Clinton McKinnon on January 9, 2013 at 6:37pm

Tried a pc based system as a baseline for our experiement. Found my eye wobbled about trying to chase the screen.  Ended up doing a program in processing to try and train the eye to stick on the crosshairs - picture of result attached.


Comment by Geoffrey L. Barrows on January 17, 2013 at 11:38pm

Clinton- This work is FANTASTIC. I am both humbled and inspired to see what you are doing here. Please do post here any questions you may have.

One thought- Individual frame-by-frame optical flow measurements can be noisy, thus the benefits of using an accumulation. Have you tried grabbing accumulations every, say, 10 frames (or some similar number) and using that as an optical flow measurement? Also, you could try taking the individual optical flow measurements and putting them in a low pass filter e.g.

     OFlp += alpha * (OFnew - OFlp);

where "alpha" is say 0.1 or 0.01 or another number. This way if you move your eye slowly, the sensor doesn't respond, but if you move it fast it responds. This might be good for detecting rapid eye motions that may indicate a "click" or a button press or something similar.

I do have another question- Do your eyes get tired when using a setup like this? I just wonder how long one can use an eye-tracker like this before they need take a break and relax the eye muscles.


Comment by Clinton McKinnon on January 18, 2013 at 12:25am

thanks for the positive comments, yes we need to do something about the filtering and thanks for the advice.  Had thought about doing a cumulative average but i think we'll put your code in action and try it on the processing display we made to train the eyes.   The eyes do get tired but once you relax and stop forcing the movement then it is a lot better.  The idea is to get maximum flow tracking for minimum movement.  I had tried a pc based system as a comparison and that was harder on the eyes than the embedded setup with flow tracking

Comment by Clinton McKinnon on February 8, 2013 at 6:39pm

 old_Filtered_OFlinear += 0.1 * (Filtered_OFlinear - old_Filtered_OFlinear);//Geof's filter 
  ODOlinearacc += old_Filtered_OFlinear;

Have added Geof's suggested filter and are working on ways of using this and other filters to gain more USB teensy/mouse functions.

Have built a mecanno gimball with camera which works to some degree.  As I have very flicky eyes more work needs to be done on smoothing.  We also noticed this on the pc eyegaze program.  Our embedded system has the advantage of being needing minimal calibration as apposed to systems that track the pupil .


You need to be a member of Embedded Eye to add comments!

Join Embedded Eye

© 2022   Created by Geoffrey L. Barrows.   Powered by

Badges  |  Report an Issue  |  Terms of Service