Embedded Eye

Give your gizmo the gift of sight

Wide field 4D optical flow odometry using Arduino and Stonyman image sensor


I've been working on a new version of our ArduEye using one of our "Stonyman" image sensor chips and decided to see if I can grab four dimensions of optical flow (X shift, Y shift, curl, and divergence) from a wide field of view. I wirebonded a Stonyman chip to a 1" square breakout board, and attached it to an Arduino Mega256 using a simple connecting shield board. I then glued a simple flat printed pinhole onto the chip using (yay!) 5-minute model airplane epoxy. With a little black paint around the edges, the result is a simple low resolution very wide field of view camera that can operated using the Arduino.

I programmed the Arduino to grab five 8x8 pixel regions- region 0 is forward while the other four regions are about 50 degrees diagonally off forward as shown. In each region the Arduino computed X and Y optical flow and odometry (essentially an accumulation of optical flow over time).

To compute X and Y shift, the algorithm summed respectively the X and Y odometry measurements from the five pixel regions. These are the first two dimensions of optical flow that most people are familiar with. To compute curl and divergence, the algorithm added the appropriate X or Y odometries from the corresponding pixel regions. For curl this results in a measurement of how the sensor rotates around it's forward axis. For divergence this results in a measurement of motion parallel to the forward axis.

In the current configuration the system operates at about 5 to 6 Hz, though when the serial dump is on that slows to about 2 Hz. Most of the delay is in the acquisition and involves wasteful array lookups to select which pixels to read out.

The video shows a few test runs where I exposed the sensor to three of the four fundamental motions. Y shift was implemented using an air track (like some of you used in physics class). Curl motion was implemented with the aid of a well-loved turntable. Divergence was implemented by hand by moving the sensor to and from clutter. The corresponding plots show the response of all four motions, with the "correct" one emphasized.

You can see that the four components are largely independent. There is some crosstalk- curl and divergence tend to be the biggest recipients of crosstalk since they are effectively a difference between signals (and getting an accurate number by subtracting two noisy numbers is not easy). Factors such as varying distances around the camera can cause uneven stimulation of the different pixel fields, resulting in phantom curl and div. There is also a little bit of drift. There is a lot of room for optimizing the system for sure.

One immediate improvement would be to use two of these Stonyman cameras back-to-back so that near omnidirectional sensing could be performed. This would give us more information to separate the different components (X,Y,curl,div) as well as allow us to separate out the other two axes of rotation from X and Y.

A setup similar to this formed the basis for our recent single sensor yaw and heave (height) stability sensor  demonstration.

Views: 2067

Comment by Randy Mackay on December 22, 2011 at 9:03pm


     Very interesting.  It's great to be able to measure more than just x/y movement.  The divergence in particular is the one I'm very interested in as a possible way to augment/replace a sonar for altitude hold on quads/helis.

     I expect the update rate could be improved once the Due (Arm based arduino) is out which should be any day now.  I believe the ARM version will be close to 10x as fast.  I think the latest Arduino 1.0 also includes non-blocking serial ports so that might get the current performance back up to 5~6hz.

     What do you think the minimum update rate is for acceptable optical flow measurements?  I guess it depends upon how fast the objects in view appear to move.  For example, if they move more than one or two pixels since the last image was captured presumably the algorithms start to be unable to find the features?  In arducopter we consume navigation data at 10~20hz.

     Also I was wondering, have you ever considered using FPGA (of which I know extremely little) to increase the update rate?  ..or do you perhaps feel that it's an unnecessary half step between a custom processor (which you've blogged about and is presumably lightening fast but) and arduino (which is slow but user friendly).

     As a side point, the mouse sensor that I've played with (ADNS3080) has the ability to upload the firmware but of course it's completely closed source so no way to write your own and emails to avagotech go unanswered...which is why we need to move on to the next generation of open source vision sensors!

     Sorry, a few random questions/comments not all completely releted to this blog post.

Comment by Geoffrey L. Barrows on December 23, 2011 at 10:55am

These are good questions and ones that I actually thought about.

Regarding what frame rate is needed- this all depends on the dynamics of the vehicle and whether it is stabilized in other dimensions. When stabilizing the micro helicopter with vision alone, we found in practice that a 50Hz update was a lower limit to keeping basic control, with 100+ Hz being more practical. We don't know for certain how much of this was due to the dynamics of the helicopter and how much was due to the need to track optical flow however. But for larger quad type platforms, I assume they already have a basic gyro to control yaw (the fastest mode) and pose. I would guess you can get by with 10-20 Hz, but this depends on how fast the pose will change in practice e.g. how quickly it goes from horizontal to a slight tilt to move sideways.

A basic rule of thumb is to first consider the angular rate you need to measure- optical flow is an angular rate. When handling angular rates- you just need the angular rate itself since this directly becomes optical flow. When handling translation- you need speed divided by distance between the sensor and the texture (here between quad and ground) since that will give you optical flow. Once you have the optical flow speed requirements of the sensor, you then consider things like the separation between pixels (also an angular distance), the maximum optical flow per frame the optical flow algorithm uses, and from that you can get the frame rate.

Here is a sample calculation- suppose pixels are 1/20 of a radian apart in the visual field, and the optical flow algorithm measures at most 1 pixel per frame. Suppose the optical flow speed requirement is 2*pi radians per second (e.g. 1 revolution per second) or 6.28 radians per second. 6.28 radians per second becomes 6.28 / (1/20) = 126 pixels per second. The algorithm needs then to run at 126 frames per second. I think it will be much less, though, for the quad.

Regarding using an FPGA vs. ARM vs. Atmel- I've always been a fan of Colin Chapman's philosophy towards power vs. minimalism- Chapman was the founder of Lotus Automobiles (British light-weight sports cars that routinely embarrass Ferraris) who basically repeated the mantra of lightness- he once said "More power makes a car faster on the straights; more lightness makes a car faster everywhere". So if we apply that analogy to image processing- using a faster processor can certainly help, but that should not be the default solution to increasing the sensor speed. The code I wrote worked at 5-6Hz, but I spent no time optimizing it. I bet it could be speed up by at least a factor of 2 to 5 if we used different data types (chars instead of shorts for the image) and streamline the acquisition (not efficient as written). That ArduEye board also has a spot to solder in an 8-pin DIP ADC that gives you 200ksps rather than the Arduino's 10ksps. The speed of the ARM should help out further- I would bet by another factor of 5.

Other ways to get speed increases may be processing some parts of the visual field at a higher frame rate and other parts at a lower frame rate- this gives you both fast-but-less-accurate information and slower-but-more-accurate information which together may be adequate. (I tend to find that for controlling a robot, speed it more critical than accuracy...) Other paths could include faster optical flow algorithms (I haven't really looked at that yet) and, for the hard-core, assembly language programming!

To go much faster than that, one could use a faster DSP- A gumstix goes up to 700MHz I believe which is another 7x factor increase. FPGAs can certainly go faster, but they are more difficult to program- we did experiment with one around 2002 and it certainly went fast but it took a loooong time to write the code. There are tools available to, say, generate FPGA code from higher level code like C and MATLAB, but I think these tools are not cheap- the MATLAB one i

Comment by Geoffrey L. Barrows on December 23, 2011 at 10:58am

I almost forgot- another path for speeding up optical flow computation is to use image sensor chips having some processing on-board, whether that processing is analog or digital. We can get another 10x increase that way if not much much more. We used to do that, but stopped when we started making use of other optical flow algorithms that were not as easy to implement with analog circuitry. But that is a valid direction- one of our old vision sensors from 2003 computed optical flow at 1400Hz that way using 88 pixels and a 10MIPS microcontroller. That one was so fast we actually had to slow it down to get the robot to use it!

Comment by Hari on February 9, 2012 at 12:51am

@ Geoffrey,

I'm interested in using these vision sensors( along with IMU) in a hand held device which can be used as 3D input device in front of  computer monitor. ( similar to WII remote )

I would like to use these sensors in sensor fusion algorithm to correct the drift in position , velocity computed from double integration of acceleration from MEMS 3-axis accelerometer.

Any possibilities or difficulties in using these vision sensors in front of bright LCD or LED monitors (0.35m to 1m)   are appreciated. 

I want to improve the working volume of device than the WII remote which only works when its in the FOV of IR Bar.


You need to be a member of Embedded Eye to add comments!

Join Embedded Eye

© 2022   Created by Geoffrey L. Barrows.   Powered by

Badges  |  Report an Issue  |  Terms of Service