I Can Read Your Pulse by Webcam

The above is the latest version of a labor of love I’ve been working on, based on research coming out of the MIT Media Lab. Turns out the veeeery minute changes in color of your skin tone can, frame-by-frame, be read and decoded to arrive at a just-fine measure of your pulse.

The general process by which it works is called photoplethysmography, and it’s the same way normal fingertip pulse readers work. Those use an LED to light up your skin and then measure how the light absorption changes whenever blood flows through.

I read through the original research paper, and then later got to peek at the code the researchers used (through the sponsor relationship of my parent company with MIT), and challenged myself to rebuild it in Flash so that it could be web-deployable (and so could be thrown into a banner or a microsite as a gimmick for one of our clients). Forgive the interface I gave it, I honestly don’t have a design bone in my body.

Want to make your own? Awesome, here’s what you need to work out:

1. First, you need to identify a region of open skin from the webcam feed, so fire up a face identification method and have it mark out a subregion of the user’s face. I used a tweaked version of the Marilena library for AS3, and had it strip out only the nose and cheeks area, so that eye and mouth movements wouldn’t screw things up.
2. Then you need to average out the red, green, and blue color values for the pixels in that subregion, and store a bunch of consecutive frames of them in a matrix. Not having found a satisfactory AS3 matrix math class, I went ahead and made my own AND BOY WAS IT A DOOZY. Turns out things like matrix multiplication, while straightforward, are not very computationally efficient, so I had to research, optimize and reoptimize before I could move very far.
3. Once you have a sufficient number of frames worth of data (this version gets by with around 70 frames, a little over two seconds’ worth, but the more you have, the more stable your readings will be), the real fun begins. You should have three signals at this point–the traces of each of the color channels of the user’s skin as they change over time–so the MIT researchers figured out that you could pass them through a blind source separation algorithm (more on that in a later post) and at least one of the outputs would contain a fairly dependable pulse reading. Again, not having found, really, ANY blind source separation libraries for AS3, I went ahead and made my own AND BOY WAS IT A DOOZY. My version of FastICA for AS3 is basically just a port of the version in the MDP parkage for Python, but with a crapload of optimizations to speed it up for Flash.
4. So by now you should have a clean pulse signal, and it’s time to extract out the period of the beats. Fire up your favorite periodogram function and pass your cleaned-up signal through it and you should be able to get a beats-per-minute measure easily. I went ahead and rolled my own implementation of the Lomb-Scargle method that was based, again, on the work from the MIT researchers and BOY WAS IT A DOOZY. Again, not a very computationally efficient process (though I recently found a better method I want to try), so this version depends on a customized Pixel Bender kernel to do all the heavy lifting outside of the Flash thread (I’ll do a whole post on it eventually).

So I plan to keep playing with it. The low frame count is primarily due to the fact that fastICA starts choking when it has to chew on too much data, so I want to get that going faster (maybe by dumping it off to a Pixel Bender kernel, too) so that I can load in more frames for it to process over. I don’t like how jumpy and off the reading can be sometimes. Work in progress!