Algorithm magnifies motions indiscernible to naked eye

For several years now, the research groups of MIT professors of computer science and engineering William Freeman and Frédo Durand have been investigating techniques for amplifying movements captured by video but indiscernible to the human eye. Versions of their algorithms can make the human pulse visible and even recover intelligible speech from the vibrations of objects filmed through soundproof glass.

Earlier this month, at the Computer Vision and Pattern Recognition conference, Freeman, Durand, and colleagues at the Qatar Computing Research Institute (QCRI) presented a new version of the algorithm that can amplify small motions even when they’re contained within objects executing large motions. So, for instance, it could make visible the precise sequence of muscle contractions in the arms of a baseball player swinging the bat, or in the legs of a soccer player taking a corner kick.

“The previous version of the algorithm assumed everything was small in the video,” Durand says. “Now we want to be able magnify small motions that are hidden within large motions. The basic idea is to try to cancel the large motion and go back to the previous situation.”

Canceling the large motion means determining which pixels of successive frames of video belong to a moving object and which belong to the background. As Durand explains, that problem becomes particularly acute at the object’s boundaries.

If a digital camera captures an image of, say, a red object against a blue background, some of its photosensors will register red light, and some will register blue. But the sensors corresponding to the object’s boundaries may in fact receive light from both foreground and background, so they’ll register varying shades of purple.