Smart necklace could track your detailed facial expressions

Human facial movements convey emotions, and assistance america pass nonverbally and execute carnal activities, specified arsenic eating and drinking.

Tracking facial movements—and perchance their cause—is 1 of the projected applications for NeckFace, 1 of the archetypal necklace-type wearable sensing technologies. A squad led by Cheng Zhang, adjunct prof of accusation subject successful the Cornell Ann S. Bowers College of Computing and Information Science, has developed NeckFace, which tin continuously way afloat facial expressions by utilizing infrared cameras to seizure images of the chin and look from beneath the neck.

Their enactment is elaborate successful "NeckFace: Continuously Tracking Full Facial Expressions connected Neck-mounted Wearables," which was published June 24 successful Proceedings of the ACM connected Interactive, Mobile, Wearable and Ubiquitous Technologies.

Co-lead authors are Tuochao Chen (from Peking University) and Yaxuan Li (from McGill University), visiting students successful the Smart Computer Interfaces for Future Interactions (SciFi) Lab, and Cornell MPS pupil Songyun Tao. Other contributors are HyunChul Lim, Mose Sakashita and Ruidong Zhang, Cornell Ph.D. students successful the tract of accusation science, and François Guimbretière, prof of accusation subject successful the Cornell Bowers College.

NeckFace is the adjacent procreation of Zhang's erstwhile work, which resulted successful C-Face, a akin instrumentality but successful a headset format. Zhang said NeckFace provides important betterment successful show and privacy, and gives the wearer the enactment of a less-obtrusive neck-mounted device.

In summation to imaginable emotion-tracking, Zhang sees galore applications for this technology: virtual conferencing erstwhile a front-facing camera is not an option; facial look detection successful virtual world scenarios; and soundless code recognition.

"The eventual extremity is having the idiosyncratic beryllium capable to way their ain behaviors, done continuous tracking of facial movements," said Zhang, main researcher of the SciFi Lab. "And this hopefully tin archer america a batch of accusation astir your carnal enactment and intelligence activities."

NeckFace, developed successful the Sci-Fi Lab of Cheng Zhang, adjunct prof of accusation science. Credit: Sci-Fi Lab

Guimbretière said NeckFace besides has the imaginable to alteration video conferencing.

"The idiosyncratic wouldn't request to beryllium cautious to enactment successful the tract of presumption of a camera," helium said. "Instead, NeckFace tin recreate the cleanable caput changeable arsenic we determination astir successful a classroom, oregon adjacent locomotion extracurricular to stock a locomotion with a distant friend."

To trial the effectiveness of NeckFace, Zhang and his collaborators conducted a idiosyncratic survey with 13 participants, each of whom was asked to execute 8 facial expressions portion sitting and 8 much portion walking. In the sitting scenarios, the participants were besides asked to rotate the caput portion performing the facial expressions, and region and remount the instrumentality successful 1 session.

NeckFace was tested successful 2 designs: a neckband, draped astir the backmost of the cervix with duplicate cameras conscionable beneath collarbone level; and a necklace, with a pendant-like infrared (IR) camera instrumentality hanging beneath the neck.

The radical collected baseline facial question information utilizing the TrueDepth 3D camera connected an iPhone X, past compared that to the information collected with NeckFace. Between the sitting, walking and remounting expressions, survey participants expressed a full of 52 facial shapes.

Using calculations involving heavy learning, the radical determined that NeckFace detected facial question with astir the aforesaid accuracy arsenic the nonstop measurements utilizing the telephone camera. The neckband was recovered to beryllium much close than the necklace, the researchers said, perchance due to the fact that 2 cameras connected the neckband could seizure much accusation from some sides than could the center-mounted necklace camera.

Zhang said the device, erstwhile optimized, could beryllium peculiarly utile successful the intelligence wellness realm, for tracking people's emotions implicit the people of a day. While radical don't ever deterioration their emotions connected their face, helium said, the magnitude of facial look alteration implicit clip could bespeak affectional swings.

"Can we really spot however your emotion varies passim a day?" helium said. "With this technology, we could person a database connected however you're doing physically and mentally passim the day, and that means you could way your ain behaviors. And also, a doc could usage the accusation to enactment a decision."

---

---

More information: Tuochao Chen et al, NeckFace, Proceedings of the ACM connected Interactive, Mobile, Wearable and Ubiquitous Technologies (2021). DOI: 10.1145/3463511