Summary: New study introduces SonicSense, a program that gives robots the ability to perceive items through vibrations. Equipped with microphones in their fingertips, robots is click, grasp, or stir objects to find audio and determine materials type, shape, and contents.
Beyond what perspective alone can provide, SonicSense enables robots to discover new objects and consider environments with greater detail and accuracy. This system represents a major advancement in mechanical sensing, enabling robots to communicate with the natural world in dynamic, unorganized environments.
Important Facts:
- Robots can “hear” items through vibration and sound using SonicSense.
- Robots can find materials, shape, and material by tapping or shaking items.
- Robots can detect unfamiliar objects with greater accuracy thanks to AI-driven technology.
Origin: Duke University
Imagine how much beer is still in your large cup in the dark movie theater. You pick up and shake the pot a little to learn how far ice is inside while prying off the cover and looking, giving you a good idea of whether you’ll need to find a free replace.
Setting the beverage up along, you wonder absent-mindedly if the seat is made of real hardwood. But, after a few plugs and a hollow sound, you decide that it must be made of plastic.
We have the ability to interpret the world through sound vibrations that emanate from objects without even considering them. And it’s something researchers are about to introduce to robots to enhance their swiftly expanding set of detecting abilities.
Set to be published at the Conference on Robot Learning ( CoRL 2024 ) being held Nov. 6–9 in Munich, Germany, new research from Duke University details a system dubbed , SonicSense , that allows robots to interact with their surroundings in ways previously limited to humans.
Jiaxun Liu, the paper’s lead author and first-year PhD, observed that “most computers now rely on eyesight to interpret the world.” Boyuan Chen, a professor of electrical engineering and supplies science at Duke, is a PhD student in his lab.
We wanted to develop a system that would work with the diverse and complex objects that are frequently encountered, giving computers a far richer understanding of the world.
SonicSense , features a mechanical hand with four hands, each equipped with a phone camera embedded in the fingers. These sensors can identify and report vibrations that a robot makes when it taps, grasps, or shakes an item. Additionally, it makes it possible for the machine to tune out ambient noises because the mics are in direct contact with the subject.
Based on the contacts and detected signs,  , SonicSense , extracts speed features and uses its previous knowledge, paired with recent advancements in AI, to figure out what materials the image is made out of and its 3D design.
It might take the system 20 different interactions to arrive at a conclusion if the image is an object it has never seen before. However, if it’s an already-existing image in its database, it is properly identify it in as little as four.
” SonicSense , gives drones a new way to hear and feel, many like humans, which may change how recent robots perceive and communicate with things”, said Chen, who also has sessions and individuals from electrical and computer engineering and computer technology.
While sound adds layers of information that the eye might miss, while sound is essential, according to the statement.
In the paper and demonstrations, Chen and his laboratory showcase a number of capabilities enabled by , SonicSense. A box of dice can be divided into various pieces, as well as the shape of the numbers inside. By turning or shaking them, it can determine the number inside as well as the shape.
It can determine how much liquid is contained inside a bottle of water by doing the same thing with it. And it can create a 3D reconstruction of the shape and material it’s made from by tapping around the outside of an object, much like how humans explore objects in the dark.
While , SonicSense , is not the first attempt to use this approach, it goes further and performs better than previous work by using four fingers instead of one, touch-based microphones that tune out ambient noise and advanced AI techniques.
This configuration enables the system to identify objects with complex geometries, transparent or reflective surfaces, and challenging materials for vision-based systems.
Our robot had to be able to interact with objects independently in an open lab environment, Liu said, “whereas most datasets are collected in controlled lab settings or with human intervention.”
” It’s difficult to replicate that level of complexity in simulations. This gap between controlled and real-world data is critical, and , SonicSense , bridges that by enabling robots to interact directly with the diverse, messy realities of the physical world”.
These abilities make , SonicSense , a robust foundation for training robots to perceive objects in dynamic, unstructured environments. The construction costs for the same contact microphones that musicians use to record sound from guitars, 3D printing, and other commercially available components are just over$ 200.
Moving forward, the group is working to enhance the system’s ability to interact with multiple objects. By integrating object-tracking algorithms, robots will be able to handle dynamic, cluttered environments— bringing them closer to human-like adaptability in real-world tasks.
The robot hand’s own design is another important development.
” This is only the beginning. In the future, we envision , SonicSense , being used in more advanced robotic hands with dexterous manipulation skills, allowing robots to perform tasks that require a nuanced sense of touch”, Chen said.
” We’re excited to explore how this technology can be further developed to integrate multiple sensory modalities, such as pressure and temperature, for even more complex interactions”.
Funding: This work was supported by the Army Research laboratory STRONG program ( W911NF2320182, W911NF2220113 ) and DARPA’s FoundSci program ( HR00112490372 ) and TIAMAT ( HR00112490419 ).
Note: The image is a representation of a robotic hand, not the SonicSense system.
About this news about robotics research
Author: Ken Kingery
Source: Duke University
Contact: Ken Kingery – Duke University
Image: The image is credited to Neuroscience News
Original Research: The findings will be presented at the Conference on Robot Learning
