2 hours ago
1
For remotely operated underwater vehicles, cloudy and turbulent waters are often a no-go. When vehicles settle on the seafloor or dig through a sandbed, they can kick up clouds of sediment that make it tough for onboard cameras to see through. Often, the only thing to do is to wait until the marine dust settles before a vehicle can safely proceed.
But a new underwater mapping technique developed by engineers at MIT and the Woods Hole Oceanographic Institution (WHOI) may allow vehicles to see through murky, low-visibility waters.
The method fuses visual images from optical cameras with acoustic data from sonar sensors. The combination enables a vehicle to quickly map the general shape of its surroundings using sonar, even in low-visibility waters. A vehicle can move toward certain shapes in the sonar-mapped environment, coming close enough for optical cameras to visually resolve specific objects in detail.
The technique is akin to pairing a dolphin’s echolocation with a sea turtle’s close-range vision to see and navigate through murky water, in real-time.
The researchers tested the method in tank experiments where they could control the water’s degree of visibility. Even in the cloudiest conditions, the system was able to see through the sediment to map the tank’s environment and visualize centimeter-scale details of objects in the tank.
The team is further improving the technique, which they’ve named Sonar-MASt3R. They envision that the mapping method could safely guide underwater vehicles through murky environments for a range of applications, including scientific exploration, underwater construction and maintenance, and deep-sea recovery.
“We hope that this work enables us to do more operations in those challenging, low-visibility environments, and helps provide more coverage in areas that are difficult to operate in today,” says Amy Phung, a graduate student in MIT’s Department of Aeronautics and Astronautics, who led the work.
Phung presented a paper detailing Sonar-MASt3R this week at the IEEE International Conference on Robotics and Automation (ICRA). The paper’s co-author is Richard Camilli, senior scientist of applied ocean physics and engineering at WHOI.
The best of both
To see underwater, scientists have generally taken an either/or approach, using either optical cameras or sonar sensors to guide the way. Optical cameras can provide detailed visual imagery of a scene, but only in waters that are relatively clear and well-lit. In contrast, sonar sensors perform just as well in clear and murky water; by emitting acoustic waves and measuring the time and angle at which they return, sonar sensors can determine the exact shape, distance, and depth of objects in the environment, though a sonar map lacks any visual detail.
To get the best of both modes, scientists have looked to combine the two in a new approach known as “opti-acoustic fusion.” In a handful of prior works, research groups have merged sonar and optical data in mapping techniques that are mostly geared toward object recognition and reconstructing workplace environments. Most techniques require time to sync and process the data and therefore do not work in real-time, while only a few can map an environment in 3D. None have been applied to high-resolution mapping underwater in murky, turbid conditions.
Phung, who is a student in the MIT-WHOI Joint Program, and Camilli, her advisor, aimed to develop an opti-acoustic fusion technique that would generate detailed 3D maps of underwater environments in real time and in low-visibility conditions. The team was motivated, in part, by challenges in safely recovering unexploded underwater mines.
“There can be old explosives in areas that make it unsafe for ships to be in, and the ability to get rid of those safely is best done by robotics,” Camilli says. “But a lot of these explosives are set in surf zone environments where visibility adds to the challenge of doing this safely. That’s one of many applications that our technique can be used for.”
Cloudy, with a chance of mapping
The new method, Sonar-MASt3R, builds on an existing technique, MASt3R, that was developed by researchers in France. MASt3R is an image matching algorithm that is trained to take in visual images of the same scene and quickly estimate the relative depth of each pixel in the scene. In this way, MASt3R can generate a 3D map of the environment in real-time, based on a camera’s 2D images.
“The downside is that there is no sense of scale,” Phung says. “It will say ‘this pixel is five units closer than this pixel,’ but it can’t say whether that’s 5 meters or 5 feet.”
Luckily, sonar provides absolute measurements of scale. The timing of sonar reflections can be translated directly into a specific depth and distance of objects that the signals bounced off, as well as their shape and contour.
In their new work, Phung and Camilli used sonar data to correct MASt3R’s scaling and generate precise 3D maps of underwater environments. Even in murky water, the method’s sonar-corrected map would enable a vehicle to know the precise location of objects, and therefore how far to safely move in for a closer inspection, which the vehicle could then do using conventional optical cameras.
The team tested Sonar-MASt3R in experiments with a tank that they filled with water, sediment, and a variety of objects such as a small boulder, a coffee mug, and a packing crate. Inside the tank, they also set up a robotic arm, onto which they mounted an underwater camera, and a sonar sensor.
For each experimental run, they first carried out a sweep trajectory, in which the robotic arm slowly swept from one side of the tank to the other to capture sonar and visual data. With this first sweep, Sonar-MASt3R quickly creates a coarse sonar-based map of the shapes and contours of the tank and its objects. The coarse map is then used to record close-up camera images of the objects, which are used to improve the map resolution. A “keyframe” approach quickly compares each new image frame to the last keyframe. If a frame provides new information not contained in the last keyframe, the image is added as a new keyframe to the map. If it is similar, it is immediately discarded. In this way, the approach can quickly fill in the map with relevant visual detail, in real-time.
The researchers tested their new approach underwater, testing eight different levels of turbidity, which they created by stirring up the tank’s sediment. Compared with other opti-acoustic fusion approaches, Sonar-MASt3R generated more accurate 3D maps and resolved smaller, centimeter-scale details, and in cloudier conditions. In the cloudiest condition, which the robotic arm’s cameras could not see through, its sonar sensors were able to generate a rough map of the tank’s hidden objects. This initial map enabled the arm to move safely through the murk and closer to specific objects, which its underwater camera could then visualize in more detail.
“An analogy would be if you were to go into a china shop in the dark, and try to pick your way around to find a specific coffee mug without knocking things over,” Camilli offers. “This would allow you to do that.”
The team plans to test the approach in natural underwater conditions, where they suspect that the mapping task should be more straightforward.
“In a tank, it’s like an echo chamber,” Camilli says. “It’s like trying to do this in a funhouse mirror setting where you get all these distortions and reverberations and ghost images that really complicates the processing. If you put it in the real world, it should be easier.”
Then, they say, Sonar-MASt3R could help scientists safely explore in cloudy, turbid, and murky underwater regions.
“The real value in this effort is so we can use this technology in mission scenarios that are untractable right now,” Phung says. “And there are plenty of untractable missions because we don’t have the observational or perception capabilities.”
This research was supported, in part, by NASA, and the National Science Foundation.
