OSCAR: artificial intelligence at your masthead
The product name “OSCAR” is an acronym for the very valuable, though challenging, collision avoidance work attempted by the multi-camera masthead pod above, along with its processing unit and display apps. Which is Optical System Cognition And Ranging. But the developers also anticipated the usage I’ve noticed in videos showing extreme offshore racing sailors using the technology: The skippers tend to call the system “Oscar” like he’s an extra crewman they’re really glad to have on board.
OSCAR combines thermal and color cameras to maintain a 24-hour masthead watch that is constantly scanned using artificial intelligence to identify floating objects and warn the crew as necessary. But can it spot a barely awash container? In confused and breaking seas? In time enough to avoid it? Well, that’s what I mean by “challenging work” and while I don’t know the answers, Oscar seems to be devoting more brainpower — artificial and otherwise — to the task than any other “smart” marine camera system I know of.
According to the OSCAR story, Raphaël Biancale returned from a six-month sailing trip and used his previous experience as an automobile engineer to invent a form of AI vision meant to help sailors “navigate waters by night in complete safety.” Of course “complete safety” is a very ambitious goal, and that’s why it’s fantastic that OSCAR systems are now being used by 18 top-notch sailors in what might be the most demanding product test imaginable — the non-stop Vendée Globe race around our planet. When you sail as fast as possible for approximately 24,000 nautical miles, the major issue may well be damaging your boat or one of its many appendages by hitting an unidentified floating object (UFO), as indicated in the Wikipedia listing of the eight Vendée races already run since 1989.
Better yet, deeply experienced ocean racers like the “Swiss army knife of Breton sailing” Vincent Riou have been helping to develop OSCAR for years, and you can watch his system explanation on YouTube. (Riou is not in this year’s Vendée, but he’s certainly not forgotten; in fact, I mentioned how Riou on PRB rescued Jean Le Cam in the 2008 race in my entry about when my new hero Le Cam rescued Kevin Escoffier after a newer Riou PRB broke badly last week!)
Another good way to get a feel for Optical System Cognition And Ranging is the OSCAR online demo, which uses actual system recordings (like the ones that hopefully get back to the developers at BSB Artificial Intelligence as part of the system’s ongoing learning aspect). In the screens above I’m trying the color and thermal video streams and note how the Racing model’s single color camera has a 120-degree horizontal field of view while the two thermal cams laced together see a 50-degree swath out to the horizon ahead. Note too how a user can click on a detected object to see its range and whatever Oscar thinks it is.
But I suspect that the preferred display is often the map view, and that the audible alarms generated by a “red dot” are the most important feature, especially for the single-handers. Note that OSCAR has a NMEA 2000 interface and in the demo you can see the many instrument values, like wind speed and direction, that can be added to that bottom data line. I also gather that Oscar can send collision avoidance instructions to an autopilot, at least on an experimental basis, and I hope to learn more about that level of system cognition soon.
So how are the OSCAR users doing on day 31 of the Vendée (tracking map here)? Well, it’s a little sad to watch the skipper interview video (screenshot above) because the great Sam Davies is now in Cape Town repairing damage sustained when her keel hit a whale or similar object at 20 knots and Escoffier recently jumped off Yes we CAM! to join a French frigate. Also, the race favorite Jérémie Beyou on Charal had to turn back 600 miles after the start because something damaged a rudder (and he’s now in last place). And that may not be all the damage that OSCAR-equipped racers have suffered so far.
But then again, Oscar certainly can’t see underwater semi-floating objects and PRB may have simply hit a wave especially wrong, plus the damage is news while the near misses are not. One racer said that the system is worth having even if it only sees 50% of the UFO’s, and I might settle for less.
At any rate, there are many Oscars still on the course and hopefully they’re helping contenders like Romain Attanasio (who is also Sam Davie’s partner) and Boris Herrmann (who could be the first Vendée winner not from France). And all the race systems include hard drives collecting video clips and object/boat data, so hopefully we’ll eventually see instances when the AI lookout made a difference (while OSCAR also improves its optical cognition).
But OSCAR is not just for extreme racing, and can display on an iPad or a familiar multifunction display like the B&G Zeus3 series. Apparently Furuno TZT3 MFDs can also run OSCAR, and Garmin support is in the works. Plus demo sample #1 looks like a common but still nerve-wracking night cruising scene to me.
Actually, OSCAR’s stabilized and augmented camera views remind me of the advanced and AI-assisted imagery seen in recent FLIR/Raymarine cams and also in Ray’s DockSense Control (and DockSense Alert) systems. So when I learned today that the next version of LightHouse3 will also support OSCAR, it seemed like a nice “tip of the hat” to a unique new marine safety idea.
And while all these advanced AI vision products are currently quite expensive for many boaters — I think an OSCAR One 320 is about $10,000 (and they still seek US distribution) — the core technologies are used in much bigger markets, and we should benefit eventually. Heck, my relatively humble Toyota RAV4 can detect highway lanes and vehicles ahead quite accurately, and even control throttle and wheel to help keep me stay safe while on cruise control. Can you too picture Oscar joining your crew?
The word from OSCAR about AP integration: “Our aim is to be connected to autopilot for very last minute detection but for now we are in early testing phases — we are working closely with autopilot manufacturers — Skipper Boris Hermann is the first to try this automatic avoidance with autopilot.”
Also, today at METS Trade Connect, OSCAR was one of five new products shortlisted for the DAME R&D Excellence in Adversity Award (which I watched “live online”). More info on the awards should start popping up soon.
And the winner was TEMO, a very interesting take on small electric outboard design that I’ll cover soon:
I come from the radar world and have had recent experience with automotive radar Vs LIDAR and camera systems. I have been using the Raymarine Virtual Reality system on my two cameras on board for over a year now. Having just retired as a Sr Fellow Engineer for Honeywell Advanced Technology I want to point out that camera systems, whether Infrared or Visible spectrum have one very serious limitation. As soon as any mist – fog, smoke, snow or rain enters the field of view camera systems are immediately rendered no better than your own eyes in the same conditions. I have personal experience with this in the commercial and military worlds. Hence, when visibility is most impaired these camera systems cannot restore the vision that has been lost due to any form of scattering particles in the atmosphere. Only radar can “see” in these conditions but it cannot provide the same level of angular resolution that a camera can. That is pure physics due to the immense difference in operating wavelength between optics and microwaves or millimeter waves or even 220GHz radar sensors. But radar does not have to provide that resolution if it can report dangers when vision is lost. Therefore as automotive tests have shown, a blend of radar and optics is needed to ultimately provide true safety in all conditions.
I retired from the Avionics world where we rate all radios, navigation sensors and radar systems with safety ratings from A down to D. A is safety of life – if the sensor fails or if it provides false information that is not detected, it could mean crashing the aircraft. A simple radar altimeter on every commercial aircraft is rated Level A. Failures of these systems are rare but they did cause one fatal crash about 6 years ago. Automotive systems that rely on vision only have already killed several drivers when they failed to correctly evaluate scenes in front of them even in excellent weather. No automotive system is rated to allow you to go to sleep and arrive at your destination – it requires your attention and cooperation. The blind spot detector on our car is fallible and generates false alerts from time to time and misses some detections other times. Rarely of course but we have noticed them. That level of reliability is NOT allowed on a commercial aircraft. It will be years before automotive systems are truly “Level A”.
AI is a remarkable technology but it has limitations. It cannot infer – it can compare and experience but it cannot forward think and infer a solution as a brain can. I worked in the world of algorithm development for many years and saw the issues involved with them regardless of the learning process that is used. I wrote Level A altitude detection algorithms for an all new altimeter flying on new business aircraft today. It took years of development, data collection and testing.
This new technology for sailing safely at night is very valuable and will no doubt aid in preventing many collisions at sea. One can imagine that ships would want this technology first so that it can be used in places like the English Channel. But it would augment AIS and Radar. Sadly the business of automating target threat detection and tracking with a marine radar may cross over into military applications so it becomes a tenuous issue for manufacturers to avoid making their system have export limitations etc. Radar can see far greater distances than camera systems so it becomes critical to prevent them from being abused by bad actors for weapons delivery. New MMW phased array radars like the RDR84K are being used today for all weather collision avoidance applications. An excellent video is available here:
I am the inventor of that radar system.
Congratulations on IntuVue RDR84K, David, looks amazing:
And of course you are correct that even high-quality thermal cameras like FLIR’s can not see well through rain, mist, or wet fog. H20 blocks the thermal waves used. Also OSCAR does not include the headers or wipers used to help some nav cams in tough conditions. But OSCAR is using FLIR Boson cam cores it turns out, and they are frank in their marketing about visibility limitations.
Merry Christmas to you too, and at least at my house we’re celebrating the beginning of Hanukkah tonight.
Thanks Ben – Shalom and Happy Hanukkah!
Just wondering if it can identify floating wooden logs? I know that object detection on a thermal camera can whereby the thermal scan waterway for non water objects.
I’m working for Madintec. We develop Autopilot MADBrain for sail boat. We equip a third of the Vendée Globe boats, including the first three. We have developed a new generation pilot with tenth of a degree precision and many high level functions.
More information https://madbrain.win (in french) and in english in seahorse.
So, we have developed and tested an automatic pilot avoidance system. Driving a sailboat is difficult: Are we going to downwind or upwind? Is the sailboat going to tack or jibe without endangering the boat.
Our system is connected to AIS and Oscar. We refused to equip a Vendée Globe boat because during our tests there were too many detection errors. We are awaiting feedback from the Vendée Globe. so far there are several collisions with ofni. It doesn’t seem conclusive.
Madintec continues to work on the subject. We connect our autopilot with other detection systems like radars. We are working to merge all detection data to minimize errors.
Madintec is continuing these tests. n 2021 we should be able to offer a product that guarantees better security. we will keep you informed.
More information in french https://voilesetvoiliers.ouest-france.fr/equipement-entretien/electronique-embarquee/collisions-en-mer-l-evitement-sous-pilote-automatique-un-algorithme-revolutionnaire-2-2-1ee97c1e-443c-11ea-bedb-f0b649de7dbb
Thanks, Matthieu. This link works well for me and I can see that you have quite a team:
I realize that OSCAR has a very challenging goal, but even partial success seems valuable. Also, I just watched Vendée Live #43 [EN] and heard Boris Herrmann report that Oscar had detected a large fender (which seems impressive) and that he sleeps better with it running.
Please do keep us informed. Thanks!