Robot chef can learn recipes by watching food videos

In brief: Robotic kitchens will probably never replace humans, but that doesn’t stop companies like Moley, Hyper Food Robotics, Xrobotics, Dexai Robotics, Miso Robotics, and Nala Robotics from trying. Meanwhile, researchers are using recent advancements in neural networks and computer vision to see if automated chefs could one day learn recipes from online cooking videos.

Robot cooks aren’t a new idea. Back in 2015, a company called Moley Robotics introduced a robot that could pick up regular kitchen utensils and use a specially-designed kitchen to prepare food. In 2020, restaurants like White Castle started testing “Flippy” cooking robots to automate the patty-flipping process at several locations.

Fast forward to the present day, and researchers at the University of Cambridge are showing off a robotic “chef” that can learn how to recreate a particular recipe by analyzing food videos. After being shown eight different salad recipes explained in video format, the new robot was able to correctly identify the recipes being prepared 93 percent of the time and was able to determine 83 percent of the actions required to prepare them.

The researchers fed the system a total of 16 video demonstrations with slight variations in the recipes and employed several publicly available neural networks to convert them into a series of tasks the robot can execute. Each video frame was analyzed with OpenPose and YOLOv5m networks to detect the cook’s hands, the ingredients, and the utensils involved. The researchers then used a statistical approach called a hidden Markov model (HMM) to discern the sequence of actions along with their length as well as the ingredient proportions.

Interestingly, the robot was able to detect when a recipe involved a higher portion size or classify small variations as human error. When the researchers fed the system a demonstration video for a ninth recipe that was truly different from the others, the robot was able to correctly identify it as new.

It’s worth noting the researchers filmed the recipes themselves and didn’t use any transition effects or fast cuts that are typical for recipes posted on platforms like YouTube or TikTok. There’s a good reason for that, as the robot needs a clear view of the ingredients involved and wouldn’t be able to follow fast movements that well.

Grzegorz Sochaki, who is one of the engineers involved in the project, explains that programming a robot to make a range of dishes is a monumental task. Over time, Sochaki and his team expect robot chefs will be able to quickly and easily learn recipes from videos posted on social media.

Despite growing concerns that robot cooks might soon replace humans in restaurant kitchens, it doesn’t look like we’ll see that anytime soon. While replacing human order-takers at drive-thru restaurants using AI chatbots seems like a feasible task, human cooks are almost impossible to replace.

If we look at the Moley robot and its competitors, they are being marketed as mechanical master chefs, but they lack the tactile, taste, and olfactory senses needed to prepare high-quality meals. At the same time, the machines need the ingredients, utensils, and appliances to be meticulously laid out for them in advance, while constant supervision is also recommended during the actual cooking process. Also, most restaurants aren’t rushing to pay $338,000 for a robot chef that needs plenty of human help to be useful and could break down during rush hours.

For those of you interested in a deep dive into how the Cambridge robot chef works, you can find the research paper on IEEE Access.

Masthead credit: Moley