IE E E R O B O T IC S & A U T O M A T IO N M A G A Z IN E B IO M IM E T IC P E R C E P T IO N , C O G N IT IO N , A N D C O N T R O L V O L U M E 2 9 • N U M B E R 4 • D E C E M B E R 2 0 2 2 Displaying the world in your hands. Inventing new ways to interact. Force Dimension designs and manufactures the finest master haptic devices for leading-edge aapppplliiccaattiioonnss iinn rreesseeaarrcchh,, mmeeddiiccaall,, iinndduussttrryy,, aanndd hhuummaann eexxpplloorraattiioonn.. The ScanTrainer from Intelligent Ultrasound is an educational tool that uses real patient scans and curriculum-based teaching across obstetrics, gynecology, general medicine, and emergency medicine. The system integrated a customized delta.3 haptic device. Force Dimension Switzerland www.forcedimension.com [email protected] Force Vol. 29, No. 4 DECEMBER 2022 ISSN 1070-9932 http://www.ieee-ras.org/publications/ram FEATURES 10 A Multimodal Fusion Model for Estimating Human Hand Force Comparing Surface Electromyography and Ultrasound Signals By Yongxiang Zou, Long Cheng, and Zhengwei Li 25 A Hybrid Visual–Haptic Framework for Motion Synchronization in Human–Robot Cotransporting A Human Motion Prediction Method By Xinbo Yu, Sisi Liu, Wei He, Yifan Wu, Hui Zhang, and Yaonan Wang 36 Toward Holistic Scene Understanding A Transfer of Human Scene Perception to Mobile Robots By Florenz Graf, Jochen Lindermayr, Çag˘atay Odaba¸si, and Marco F. Huber 50 Biomimetic Electric Sense-Based Localization A Solution for Small Underwater Robots in a Large-Scale Environment By Junzheng Zheng, Jingxian Wang, Xin Guo, Chayutpon Huntrakul, Chen Wang, and Guangming Xie 66 Biomimetic Force and Impedance Adaptation Based on Broad Learning System in Stable and Unstable Tasks Creating an Incremental and Explainable Neural Network With Functional Linkage By Zhenyu Lu and Ning Wang 78 Controlling Maneuverability of a Bio-Inspired Swimming Robot Through Morphological Transformation Morphology Driven Control of a Swimming Robot By Kai Junge, Nana Obayashi, Francesco Stella, Cosimo Della Santina, and Josie Hughes 92 Simulation to Real Learning Energy-Efficient Slithering Gaits for a Snake-Like Robot By Zhenshan Bing, Long Cheng, Kai Huang, and Alois Knoll OONN TTHHEE CCOOVVEERR This issue of IEEE Robotics and Automation 104 A Robust Visual Servoing Controller Magazine focuses on the state of the art for Anthropomorphic Manipulators With in biomimetic perception, cognition, and control research. Field-of-View Constraints and Swivel-Angle Motion Overcoming System Uncertainty and Improving Control Performance ©SHUTTERSTOCK.COM/MONIKA WISNIEWSKA By Jiao Jiang, Yaonan Wang, Yiming Jiang, He Xie, Haoran Tan, and Hui Zhang (Continued) If you like an article, click this icon to record your opinion. This capability is available for online Web browsers and offline PDF reading on a connected device. Digital Object Identifier 10.1109/MRA.2022.3218353 DECEMBER 2022 • IEEE ROBOTICS & AUTOMATION MAGAZINE • 1 FEATURES (Continued) 115 A Small-Scale, Rat-Inspired Whisker Sensor for the Perception A Publication of the IEEE ROBOTICS AND AUTOMATION SOCIETY of a Biomimetic Robot Vol. 29, No. 4 December 2022 ISSN 1070-9932 http://www.ieee-ras.org/publications/ram Design, Fabrication, Modeling, and Experimental Characterization EDITORIAL BOARD Humanitarian Technology: Vacant By Yulai Zhang, Shurui Yan, Zihou Wei, Xuechao Chen, Editor-in-Chief Standards: Craig Schlenoff (USA) Yi Guo Toshio Fukuda, and Qing Shi President’s Message: Frank Park (Korea) ([email protected]) Regional Spotlight: Megan Emmons (USA) Stevens Institute of Technology (USA) 127 Contact Shape and Pose Recognition Student’s Corner: Francesco Missiroli (Germany) Editors Utilizing a Multipole Magnetic Tactile Sensor Elena De Momi TC Spotlight: Yasuhisa Hirata (Japan) With a Metalearning Model Politecnico di Milano (Italy) Women in Engineering: Karinne Ramirez By Ziwei Xia, Bin Fang, Fuchun Sun, Huaping Liu, Jindong Tan Amaro (Sweden) Weifeng Xu, Ling Fu, and Yiyong Yang University of Tennessee (USA) IEEE RAS Vice-President Associate Editors of Publication Activities Todd Murphey (USA) Ming Cao University of Groningen (The Netherlands) RAM home page: http://www.ieee-ras.org/publications/ram Feifei Chen Jiao Tong University (China) IEEE Robotics and Automation Society Executive Office Carlos A. Cifuentes Amy Reeder Bristol Robotics Laboratory (UK) Program Specialist Kingsley Fregene [email protected] Lockheed Martin (USA) Advertising Sales Antonio Frisoli Mark David Scuola Superiore Sant’Anna (Italy) Director, Business Development—Media & Jonathan Kelly Advertising University of Toronto (Canada) Tel: +1 732 465 6473 Fax: +1 732 981 1855 Ka-Wai Kwok [email protected] The University of Hong Kong (Hong Kong) IEEE Periodicals Surya G. Nurzaman Magazines Department Monash University (Malaysia) Kristin Falco LaFleur Weihua Sheng Senior Journals Production Manager COLUMNS & DEPARTMENTS Oklahoma State University (USA) Patrick Kempf Yue Wang Senior Manager Journals Production Clemson University (USA) Janet Dudar 4 FROM THE EDITOR’S DESK Enrica Zereik Senior Art Director CNR-INM (Italy) Gail A. Schnitzer 6 PRESIDENT’S MESSAGE Houxiang Zhang Associate Art Director Norwegian University of Science Theresa L. Smith 8 FROM THE GUEST EDITORS and Technology (Norway) Production Coordinator 138 WOMEN IN ENGINEERING Past Editor-in-Chief Felicia Spagnoli Bram Vanderborght Advertising Production Manager 141 INDUSTRY ACTIVITIES Vrije Universiteit Brussel (Belgium) Peter M. Tuohy RAM Column Manager Production Director 145 TC SPOTLIGHT Amy Reeder (USA) Kevin Lisankie Editorial Services Director RAM Editorial Assistant 148 COMPETITIONS Joyce Arnold (USA) Dawn M. Melley Staff Director, 158 SOCIETY NEWS COLUMNS Publishing Operations Competitions: Hyungpil Moon (Korea) IEEE-RAS Membership 164 CALENDAR Education: Andreas Müller (Austria) and Subscription Information: +1 800 678 IEEE (4333) From the Editor’s Desk: Yi Guo (USA) Fax: +1 732 463 3657 Industry News: Tamas Haidegger http://www.ieee.org/membership_services/ (Hungary) membership/societies/ras.html IEEE prohibits discrimination, harassment, and bullying. For more information, Digital Object Identifier 10.1109/MRA.2022.3218352 visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html. IEEE Robotics and Automation Magazine (ISSN 1070-9932) (IRAMEB) is published quarterly by photocopy beyond the limits of U.S. Copyright law for the private use of patrons 1) those post-1977 the Institute of Electrical and Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th Floor, articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the New York, NY 10016-5997 USA, Telephone: +1 212 419 7900. Responsibility for the content rests code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 upon the authors and not upon the IEEE, the Society or its members. IEEE Service Center USA; 2) pre-1978 articles without a fee. For other copying, reprint, or republication permission, write (for orders, subscriptions, address changes): 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ 08855 Copyrights and Permissions Department, IEEE Service Center, 445 Hoes Lane, Piscataway, USA. Telephone: +1 732 981 0060. Individual copies: IEEE Members US$20.00 (first copy only), NJ 08854. Copyright © 2022 by the Institute of Electrical and Electronics Engineers Inc. All rights non-Members US$140 per copy. Subscription rates: Annual subscription rates included in IEEE reserved. Periodicals postage paid at New York and additional mailing offices. Postmaster: Send Robotics and Automation Society member dues. Subscription rates available on request. Copyright address changes to IEEE Robotics and Automation Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ and reprint permission: Abstracting is permitted with credit to the source. Libraries are permitted to 08854 USA. Canadian GST #125634188 PRINTED IN THE U.S.A. 2 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022 Say Hello_to the next robotic innovator Research teams from around the globe were asked to submit their concepts for the “Open Platform Challenge”. Three teams made it to the finals and will be presenting their projects on automatica fair in June 2023. The award comes with a 20,000-euro prize. Meet our finalists. KUKA Innovation Award 2023 MeRLIn Team SPIRIT, Team Fashion & Robotics, Team JARVIS, German Aerospace Center, Germany University of Arts and Industrial Design Politecnico di Milano, Italy A single large refinery often requires Linz, Austria The goal of the team from Politecnico more than 50,000 maintenance and The fashion and textile industry are di Milano is to develop a complete inspection routines. The team from the under great pressure to reduce the plug-and-play method for programming German Aerospace Center is working on a environmental footprint by producing collaborative robotic applications telepresence solution that enables safe and more durable, high-quality products and fully compatible with the new iiQKA intuitive operation of aerial manipulators developing circular material flows. The ecosystem. This will facilitate their in- for industrial applications in the oil and gas departments for “Creative Robotics” and troduction in small and medium-sized industry. “Fashion and Technology” at UfG Linz are enterprises, enabling unskilled working on creating a way for textile operators to program the robot for a companies and designers to increase their new task and, thanks to the system production by setting up micro factories AI-enhanced capabilities, generalize to with collaborative robot systems. unknown situations, new tasks, and product variants. www.kuka.com/InnovationAward2023 FROM THE EDITOR’S DESK Humans and the Environment By Yi Guo T he COVID-19 pandemic has We need to nurture the envi- concepts involve principles changed a lot of things, one ronment we live in and found in nature for creating of which is human behavior. avoid environmental dam- robot locomotive capabili- For me, I found a new hobby age that essentially endan- ties; multirobotic systems of hiking during the first year of the gers human health. study how animal swarms pandemic. I hiked in dozens of state The role nature plays in communicate and exchange parks around me during the fall and human welfare and economic information and the hierarchical winter sea- activity has been overlooked. Complex and territorial structure of animal societ- sons, some of natural systems can flip from one equi- ies; and robotics learning draws inspira- We need more and which I did librium to another when under pres- tion from human learning and neural not even know sure. The demands humans currently network-based brain cognition capabil- more innovative existed before place on nature may not be sustainable ity. The field of soft robotics stems from robots to join the pandemic. by Earth’s ecosystem. Natural capital bioinspired concepts and research on I felt relieved was recently included in an analysis of deformable structures, soft materials, humans’ effort both physical- the sustainability of current rates of and morphological computation. While to protect the ly and mental- economic growth. As we learn more bioinspired robotics is a very broad ly after the through interaction with nature, it is research area, challenges exist in many environment. weekend hik- necessary to be aware of the sustain- frontiers, including the integration of sci- ing trips, and ability of the ecosystem around us and ence and technology advances toward it was helpful pay attention to the impact of human enhanced robotics capability in percep- for me to reduce the Zoom fatigue behavior and what new technology will tion, cognition, learning, and control. built up during work days. impose on the environment. This December issue is a special issue Nature creates the environment that Robots can help protect the environ- (SI) on biomimetic perception, cogni- we live in. Environmental damage can ment by adding monitoring capabilities tion, and control: “From Nature to lead to new diseases. Nearly two-thirds in extreme conditions, such as robotic Robots.” We received more than 30 sub- of the hundreds of diseases that emerged vehicles for supply transport, under-ice missions to this themed issue, which is in the past century were transmitted exploration to support climate change the highest number of SI submission in from animals to humans, including studies in the Arctic, and surface and the past two years. I’d like to thank the HIV/AIDS and probably COVID-19. underwater robots to monitor chemical lead guest editor, Dr. Chenguang Yang, Scientists also found a correlation and oil pollution. Robotics can also and his team, Dr. Shan Luo, Dr. Nathan between the loss of forests in Africa and help fight climate change by reducing Lepora, Dr. Fanny Ficuciello, Dr. Dong- the outbreaks of Ebola. Altering land use carbon emissions through renewable heui Lee, Dr. Weiwei Wan, and Dr. threatens biodiversity, and deforestation energy resources, helping crops survive Chun-Yi Su, for their hard work manag- and intensive farming are linked to out- droughts, and planting trees. We need ing the peer review process within the breaks of transmitted diseases. Also, sci- more and more innovative robots to constrained time frame of this SI. I’d also entists have studied how people have join humans’ effort to protect the envi- like to thank the assisting IEEE Robotics altered living organisms and exerted an ronment. We have no choice. and Automation Magazine associate edi- evolutionary pressure on other species. Bioinspired robotics is a field of tor, Surya Nurzaman, for his help on the robotics that studies biological systems SI. I hope you find inspiration from the to achieve engineering goals. Among feature articles. Enjoy reading! Digital Object Identifier 10.1109/MRA.2022.3213198 Date of current version: 2 December 2022 the well-studied topics, locomotion 4 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022 2 2 0 2 PRESIDENT’S MESSAGE Rethinking the Research Paper By Frank Park L ike many of you, I’ve spent ing as a technical field. Our details) and performed the a good part of the summer work is more and more comparison experiments as months harried by paper dead- embodied and validated by faithfully as we could. We’ve lines. For each paper, the story code, data, and bench- not yet received feedback on unfolds something like this: the tri- marks, and like the machine these latest set of experiments, umph and euphoria that one experi- learning and data science but already I can anticipate ences immediately after submitting the research communities, there is heated discussions on the experi- paper quickly wears off, to be replaced growing acceptance within our com- mental setup, algorithm implementa- by an encroaching regret that with munity on the need to make code and tion, and how truly meaningful these more time, the data available. comparisons are in the absence of acces- paper could What we still seem to lack in robot- sible, reproducible benchmarks. While more evidence have been so ics, however, are benchmarks. To be fair, I have also noticed that several our much better. benchmarks have been developed for conferences are now experimenting needs to be collected Then the re- grasping, pick-and-place, and various with double-anonymous (or double- before any definitive views arrive: other robotics tasks, but these tend to be blind) reviews, open discussion phases poring over the narrowly defined for specific hardware between reviewers and authors, and conclusions can be comments, de- and environment requirements and are several other new practices. All these drawn, experimenting cisions are difficult to implement. Simulation efforts to try to reduce bias in our quickly made benchmarks have been proposed as an reviews—be it gender, nationality, with new review on whether to alternative, but even the best simulators author or institutional reputation, or practices is a very accept or reject today lack the ability to model friction, any number of factors— are highly wel- a criticism— contact, deformations, and other com- come and in keeping with our commu- welcome development. should I hold plex physical interactions in a realistic nity’s spirit of innovation and fairness. firm or show way. I think it’s fair to say that the Some members of our community are some contrition for not explaining inherent challenges of developing currently studying the advantages and more clearly?—and whether to comply benchmarks for robotics are much challenges of implementing double- with all the requests (demands?) for ad- harder than, say, those for vision or nat- anonymous reviews, as well as the ditional experiments and comparisons ural language. experiences of other research commu- to existing work. The road to publica- Returning to the review of our paper, nities. While more evidence needs to be tion is always a bumpy one, it seems. not surprisingly a reviewer had asked for collected before any definitive conclu- Lately I’ve noticed that reviewers comparisons of our method against sions can be drawn, experimenting seem to be asking for more and more another recently published method, with new review practices is a very wel- elaborate experimental comparisons claiming this to be the state-of-the-art. come development. against the state-of-the-art. At one Since no code was provided for this I encourage you to let your voices be level, it’s a sign that robotics is matur- state-of-the-art method, we spent a great heard in the ongoing community dis- deal of effort implementing the algo- cussion on how we measure research rithm (whose description turned out to progress, and exploring ways to further Digital Object Identifier 10.1109/MRA.2022.3214390 Date of current version: 2 December 2022 be lacking some small but crucial reduce bias in our review process. 6 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022 FROM THE GUEST EDITORS Biomimetic Perception, Cognition, and Control: From Nature to Robots By Chenguang Yang , Shan Luo, Nathan Lepora, Fanny Ficuciello, Dongheui Lee, Weiwei Wan, and Chun-Yi Su Awide range of technological surface electromyography signals and using the detachment of feathers while developments are inspired by A-mode ultrasound signals of forearm maintaining the same low-level control- biological individuals and ad- muscles. Yu et al. introduce a hybrid ler. Bing et al. apply reinforcement learn- vanced synthetic materials, visual–haptic framework enabling a ing (RL) on snake-like robot control that cognitive sensors, control algorithms, robot to achieve motion synchroniza- can fully exploit the hyper-redundant artificial intelligence technology, and in- tion in human–robot co-transporting. bodies of the robot. Simulations and telligent systems. One of the major chal- Graf et al. build a framework for experiments show that RL can generate lenges is to create a comprehensive transferring ideas from human scene substantially more energy-efficient gaits study by integrating different tech- perception to robot scene perception than those generated by conventional niques into ro- to contribute toward robots’ holistic model-based controllers. Jiang et al. pro- botic systems so scene understanding, based on a wide pose a humanoid control method based On the topic of that the perfor- survey and comparison of robotic on visual servoing by utilizing a swivel mance of ro- scene perception approaches with angle derived from the human arm to robot system and bots can be neuroscience theories and studies of realize the human-like behavior of control, Stella et al. improved and human perception. anthropomorphic robot manipulators. applied to more In the area of intelligent cognition The proposed method is substantiated develop a feather complex and and learning, Zheng et al. propose a based on a 7-degree-of-freedom Sawyer star-like robot that diverse scenari- biomimetic electric sense-based local- robot and a constructed visual servo os. The articles ization scheme, including an electric physical platform. can actuate layers of in this issue sense-based localization scheme and In the field of biomimetic sensors, flexible feathers and focus on the three model-based perception meth- Zhang et al. design a rat-inspired state of the art ods for large-scale underwater local- whisker sensor for a biomimetic ro- detach them at will. in biomimetic ization. Lu and Wang introduce a botic rat and demonstrate its superior perception, novel biomimetic force and impedance tactile perception performance. Ex- cognition, and control research and aim adaption framework based on Broad perimental results demonstrate its to explore related technical avenues in Learning System for robot control in outstanding texture discrimination the multimodal bioinformation percep- stable and unstable environments. The ability and excellent performance on tion framework, intelligent cognition connections of created neural network contour reconstruction. Xia et al. de- and learning, robotic systems and con- layers and the settings of the feature velop a soft magnetic tactile sensor trol, and new biomimetic sensors. nodes are explainable by human motor and a multipole magnetization meth- In the research of multimodal bio- control and learning principles. od for extracting contact surface fea- information perception framework, On the topic of robot system and tures. We sincerely thank all the Zou et al. propose the clinical readi- control, Stella et al. develop a feather authors for their contributions and ness of a multimodal fusion model star-like robot that can actuate layers of the editors for their efforts and hope to estimate hand force based on flexible feathers and detach them at will. that the contents of this issue can Based on this optimized feather and bring information and inspiration to theoretical framework, the new robotic researchers in related fields. Digital Object Identifier 10.1109/MRA.2022.3213199 Date of current version: 2 December 2022 setup can change its motion path by 8 • IEEE ROBOTICS & AUTOMATION MAGAZINE • DECEMBER 2022