Bhabatosh Chanda · Subhasis Chaudhuri Editors Santanu Chaudhury Heritage Preservation A Computational Approach Heritage Preservation Bhabatosh Chanda Subhasis Chaudhuri (cid:129) Santanu Chaudhury Editors Heritage Preservation A Computational Approach 123 Editors Bhabatosh Chanda SantanuChaudhury Electronics andCommunication CSIR-CEERI SciencesUnit Pilani,Rajasthan Indian Statistical Institute India Kolkata, West Bengal India SubhasisChaudhuri Department ofElectrical Engineering Indian Institute of Technology Bombay Mumbai,Maharashtra India ISBN978-981-10-7220-8 ISBN978-981-10-7221-5 (eBook) https://doi.org/10.1007/978-981-10-7221-5 LibraryofCongressControlNumber:2018935209 ©SpringerNatureSingaporePteLtd.2018 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodologynowknownorhereafterdeveloped. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations. Printedonacid-freepaper ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd. partofSpringerNature Theregisteredcompanyaddressis:152BeachRoad,#21-01/04GatewayEast,Singapore189721, Singapore Preface Heritage records the history of our world—social, religious, ecological, cultural, and political. It represents the footprint of the temporal evolution of civilizations andecosystems.Asitisinheritedfromthepast,itshouldbebestowedonthefuture for thebenefitof subsequent generations. It isthe legacy oftangibleartifacts (e.g., monuments, statues, manuscripts, and paintings) and intangible elements (e.g., customs,religion,culture,legends,andhistory).Thevulnerabilityofheritageitems totheharmfuleffectsofclimatechange,pollution,naturalandman-madedisasters, neglect,infiltrationofnewideasandpractices,vandalism,andevenmasstourismis becoming more and more evident over the years. Thus, preservation, renovation, restoration, and management of heritage items have become an important task nowadays. However, the physical renovation and restoration work may cause damage to otherwise undamaged part of the artifacts and may fail to provide the flavor of history. Sometimes, this loss may be irrecoverable. Second, the heritage artifacts may not be physically accessible for renovation and restoration work due to various reasons. So, a plausible solution might be restoring and reconstructing the damaged artifacts digitally using computational methods without any physical embodiment. Being re-rectifiable as well as inexpensive, the digital preservation, restoration, and reconstruction of heritage items have become an active research field.Heritage,bothtangibleandintangible,mayalsoberecorded,represented,and archived in digital space conveniently by means of appropriate computational methods. This handling of heritage items in digital space is popularly known as digital heritage preservation and is an emerging area of research incomputational techniques. This book presents a balanced treatment of heritage preservation problems and state-of-the-art digital techniques to achieve their novel and useful solutions. It covers the methods starting from data acquisition and digital imaging to compu- tational methods forrecreationoforiginal(pre-damaged)look ofheritageartifacts. Case studies are almost equally drawn from the tangible and non-tangible heritage which is multi-dimensional as well as rich. Authors of the articles have been working in their respective fields for years and have presented their working v vi Preface experience and methods so lucidly that these can be easily implementable. Both methods and results are illustrated with a large number of color images. This book consists of sixteen articles covering five major aspects of digital her- itage research, namely (i) digital system for heritage preservation including data acquisition, (ii) signal and image processing, (iii) audio and video processing, (iv)imageandvideodatabase,and(v)architecturalmodelingandvisualization.The firstaspectiscoveredbythefirstthreearticleswrittenbyGeorgopoulos(Article1), MacDonald (Article 2), and Yao et al. (Article 3). In Article 1, technological advancesin digitalheritagepreservationarebriefly presented andexplained.It also presents the currently available digital documentation techniques along with the contemporary nondestructive techniques for cultural heritage pathology diagnosis and conservation. Digital heritage requires the represented physical objects to look realistic when viewed on a display screen. Article 2 presents a new method for estimating the reflectance distribution function from a set of images taken in an illumination dome. Surface normal and albedo are calculated by regressing over a subset oftheintensityateachpixel. Ground truth collectionandlargedata analysis and evaluation are the important tasks for developing methods for digital heritage. Article 3 presents an open,cross-platform, effective,andextensible GUI annotation tool, known as Epix, for that purpose. This tool can also be used for multiple researchpurposes,includingEuclideanepipolarerrormeasurementand3Ddistance measurement.Signalandimageprocessingaspectsaredealtinthenextfivearticles written by Madhu and Joshi (Article 4), Kumar et al. (Article 5), Frohlich et al. (Article 6), Sreeni and Chaudhuri (Article 7), and Mukherjee and Sural (Article 8). The problems dealt here range from single-image super-resolution (SR) using deep learning framework (Article 4) to source-constrained exemplar-based inpainting for restoring images of old degraded cave paintings (Article 5), from fusion of 3D and 2Dvisualdataforcolorizingthepointcloudfrommultiplecameraswithapplication to old churches (Article 6) to haptic rendering of object described by a dense, oriented point cloud data without a pre-computed polygonal mesh (Article 7). Finally, Article 8 presents an ontology-based approach to retrieve structurally and semantically similar images from heritage image dataset. Articles 9, 10, 11, and 12 covertheaudioandvideoprocessingaspectsofdigitalheritage.Article9(writtenby Mandal et al.) develops a formal model for the Ragas, consequently classifies the Ragas, and evaluates music scores based on the Ragas. Like music, films are also important media for cultural heritage. A unified approach for detecting some of the most commonly appearing noise artifacts in heritage movies and restoring them to achieve a superior visualqualityforviewingandarchivalispresented inArticle 10 (writtenbyBhattacharyaetal.).NexttwoarticleswrittenbyMohantyetal.(Article 11)andMulliketal.(Article12)dealwithdance,morespecificallyBharatanatyam, and its various components such as hand gestures, facial expressions, and dynamic body postures are called Adavus. Benchmark dataset is an essential part of evalu- atingcomputervisionalgorithms.Articles13and14,respectively,presenttwonovel datasets: (i) an image dataset of different temples in Bishnupur suitable for evalu- ating computer vision techniques such as 3D reconstruction, image inpainting, texture classification, and content-specific figure spotting and retrieval (written by Preface vii Ghorai et al.) and (ii) a novel dataset for evaluating computer vision method for recognitionofSattriyadancehandgestures(writtenbyDevietal.).Finally,lasttwo articles cover architectural modeling and visualization aspects. Article 15 outlines the process and methodology of arriving at visualizations of clothing style of Vijayanagaraperiodthroughstudyandanalysisoftextualnarratives,murals,stucco work, and relief work of this period, while Article 16 presents a thorough study, documentation, analysis of design elements, and 3D virtual modeling as well as reconstruction of the Krishna temple complex at Hampi, Karnataka, India. In essence, this book is a comprehensive representation of various aspects and state-of-the-art methodologies suitable for heritage preservation through computa- tional approach. We express our utmost gratitude to the contributors for sharing their experience through these articles. We are thankful to the reviewers without whose support it would not be possible to select the high-quality works for this book.WearethankfultoSaurabhKumarofIITBombayforhelpingusonhandling the Web interface for all communication and review activities of the submitted manuscripts.Someofthesearticlesareextendedversionsofthepaperspresentedin theWorkshoponDigitalHeritageorganizedinassociationwithIndianConference on Computer Vision, Graphics and Image Processing held at Guwahati in December 2016. We are also thankful to the organizers of the conference and the workshop. Lastbutnot least, we are grateful to the editorial team of Springerwho has persuaded and helped to bring this book into existence. Editors gratefully acknowledge the support of DST, Ministry of Science and Technology, Government of India, in this endeavor. Kolkata, India Bhabatosh Chanda Mumbai, India Subhasis Chaudhuri Pilani, India Santanu Chaudhury Contents Contemporary Digital Technologies at the Service of Cultural Heritage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Andreas Georgopoulos Visual Realism in Digital Heritage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Lindsay W. MacDonald EpiX: A 3D Measurement Tool for Heritage, Archeology, and Aerial Photogrammetry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Shizeng Yao, Hang Yu, Hadi AliAkbarpour, Guna Seetharaman and Kannappan Palaniappan Digital Heritage Reconstruction Using Deep Learning-Based Super-Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Prathmesh R. Madhu and Manjunath V. Joshi Restoration of Digital Images of Old Degraded Cave Paintings via Patch Size Adaptive Source-Constrained Inpainting. . . . . . . . . . . . . 87 Veepin Kumar, Jayanta Mukherjee and Shyamal Kumar Das Mandal 3D–2D Data Fusion in Cultural Heritage Applications. . . . . . . . . . . . . . 111 Robert Frohlich, Stefan Gubo, Attila Lévai and Zoltan Kato Haptic Rendering of Oriented Point Cloud of Heritage Objects Using Proxy Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 K. G. Sreeni and Subhasis Chaudhuri Ontology-Driven Content-Based Retrieval of Heritage Images. . . . . . . . 143 Dipannita Podder, Jit Mukherjee, Shashaank Mattur Aswatha, Jayanta Mukherjee and Shamik Sural On the Deep Structure of Ragas and Analytic Rating of Music Scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Sudipa Mandal, Shilpi Chaudhuri, Antonio Anastasio Bruto da Costa, Gouri Karambelkar and Pallab Dasgupta ix x Contents Restoration of Archival Videos for Preserving Digital Heritage of India . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Saumik Bhattacharya, K. S. Venkatesh and Sumana Gupta Nrityamanthan: Unravelling the Intent of the Dancer Using Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Aparna Mohanty, Kankana Roy and Rajiv R. Sahay Characterization, Detection, and Synchronization of Audio-Video Events in Bharatanatyam Adavus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Tanwi Mallick, Partha Pratim Das and Arun Kumar Majumdar An Image Dataset of Bishnupur Terracotta Temples for Digital Heritage Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Mrinmoy Ghorai, Sanchayan Santra, Soumitra Samanta, Pulak Purkait and Bhabatosh Chanda A Dataset of Single-Hand Gestures of Sattriya Dance . . . . . . . . . . . . . . 293 Mampi Devi, Sarat Saharia and D. K. Bhattacharyya Analysis of Murals, Stucco Work, Relief Work and Textual Narratives for Understanding Clothing Styles of Vijayanagara Period . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Mamata N. Rao Visually Reconstructed Krishna Temple, Hampi, Karnataka. . . . . . . . . 333 Meera Natampally About the Editors BhabatoshChanda isaProfessorattheIndianStatisticalInstitute,Kolkata,India. His research interests include image and video processing, pattern recognition, computer vision, and mathematical morphology. He has published more than 200 articlesinrefereedjournalsandconferences,authored2books,andedited5books. He received the Indian National Science Academy ‘Young Scientist Medal’ in 1989, the Institution of Engineers (India) ‘Computer Engineering Division Medal’ in 1998, the ‘Vikram Sarabhai Research Award’ in 2002, and the ‘IETE-Ram Lal Wadhwa Gold Medal’ in 2007. He is a Fellow of the Institute of Electronics and Telecommunication Engineers (FIETE), National Academy of Science, India (FNASc.),IndianNationalAcademyofEngineering(FNAE),andtheInternational Association of Pattern Recognition (FIAPR). Subhasis Chaudhuri holds the KN Bajaj Chair Professorship at the Indian InstituteofTechnologyBombayandisaJ.C.Bosenationalfellow.Heworksinthe area of image processing, computer vision, and haptics. He has published 300 papers in various journals and conferences, as wellas 6 monographs in his area of research.Hehasalsoservedontheeditorialboardofseveralinternationaljournals, includingIEEET-PAMIandIJCV.HeisarecipientoftheShantiSwarupBhatnagar Prize in Engineering Sciences. He is a Fellow of the IEEE and the Science Academies in India. Santanu Chaudhury is currently the Director of the CSIR-CEERI, Pilani, and a Professor in the Department of Electrical Engineering, Indian Institute of Technology Delhi (IIT Delhi). He was also the Dean of Undergraduate Studies at IITDelhi.HewasSchlumbergeraswellasDhananjoyChairProfessoratIITDelhi. His research interests include computer vision, robotics, embedded systems, and machinelearning.Hehaspublishedmorethan250researchpapersininternational journals and conference proceedings and a number of patents. He has been on the program committee of a number of international conferences like ICCV, ACCV, xi