Sensors 2012, 12, 5705-5724; doi:10.3390/s120505705 OPEN ACCESS sensors ISSN 1424-8220 www.mdpi.com/journal/sensors Article Automatic Construction of 3D Basic-Semantic Models of Inhabited Interiors Using Laser Scanners and RFID Sensors Enrique Valero 1,2,*, Antonio Adan 2 and Carlos Cerrada 1 1 School of Computer Engineering, Universidad Nacional de Educación a Distancia (UNED), C/Juan del Rosal, 16. 28040 Madrid, Spain; E-Mail: [email protected] 2 3D Visual Computing and Robotics Lab, Universidad de Castilla-La Mancha (UCLM), Paseo de la Universidad, 4. 13071 Ciudad Real, Spain; E-Mail: [email protected] * Author to whom correspondence should be addressed; E-Mail: [email protected]. Received: 22 March 2012; in revised form: 16 April 2012 / Accepted: 2 May 2012 / Published: 3 May 2012 Abstract: This paper is focused on the automatic construction of 3D basic-semantic models of inhabited interiors using laser scanners with the help of RFID technologies. This is an innovative approach, in whose field scarce publications exist. The general strategy consists of carrying out a selective and sequential segmentation from the cloud of points by means of different algorithms which depend on the information that the RFID tags provide. The identification of basic elements of the scene, such as walls, floor, ceiling, windows, doors, tables, chairs and cabinets, and the positioning of their corresponding models can then be calculated. The fusion of both technologies thus allows a simplified 3D semantic indoor model to be obtained. This method has been tested in real scenes under difficult clutter and occlusion conditions, and has yielded promising results. Keywords: 3D modeling; RFID; laser scanner; 3D data processing 1. Introduction The concept of “intelligent environment” arose in the late 90s to characterize a set of technological elements which were capable of communicating with the user and making decisions by means of an intelligent process. This interaction between user and device is carried out in a transparent manner, in the sense that the presence of these elements is sufficient to establish communication between them. Sensors 2012, 12 5706 In [1], the authors afford a solution to link contextual information with user interactions from daily activities, in an implicit and transparent way. One of the best known examples concerning intelligent environments is radio frequency identification (RFID) technology. RFID systems are able to store and retrieve significant data from identified items by means of small tags in which this information can be easily read and written. These small tags can be attached to or incorporated into products, animals or people, and contain relevant information about the associated item, thus permitting their identification and the control of some of their features. The relative low cost of this technology, together with its rapid development and the tags’ adaptability for use on a great variety of surfaces have led to the constant expansion of RFID technology over the last few years. There are diverse fields of application of this technology. In the fields of manufacturing and distribution, RFID systems have been used in the identification of packing and containers and for component tracking in factories [2,3]. Moreover, savings of time and paper are achieved in these cases because the inspection registers can be certified by writing on the tags which are attached to the elements [4]. Other authors [5] propose an application for a conference site through identification by RFID. In this work, people attending the conference wear RFID tags and the session room is equipped with antennas which detect the presence of the speakers in the room and facilitate the proceedings. This intelligent technology is also used in medical environments to locate the hospital equipment and improve its management, control patients' activities in hospitals, or help people who need assistance [6]. RFID technology has also been applied in the field of robotics where it has been used as a complementary technology to laser sensors for robot localization and for environmental map construction, for example in [7]. The robotic platform described in [8] has been built to assist the visually impaired in supermarkets or airports. The intelligent system helps them to avoid collisions or to indicate how to follow itineraries. Tong and Zekavat [9] propose the use of tags and readers for the safer navigation of mobile robots, by avoiding collisions and estimating times and directions of arrival. Other authors [10] combine RFID technology with GPS to improve the precision of the navigation systems in road vehicles. RFID and 3D scanners has been already used by other authors [11] in computer vision field with recognition purposes. They detect the presence of a given object in a scene by reading the RFID-EPC code of the tag attached to it. Presence information helps to alleviate the searching processes, achieving significant computation time reduction in the object recognition algorithms. However, in this work only an identification code for every object is written in the tags but information concerning to dimensions, type of element or color is not provided. 2. RFID and Laser Scanning Technologies in Buildings 2.1. RFID Applications Our work with RFID technologies is framed in the area of construction, particularly in civil and industrial buildings. RFID technology fits well into the Building Information Modeling (BIM) standard that won general acceptance in the Architecture Engineering and Construction (AEC)-field several years ago. The advantage of BIM is that the entire life-cycle of the building is covered by Sensors 2012, 12 5707 a unified numerical model. RFID tags help to identify elements of this model on-site. A wide number of references exist in the AEC field. RFID technology is becoming a very useful tool in this research field because of its rapidity in identification and information localization, together with the tags’ durability and the high range of action of these systems. Linking the RFID technology to the construction industry is a challenge which began the late 90s, particularly in areas such as vehicle access control, personnel control, tool maintenance, or cycle times and resource tracking. In fact, the principal application of the RFID in the construction processes is currently the tracking of materials. Several authors [12,13] report endowing materials with tags in order to know their movements within the area of construction from reception to installation. Other authors propose the fusion of technologies by equipping the tag readers with GPS antennas to obtain the precise position of the constructive elements [14,15], or by verifying the progress of the activities by means of laser sensors and photogrammetric techniques in addition to RFID [16]. Works linked to the study of the traceability of pipe spools also exist [4,17,18]. An interesting application presented in [19] refers to the prevention of accidents caused by collisions in places under construction. Structures, machinery and workers are provided with tags and readers that allow their risk in the scenarios to be evaluated. This technology is also used for the management and control of workers’ punctuality [4,20], and the access control of vehicles to the construction area [21]. Numerous works can also be found in the field of maintenance and inspection. Kondo et al. [22] present a study on the draining of a system of pipelines. They use a set of tagged balls both to automatically verify the evacuation time of each one and to check the correct operation of the system. Information management in the areas of civil engineering and construction goes beyond the constructive process. It is also necessary to record certain data during the lifespan of the building. The safety of a building can be improved, for example, with a good management of its set of fire-extinguishers, by indicating on their tags whether any inspection is required, whether they work properly or where they are located [23]. Moreover, the application proposed in [24] may assist rescue teams in a fire. Information from different tags relevant to a building’s fire control (maps, number of plants, location) is gathered and sent by means of a PDA system with RFID. With regard to maintenance and inspection, Cheng [25] proposes an information system for the maintenance of open buildings. The tracking of the elements comprising the building that must be evaluated for their replacement is carried out with this application. 2.2. Laser Scanning Applications Our work is focused on the automatic construction of 3D basic semantic models of inhabited interiors using laser scanners with the help of RFID technologies. An inhabited environment involves certain disorder in the scene: there are objects on tables, in racks, papers stuck to walls and to windows, etc. Moreover, some elements in a scene may occlude others, signifying that they cannot be entirely sensed by a laser scanner. In the field of laser scanning applied to buildings, there are some lines of work particularly developed during the last years. Different techniques for the automatic reconstruction of as-built facilities are presented in [26]. The authors review several previous works focused on basic structures recognition, geometric modeling and constructive elements relationship. Sensors 2012, 12 5708 As regards the detection and modeling of single objects or parts of large scenarios, Kwon et al. [27] introduce a set of algorithms which fits sparse point clouds to single volumetric primitives (cuboids, cylinders and spheres). The algorithm is extended to groups of primitives belonging to the same object. Another work [28] identifies and localizes relevant kitchen objects including cupboards, kitchen appliances, and tables. They interpret the point clouds in terms of rectangular planes and 3D geometric shapes. Valero et al. [29] focus on the modeling of those linear moldings that typically surround doorways, windows, and divide ceilings from walls and walls from floors. In [30] an automated recognition/retrieval approach for 3D CAD objects in construction context is presented. Bosche [31] proposes a semiautomatic method to match 3D existing models to the collected data in industrial building steel structures. The author develops a variant of the ICP algorithm to recognize CAD models objects in large site laser scans. The same author presents in [32] a plane-base registration system to coarsely align laser scanners data with the project 3D models in the AEC/FM industry context. As regards walls and façades detection, detailed models of part of these components are obtained in [33–36]. Bohm et al. [33,34] propose a method where the data processing goes from detecting windows through low data density regions to discover other data patterns in the façade. Important façade elements such as walls and roofs are distinguished as features in [35]. Thrun et al. developed a plane extraction method based on the expectation-maximization algorithm [37]. Other researchers [38,39] have proposed plane sweep approaches to find planar regions. Früh et al. [36] develop a set of data processing algorithms for generating textured façade meshes of cities from a series of vertical 2D surface scans. To perform the automatic reconstruction of a scene, we process a dense cloud of points to identify certain objects whose 3D models are stored in a knowledge base. Owing to the occlusions and the complexity of the geometry of the scanned elements, the identification of objects from the cloud of points generated by the scanner is a very difficult task. To date, few papers have tackled the occlusion problem by using range data for indoor reconstruction. Papers which process laser scanner data to model predominantly planar surfaces, such as walls, floors, and ceilings, can be found in [40]. In [41], the authors are also able to detect and model openings, and fill in the occluded regions. Of course, many more related papers exist which use range images and deal with specific topics such as plane detection, wall identification, 3D scene segmentation, the reconstruction of surfaces, detailed, precise, momentary occlusion and hole-filling, but none of them provide an extended solution in complex scenes covering all those aspects. Adan [42] presents a method to automatically convert the 3D point data from a laser scanner positioned at multiple locations throughout a building into a compact, semantically rich model. In this work, the main structural components of an indoor environment (walls, floors, ceilings, windows and doorways) are identified despite the presence of clutter and occlusion. As has already been mentioned, this article proposes a solution based on the combination of two technologies of a different nature, these being laser scanners and radio frequency identification. RFID tags adhered to different objects provide valuable information about them, which makes the identification and positioning of basic elements in the scene faster and easier. In this work, we propose an innovative approach in this challenging research field. Obtaining an accurate model of an interior can be very useful in applications related with exploration, manipulation and autonomous navigation Sensors 2012, 12 5709 with mobile robots or vehicles. In these cases the robot needs to know the principal constructive elements of the interior, the objects in the scene and their position. The paper is organized as follows: Section 3 presents the structure of the proposed system, based on two different sensorial technologies. In Section 4 we show the information that is available from both the laser scanner and the RFID sensor. Section 5 proposes a solution by which to identify and calculate the pose of basic elements (both static and dynamic) in a facility. Finally, Section 6 shows the experimental results obtained with our proposal. 3. Overview of the System The outline of the proposed system is depicted in Figure 1. The system is based on two technologies connected by a specific integration block. The treatment of 3D information in the scene, represented by the 3D Vision Techniques block, considers both the acquisition and the processing of 3D data. A FARO Photon 80 scanner has been used in this block, and the data obtained from this 3D sensor provides geometric and color information of the scene. Bearing an automatic generation of the model of the scanned scene in mind, it is necessary to recognize the elements that are present in the scanned zone. RFID technology can assists vision technologies in this task by providing relevant information to identify and pose the elements in the scene. The RFID System block represents the treatment of the information stored in the passive RFID tags. An OBID sensor, model LRU 3500 from FEIG Company, connected to an ultra-high frequency (UHF) antenna has been used in this block. This sensor allows information to be read and written on the tags. Both, 3D scanner and RFID system, work together during the scanning process. The RFID reader is onboard the scanner platform. Thus 3D data and tag detection can be obtained at the same time. We denote this system in Figure 1 as “Intelligent 3D Vision System in a Tagged Universe”, in which data from the two sensors is used to identify elements and to reconstruct the 3D model of the scenario. Figure 1. Overview of the system proposed in this work, where 3D vision and RFID technologies are fused. Sensors 2012, 12 5710 With the purpose of obtaining a general view of the process carried out in this work, a flowchart is presented in Figure 2. This diagram shows the different methods executed during the data acquisition by means of the above mentioned sensors. Figure 2. Flowchart of the acquisition data process. At first, the two sensors (scanner and RFID reader) are placed together in several strategic locations of the scene to cover the whole indoor environment. For each of these positions, a data acquisition process is carried out. A set of 3D points is provided by the laser scanner and at the same time, certain geometry and color information of the different elements is obtained from the tags by means of the RFID sensor. The 3D point clouds, which are acquired from different positions of the inhabited interior, are pre-processed and registered in accordance with a universal coordinate system (UCS). These two datasets, corresponding to 3D points and object information (in purple in Figure 2), are the inputs of the algorithms which identify and pose the basic elements in the scene. These algorithms will be explained in Section 5. 4. Available Information from Laser Scanner and RFID 4.1. Laser Scanner 3D information of the environment is obtained by virtualizing the scenario by means of the laser scanner sensor mentioned in the previous section. Its field of vision allows it to be employed in scenes Sensors 2012, 12 5711 under study up to 80 m. The information provided by the sensor is associated with the geometry and the texture of the scanned room. Several acquisitions of the same room are carried out from different points of view. Then, all the range images provided by the sensor are filtered and transformed to a universal coordinate system in a first pre-processing phase. For a given acquisition, several fiducials are identified in a planar image of the scene and the user marks their centers of mass. A set of correspondences between the center points from two different sensor positions are manually established, and a transformation matrix is obtained between both reference systems by means of commercial scanning software. The same procedure is followed with the next sensor positions, having considered the first sensor position as the universal coordinate system. All the data thus remain registered in a common reference system, as do all the sensor positions considered. This registering process is habitual in the field of the digitalization of interiors or exteriors in buildings. Figure 3 shows the registration of two digitalizations of an inhabited interior. Figure 3. Joint registering of two digitalizations of a scanned inhabited interior. All the available geometric information of the scene consists of an unconnected set of points which is gathered after registering all the 3D data under a common reference system. Certain geometric features can be extracted from the treatment of these data which allow the automatic segmentation and recognition of certain components. Walls, ceilings, floors, doorways and windows or the frames around them can be obtained in this manner. Nevertheless, these recognition processes are extremely complex in inhabited interiors. 4.2. RFID RFID technology is an information exchange system for automatic identification inspired by radio frequency waves. The information is stored in devices named tags or transponders. Three types of tags exist: active, passive and hybrid. Active tags incorporate the power supply for their circuits and propagate the signal to the reader. Passive tags obtain the required energy by induction from the readers. Passive tags can be used for distances of up to approximately 15 m, whereas active tags have Sensors 2012, 12 5712 a much bigger range of action (up to 500 m) and can store a large quantity of information. Hybrid tags can transmit, but they have to be told to transmit. They need to be turned on by a signal, as could be a satellite. The system’s range of action is also influenced by the type of antenna installed in the reader. The use of a low frequency (LF) antenna or a high frequency (HF) one thus allows operating distances in the range of dozens of centimeters, whereas the installation of an ultra-high frequency (UHF) antenna allows communication in the range of meters. The previous information will allow the reader to deduce that the greatest range of communication is achieved by the use of UHF antennas and active tags. In our particular case, where the required working range is of dozens of meters and where it is necessary to modify the tags’ content, we have used UHF antennas (865–868 MHz) and passive tags with a storage capacity of up to 512 bytes. Figure 4 shows that the tags are located in a disperse manner in the scene and, as has been mentioned previously, they contain information about attributes corresponding to the objects to which they are adhered. One advantage of this technology as opposed to others such as bar codes or infrared light is that the tags do not need to have a direct vision of the reader to gather the information. They can therefore be attached to any of the surfaces of the objects located in the scene. Nevertheless, if the tag is attached over a metallic surface, it must be placed approximately 1 cm away from the surface to be properly detected. Figure 4. (a) Diagram representing the laser scanner and the RFID system in a room. (b) Several tags placed on dynamic elements of the room. (a) (b) In our work, tagged elements are classified into static (structural) and “dynamic” (equipment). Until now, the model of the scene has been reduced to a set of static and dynamic basic elements. Sensors 2012, 12 5713 The essence of our research is to obtain the complete model of the scene. We define some objects as “dynamic” because they can be moved or even removed in the same scenario, depending on the time in which the scenario is sensed. The objects considered and the information that is stored for each one are shown in Table 1. Static elements concern those which can be found in an empty room before it is furnished. They are basically: walls, openings, moldings and columns. In the following paragraphs we present the information contained in each particular RFID tag. Table 1. Summary of tag localization, and the data stored in each of them. Type of element Tag Data Stored type (wall, ceiling, floor),dimension, paint Wall color and number of openings type (door, window, closet, pure hole), Static Opening dimension and color Molding type (door, window), main dimensions, color Column cylinder/square, main dimensions Table high, area of table top, color Wardrobe/Filling cabinet dimension of three planes, color Dynamic type( chair, sofa), chair’s legs length and Chair separation, color, surface and volume (a) Wall tag. This contains the properties of the wall at the moment at which the scene is scanned: wall type (ceiling, floor, wall), wall dimension (length, width), paint color, number of openings on the wall. (b) Opening tag. This concerns the holes in the walls which are usually covered by other structures. An opening is characterized by: type (door, window, closet and pure hole), opening dimension ((length, width) and color. (c) Molding tag. It may be a molding on the opening or a molding on the wall. The information is as follows: type (wall molding and opening molding), molding dimension (length) and color. (d) Column tag. The columns correspond to the pillars which can be found in the scene but not next to the walls. We distinguish between cylinders and square pillars. The so called “dynamic” elements, are those which can be added, removed or changed with regard to its initial position in the scene. The approach presented in this paper considers the minimum number of dynamic elements which can be found in an inhabited scene. To date we have distinguished three dynamic elements: tables, chairs and wardrobe/filling cabinet. There are a wide variety of shapes for each type of dynamic element. In order to facilitate the matching process between the model corresponding to the tag recognized by the RFID and the 3D data provided by the laser scanner, we include basic information of the models. The main idea is to carry out a model based segmentation of the 3D data in order to carry out a further model-data matching process. For example, for table tags we include only the information which is related to the table top (area and height) so that we can search for 3D data fitted to a plane for a particular distance from the floor. A description of dynamic tags is as follows: Sensors 2012, 12 5714 (a) Table tag. Any type of table is characterized solely by the height and area of the top. It does not make sense to store more detailed information because, owing to the occlusion and the incomplete scanning of the scene, information concerning a part of the table’s surface might not be available. Figure 4 shows an example in which part of the table is missing. (b) Chair tag. Here we distinguish between simple chair and sofa elements. Chair tags only contain information concerning the chair’s legs (length and separation) whereas sofa tags store information about surface and volume. (c) Wardrobe/filling cabinet tags. These elements tend to be parallelepipeds and the scanner usually provides vertical or horizontal flat faces. We therefore store the dimensions of the element’s three planes. Although color information is also stored in all the aforementioned tags, in this paper it is not used as discriminatory feature. This is owing to the fact that color might be a highly variable property which strongly depends on the illumination of the scene. This challenging issue will be tackled in the future. Note that tags coordinates in the world reference system are not necessary in this work since we only need to identify the object in which the tag is placed. 5. Identifying and Positioning Basic Elements of the Scene The general strategy is presented in the diagram of Figure 5. Figure 5. Flowchart of the data segmentation process.