LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h Use of QGIS Geographical Information System in Basic Field Archaeology and LIDAR Processing LIDAR data and other data sources are processed to describe the basic use of QGIS, and how to process the UK Open Data LIDAR files into 2D images and 3D models. Stephen Eastmead First published November 2017 Current Revision 4 February 2018 Chapter 1: LIDAR and Mapping in QGIS Page 1 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h Introduction Fair Use It has taken hundreds of hours of work, experimentation and recording to assemble the information in these pages. I would be grateful if you could recognise this by making a small donation to Altogether Archaeology, which is a small community archaeology group that relies heavily on membership subscriptions to fund archaeological fieldwork. You can simply make the donation either by bank transfer or by post, using this webpage: https://altogetherarchaeology.org/DonationsMenu.php or by using PayPal sending the donation to this email address: [email protected] Suggested donation: £10-£15 in the UK or $20-$25 using PayPal from other countries. Copyright © Stephen Eastmead 2017 Please do not pass the document on to others, but tell them where it can be downloaded from by sending them this download link: https://eastmead.com/QGIS-LIDAR.htm I would be grateful if you would acknowledge my efforts if you use any of the information to provide maps used within a presentation or publication. Thanks. Acknowledgements My thanks to Sue Goldsborough for spending considerable time in testing these procedures and making valuable suggestions to the text, and to my wife Pam, for trying to locate the inevitable typos that escaped our scrutiny. All the rest of the typos are 100% my own work. General Information Originally this was seven separate documents, but I have since amalgamated them into this seven-chapter document. Each chapter is a natural follow-on from the previous chapter, except for the concluding chapter, which is a miscellany of useful QGIS and LIDAR related snippets. The Contents page indicates the scope of each chapter. The data downloads described in Chapter 1 are used throughout the rest of the chapters, so make sure you have downloaded them first. This document contains both internal (document) and external (Internet) links. Each internal link has a [BackLink] which returns you either to the original link page, or to the next section in the described process. If every minute step was described, this would be a long manuscript. So, I have assumed that readers have a basic knowledge of how to use Microsoft Windows software. The described procedures have all been developed on a Windows 10 Home 64-bit OS computer. I haven’t been able to test them on earlier versions of Windows. MAC OS X Sue Goldsborough has tested the instructions on her MacBook Pro and has found that they can easily be followed. There are specific MAC OS X instructions in addition to the Windows 10 procedures. As of September 2017, it was necessary to use the Firefox browser instead of Safari or Chrome. Edition / Revision update history See this link Chapter 1: LIDAR and Mapping in QGIS Page 2 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h Contents Introduction ....................................................................................................................... 2 Fair Use ......................................................................................................................... 2 Copyright © Stephen Eastmead 2017 ............................................................................ 2 Acknowledgements ........................................................................................................ 2 General Information ....................................................................................................... 2 MAC OS X ..................................................................................................................... 2 Edition / Revision update history .................................................................................... 2 Contents ............................................................................................................................ 3 Chapter 1: LIDAR and Mapping in QGIS ............................................................................... 8 Chapter 1: Introduction ...................................................................................................... 8 Step 1 The Site .............................................................................................................. 8 Step 2 Create some folders ......................................................................................... 10 Step 3 LIDAR data - Background ................................................................................. 10 Step 4 LIDAR data - Downloads .................................................................................. 11 Step 5 Raster and Vector Maps – Background ............................................................ 14 Step 6 Open Data Maps – Background ........................................................................ 14 Step 7 Ordnance Survey Open Data Maps - Background ............................................ 15 Step 8 Vector and Raster Map – Downloads ................................................................ 15 Step 9 QGIS - Download .............................................................................................. 16 Step 10 QGIS Main Window ........................................................................................ 17 Chapter 1: QGIS Main Window General Notes: ........................................................... 18 Chapter 1: Appendix 1 OS Open Data Download Examples ............................................ 19 Chapter 2: LiDAR Processing in QGIS - 2D ........................................................................ 20 Editor’s Note: New Chapter 2 functionality ....................................................................... 20 Digital Elevation Model creation (DEM) ........................................................................... 20 ASC Files ..................................................................................................................... 20 GDAL: Geospatial Data Abstraction Library ..................................................................... 21 LiDAR General Processing Procedures ........................................................................... 21 Hillshade Processing Methodology .............................................................................. 21 Step1 – Single and multiple merged asc files into a DEM layer ....................................... 23 Step 1 Option 1: Single asc file DEM: Using the ‘Add Raster Layer’ icon ..................... 23 Step 1 Option 2: Merging multiple raster (asc) files in a DEM ....................................... 23 Step 1 Option 3: Virtual DEM alternative method ......................................................... 24 Step 2 – Filling nodata -9999 ‘holes’ in LiDAR data ......................................................... 25 Method ......................................................................................................................... 26 Chapter 1: LIDAR and Mapping in QGIS Page 3 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h Step 3 – Applying Hillshade to the DEM file ..................................................................... 27 Step 3 Option 1 ............................................................................................................ 27 Step 3 Option 2 ............................................................................................................ 28 Chapter 2 Appendix 1 – Previous Version of Chapter 2 ................................................... 30 Introduction .................................................................................................................. 30 DEM Creation Procedure ............................................................................................. 30 Single asc file DEM: ..................................................................................................... 30 Creating a multiple asc file DEM .................................................................................. 31 Creating Hillshade Images ............................................................................................... 32 Hillshade Procedure ..................................................................................................... 32 Chapter 3: LIDAR Processing – Colour Ramps ................................................................... 34 Installing the Plugin ......................................................................................................... 34 Creating a LIDAR image with colour banded elevations .................................................. 36 Preparation: ................................................................................................................. 36 Colour banding elevation: organising the Layers Panel: ............................................... 37 Colour banding elevation: DEM layer processing ......................................................... 37 Colour banding elevation: Hillshade Layer processing ................................................. 38 Colour Banding Elevation with OS CONTOURS-terr50 Data ....................................... 39 Chapter 4: LIDAR Processing – 3D Models ....................................................................... 40 Chapter 4: Introduction .................................................................................................... 40 Install Plugin .................................................................................................................... 40 Clipping Raster Files ....................................................................................................... 40 Clipping Method for LIDAR during 3D model generation ................................................. 41 3D models – Options ....................................................................................................... 43 Software for the Sketchfab option .................................................................................... 43 Saving the Python Code to your Windows PC ................................................................. 44 Generating a HTML model from your DEM and Hillshade .tif files ................................... 44 Simple ‘White Space’ Removal .................................................................................... 45 Alternative method of eliminating ‘white space’ ............................................................ 47 Generating a 3D model from your DEM and Hillshade tif files and importing it into Sketchfab ........................................................................................................................ 48 Generating the Python command line .............................................................................. 49 Concise Procedure Windows 10 computers .................................................................... 49 Completing the upload to Sketchfab ................................................................................ 50 Chapter 4: Appendix 1 - Python Code ............................................................................. 53 Chapter 4: Appendix 2 - Command text editing in detail Windows 10 .............................. 56 Chapter 4: Appendix 3 - Notepad Python command text ................................................. 58 Chapter 4: Appendix 4 - Sketchfab Post Processing........................................................ 59 Chapter 1: LIDAR and Mapping in QGIS Page 4 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h Chapter 4: Appendix 5 - ‘White Space’ Removal using MeshLab .................................... 60 Chapter 4: Appendix 6 - Saving the Python Code to your MAC OS X Computer ............. 61 Chapter 4: Appendix 7 - Concise Procedure MAC OS X Computers ............................... 62 Screen Images ............................................................................................................. 63 Chapter 5: Using GPS data, Contours, Raster, Vector and On-line Maps ........................... 65 Downloading the GPS Data ............................................................................................. 65 GPS Tracks Line: Style ................................................................................................ 67 Labels .......................................................................................................................... 69 Raster Maps .................................................................................................................... 72 Vector Maps .................................................................................................................... 75 Raster Tile File Sizes ................................................................................................... 75 Vector Files Sizes ........................................................................................................ 75 Online Maps .................................................................................................................... 78 Contour Lines .................................................................................................................. 79 Chapter 6: QGIS Print Composer ........................................................................................ 80 Chapter 6: Introduction .................................................................................................... 80 Procedure ........................................................................................................................ 80 Step 1: Configuring the data to map ............................................................................. 80 Step 2: Creating a new Print Composer Map ............................................................... 80 Step 3 Viewing the map in Print Composer .................................................................. 81 Step 4 Adding: Border, Grid and Coordinates .............................................................. 82 Step 5: Adding a Scalebar ........................................................................................... 83 Step 6: Adding: Title, copyright and feature text ........................................................... 84 Completed Map............................................................................................................ 85 Locating an Agisoft trench plan image on a map using the trench’s GPS coordinates . 85 Map Output ..................................................................................................................... 88 Landscape Map Features ................................................................................................ 88 Chapter 7: QGIS, LIDAR & Other – Miscellaneous Snippets ............................................... 89 Chapter 7: Introduction .................................................................................................... 89 1 QGIS Decorations ........................................................................................................ 89 2 Processing LIDAR Point Cloud Data using LAStools .................................................... 89 LAStools Installation .................................................................................................... 90 Point Cloud (.laz) QGIS Procedure .............................................................................. 90 Blast2dem - Creating New Classification Combinations ............................................... 92 3 Planlauf TERRAIN ........................................................................................................ 92 4 Relief Visualization Toolbox version 1.3 ....................................................................... 93 5 QGIS Software Bugs .................................................................................................... 95 Chapter 1: LIDAR and Mapping in QGIS Page 5 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h 6 Plugins installed ........................................................................................................... 95 7 Google Earth ................................................................................................................ 96 8 Google Earth and the British National Grid ................................................................... 97 GPS Utility Preliminary Procedure ............................................................................... 98 Google Earth Error Assessment ................................................................................... 98 GPSU Data Correction Procedure ............................................................................... 99 9 Add a Geology Layer in QGIS using WMS/WMTS ...................................................... 101 Geology Example ....................................................................................................... 101 10 QGIS LIDAR 1M DSM & 1M DTM England Zoomable using WMS/WMTS ............... 103 11 Ordnance Survey Open Data Raster and Vector Maps Issue ................................... 106 12 Ordnance Survey OS Net ......................................................................................... 106 13 IrfanView Windows Image Editor .............................................................................. 107 14 Transferring Large Files for Free using the Internet .................................................. 107 15 Editing Navigation Controls in Sketchfab .................................................................. 108 Sketchfab Navigation Editing Procedure .................................................................... 108 16 QGIS SAGA LIDAR Tools ........................................................................................ 111 17 LIDAR Resolution Comparison: 200cm - 100cm - 50cm - 25cm ............................... 112 18 GPS Software: GPSU and GPS TrackMaker ............................................................ 114 19 QGIS Website links .................................................................................................. 114 Gentle Introduction to GIS .......................................................................................... 114 Training and User Guides both as webpage and pdf download .................................. 114 QGIS Current versions ............................................................................................... 114 QGIS Previous versions ............................................................................................. 114 QGIS Development .................................................................................................... 114 QGIS Issue Tracking .................................................................................................. 114 QGIS Bug Reporting .................................................................................................. 114 Chapter 8 planlauf/TERRAIN ............................................................................................ 115 Introduction ................................................................................................................... 115 Procedures .................................................................................................................... 115 Methodology .................................................................................................................. 115 Files ........................................................................................................................... 115 Menu Options ............................................................................................................ 116 Basic Processing Steps ................................................................................................. 117 Stage 1 Merging LiDAR asc files ................................................................................ 117 Stage 2 Creating the plfter file .................................................................................... 118 Step 3 Creating the Local Relief and Slope Models ....................................................... 119 Step 4 Color Ramps ................................................................................................... 119 Chapter 1: LIDAR and Mapping in QGIS Page 6 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h Step 5 Export Options ................................................................................................ 122 Summary ................................................................................................................... 123 Acknowledgments ............................................................................................................. 123 Feedback ...................................................................................................................... 124 [BackLinkToChapter7] Chapter 1: LIDAR and Mapping in QGIS Page 7 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h Chapter 1: LIDAR and Mapping in QGIS Chapter 1: Introduction If you intend following this series of basic use chapters to record archaeology in QGIS, or to manipulate LIDAR data, then follow Chapter 1 so that you can obtain all the necessary data files. You will be downloading: LIDAR tiles, OS Open Data: raster and vector maps, and a GPS (gpx) file type. The data downloads are of How Hill, near Downholme in Swaledale, North Yorkshire. By the end of the document you will be able to obtain and process your own maps and LIDAR data, GPS waypoint or track data as well as dipping your toe into 3D modelling. I have chosen the How Hill site as it demonstrates interesting multi-period LIDAR features and produces nice LIDAR colour coded elevation bands. Whilst this series of chapters are LIDAR based, all that you learn will be applicable using QGIS for non-LIDAR archaeological data. Note American spelling is used when referring to software features e.g. color. Step 1 The Site The site chosen to illustrate the use of QGIS in this document can be seen on the 1:25,000 OS map below. Note: the blue grid lines denote 1 km by 1 km squares. a) Decide which 1km squares you require data for. Obviously How Hill is divided across the left-hand pair and I know there is archaeology on the opposite hillside across both right-hand squares. b) Having established that four squares of LIDAR and Map data will be used, you then need to identify them. Memory-Map will indicate the coordinates as you move the cursor over each square see above. If you do not have access to a digital map or a paper map, then go to: https://www.gridreferencefinder.com/bing.php If it is not set to Aerial View, click on ‘Aerial View’ and select Ordnance Survey. Zoom into the site, Chapter 1: LIDAR and Mapping in QGIS Page 8 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h and then right-click on the map to display British National Grid Coordinates. See red pins on the map (above), where I have right-clicked on all four 1km squares. NOTE: The difference between the Grid Reference and the X(Easting) and Y(Northing) which has the 2-letter grid prefix replaced by 4’s in this case. This is explained in Paragraph e) below. c) Starting with the bottom left grid record each of the four grids’ 2-digit coordinates like this: a. Bottom left SE 10 97 (Point A) b. Top left SE 10 98 (Point B) c. Top right SE 11 98 (Point C) d. Bottom right SE 11 97 (Point D) and note of the 10km by 10km grid co-ordinate which is the 1-digit co-ordinate i.e. SE 19. If you are unlucky, the project may cross over more than one 10km grid. d) You will see when using QGIS that the British National Grid (BNG) or EPSG:27700 doesn’t display the leading grid letters like SE for How Hill, but a numeric equivalent. SE uses a 4 before each easting and northing, so the reference shown on the map of SE 10766 97945 becomes 410766 497945. See how 4 and 4 for grid SE is derived on the map below. Chapter 1: LIDAR and Mapping in QGIS Page 9 LIDAR and Mapping in QGIS Ed1 Rev4.4.docx 05/03/2018 05:53h e) Note: Remember that the OS British National Grid’s datum is known as OSGB36, and in QGIS the BNG is called EPSG:27700. f) The yellow highlighted data in paragraphs: c) and d) above is used to obtain the LIDAR and Map data. Step 2 Create some folders You could potentially create a lot of new files. I recommend creating either C:\LIDAR\projectname\ folder system, or C:\QGIS\projectname\ system, and then creating suitable sub-folders as suggested in this document. Remember not to use spaces in- between folder names and file names, hence using projectname rather than project name, and keep names short as possible. Following these recommendations will eliminate potential problems as you proceed through this document. Secondly, create a master map folder containing sub-folders for your various map downloads e.g. C:\MAPS\. Step 3 LIDAR data - Background Historic England has a free pdf document on LIDAR. The first two images below are taken from that publication. They provide useful background information. The pdf can be found here: Historic England LIDAR publication: https://content.historicengland.org.uk/images- books/publications/light-fantastic/light-fantastic.pdf/ Chapter 1: LIDAR and Mapping in QGIS Page 10