Archive for the ‘Public Domain Data’ Category

Ordnance Survey GB to provide OS MasterMap data for free.

June 19, 2018 Leave a comment

The UK government announced last week that key parts of the OS MasterMap dataset (OSMM) are to be made available free of charge (see full announcement from OS). The following two datasets are due to be released under the Open Government licence (OGL) agreement:

  • OS MasterMap Topography Layer property extents
  • OSMM Topography Layer TOIDs (TOpographic IDentifiers), built into the features in the OS OpenMap-Local dataset.

In addition, a number of datasets will be made available (through an API) for free, up to a threshold, including:

  • OS MasterMap Topography Layer, including building heights and functional sites
  • OS MasterMap Greenspace Layer
  • OS MasterMap Highways Network
  • OS MasterMap Water Network Layer
  • OS Detailed Path Network

The announcement didn’t included any information on what the threshold for free access was, but no doubt details will start to filter out shortly as organisations start making use of these new data assets.


Imaging Spectrometer data from NASA AVIRIS

The NASA AVIRIS mission has generated imaging spectrometer data for many areas of the USA since the 1990s.  The AVIRIS download portal for data from 2006 onward is on a node at NASA, here.  The AVIRIS sensor collects data that can be used for characterization of the Earth’s surface and atmosphere from geometrically coherent spectroradiometric measurements. This data can be applied to studies in the fields of oceanography, environmental science, snow hydrology, geology, volcanology, soil and land management, atmospheric and aerosol studies, agriculture, and limnology.  Applications under development include the assessment and monitoring of environmental hazards such as toxic waste, oil spills, and land/air/water pollution. With proper calibration and correction for atmospheric effects, the measurements can be converted to ground reflectance data which can then be used for quantitative characterization of surface features. In short, AVIRIS can collect in over 200 bands and therefore it can help analysts work out details such as vegetation health, or even species type, from the data.

The AVIRIS portal, presented in a Google map with popups with download links, as well as the  metadata file (in plain text format, available here)  both look very dated.  But this is a case where we encourage the user to give it a try–the portal may not look modern, but the data behind the portal is incredibly useful.   One can toggle data layers in the right hand corner of map to show All AVIRIS data or the Attrib. Filtered data (data that meets the attribute criteria but ignores the spatial filter).  One can also bound a box on the map, which has long been a favorite feature of mine on data portals, using the red rectangle to activate.  To update the spatial filter, click the “Update Map” button below the map.   The files are not streamed, but must be downloaded; perhaps because of their large size (typically over 1 GB).  Again, think “old school” formats – zip files and TAR files, but again, the data are plentiful and useful.   A set of previews are available, for example, here, and shown below.  For more information about AVIRIS data, see this link and this link.



The AVIRIS data portal.


A sample AVIRIS image. 

An update on the World Bank’s Spatial Agent

May 28, 2018 4 comments
It sounds like a modern detective novel, but the Spatial Agent is actually a new, free app from the World Bank that offers one-stop access to interactive maps and charts of national, regional, and global datasets.  Jill Clark reviewed this site on our blog here. As we have written about data sources on this blog for nearly six years, and covered this topic in our book, the phrase “one-stop access” naturally caught my attention.  Could the Spatial Agent truly be all that it claims to be?


To find out, I began by watching a webinar that Mr Harshadeep recently conducted, which is as of this writing, still available online, here, after a short registration process.  In the webinar, and after my subsequent investigations, I was amazed at how the Spatial Agent as an app could bring together on-demand thousands of free, public-domain spatial data and analytical services (from in-situ and earth observation sources and also live cloud computing services).  It represents the data from sources such as the UN, NASA, NOAA, ESA, World Bank, many universities, and thousands of other sources, covering themes such as social (poverty, water supply), environmental (land use, biodiversity), economic (GDP, energy), and climate (snow cover, precipitation, for example).

The goal of the Spatial Agent is to offer solutions to many of the development challenges faced across the globe, which are often hampered by the poor availability of spatial data. For example, the app can be used to determine the areas in Madagascar that are susceptible to cyclones, or the areas in India that have high child malnutrition, or discovering the major exports of Vietnam, or determining how fast the population in Lagos is rising.  As these examples show, the Spatial Agent’s data cross boundaries, disciplines, and cover many different scales.  The Spatial Agent is the creation of Nagaraja Harshadeep, the lead environmental specialist and global lead for watersheds at the World Bank.  Mr Harshadeep has decades of experience working with spatial data and the application reflects his knowledge and passion.  There is much more than maps and imagery here, but rich tabular databases and other services, and the metadata for each of the data sets is quite robust.

I have been a long time fan of the spatial data from the World Bank, and use their data in several systems, including many layers available in ArcGIS Online.  The major limitation with the Spatial Agent app at this point that I can see is that it is just that — an app.  Therefore it only works on mobile phones and tablets.  I understand in part why it is focused on these devices–these are what many people are using day to day in their work.  Still, to bring the data sets into a GIS and more fully use them, I would love to see its capabilities inside of a series of user-driven interfaces that could be run in a standard web browser on a computer where I also have GIS and statistics tools available to me.  But I was glad to see this note about this very thing on the project’s site:  “The web version is being developed with the Bank’s Global Reach effort for launch later this year.”  Since the data and documentation are so rich on this site, I look forward to finding out how we will be able to use the services in a GIS.  Even without a GIS, the Spatial Agent is already very useful, because it is helping to bring data-driven decisions to daily decision making.


Two views of the hundreds of data layers and statistics available via the Spatial Agent.

For more information, including the links to access the apps, and the tutorials, see this page.

A review of the ArcGIS Pro Cookbook

May 14, 2018 4 comments

GIS Professional Tripp Corbin’s book, the “ArcGIS Pro 2.x Cookbook” (2018, Packt Publishing) is new but I believe will quickly become a valued and oft-used resource. Mr Corbin’s goal in writing this extensive (694 pages) resource is to help GIS professionals “create, manage, and share geographic maps, data, and analytical models using ArcGIS Pro.” The audience for this book includes all who are learning GIS, or learning Pro, as well as those migrating from ArcMap to Pro.

Tripp’s “cookbook” theme is evident throughout the book’s format, where in each section and problem to be solved, he shows how to get ready, how to do it, how it works, and … “there’s more” (additional resources). That the book is from Packt is excellent, because Packt ( offers eBook versions of every one of its books, and also offers newsletters and tech articles. That Tripp is a full time trainer and instructor is evident–he understands the challenges in learning a rapidly-changing and complex technology inherent in GIS with just enough tips to keep the reader engaged. He also encourages the reader to think about how to apply each tool and method to his or her own work. He offers the reader the ability to download the sample data for the book, and the data bundle is also on GitHub. He also includes PDFs of all images of screen shots and diagrams.

I like Tripp’s approach because, similar to my own instruction, and central to the theme of this blog, he starts with data. He’s not hesitant to discuss the benefits but also the limitations of each data format such as shp, gdb, and CAD files. He spends quality time in the book helping the reader understand how to convert data to the format that best fits his or her needs. His sections on linking tables from outside sources to existing data, on editing (in particular, a focus on topologies to improve data accuracy and increasing editing efficiency), and on 2D and 3D analysis are very helpful. I was pleased to see much attention to what I consider to be a chief advantage of Pro–the ability to more easily share content from Pro to ArcGIS Online and hence the wider community. Another wonderful new function in ArcGIS Pro is also included in the book–writing and using Arcade scripts, applied to symbology, classification, and analysis.

As a GIS book author, I know the challenges faced in writing such a book–what should be included, and what should be left out? Tripp does a nice job here as well, including the fundamentals that most users will touch. The book’s chapters include: 1: Capabilities and terminology. 2: Creating and storing data. 3: Linking data together. 4: Editing spatial and tabular data. 5: Validating and editing data with topologies. 6: Projections and coordinate systems. 7: Converting data from one format to another. 8: Proximity analysis. 9: Spatial statistics and hot spots. 10: 3D maps and 3D analyst. 11: Arcade, labeling and symbology expressions. 12: ArcGIS Online, 13: Publishing your own content to ArcGIS Online. 14: Creating web apps using ArcGIS Online.

These chapters cover a great deal of ground. In the editing chapter, for example (Chapter 4), configuring editing options, reshaping existing, splitting, merging, aligning, creating new point line polygon features, creating new polygon feature using autocomplete, and editing attributes using attribute pane and in the table view, are all examined. The examples in the book are interesting and relevant, and not without some humor (Trippville is a community that is often studied). In my view, the book contains just the right amount of graphics. Tripp provides answers to the questions he poses, and then gives the explanation for each answer. Despite the “recipes” provided in the cookbook, not all of them require the previous recipe to be used, which is excellent for all of us in GIS who have limited time and want to select sections in a non-sequential order.

I highly recommend using this book in conjunction with Tripp’s other book on this topic, “Learning ArcGIS Pro.” The Learning book focuses on installing, assigning licenses, navigating the interface, creating and managing projecrts, creating 2D and 3D maps, authoring map layouts, importing existing projects, creating standardized workflows using tasks, and automating analysis and processes using modelbuilder and python. The Learning ArcGIS Pro book ideally should be used first, before the ArcGIS Pro 2.x Cookbook, but if you are pressed for time, these two books could be used in tandem. Keep both of them handy–they will be very useful to you.


Covers of Tripp Corbin’s ArcGIS Pro books. 


An example of the detail provided in Tripp Corbin’s ArcGIS Pro 2.x Cookbook. 

Need access to thousands of historical aerials and topographic maps at your fingertips? Try

April 30, 2018 2 comments

Imagine having instant access to thousands upon thousands of historic aerial photographs and topographic maps to be able to examine change over time.  Thanks to a resource called Historic Aerials, you do have this wealth of information at your fingertips.  These aerials and maps, which go back 50, 60, and even 70 years or more, can be used for research, for instruction, for planning, and for other purposes.  Being in the field of geography and GIS education, I can think of many disciplines in which this can be used — urban and rural geography most certainly, but also biology, environmental science, city planning, history, agriculture, and also in GIS courses.  These resources foster spatial thinking about changes in time and space, from natural causes, such as volcanic eruptions or changes in river meanders, or from human causes, such as urbanization or the construction of reservoirs.  And given the connection that often exists between human and natural changes, sometimes these causes are intertwined–the construction of jetties along barrier islands often influences the naturally occurring longshore sediment transport and the migration of the islands themselves, as can be seen by comparing the historic to modern aerials of Ocean City, Maryland, for example.

The interface for the Historic Aerials is intuitive and provides tools that allow the user to compare USGS topographic maps of various years as well as the aerials themselves.  In the example below, I compare a 1958 with a 2009 aerial of a section of Grand Junction Colorado, before and after Interstate 70 and some surrounding housing was constructed.  You can also use the spotlight tool to “see back in time” for wherever you pan your cursor, and you can turn on the streets to see where  streets would one day be constructed on top of historical imagery.

The site comes from the Nationwide Environmental Title Research group, which has spent over 20 years collecting the worlds largest database of historical aerial images and topographical maps of the USA.  Their sources include USGS and USDA imagery, several private collections, and they are continually acquiring more. All the imagery they collect is orthorectified to provide the data in a searchable and precise geo-locatable format.  A print or digital image (for GIS users, GeoTIFF will be especially useful, delivered in lat-long coordinates, but JPG and PNG are also offered) of any of the maps or aerials is available.  In addition, a subscription service allows anyone to access the site with the following advantages: Full screen viewer, no advertising, PDF builder, quick JPEG downloads, and multiple user accounts.   I had a pleasant chat with the good people behind the site and found that their prices are quite reasonable.  Their FAQ, forum, and tutorials make it clear that they are committed to user success with these resources, and there are human beings behind this site to help as well.

Quite frankly, during my years working at the USGS, I always dreamed that my agency would create something like this.  Kudos to the HistoricAerials staff for making this a reality!


The interface for is quite intuitive and allows for fascinating investigations back in time for lands across the USA. 

General Data Protection Regulation comes to the EU … and it covers location.

April 23, 2018 1 comment

Legislation protection typically lags behind technological innovation until a clearer picture of how a particular technology is used and abused emerges. The issues associated with personal location data privacy and the often unintentional disclosure of personal and sensitive information, are subjects we have discussed many times before on Spatial Reserves (Always On: The analysts are watchingPrivacy concerns from fitness maps and apps).  However, things are set to change in the European Union (EU) with the introduction of new legislation covering how personal location information is collected, used and stored.

On 25th May 2018, General Data Protection Regulation (GDPR) will come into effect in the EU, replacing an earlier data privacy directive introduced in 1995. Aimed at protecting the personal data of all EU citizens, GDPR establishes new safeguards to minimise data breaches and misuse. It introduces a number of Data Subject Rights including right of access, right to be forgotten and privacy by design. Although such regulatory initiatives are often associated with the data collecting activities of larger companies and organisations and the type of data they collect, such as those in the financial and marketing sectors, GDPR extends the definition of personal data in Article 4 (1) to include:

‘… any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person

This legislative acknowledgement recognises location as an equally important component of personal information and an indicator of what GDPR defines as sensitive traits (political affiliation, religious beliefs and so on). Companies, organisations and institutions in the EU will be required to ask permission to use personal location information, be transparent about the use of this information and delete it if requested to do so.



Using the Data Interoperability Extension to import SDTS DLG files into ArcGIS Pro

April 16, 2018 Leave a comment

One of the themes of this blog and our book has been the wide variety of spatial data formats in existence.  Some of these spatial data formats have remained challenging to import into a GIS right up to the present day.  To meet this challenge, Esri’s Data Interoperability Extension has been a longstanding and useful set of tools that enables a wide variety of spatial data formats to be imported for use in a GIS.  It is an integrated spatial ETL (Extract, Transform, and Load) toolset that runs within the geoprocessing framework using Safe Software’s FME technology. It enables you to integrate data from multiple sources and formats, use that data with geoprocessing tools, and even publish it with ArcGIS Server.

I recently tested the Data Interoperability Extension in ArcGIS Pro and was thrilled with the results.  Read about how to install and authorize the extension here.  The extension does many things, but one that is particularly useful is that the extension creates a toolbox directly in ArcGIS Pro (graphic below).  I used this toolbox’s Quick Import tool to import a SDTS Format DLG (USGS Digital Line Graph) file directly to a file geodatabase.  The tool, like other ArcGIS Pro geoprocessing tools, walked me right through the process:  Data Interoperability > Quick Import > I then pointed to my DLG files in SDTS format > I named the resulting gdb (file geodatabase).  Once imported, I was then able to work with my hydrography, hypsography, roads, boundaries, and other data.

DLG files have existed since the early 1990s.  Why are we still working with them?  The reasons are that (1) They are dated but still useful vector data sets; (2) Many geospatial data portals still host data only in this format, such as the USGS Earth Explorer.  Another way to import these DLG files into ArcGIS Pro or ArcMap is to use the DLG2SHP tools that I wrote about in this set of guidelines using a standalone program.  See below for step-by-step instructions with the Data Interoperability Extension with screen shots.


1. Use Toolboxes > Data Interoperability Tools > Quick Import, as shown above.data_interoperability_use_for_dlg_screen1

2.  Using QuickImport pulls up a “specify data source” dialog box, as shown above.


3.  In the specify data source dialog box, use “find other source” and then specify SDTS format.


4.  Selecting SDTS format.


5.  Pointing to the SDTS file (after it has been unzipped and un-TAR’d) and saving it into a geodatabase.


6. Once the file has been imported into a geodatabase, it can be added to a new map in ArcGIS Pro.  The data is now ready for use, as shown for this hydrography example, above.