Archive for the ‘Public Domain Data’ Category

Key statements about the importance of spatial data

August 6, 2018 2 comments

Sometimes it is helpful to have some research results and quotes in support your data advocacy efforts at your own organization–in your promotion of  “why this all matters!”.   And, of course, why your efforts need to be funded and supported!  Here are a few key quotes about the importance of spatial data–and what happens when the data doesn’t exist.

Kathryn Sullivan, former NASA astronaut, recently commented, The power of a map to put time and place and phenomena together, to give it to our brains through the most potent input sensor human beings have — our eyes — is a remarkable accelerator for the comprehension and engagement and use of the data that tell us what’s on Earth, where are things happening on our planet …   as reported in Cheney, S. (2017). How GIS can help us understand our changing oceans.  (quoted on the 2017 Esri Ocean GIS Forum on

Another extremely useful statement is, “Advances in research on resilience and vulnerability are hampered by access to reliable data” can be found in Barrett, C. B. and D.D. Headey. 2014. Measuring resilience in a risky world: Why, where, how, and who? 2020 Conference Brief 1. May 17-19, Addis Ababa, Ethiopia. Washington, D.C: International Food Policy Research Institute.

“The lack of data is one of the biggest obstacles to progress toward development goals” was a part of the statements from the United Nations Independent Expert Advisory Group (UN). 2014. A World That Counts: Mobilising the Data Revolution for Sustainable Development. A Report to the UN Secretary General. New York, NY: United Nations, pp.28.

Perhaps the strongest argument for more and better data comes from ODI’s report The Data Revolution:  Finding the Missing Millions, where the authors cite numerous cases where inadequate data leads to sub-optimal policy decisions.  These cases “confirm some of the anecdotal evidence about the lack of good data in developing country ministries.”    The full citation is:  Stuart, E., E. Samman, W. Avis, and T. Berliner. 2015. The data revolution: finding the missing millions. ODI Research Report 03. London: Overseas Development Institute, pp. 51. (

Based on the research that Jill Clark and I have done in this area over the past decade, I would add to the ODI statement that in developed countries, some similar challenges exist.  We have documented those in these blog essays for six years.

I sometimes use the statements from this National Academies of Sciences report and these written by  I have created videos on this topic such as here and articles such as in Directions Magazine here.

My own contribution to these quotes is, “We have made much progress, to be sure, but the world’s increasingly complex and serious issues are not going to wait around another generation for us to get our data act together.”

What are the quotes and studies you are using in your own data advocacy efforts?  Please share those in the comments section.

Back CameraPhotograph by Joseph Kerski.



Using Kaggle big data in a GIS

July 9, 2018 1 comment

Kaggle is a platform for predictive modelling and analytics competitions in which statisticians and data miners compete to produce the best models for predicting and describing the datasets uploaded by companies and users. This crowdsourcing approach relies on the fact that there are countless strategies that can be applied to any predictive modelling task and it is impossible to know beforehand which technique or analyst will be most effective. [Wikipedia].   Over a half million people are in the Kaggle community, from nearly every country in the world.   Kaggle was acquired by Google a few years ago.  You can also learn about R, SQL, machine learning, and other topics on the site.  Why mention Kaggle in our geospatial data blog?  Kaggle hosts data sets on their site, some of which are spatial in nature, and some of which are truly “big data” (such as 9 million open images URLs), and as such, it represents a source of information for the GIS analyst, researcher, and instructor.

Because the data posted to Kaggle comes from a global community with diverse interests, expect an unusual array of data sets, from chest x-rays, superheroes, air quality, to birdsongs.  Some data are from surveys.  Many intriguing gems exist; for example, one of the data sets of interest to me as a geographer on the Kaggle site is the world happiness data .  It is available as a CSV for three different years.  The only unfortunate aspect of these tables is the lack of a country code; and relying only on name of country could present problems in joining the data to a map.

One can also learn about data sources by spending time on the Kaggle site.  For example, I learned about Uber Movement that contains data from selected cities and points of departure, Sports Reference that someone used to scrape 120 years of Olympic history data from, and a cancer imaging archive that someone used to obtain disease type and location.    Given the nature of the site, expect all sorts of oddities: My search on mountains of the world resulted in lots of “404 Not Found” errors; some data is documented and others not so much; and obtaining some of the data requires the user to be a programmer.  Still, Kaggle is a useful and unusual source worthy of attention, and given the rapid evolution in big  data and crowdsourcing, as we frequently write about on this blog, I expect that we will be seeing many more sites like this in the future.


A section of the Kaggle listing of data sets, showing the diversity of themes, scales, and sizes. 

Historical Imagery for the entire world now available via Wayback Service in ArcGIS from Esri

July 5, 2018 2 comments

I know that many of you regularly want to examine changes-over-space-and-time with imagery and GIS for research or instruction purposes.   As of last week, 81 different dates of historical imagery for the past 5 years now reside in ArcGIS via the World Imagery Wayback service.   For more information, see:

You can access this imagery in ArcGIS Online, ArcMap, and ArcGIS Pro.  A great place to start is the World Imagery Wayback app – just by using a web browser  –    A fascinating and an incredible resource for examining land use and land cover change, changes in water levels of reservoirs, coastal erosion, deforestation, regrowth, urbanization, and much more.  This resource covers the entire globe.

However, in keeping with the theme of our book The GIS Guide to Public Domain Data and this blog of being critical of the data, caution is needed.  The dates represent the update of the Esri World Imagery service.  This service is fed by multiple sources, private and public, from local and global sources.  Thus, the date does not mean that every location that you examine on the image is current as of that date.  I verified this in several locations where my ground observations in my local area show construction as of June 2018, for example, but that construction does not appear on the image.  In addition, several other places I examined from wintertime in the Northern Hemisphere were clearly “leaf-on” and taken during the summer before, or even from the summer before that.  Therefore, as always, know what you are working with.  Despite these cautions, the imagery still represents an amazing and useful resource.


Sample from this imagery set for 30 July 2014 (top) and four years later, 27 June 2018 (bottom) for an area outside Denver, Colorado USA. 

Access data and analytical tools online from EOS Data Analytics

A new platform from EOS Data Analytics ( allows data users not only to access data, but even perform online image processing in a web browser.  It is a set of mutually integrated cloud products for searching, analyzing, storing, and visualizing geospatial data.  This is a representation of what we have been discussing on this blog, namely, the increasing adoption of Software as a Service, and in addition, the combination of SaaS with data services and analytical services.  Thus, GIS professionals can search for, analyze, store, and visualize large amounts of geospatial data in one platform.  Doing all this in one system and also in a browser is, quite frankly, quite amazing.

With the EOS Platform, GIS users have access to an ecosystem of four mutually integrated EOS products, which together provide a powerful toolset for geospatial analysts. Image data is stored in cloud-based storage and is available for image processing or remote sensing analysis at any time; this can be a raw user file, an imagery obtained from their LandViewer data portal, or an output file from their online EOS Processing tools.  The EOS Platform is currently available for free during an open Beta.   The LandViewer tool has been freely available for some time and will continue to be.


There are at least two reasons why image processing is the platform’s major asset: the processing of large data amounts runs online and offers as many as 16 workflows with even more coming soon. On top of that, users can get the best cartographic features of EOS Vision for vector data visualization and soon to come, analysis.


A list of EOS Processing workflows that can be filtered by industry and input data type.

I found the LandViewer tool easy to use, with a wide variety of data sets to choose from, including Landsat, MODIS, NAIP, and others.


The LandViewer tool interface.


Choosing one’s own area of interest via the LandViewer interface.

Ordnance Survey GB to provide OS MasterMap data for free.

June 19, 2018 1 comment

The UK government announced last week that key parts of the OS MasterMap dataset (OSMM) are to be made available free of charge (see full announcement from OS). The following two datasets are due to be released under the Open Government licence (OGL) agreement:

  • OS MasterMap Topography Layer property extents
  • OSMM Topography Layer TOIDs (TOpographic IDentifiers), built into the features in the OS OpenMap-Local dataset.

In addition, a number of datasets will be made available (through an API) for free, up to a threshold, including:

  • OS MasterMap Topography Layer, including building heights and functional sites
  • OS MasterMap Greenspace Layer
  • OS MasterMap Highways Network
  • OS MasterMap Water Network Layer
  • OS Detailed Path Network

The announcement didn’t included any information on what the threshold for free access was, but no doubt details will start to filter out shortly as organisations start making use of these new data assets.

Imaging Spectrometer data from NASA AVIRIS

June 11, 2018 1 comment

The NASA AVIRIS mission has generated imaging spectrometer data for many areas of the USA since the 1990s.  The AVIRIS download portal for data from 2006 onward is on a node at NASA, here.  The AVIRIS sensor collects data that can be used for characterization of the Earth’s surface and atmosphere from geometrically coherent spectroradiometric measurements. This data can be applied to studies in the fields of oceanography, environmental science, snow hydrology, geology, volcanology, soil and land management, atmospheric and aerosol studies, agriculture, and limnology.  Applications under development include the assessment and monitoring of environmental hazards such as toxic waste, oil spills, and land/air/water pollution. With proper calibration and correction for atmospheric effects, the measurements can be converted to ground reflectance data which can then be used for quantitative characterization of surface features. In short, AVIRIS can collect in over 200 bands and therefore it can help analysts work out details such as vegetation health, or even species type, from the data.

The AVIRIS portal, presented in a Google map with popups with download links, as well as the  metadata file (in plain text format, available here)  both look very dated.  But this is a case where we encourage the user to give it a try–the portal may not look modern, but the data behind the portal is incredibly useful.   One can toggle data layers in the right hand corner of map to show All AVIRIS data or the Attrib. Filtered data (data that meets the attribute criteria but ignores the spatial filter).  One can also bound a box on the map, which has long been a favorite feature of mine on data portals, using the red rectangle to activate.  To update the spatial filter, click the “Update Map” button below the map.   The files are not streamed, but must be downloaded; perhaps because of their large size (typically over 1 GB).  Again, think “old school” formats – zip files and TAR files, but again, the data are plentiful and useful.   A set of previews are available, for example, here, and shown below.  For more information about AVIRIS data, see this link and this link.



The AVIRIS data portal.


A sample AVIRIS image.