Modern strategies for finding geospatial data

What are modern best strategies for finding geospatial data?  Despite the increase in volume and variety of spatial data even in the 8 years since we have been writing this column, I still receive this question several times each week.  And for good reason–(1) Data needs are often very specialized, in theme, scale, region, attributes, and other characteristics, and often it is difficult to find a thread of a similar search by someone else online; (2) The ephemeral nature of the web makes it challenging to learn from others’ strategies (hence the reason for this blog).  My advice is still specifically tailored to the data requester, but there are some elements that are common to many of my responses, which I share below in the hopes it will be helpful to others.

10 Strategies.  (1) First and foremost, have a clear vision of the data you need, the scale, the date, the extent, the resolution, and other characteristics before you spend time searching.  Also set yourself a limit on how much time you will devote to processing the data before you can use it for analysis.  This will reduce the behavior that I think is similar to “wandering through a grocery store without a clear idea of what you are looking for”.  Walking up and down aisles in a grocery store, you might take something that you really didn’t want or originally intend to pick up, and waste valuable time with it in your project.  But once you’ve thought carefully about your needs and what you will and will not accept, then you are ready to search.

(2)  While you are searching, here is a useful aid that we reviewed to help you decide whether the data will meet your needs.

(3) Start your search with the ArcGIS Living Atlas of the World. In my judgment, the 8,000 + layers, maps, apps, and services represent the single most useful set of curated geospatial resources in formats that are easily ingestible into GIS software, especially the ArcGIS platform.  The easily ingestible is key–one noble goal is to minimize the data search component of your project, so that you can focus on analysis, and then communicating your results.  There are many stories of GIS users throughout the decades spending 90% of the project time manipulating and processing the data, leaving only a small amount of time for analysis, assessment, and evaluation.  If the data you are seeking is not in Living Atlas or not in the format that you desire, then move on to the next steps.

(4) Investigate ArcGIS Online.  The geospatial community has been wonderfully sharing content in ArcGIS Online since the platform launched in 2010.  Thousands of content items are uploaded daily, so there is no shortage of volume, and you can filter on data type, owner name, and other items to narrow your search, as explained here.  For example, I make use of the owner: ___ < owner name> search strategy quite often to find my own content and content from authors and organizations that I respect most.

(5)  Investigate ArcGIS Hub sites.  These have rapidly expanded over the past two years in breadth and depth and are an incredibly useful way of accessing data, especially if you are an ArcGIS user, but even if not.  Hub sites usually offer an array of download and streaming data options.  Starting points include searching on “Your area of interest” and then “GIS data portal” or “ArcGIS Hub”.  As one of hundreds of examples, here is the Boulder County ArcGIS Hub.  There are also useful starting points such as ArcGIS Hub search tools and galleries.

(6)  Visit and investigate several of the selected “top 10” (such as satellite imagery, Landsat, and vector data portals) lists that we have reviewed and compiled for this blog.

(7) If these do not net fruitful results, then dig into portals organized by theme that would cover your topic (such as WRIs on land use and natural resources).

(8) Next, investigate portals organized by location, that would cover at least in part, your study area (such as Ordnance Survey for the UK, and state, province, region, or city (examples here for Oregon, here for a large city of Los Angeles, here for the small city of Oak Hill West Virginia.

(9)  Scholarly research.  Because more research is tied to geospatial data than ever before (see our post here on this topic:  Part 1 and Part 2), check Google scholar, library databases, and other sources of peer-reviewed research, as well as conference proceedings from Esri, AGU, AAG, and other major events, to investigate the sources identified in the research.  Be tenacious: Often the data sources are listed at the end; sometimes the research will be published in a poster or a story map or other form of media; sometimes it may require you to contact the author to dig out the sources.

(10) Use ChatGPT!   My recent inquiry to ChatGPT about data for eastern Nebraska was fairly useful, even though there were no hyperlinks available, and I am confident it will improve over time:

It yielded these results:  Here are a few options for finding geospatial data for Eastern Nebraska:

1. USGS EarthExplorer : The USGS EarthExplorer website offers a wide range of geospatial data, including aerial imagery, satellite imagery, and topographic maps. You can search for data by location and download it for further analysis.
2. Nebraska Department of Natural Resources : The Nebraska Department of Natural Resources (NDNR) provides geospatial data related to natural resources in Nebraska. Their website offers datasets such as hydrology, land cover, and geology.
3. OpenStreetMap : OpenStreetMap is a collaborative mapping project that allows users to contribute and access geospatial data. You can search for specific areas in Eastern Nebraska and download data in various formats.
4. Nebraska GIS Data Portal : The Nebraska GIS Data Portal is a platform that provides access to geospatial data from different sources. You can search for datasets by location or topic and download them for your specific needs.

This additional note from ChatGPT was also a good reminder:  Remember to review the terms of use and licensing requirements for each data source before using the data for your project.

Words of advice.  Just like there is often no “single best source” for one specific topic on the web, no single “geospatial gold mine” exists that is the end-all site for spatial data.  However, this is actually a good thing, in my opinion.  Why?  It forces the user to think critically about the data sets that user may or may not choose.  As is clear on the Spatial Reserves data blog, choices abound.  Those choices require the user to investigate and evaluate each of them.

Example of these strategies in action.  Here is a sample data request I recently received:   We are working on a project involving drought in Colorado and we are looking for the following data:

• Soil Moisture: Map of Soil Moisture for April 2013 – we found this source but the pixel size might be too large to be meaningful for a storymap that focuses on Colorado.
• Precipitation: Is there a map of total precipitation for Colorado, 2013 water year (October 2012-September 2013)?
• Streamflow: Time series of streamflow of Colorado River October 2012-September 2015 (with average streamflow marked).
• Groundwater: Is there a spatial map of groundwater availability?
• Reservoir Levels: Time series of Lake Powell water level for October 2012-September 2015.
• Temperature – is there a map of temperature in April 2013 for across Colorado?
• Agricultural Health: Is there an index for agricultural health? Would there be a time series or some data for 2013?
• Water use: Is there a map that shows water use?

My reply included the following:  A search for spatial data in a political unit such as Colorado USA, will yield many sites, from which you will need to evaluate and make choices.  There are data portals via the Geospatial Centroid at Colorado State University,  via the Colorado Information Marketplace (data.colorado.gov), the OpenColorado data catalog, ColoradoView, and also those portals hosted by individual state agencies such as the Department of Natural Resources, the Colorado Geological Survey, and the Department of Transportation.   The sources require close evaluation to determine if they meet your needs.  Some sources are formatted as lists of data or map-based clickable links that require you to download the data to your device, while increasingly, others provide data as streaming services.  For example, this map service “DNR Viewer” from the Department of Natural Resources does not allow for the downloading of geodatabases, imagery, or shapefiles, but this data portal from the same agency does.

On the topic of streamflow:  From the USGS, this Current Water Data resource has been a longstanding favorite of mine:  https://waterdata.usgs.gov/nwis/rt.  Nowadays, however, it is also a live data feed in ArcGIS Online:  The metadata is here:  https://www.arcgis.com/home/item.html?id=4d8b9ff2d9e74ad18daecb2db7eaa87f    Examine the metadata and open it in ArcGIS Online or in ArcGIS Pro.  This is a premium service so you need to sign in to ArcGIS Online to access it, but once you do, it looks like this, below (there is also a global stream gauge layer, here):

On the topic of groundwater:  See this report and sets of data.  Reservoirs are available here at USGS and here in the ArcGIS Living Atlas of the World, but reservoir levels are more difficult to obtain:  It may involve calling or emailing water resources agencies.  Yes, sometimes these “20th Century methods” are best!

On the topic of Agricultural Health:  I am not certain if there is a globally accepted measure for it analogous to the HDI for education-health-life expectancy from the UN.  There is, however, an “agricultural stress index” map and measure, from the FAO, here.  You might have to speak with someone at the state department of agriculture.  Here is one of my favorite agriculture-related story maps entitled “Farming for the Future“.   This story map contains links to individual maps with data layers for the type of crop.

On the topic of temperature, I recently worked with an Esri colleague on the data team, and, good news!… he created this average global air temperature layer.

On the topic of precipitation:  In conjunction with an educational project, I compiled this layer as well as other Colorado layers, and display them in this gallery.  The precipitation data came from COCORAHS – the Community Collaborative Rain Hail & Snow Network.  In terms of water use, this remains challenging, but this Living Atlas collection contains water related items.  When I worked at USGS, I made extensive use of this water use report:  https://waterdata.usgs.gov/nwis/wu.   From this report, you may need to go through this workflow: Query the system > obtain a CSV table > bring that table into ArcGIS Online or ArcGIS Pro to create a map from it.

I recently taught a course with the content described in this essay.  The course is available in video form, here:

Part 1:   https://youtu.be/eBEWQRFYQBE?si=YwkLGw7TWUyIgjfO

Part 2:  https://youtu.be/UyG7TEdSv8I?si=UCSflXx16Om-zRhM

Part 3:  https://youtu.be/JDH8LP5Vy1Y?si=GcaK3cVGEe7I92my

Part 4:  https://youtu.be/XSCCq-IrYyY?si=ZSRekabrhCbftkVZ

I hope the above content is useful and relevant, and I look forward to your questions and comments!

Feel free to share your strategies, in the comments, below.

–Joseph Kerski.

Updated:  January 2024.