Welcome

April 16, 2012 5 comments

Welcome to the Spatial Reserves blog.

The GIS Guide to Public Domain Data was written to provide GIS practitioners and instructors with the essential skills to find, acquire, format, and analyze public domain spatial data. Some of the themes discussed in the book include open data access and spatial law, the importance of metadata, the fee vs. free debate, data and national security, the efficacy of spatial data infrastructures, the impact of cloud computing and the emergence of the GIS-as-a-Service (GaaS) business model. Recent technological innovations have radically altered how both data users and data providers work with spatial information to help address a diverse range of social, economic and environmental issues.

This blog was established to follow up on some of these themes, promote a discussion of the issues raised, and host a copy of the exercises that accompany the book.  This story map provides a brief description of the exercises.

pdd

Tripp Corbin’s Learning ArcGIS Pro 2 book enables you to quickly learn and effectively use ArcGIS Pro

October 19, 2020 Leave a comment

Tripp Corbin’s new book Learning ArcGIS Pro 2, Second Edition, from Packt Publishing, is a resource that I will be using long into the future and I know many in the GIS professional community will be doing the same.  As ArcGIS Pro replaces ArcMap in government, nonprofit, academic, and private industry, and as ArcGIS Pro continues to evolve, this book is an extremely useful resource either to follow along in its entirety, or to tap into specific chapters to hone specific skills.  University and college professors and students will also find this book very useful. 

I have long been an admirer of the instructional style of Tripp’s books, and wrote a review of his ArcGIS Pro Cookbook here in this blogLearning ArcGIS Pro 2 follows Tripp’s excellent balancing of theory and application and enables anyone regardless of how much background they have in GIS to be using this powerful set of geospatial tools quickly.  Tripp’s deep and rich background with the GIS community and in education in particular over many decades is manifest in the book’s careful attention to helping users through the things he knows will cause them the most difficulty. 

Tripp covers the spectrum from technical requirements, installing the software, managing licenses, starting projects, editing, performing analysis, creating maps, 3D scenes, and layouts, automating processes with ModelBuilder and Python, using Arcade scripts, and, appropriate to today’s cloud-based workflows, how to share results with others via layers and maps in ArcGIS Online.  Germane to this blog, Tripp also touches on data issues. Plus, the data for the hands-on activities in the book is easily accessed and interesting to use, covering parcels, floodplains, and much more, at a variety of scales.  

As a GIS instructor I appreciate the graphics he has included—they’re not in color, but they are large and legible, and that’s I think even more important.  He also has the right number of screen shots—not too many, but just enough to keep the learner moving forward.  In short, you won’t get “tripped up”—you will be able to keep moving forward.  Packt does a very nice job with their digital editions, which for GIS professionals might be the most useful format, though the printed version is nicely laid out as well.   I salute Tripp Corbin and Packt Publishing for this excellent resource for the community.

A few pages from Tripp Corbin’s new Learning ArcGIS Pro 2, Second Edition, book.

–Joseph Kerski

Categories: Public Domain Data

Everyday examples of being critical of the data

October 5, 2020 4 comments

Each day presents new examples of the central theme of this blog–the importance of being critical of the data, including spatial data. Some of the most effective examples are those rather odd bits of geospatial information, and I have included some of my recent favorites here. These examples are intriguing; some are even fun. They might serve as attention-getting images as you teach students or colleagues about data quality on maps, visualizations, and other forms of communication. If a picture is worth 1,000 words, I say that a map or image is worth 1,000 pictures.

At first glance, from the following result from a phone map search, George Mason University has become a simple Chinese fast-food chain! It begs the question: Is the 4-star rating reflective of the university or of the chicken chow mein?

George Mason University — a Chinese fast food chain?

See this example below from a catalog that I have treasured for a few years. Map orientation matters! This speaks to one of the central goals of my entire career, which is to do all I can to increase geographic literacy. We still have a long way to go! In fairness to this catalog, though, this image has been corrected after I and a few colleagues wrote to them.

Yes, Virginia, map orientation does matter!

Check out the satellite image below. You have heard of life imitating art and vice versa. Is this a case of maps imitating the Earth, or vice versa? I followed the advice we promote in this blog; i.e., I checked several other sources, and it does appear that the pushpin-looking feature in California is legitimate! Notice the dry pushpin-like feature facing the one that contains what appears to be water, and the boat on the desert sands to the southwest of the water feature. The Esri Wayback imagery shows the feature under construction from a image dated February 2014. Still, it is a puzzle what the purpose of a water feature in the middle of the desert is.

Maps imitating the Earth, or vice versa?

“What’s wrong with this picture?” My recent search to fill out a proper bibliographic citation for an article I wrote for the location of the National Academy of Sciences headquarters netted me a location on a house on a cul-de-sac in suburban Denver. How can that be? Could it perhaps be because I was doing the search from a computer in the Denver area?

Is this really the headquarters of the National Academy of Sciences?

This errant weather feed below existed online for nearly a year before it was corrected. I know it gets hot in Texas, but really! If the heat doesn’t do you in, the rainfall deluge, impossibly high humidity, and the ferocious winds will!

Errant weather feed. I know it gets hot in Texas, but this is ridiculous.

But even outside of maps, examples are just as numerous. Take a careful look at the Beatles tunes listed below. Funny, I’ve never heard the songs “Penny Lance”, “A Day in the Sky”, or “Can’s Buy Me Love” before! In addition, this isn’t an “album” at all, but rather a user-created playlist but appears on a host of music-related websites.

Beatles playlist – funny, I’ve not heard some of these odd song titles!

Feel free to share some additional examples that you have found in the comments section below!

Modern strategies for finding geospatial data

September 21, 2020 Leave a comment

What are modern best strategies for finding geospatial data?  Despite the increase in volume and variety of spatial data even in the 8 years since we have been writing this column, I still receive this question several times each week.  And for good reason–(1) data needs are often very specialized, in theme, scale, region, attributes, and other characteristics, and often it is difficult to find a thread of a similar search by someone else online; (2) the ephemeral nature of the web makes it challenging to learn from others’ strategies (hence the reason for this blog).  My advice is still specifically tailored to the data requester, but there are some elements that are common to many of my responses, which I share below in the hopes it will be helpful to others.

(1) First and foremost, have a clear vision of the data you need, the scale, the date, the extent, the resolution, and other characteristics before you spend time searching.  Also set yourself a limit on how much time you will devote to processing the data before you can use it for analysis.  This will reduce the behavior that I think is similar to “wandering through a grocery store without a clear idea of what you are looking for”.  Walking up and down aisles in a grocery store, you might take something that you really didn’t want or originally intend to pick up, and waste valuable time with it in your project.  But once you’ve thought carefully about your needs and what you will and will not accept, then you are ready to search.

(2)  While you are searching, here is an aid that we reviewed to help you decide whether the data will meet your needs.

(3) Start your search with the ArcGIS Living Atlas of the World. In my judgment, the 8,000 + layers, maps, apps, and services represent the single most useful set of curated geospatial resources in formats that are easily ingestible into GIS software, especially the ArcGIS platform.  If the data you are seeking is not in Living Atlas or not in the format that you desire, then:  (4) Go to selected “top 10” (such as satellite imagery, Landsat, and vector data portals that we have reviewed in this blog.   These include such useful tools such as  ArcGIS Hub search engines.  If these do not net fruitful results, then dig into (5) portals organized by theme that would cover your topic (such as WRIs on land use and natural resources),  (6) portals organized by location, that would cover at least in part, your study area (such as Ordnance Survey for the UK, and state, province, region, or city (examples here for Oregon, here for a large city of Los Angeles, here for the small city of Oak Hill West Virginia.

Just like there is often no “single best source” for one specific topic on the web, no single “geospatial gold mine” exists that is the end-all site for spatial data.  However, this is actually a good thing, in my opinion.  Why?  It forces the user to think critically about the data sets that user may or may not choose.  As is clear on this data blog, choices abound.  Those choices require the user to investigate and evaluate each of them.

Here is a sample data request I recently received:   We are working on a project involving drought in Colorado and we are looking for the following data:

  • Soil Moisture: Map of Soil Moisture for April 2013 – we found this source but the pixel size might be too large to be meaningful for a storymap that focuses on Colorado.
  • Precipitation: Is there a map of total precipitation for Colorado, 2013 water year (October 2012-September 2013)?
  • Streamflow: Time series of streamflow of Colorado River October 2012-September 2015 (with average streamflow marked).
  • Groundwater: Is there a spatial map of groundwater availability?
  • Reservoir Levels: Time series of Lake Powell water level for October 2012-September 2015.
  • Temperature – is there a map of temperature in April 2013 for across Colorado?
  • Agricultural Health: Is there an index for agricultural health? Would there be a time series or some data for 2013?
  • Water use: Is there a map that shows water use?

My reply included the following:  A search for spatial data in a political unit such as Colorado USA, will yield many sites, from which you will need to evaluate and make choices.  There are data portals via the Geospatial Centroid at Colorado State University,  via the Colorado Information Marketplace (data.colorado.gov), the OpenColorado data catalog, ColoradoView, and also those portals hosted by individual state agencies such as the Department of Natural Resources, the Colorado Geological Survey, and the Department of Transportation.   The sources require close evaluation to determine if they meet your needs.  Some sources are formatted as lists of data or map-based clickable links that require you to download the data to your device, while increasingly, others provide data as streaming services.  For example, this map service “DNR Viewer” from the Department of Natural Resources does not allow for the downloading of geodatabases, imagery, or shapefiles, but this data portal from the same agency does.

On the topic of streamflow:  From the USGS, this Current Water Data resource has been a longstanding favorite of mine:  https://waterdata.usgs.gov/nwis/rt.  Nowadays, however, it is also a live data feed in ArcGIS Online:  The metadata is here:  https://www.arcgis.com/home/item.html?id=4d8b9ff2d9e74ad18daecb2db7eaa87f    Examine the metadata and open it in ArcGIS Online or in ArcGIS Pro.  This is a premium service so you need to sign in to ArcGIS Online to access it, but once you do, it looks like this, below (there is also a global stream gauge layer, here):

streamflow

On the topic of groundwater:  See this report and sets of data.  Reservoirs are available here at USGS and here in the ArcGIS Living Atlas of the World, but reservoir levels are more difficult to obtain:  It may involve calling or emailing water resources agencies.  Yes, sometimes these “20th Century methods” are best!

On the topic of Agricultural Health:  I am not certain if there is a globally accepted measure for it analogous to the HDI for education-health-life expectancy from the UN.  There is, however, an “agricultural stress index” map and measure, from the FAO, here.  You might have to speak with someone at the state department of agriculture.  Here is one of my favorite agriculture-related story maps entitled “Farming for the Future“.   This story map contains links to individual maps with data layers for the type of crop.

On the topic of temperature, I recently worked with an Esri colleague on the data team, and, good news!… he created this average global air temperature layer.

On the topic of precipitation:  In conjunction with an educational project, I compiled this layer as well as other Colorado layers, and display them in this gallery.  The precipitation data came from COCORAHS – the Community Collaborative Rain Hail & Snow Network.  In terms of water use, this remains challenging, but this Living Atlas collection contains water related items.  When I worked at USGS, I made extensive use of this water use report:  https://waterdata.usgs.gov/nwis/wu.   From this report, you may need to go through this workflow: Query the system > obtain a CSV table > bring that table into ArcGIS Online or ArcGIS Pro to create a map from it.

Feel free to share your strategies, in the comments, below.

–Joseph Kerski

Categories: Public Domain Data

A review of the ArcGIS Hub Gallery

September 7, 2020 Leave a comment

As we have described in this blog, ArcGIS Hub is a set of templates and tools that allow organizations to more easily share their geospatial data holdings and also to engage the community. ArcGIS Hub has emerged over the past few years as a primary way that organizations are sharing geospatial data, and thus is germane to the theme of this blog and our book.

One of the resources that is worth frequently checking is the ArcGIS Hub gallery. The sites in this gallery range from those created by local, regional, and national governments, international agencies, nonprofits, universities, to those created by private companies. At the current time, you can search by these “sectors” but also on Focus Areas:  Transportation  |  Elections  |  Census 2020  |  Financial Transparency  |  Volunteerism  |  Open Data.

I advise “frequently checking” because the gallery is actively curated and changes often. But the resources are vast and varied!

A portion of the ArcGIS Hub Gallery.

–Joseph Kerski

A review of the Arizona AZGeo Data Hub

August 24, 2020 1 comment

Recently, the Arizona Geographic Information Council (AGIC) and the Arizona State Land Department (ASLD) launched a new data portal named the AZGeo Data Hub. AZGeo Data Hub is a cloud-based platform built on Esri’s ArcGIS Online functionality.  We have written about other platforms based on ArcGIS Hub here and here. Together with the AZGeo Data Hub, these articles show that in a few short years, ArcGIS Hub has been adopted by many organizations as a viable, easy-to-configure way of serving their data and related resources.

The AZGeo Data Hub is designed to provide GIS users with links to online map services, FGDC compliant metadata, and geospatial data downloads. Data on AZGeo includes GIS layers for administrative boundaries, demographics, environmental factors, hydrology, imagery, indices, mining, natural features, transportation, and more. AZGeo is hosted by the Arizona State Land Department and staffed by the Arizona Land Resources Information System (ALRIS).

As we have documented in this blog, sharing data is more than just setting up and maintaining a website. Today’s modern web GIS architecture is very much tied to a vibrant community, of which AZGeo Data Hub is an excellent example. I have been a member of the Arizona GIS community since the mid-1990s and they have long been a wonderful source of technical assistance and are lifelong colleagues. Their schools have been among the USA’s most vibrant in terms of promoting geographic literacy, and their universities and community, technical, and tribal colleges have long been trailblazers in the use of GIS in GIScience, geography, and environmental science programs, and more recently in university libraries, schools of education, and in schools of business.

Anyone can view, search and download the data via the AZGeo Data page as a Public User with no login required for access. But in addition, AZGeo utilizes groups to help GIS professionals throughout the state streamline their work processes and share data layers that might be restricted from public access. People can join these groups and then take advantage of the data and applications shared within groups, as well as gain access to ArcGIS Online applications via an AZGeo Hub account. An AZGeo User account is required for this level of access. This is detailed in a helpful graphic, shown below.

Graphic showing ways to access and interact with the Arizona geospatial data and the community.

My test of the AZGeo Hub’s data page was fruitful. I was able to find a variety of data sets on hydrography, population, land use, and other themes, at a variety of scales. I intend to check back often to see additional data sets that get added over time, and I encourage you to do the same.

Perhaps more important than the data serving is that these Hubs help foster community amongst their users. Many have feedback loops and forums and other means by which the community can interact, strengthening the notion that GIS is not just a set of data and tools, but a system of engagement.

–Joseph Kerski

The Landsat Explorer Web Mapping Application, Revisited

August 10, 2020 Leave a comment

In four years since we last wrote about what was then the “Landsat thematic bands application,” many features have been added to this tool. Furthermore, as it is, I believe, one of the very most useful mapping tools for research and instruction, ’tis time for an updated review. Now known as the Landsat Explorer app, this web mapping application is now part of the Living Atlas set of apps, a very useful set of tools that includes the Wayback imagery (which we have written about here), the water balance app, and others. I have used the Landsat Explorer app for research and also in a wide variety of instructional settings, from upper primary to university level, and in remote sending and GIS courses, but also in geography and environmental science courses. You could also use it for research purposes. With this app, Landsat images for the whole world are available to you at your fingertips, over a 45 year span of time. Truly amazing.

The Landsat Explorer app’s main tools are its renderer, identify, time-swipe-change, mask, save layer to ArcGIS Online, export, and add data from ArcGIS Online.

The renderer allows you to choose from about 10 band combinations, from agriculture to vegetation index to geology and more. This is incredibly responsive and useful. The only slight misgiving I have from an instructional standpoint is that I wish the band combinations would be visible, so students and others understand some of remote sensing theory. To the app’s credit, though, you can choose your own custom bands and indicate the band number you wish to display, shown below.

Identify allows you to learn the image scene ID, acquisition date, and cloud cover.

Most likely, most users will use this application for understanding change over time via imagery. You can use the swipe tool and the change detection tool for this. Before you can compute change, you will need to select Primary and Secondary Dates. Use the Time Slider to select an earlier date and click the Set as Secondary Layer button. Then, select a later date and move to the Swipe tool or the Change Detection tool to compare them. The Swipe tool for the Mt St Helens area for 1990 and 2019 are shown below. The swipe tool is my favorite but admittedly sometimes a frustrating tool to use, to get exactly the images you wish to have on the left and right side. It is also a bit confusing in the app to understand which image corresponds to which dates. All I can say here is to go through the very helpful tutorial, available on the lower left via the ‘graduation cap’ symbol, and carefully follow the steps.

The change detection tool can calculate changes in vegetation health (NDVI or SAVI), burned area (Burn Index), water content (Water Index), and urban area (Urban Index). Three items of interest: First, I am using the vegetation index, but you can use this drop-down to change to the soil adjusted vegetation index, the burn index, the water index, and the urban index. Second, the sliders can be used in your change detection. Third, you can define your own area of interest rather than take the default scene that is in your browser window.

Save allows you to save the top layer to your ArcGIS Online account (a log in to your own account is required). Export allows you to extract whatever map (the top layer in the app) you have built as a local file–a high resolution TIFF with customizable pixel sizes and spatial reference. This is a file that you can then bring into ArcGIS Pro or any GIS software for further analysis.

As the name implies, “Add data from ArcGIS Online” allows you to bring in your content from your own ArcGIS Online account into the Landsat viewer (a log in to your own account is required). After doing so, however, it wasn’t clear how to obtain a list of layers so I could turn my added layer off, so in my view this is the only tool I would probably would not use much. “Stories” shows a selection of 5 areas around the world where Landsat imagery is applied to solve real-world challenges, including in flood analysis. The “points of interest” choice here is particularly useful for instruction, guiding you to numerous places around the planet with explanations of each human-built and natural feature with a specific combination of Landsat bands.

Finally, if you wish to create your own similar type of app in the ArcGIS platform, go to the GitHub imagery zone and to the ArcGIS Developers site. Give this tool a try and I look forward to your comments.

–Joseph Kerski

Teaching about spatial data quality

July 26, 2020 4 comments

It is important to be critical of data, including, and perhaps especially, spatial data. I frequently receive inquiries from professors seeking resources on the best resources to teach about data quality and foster related discussions with students. Here is one of my recent responses to such an inquiry.

  1. If you need a fun and engaging set of maps and discussion, use my presentation on Good Maps, Bad Maps, and why it all matters: https://sway.office.com/HqKUCu2ib60rkijh?ref=Link. Scroll down to 30% of the way down in this presentation, in the “Maps Tend to Believed” section.  Here you will see a whole set of BAD MAPS. They are misleading, erroneous, or just plain bad for many different reasons – the projection is unsuitable, the data is questionable or impossible (for example, I know it gets hot in Texas but there is a temperature reading from a data feed that is over 3,000 degrees as one example), the legend is misleading, places are blatantly shown in the wrong location, or some other reason.
  2. A short reading on the above topic, why data quality still matters now more than ever:  https://spatialreserves.wordpress.com/2017/12/04/why-data-quality-still-matters-now-more-than-ever/
  3. More food for thought, presented as the “best available data” “BAD data”:  https://spatialreserves.wordpress.com/2017/08/14/best-available-data-bad-data/
  4. A guide for deciding which data will be useful for your needs:  https://spatialreserves.wordpress.com/2018/11/26/a-graphical-aid-in-deciding-whether-geospatial-data-can-be-used/
  5. An essay reflecting on the 30 checks for data errors:  https://spatialreserves.wordpress.com/2015/03/22/gis-gigo-garbage-in-garbage-out-30-checks-for-data-errors/

I hope these resources will be useful to many!

Data quality is a major theme of this blog.

–Joseph Kerski

Categories: Public Domain Data Tags:

Ordnance Survey’s new data hub and APIs

July 20, 2020 1 comment

Ordnance Survey, Great Britain’s national mapping agency, recently launched a new OS Data Hub, replacing their existing online data portals.

Screenshot 2020-07-17 10.33.24.png

OS Data Hub

In addition to providing up to £1,000 per month of free access some datasets, including 1:25 000 leisure mapping and road network data, through a series of new APIs, the new hub will also provide free access (up to a monthly threshold) to their flagship 1:1250 MasterMap product. 

The new APIs include:

  • OS Maps API: Integrating OS mapping into apps and websites (includes OS MasterMap Topography Layer and 1:25 000 Scale colour raster data)
  • OS Vector Tile API: Fully customisable vector maps 
  • OS Features API: Numerous layer of OS MasterMap data layers including greenspace, highways network, topography and water network.
  • OS Names API: Look-up service including place names, postcodes and roads.
  • OS Linked Identifiers API: Linking properties, streets and OS MasterMap features.

The release of these new APIs and the data hub are part of a wider objective, outlined in the UK’s Geospatial Strategy, to create a national location data framework by 2025.

 

 

 

Be Critical of the Data: Imagery too!

July 12, 2020 2 comments

As we have written about many times in this blog and in our book, being a critical consumer of data is essential for successful, wise decision making in the modern world.  Geospatial data offers a vast array of capabilities, but also offers numerous examples of discrepancies that often are found only through paying attention to details.  Even imagery is not immune to offsets and discrepancies, as we have detailed here and here.  The following represents another example of why this all matters.

A colleague of mine was working on a project outside of the town of Redcliff, in western Colorado USA.   In the first image, the red point on the older World Imagery (Esri Clarity) layer is at the corner of the house, and its coordinates (in WGS84) were calculated on that image.  In the second image, the white point at the corner of the house is on the current Esri World Imagery layer, and the coordinates (in WGS84) that were calculated based on that image.

bonar1
bonar2

Below are the same points on Google Earth. They match up pretty well with the World Imagery (Clarity) imagery.

bonar_google_earth

Further investigating the extent of what seems to be an offset between the two image layers are three screen captures below in the Paonia, Colorado area. These show a Delta County local road crossing the North Fork Gunnison River.   The map is cast in NAD83 UTM Zone 13.  The roads shapefile is from the Colorado Department of Transportation, Delta County Local and Major Roads (2019), and also in NAD83 UTM13.  The intersection point shows WGS1984 Lat-Long coordinates.   The two Esri World Imagery layers are both in WGS_1984_Web_Mercator_Auxiliary_Sphere.   The latest USDA NAIP (2019) is in NAD83 UTM13.   The World Imagery (Clarity) basemap and the NAIP both line up well with the Colorado DOT roads shapefile. The Esri World Imagery basemap does not.

bonar5
bonar6
bonar7

After reading these notes from my colleague, I spent some time looking at this, and then, along the lines of “being critical of the data” I did some of my own investigating.  I drew the yellow line in ArcGIS Online on the image along the bridge following the Clarity layer, and a cream-colored line along the bridge following the base imagery layer, noting that about a 12 meter offset exists, largely along a north-south axis, here.  But, I panned to other locations in Colorado and outside Colorado and found no offset.

bonar3
bonar4

I relayed this to my Esri colleagues who upon investigation, told me that this is not a projection issue, but likely a shift introduced in Maxar’s orthorectification process (related to ground control or DEM or both).   They told me that they will be publishing CONUS wide updates over the next several months which should resolve this particular localized shift.   Hence, repeat my above experiment whenever you happen to be reading this essay, and you may find different (and hopefully better) results!

I think at least five take-away points are important here (feel free to add more in the comments section).

(1)  GIS and remote sensing tools and data are continually evolving particularly in our data-as-services world.

(2)  Sometimes a wealth of information can be obtained by asking questions of the data and services providers.  Sometimes they can even hasten changes that need to be made.

(3)  The user needs to determine “fitness for use” regarding the spatial accuracy, completeness, date, and other characteristics for their project:  The project’s needs determine whether the data under consideration will be useful.  In the above example, if you were laying water or gas pipe or conducting land surveying for the assessor’s department, the above offsets would likely require you to use the Clarity layer or seek another source.  But if you are assessing regional or even local land use patterns, either layer would likely be just fine.

(4) Be critical of the data.   Be curious and ask questions.

(5)  Pay attention to detail.  Investigate.

–Joseph Kerski

Categories: Public Domain Data Tags: ,

Best practice guidance and tools for geospatial data managers and onwards to 2025…

Following on from their search engine optimisation recommendations for data publishing, the UK Government Cabinet Office and Geospatial Commission have combined forces to produce a comprehensive set of best practice guides and tools for geospatial data managers. Included in the set are:

Also recently published by the Geospatial Commission is the UK’s geospatial strategy 2020 to 2025 – Unlocking the power of location. The report identifies four critical mission objectives and nine opportunities to pursue with the end goal of a  ‘coherent national location data framework‘ by 2025. Highlighted in the report is the importance of open data sources/formats and collaboration with organisations such as the Open Geospatial Consortium, as key components for meeting those objectives.