Welcome

April 16, 2012 5 comments

Welcome to the Spatial Reserves blog.

The GIS Guide to Public Domain Data was written to provide GIS practitioners and instructors with the essential skills to find, acquire, format, and analyze public domain spatial data. Some of the themes discussed in the book include open data access and spatial law, the importance of metadata, the fee vs. free debate, data and national security, the efficacy of spatial data infrastructures, the impact of cloud computing and the emergence of the GIS-as-a-Service (GaaS) business model. Recent technological innovations have radically altered how both data users and data providers work with spatial information to help address a diverse range of social, economic and environmental issues.

This blog was established to follow up on some of these themes, promote a discussion of the issues raised, and host a copy of the exercises that accompany the book.  This story map provides a brief description of the exercises.

pdd

Advertisements

The Top 10 Most Useful Geospatial Data Portals, Revisited

February 18, 2019 7 comments

We have been writing this geospatial data column for 7 years now, beginning when our book The GIS Guide to Public Domain Data, was published.  Over those years, in addition to keeping issues such as data quality, copyright, privacy, and fee vs. free at the forefront of the conversation, we have tested and reviewed many geospatial data portals.  Some of these portals promise more than they deliver, some have been frustrating, but many have been extremely valuable in GIS work.  Back in 2017 we listed 10 of those that we have found most useful, rich with content, easy to use, and with metadata that is available and understandable.  A few are no longer functioning, and a few have emerged that merit inclusion in the top 10 list.  In creating such a list, we realize that “most useful” really depends on the application that one is using GIS for, but the list below should be useful for GIS users across many disciplines. Some allow for data to be streamed from web servers into your GIS software, and all allow data to be downloaded.

  1. The Open Data portal based on ArcGIS Hub technology.   This portal’s simple “what” and “where” interface is the entry point to a vast, curated, and growing list of valuable open data sites, along with a helpful story map described here.
  2. The Esri Living Atlas of the World is an expanding, curated set of data and maps on thousands of topics that can be used and also contributed to by the GIS community.
  3. The European Space Agency’s Sentinel Online data portal includes a wide variety of image-related data sets on the five themes of land, marine, atmosphere, emergency, and security.
  4. CIESIN at Columbia University has been serving data for over 20 years on climate, population, soil, econonics, land use, biodiversity, and other themes, including its Socioeconomic Data and Applications Center (SEDAC).
  5. The Atlas of the Biosphere serves global data, largely in grid format, of human impact, land use, ecosystems, and water resources themes.
  6. Natural Earth is a public domain dataset at small scale (1:10,000,000, 1:50,000,000, and 1:110,000,000) for the globe, in vector and raster formats that are easily ingestible in GIS software.
  7. The World Resources Institute hosts a variety of data geospatial data sets for specific areas of the world, such as Kenya and Uganda.
  8. The FAO GeoNetwork.  This portal contains global to regional scale data from administrative  boundaries and agriculture to soils, population, land use, and water resources.
  9. OpenTopography.   This NSF data facility from UC San Diego focuses on “Earth science-related, research-grade, topography and bathymetry data”, including a mountain-load of Lidar data.
  10. Many “lists of data sites” have appeared over the years.  Most are not kept up to date and end up being less useful over time.  However, those that are still quite helpful that we have reviewed are Dr Karen Payne’s list from the University of Georgia, and Robin Wilson’s list of free spatial data.  A few others that are useful are this list from the USGS that I started back when I worked at that organization, and this list from Stanford University.

A few others “almost make the top 10” :  The National Map from the USGS, data.gov from the US Government (though I am still frustrated that they removed the zebra mussels data that I used to access all the time), environmental and population data from TerraPopulusDiva-GIS’s data layers for each country, the UNEP Environmental Data Explorer, the NEO site at NASA Earth Observations, and OpenStreetMap (which besides roads, also includes buildings, land use, railroads, and, waterways)

For more details on any of these resources, search the Spatial Reserves blog for our reviews, remain diligent about being critical of the data you are considering using, and as always, we welcome your feedback.

lidar

Working with Lidar data obtained from the USGS National Map data portal. 

–Joseph Kerski

Track on Track, Revisited: Spatial Accuracy of Field Data

February 4, 2019 3 comments

Back in 2014, I tested the accuracy of smartphone positional accuracy in a small tight area by walking around a track.  During a recent visit to teach GIS workshops at Carnegie Mellon University, I decided to re-test, again on a running track.  My hypothesis was that triangulation off of wi-fi hotspots, cell phone towers, and the improved GPS constellation would have improved the spatial accuracy of my resulting track over those intervening years.

After an hour of walking, and collecting the track on my smartphone with a fitness app (Runkeeper), I uploaded my track as a GPX file and created a web map showing it in ArcGIS Online.  Open this map > use bookmarks > navigate to the Atlanta and Pittsburgh (Carnegie Mellon University) locations (also shown on the graphic below on the left and right, respectively).   Once I mapped my data, my hypothesis was confirmed:  I kept to the same lane on the running track, and the width of the resulting lines averaged about 5 meters, as opposed to 15 meters on the track from four years ago.  True, the 2014 track variability was no doubt in part because I was surrounded by tall buildings on three sides (as you can see in my video that I recorded at the same time) , while the building heights on the Carnegie Mellon campus were much lower.  However, you can measure for yourself on the ArcGIS Online map linked above and see the improvement over those two tracks taken just 4 years apart.

I did another test while at Carnegie Mellon University–during my last lap on the track, I moved to the inside lane.   This was 5 meters inside the next-to-outer lane where I completed my other laps.  I wanted to see whether this shift would be visible on the resulting map.  It is!  The lane is clearly visible on the map and on the right side of the graphic below, marked as “inside lane.”

To explore further, on the map above, go to > Contents, to the left of the map, and turn on the World Imagery Clarity layer.   Then use the Measure tool to determine how close the track is to the satellite imagery (which isn’t perfect either, but see teachable moments link below).  You will find that at times the track was 0.5 meters from the image underneath Lane 1, and at other times 3.5 meters away.

Both tracks featured “zingers” – lines stretching away from the actual walking tracks, resulting from points dropped as I exited the nearby buildings and walked outside, as my location based service first got its bearing.  But again, an improvement was seen:  The initial point was 114 meters off in 2014, but in 2018, only 21.5 meters.  In both cases, as I remained outside, the points became more accurate.  When you collect data, the more time you spend on the point you are collecting, typically the more spatially accurate that point is.

tracks_comparison

Comparison of tracks taken with the same application (RunKeeper) on a smartphone four years apart illustrate the improvements in positional accuracy over that time. 

To dig deeper into issues of GPS track accuracy and precision, see my related essay on errors and teachable moments in collecting data, and on comparing the accuracy of GPS receivers and smartphones and mapping field collected data in ArcGIS Online here and here.

Location based services on the smartphone still do not yet deliver the spatial accuracy for laying fiber optic cable or determining differences in closely-spaced headstones in cemeteries (unless a device such as Bad Elf or a survey-grade GPS is used).  Article are appearing that predict spatial accuracy improvements in smartphones.  Even today, though, I was quite pleased with my track’s spatial accuracy, particularly in 2018.  I was even more pleased considering that I had the phone in my pocket most of the time I was walking.  I did this in part because it was cold, and cold temperatures tend to rapidly deplete my cell phone’s battery (which is unfortunate, and the subject of other posts, many of which sport numerous adds, so they are not listed here).   Happy field data collection and mapping!

–Joseph Kerski

Your location information is for sale–Article

January 21, 2019 Leave a comment

A recent article in the New York Times reinforced a major theme of our book and this blog:  That your location is being captured, bought, sold, and used.  The article begins with a compelling image of over 235 million locations captured from more than 1.2 million unique devices during only a three-day period in 2017.  The article also features an animation of a smartphone user’s journey from home to work and to other locations atop a 3D satellite view of that person’s neighborhood.  It is my hope that this compelling and high-resolution animation, as well as those featured as streaks and points of lights in an even more compelling follow up article from the New York Times, will help people to pause and consider location privacy in a way that text-based posts like ours on Spatial Reserves might not adequately do.

The article also includes the results of a test that the staff had run:  To evaluate location-sharing practices, The NY Times tested 20 apps, most of which had been flagged by researchers and industry insiders as potentially sharing the data. Together, 17 of the apps sent exact latitude and longitude to about 70 businesses.  Having written the Public Domain Data book with Jill Clark and co-authoring this blog for 7 years, I’m not at all surprised by these figures.  Are you?

Geospatial technology is key to the efficient management of our complex world, and location is fundamental to that technology.  These tools and data have long enabled decision-makers to plan for a more sustainable and healthier future.  They are also becoming increasingly used by individuals to save time, save fuel, become more physically fit, and make smarter decisions in their daily lives.  These tools are not–and should not–go away.  But as we have stated in these essays–people should be aware of the information they are sending, on purpose and inadvertently, and why it all matters.

–Joseph Kerski

Global Land Cover Facility goes offline

January 7, 2019 Leave a comment

The world of geospatial data portals is dynamic; new sites appear and others disappear.  Sites are shut down due to the end of a funding period, changes in technologies, or as a result of mission or personnel changes. One of the earliest and most useful sites particularly for remotely sensed imagery recently went off-line–the Global Land Cover Facility from the University of Maryland.  Their notice said, “The GLCF has had a very good run since 1997! Originally it was funded under NASA’s Earth Science Information Partnership (under the inspired leadership of Martha Maiden of NASA). Subsequently it was maintained to support our NASA-funded research activities especially those concerned with Landsat data.   We feel we have attained what we wanted to accomplish, and now it’s time for us to move on and explorer other ventures. The data and services provided by GLCF are now mostly available via government agencies, especially USGS and NASA.”

To expand on the last note above, what should you, the GIS user who loves imagery, do?  For the time being, the GLCF data are still on a no-graphics FTP site, here:  ftp://ftp.glcf.umd.edu/.   But better yet, we have examined numerous functioning imagery portals on this blog; start here.  These include, for example, LandViewer, EOS Data Analytics, NASA AVIRIS, the GeoPortal, Lidar from USGS, DevelopmentSeed, Sentinel-2, and many others.

glcf

Thank you, GLCF!  You provided a wonderful service, and will be missed. 

–Joseph Kerski

Finding data via ArcGIS Hubs around the world

December 24, 2018 5 comments

To date, over 10,000 open data sites have been published by governments all over the world using ArcGIS Hub technology, as described here.  I recently wrote about how to use the ArcGIS Hub open data portal to search for and find geospatial data.  This story map is another way to find data, because it lists and provides access to some of these hubs.  The story map shows 1,500 sites created in 2017, so while only a fraction of the total, the map provides an excellent way to browse and find spatial data.  I used it recently to easily find transportation spatial data in Victoria, Australia, for example, for the purpose of constructing a GIS lesson around.

Equally important as accessing the data is that the story map raises the important point that the open spatial data movement is more than just about sharing data.  As we have written about extensively in this blog, the open spatial data movement is a reflection of the transformation that GIS is bringing in the way government, private, and nonprofit organizations work.  This increased collaboration between departments and engagement with citizens is a long-term and oft-difficult effort, but is resulting in increased efficiency, and yes, in data access for all.

Try this story map as a way to discover spatial data, particularly by region, and I think you will find it to be a useful tool.  For more information and background on ArcGIS Hub, see this essay by my colleague Andrew Turner.

hubs.JPG

Hubs around the world story map.  This map provides a description and access to dozens of data sets served via ArcGIS Hub portal technology.

Quality Matters…

December 17, 2018 Leave a comment

When Apple Maps was launched six years ago it was not a resounding success, by any measure. Although much of the criticism levelled at Apple focussed on the application interface and the lack of some keys features Google Maps users took for granted, for many the main issue was the quality of the map data. Apple Maps was originally delivered on a platform of third party map data, including TomTom and OpenStreetMap, with the majority of the satellite imagery sourced from DigitalGlobe. In response to the criticism, Apple vowed to do better and set off on a mission to improve the application and challenge the dominance of Google Maps.

Many application upgrades later, the map data is still not considered to be of the same quality as Google Maps. For example, zoom into a location in Queens, New York and compare the quality and range of information reported for local transport services in Google Maps compared to the same site and services reported in Apple Maps. Both Apple and Google Maps provide the number of the bus service using a particular stop but Google Maps provides more … street view data to visualise (and confirm) the location of the bus stop and better integration of supplementary traffic and transport service information. 

Queens, New York – Google Maps

The same bus stop in Apple Maps is shown at a slightly different location (further to the east along 48th Ave) and lacks the integrated street view. 

Queens, New York – Apple Maps

All that is set to change with an ambitious plan from Apple to rebuild their map data platform (see reports in TechCrunch and Medium). Taking a leaf out of the Google handbook on data collection, Apple have invested in specially equipped vans and drones, decked out with GPS, LiDAR, high resolution cameras and other data capture tools. In addition, Apple is also generating map information from anonymised iOS device data, adopting a strict ‘privacy-by-design’ methodology, to improve road network and pedestrian traffic information. 

The new in-house Apple Maps service has been available on a limited basis in California, USA for a few months now and there are plans to roll the service out to the whole USA over the next year. No word yet on when it will be available further afield.

Finding Data on ArcGIS Hub Open Data Portal

December 10, 2018 2 comments

The ArcGIS Hub Open Data Portal  has in a short period of time become a very useful means by which geospatial data can be searched, found, and used.   I believe that there are two main reasons why:  The ArcGIS Hub (1) allows organizations to easily host their own data, and (2) provides an easy to use but powerful set of tools for users to find data.  At the time of this writing, nearly 111,000 data sets were linked to the ArcGIS Hub Open Data Portal from nearly 6,000 organizations worldwide.

In keeping with the theme of our book and this blog, pay close attention to each of the data sets listed here that you are interested in using, and make sure you understand the usage restrictions, if any.  Not all data sets listed are necessarily “open” for any conceivable use, so again, understand the licensing and usage for your desired data set.

One advantage to using the ArcGIS Hub Open Data Portal from the user’s perspective is its simple layout (Figure 1):  The user is presented with a search category box along with a location box; i.e. “near <location x>.”  This surprisingly straightforward interface reminds me of how simple I found Google search to be nearly 20 years ago after years of using WebCrawler, AltaVista, and other search engines.

My education outreach team recently used the ArcGIS Hub in an educational context, in our Esri MOOC entitled “Do It Yourself Geo-Apps”.  In the MOOC, we had participants leverage open data to build web apps using Washington D.C.’s Vision Zero Safety data to help people learn more about pedestrian and bicyclist safety within the community. Specifically, students in the course searched and found data on commuting in Washington DC, downloaded the data as a shapefile, and uploaded it to their ArcGIS Online account (Figure 2) and began analyzing it.

An alternative workflow becoming rapidly adopted, as we have documented in this blog, rather than download and upload, is to obtain the link for the data as a REST endpoint and add it directly into a working session in ArcGIS Online, from which analysis tools can be run.  To do this using the ArcGIS Hub, use the APIs link on the right side after you find your desired data set, with one modification:  The GeoService full dataset are often tagged with a query statement.  For example, the Michigan hydrography polygons are listed as:  https://gisago.mcgi.state.mi.us/arcgis/rest/services/OpenData/hydro/MapServer/17/query?outFields=*&where=1%3D1.  To view the data in ArcGIS Online, remove everything after MapServer/, as shown in Figure 3.

Another fascinating feature in that same right-hand zone on the metadata results page is “create story map”, which, as the name implies gets you started right away creating and displaying the data in a story map (Figure 4) – in my case, a map series story map.  From this point, you could add additional layers, audio, video, photographs, and narrative to this same story map.

It is understandable with any open portal such as this, with contributions from a wide variety of organizations, that some challenges will exist.  From the perspective of the data user, one of those current challenges is finding results to searches on medium sized polygon areas, such as “Colorado” or “Platte River drainage”.   However, in the above Washington DC example, even if you did not know the term “Vision Zero”, a search on bicycle safety near Washington DC would provide you the result you are seeking.  The data extent for the Washington DC Vision Zero covered the entire North Atlantic Ocean, but that’s no doubt the result of an improperly encoded data point.

There is much more to ArcGIS Hub than this open data portal.  ArcGIS Hub includes community engagement tools such as event management, comment management, engagement dashboards, and initiatives.  One of the most appealing things about the ArcGIS Hub is that if you have an ArcGIS Online subscription, you can share your own authoritative open data with ArcGIS Hub.  By using your existing ArcGIS Online groups to identify data to share, you can set up public-facing websites for people to easily find and download your data in a variety of open formats. Your open datasets are connected to the source and are automatically updated.   I highly recommend spending time with the ArcGIS Hub, beginning with the open data portal.

openhub.JPG

Figure 1.  ArcGIS Hub Open Data interface, a very useful tool for finding geospatial data. 

hub1

Figure 2.  Vision Zero safety data for Washington DC from the ArcGIS Hub streamed into ArcGIS Online. hub2

Figure 3.  Michigan hydrography data from the ArcGIS Hub streamed into ArcGIS Online.

hub4

Figure 4.  Story map from Michigan hydrography polygons.