Archive

Author Archive

A review of the Oak Hill West Virginia Open Data Site

September 1, 2019 Leave a comment

Recently at the Esri User Conference, I met the amazing and innovative GIS coordinator for the city of Oak Hill West Virginia.  The open data portal that this coordinator created represents an excellent example of what we have been describing in this blog–the open data movement combining with tools that enable GIS administrators to create and maintain the resources that will serve their internal and external data users.

Oak Hill Open Data works alongside its main website to provide information and enhance transparency to constituents through the power of GIS. Oak Hill Open Data is the repository for data, maps, and apps being generated by the City of Oak Hill. It features the city’s zoning map viewer, “Oak Hill CPR” for submitting citizen complaints directly to the city, an Operations Dashboard for dilapidated structures, and Story Maps about Needleseye Park and the city’s monthly city council meetings.

The site is easy to use, and includes web applications, pages, and even social media feeds and videos.  A search in the top search box on zoning, structures, transportation, and other words and phrases immediately netted me exactly what I was looking for, as streaming data services or as downloadable files in a variety of formats.  I have already used it to create several GIS-based lessons, such as investigating traffic accidents, and intend to do so in the future.  I salute those involved with putting this resource together and I encourage other governmental, nonprofit, academic, and private organizations to make use of the tools such as ArcGIS Hub to do something similar.  Fortunately, many are doing just that!

oak_hill_open_dataA section of the open Data Portal for Oak Hill, West Virginia (https://gis-cityoh.opendata.arcgis.com/).  

–Joseph Kerski

Data.census.gov: A new US Census Bureau resource for data

August 18, 2019 Leave a comment

After reviewing the US Census Bureau’s new data site (https://data.census.gov) I find it more intuitive than the standard way of accessing Census data.  It was created, according to the site, “based on overwhelming feedback to streamline the way you (the data user) gets data.”   Aligned with other data sites that we have reviewed in this blog, the US Census Bureau is drifting away from delivering data through a series of individual tools (such as American FactFinder, OnTheMap, and My Congressional District) to a model where those tools’ capabilities are integrated and served on a single platform, eventually accessible by a single search bar on Census.gov.  The US Census Bureau’s vision for data dissemination is to “improve the customer experience so users spend less time searching for data products and more time using them.”  That phrase should bring joy to every GIS analyst’s heart.

For those familiar with Census data table numbers, you can search for table “B01001” to view the “Sex by Age” tables.  Or, start typing a Table ID and then select the options that appear below the search bar.  You can also transpose columns and rows in data tables.  You can download data tables in .CSV format and download multiple years in one ZIP file.   This site is a work in progress.  Don’t expect miracles and everything in the immense Census archive to suddenly become easy to access.  All of my searches resulted in hundreds or even thousands of hits, for example, a search for “census tract boundaries orange county california” netted over 1,000 entries, most of which had nothing to do with my search.  Thus, (1) the site still needs work but also (2) I need to spend more time with it.  You may need to bail out to the tried-and-true FTP site for some data.  To their credit, creating an easy-to-use resources for a data archive for the US Census Bureau has got to be a challenge.  After working over 4 years as a Census Bureau geographer, helping create the TIGER system, I have great respect for the agency.

While the site represents quite a bit of information to deal with, the release notes (https://data.census.gov/assets/releasenotes/faqs-release-notes.pdf) are helpful.  The section on known limitations, part of the release notes, are particularly useful.

census_data-site

Conducting a search with the US Census Bureau’s new data resource.

Joseph Kerski 

The Top 12 Most Useful Landsat Image Sites

August 4, 2019 4 comments

Recently, I wrote an essay about the sites that are, in my judgment, the top 10 in terms of containing useful geospatial data.   Now, I would like to describe what I consider to be the top sites for Landsat satellite imagery in terms of content and ease of use.  Let’s limit it to the Top 12.  Why might such a list be helpful?  First, there is no “one single site” to obtain Landsat data, and second, the sites are in continual flux, with some such as the Global Land Cover Facility disappearing and some having recently been created.  As with any consideration of data portals, make sure you have done a careful assessment of your data needs–band combinations, resolutions, formats, streamed services vs. downloaded files, dates, how many files you need, and so on, to guide you before you start searching.

(1)  The DevelopmentSeed’s Libra Portal.  I recently used this resource to include in the update (to ArcGIS Pro) for the Brazil land use change lesson that we host on the Spatial Reserves set of 10 hands-on exercises.  We wrote about the Libra portal here, and it remains in my judgment a no-nonsense resource that is easy to use with a wealth of options and data.

(2)  The EOS Data Analytics Landviewer, as we described here, is very useful and user friendly.   The EOS staff also wrote this helpful review of imagery sites.  Like the DevelopmentSeed portal, I find its user interface to be very straightforward.   The Landviewer includes Sentinel-2 and other imagery, as well.

(3)  Esri’s ArcGIS Living Atlas of the World has made amazing strides in content and usability since we first wrote about it here.  Most of Esri’s ArcGIS Living Atlas data is provided as streaming services instead of download, but for an increasing number of workflows, this is actually perfect.  The Living Atlas has in a few short years become probably the largest collection of spatial data on the planet, and so I recommend keeping it in mind not just for satellite imagery, but vector data as well, some of which can be downloaded, and all of it can be streamed.  Plus, you can contribute your organization’s data to the Living Atlas.  On a related note, be sure to check ArcGIS Online for imagery as well, the web GIS platform that Esri’s ArcGIS Living Atlas is based on.

(4)  The Esri Landsat Thematic Bands Web Mapping Application.   As we described in this post, through this application, you can access a variety of up-to-date and historical images in various band combinations, and save specific configurations and locations to share with others.

(5)  The USGS Earth Explorer.  While the Earth Explorer is in my view in need of improvement from the user’s perspective, it is functional and does contain a wealth of data, and sometimes is the best source for specific image sets.

(6)  The USGS Landsat Look Viewer.  I prefer the Landsat Look viewer’s interface over the Earth Explorer, as I described here.

(7)  The USGS GloVIS viewer.  I also prefer this interface over Earth Explorer.  GloVIS dates back to 2001 and was redesigned in 2017.

(8)  Landsat 8 archive on Amazon AWS.  As we described here, this has emerged as an amazing archive of data.  The user, as one might expect, is faced with a list of files rather than a fancy User Interface, but sometimes accessing specific files is exactly what one needs.

(9)  Landsat archive in Google Cloud.   Like the AWS experience, the UI is spartan but its data sets are vast, which is what one would expect from Google.

(10)  The FAO GeoNetwork.  This site focuses on vector data sets, but its raster holdings include many useful Landsat mosaics for specific geographic areas such as countries.

(11) Remote Pixel.  This is incredibly easy to use and blazing fast to zoom, pan, and query, and largely the work of a single individual.  It is my hope that if its developer does not maintain it in the future, that someone else will, because it is so marvelous.  Fortunately, the developer shows others how to host something like this themselves.

(12)  The Copernicus Open Data Access Hub, as its name implies, focuses on Sentinel data, but if you are interested in Landsat imagery, you probably are interested in other imagery as well.

satellite_image_portals_collageA few of the image portals described in this article.   We look forward to your feedback!

–Joseph Kerski

Climate GIS Data from WorldClim.org

Some of the most sought-after GIS data sets are those on climate, and rightly so, given its importance.  Worldclim.org is one of my favorite sources.  WorldClim’s data sets include minimum and maximum temperature, average temperature, precipitation, solar radiation, wind speed, water vapor pressure, plus 19 bioclimate variables (including such items as minimum temperature of the coldest month).  The following link explains the variables:

https://www.worldclim.org/bioclim

The following link provides access to the data, at a variety of spatial resolutions from 30 seconds to 10 minutes, all in grid format, as zipped geoTIFF files:

http://worldclim.org/version2

WorldClim is supported by Feed the Future to the Geospatial and Farming Systems Consortium of the Sustainable Intensification Innovation Lab.  However, you will need to dig for metadata on WorldClim–the site is extremely spartan, and take note – contains some ads – but don’t let that put you off — if you want a no-nonsense, quick way of accessing specific types of climate data, this is a valuable resource.

aa_wellington

Speaking of climate, ah! – the skies above Wellington, New Zealand, on the Autumnal Equinox there, March 2019.  Photo by Joseph Kerski. 

–Joseph Kerski

Categories: Public Domain Data Tags: , , ,

A review of the Humanitarian Data Exchange

July 7, 2019 2 comments

The Humanitarian Data Exchange, https://data.humdata.org/, has as its stated goal offering users to “find, share, and use humanitarian data all in one place”.  This seems like a tall order, but the resource in my judgment does an admirable job of meeting this goal.  What constitutes “humanitarian data”?  Broadly defined, it is any data that could be used to benefit humanity–population, health, environmental, and much more.  At the time of this review, the site contained 9,673 datasets in 253 locations from 1,189 sources.  HDX is managed by OCHA’s Centre for Humanitarian Data, located in The Hague. OCHA is part of the United Nations Secretariat and is responsible for bringing together humanitarian actors to ensure a coherent response to emergencies. The HDX team includes OCHA staff and a number of consultants who are based in North America, Europe, and Africa.  The site seems to focus on geospatial data.

One unique and interesting aspect of the Humanitarian Data Exchange as befitting the name of “exchange” is that the site offers users the ability to clean and visualize their data with easy-to-use tools that work with the Humanitarian Exchange Language (HXL) standard.  The site offers assistance in tagging one’s data, spotting for potential errors, and creating charts and graphs from data. Indeed, the resource is more than a geodata portal–it is a community of data sharing and sharers, including a blog.

I recently used the Humanitarian Data Exchange to obtain data in Brazil for revising some GIS-based lessons on for our set of exercises for ArcGIS Pro.  I found the site to be straightforward and easy to use, with results that were actually incredibly pertinent to my search terms of such things as states of Brazil political boundaries, hydrography, and the Human Development Index.  I also appreciated the clear graphics that told me right away what format the data were in–zipped shapefile, XLS table, and so on.   One can filter by location, file type, license type (a very nice feature), and organization, and even follow organizations of interest.

I highly recommend using the Humanitarian Data Exchange in your day to day GIS work.

hdx

The Humanitarian Data Exchange.

–Joseph Kerski

Categories: Public Domain Data

Using GeoSeer to find geospatial data

GeoSeer (https://www.geoseer.net) is a search engine for spatial data covering (at the time of this writing) over 1.2 million distinct spatial datasets from over 180,000 public OGC services (Web Map Services (WMS), Web Feature Services (WFS), Web Coverage Services (WCS), and Web Map Tile Services (WMTS)).

There are a huge number of OGC services online but they’re largely invisible. GeoSeer is designed to solve this “discoverability problem”, similar to how regular search engines like Google, Bing, and DuckDuckGo find web pages, but GeoSeer is focused on OGC services.  In fact, I was originally drawn to GeoSeer because of their statement, “We created GeoSeer to solve a problem: it’s an absolute pain to find spatial data.”  Indeed, this was one driving force for our book and this blog!  Another thing that attracted me was its simple interface (see below), which reminded me, after years of using 37.com, Webcrawler, AltaVista, and other web search tools over the 1990s, the first time I saw the simple but powerful Google search interface.

The GeoSeer bot scrapes over 350 Open Data portals looking for OGC services to add to the index, including the ArcGIS Hub Open Data Portal [1], the Global Earth Observation System of Systems (GEOSS) Portal [2], and many others. By scraping all of these portals and combining all of the discovered services into a single search engine, GeoSeer makes it easy to find open and public spatial data.  One can search by bounding box, lat-long, and service type, as explained here.

For end users, the benefits are a much easier data discovery process, while for the data providers it improves uptake of services and data that would otherwise be invisible and unused.   GeoSeer also includes an API to allow organisations to use the search functionality in their own WebGIS or application, allowing non-expert users to easily find and use these services.  I also liked working with the map-based interface to find data (see below).

How do you obtain data once you have found it?   GeoSeer is designed to demonstrate how the API can be integrated into a webGIS. Rather than trying to be a full webGIS, it was created to demonstrate how smooth the entire search-add process can be for end-users with the API.  It is a search engine to data, but unlike some of the other resources we have reviewed here, itself does not contain data.  Data can be downloaded via WFS or WCS.  WFS are raw vector/Feature data, while WCS are raw raster/Coverage data. If data is WMS/WMTS, then what the user sees is a pre-rendered map only.  Some datasets are available via multiple services, which is why GeoSeer says “distinct” in its “1.2 million distinct spatial layers” statements.  A statistics page shows how many of each data type GeoSeer has in its index: https://www.geoseer.net/stats/

To properly interact with the resulting data, the user will need to load the data into a proper (web) GIS.  The simplest way to do that is to use the regular search, which for me was to search for trails along the Front Range of Colorado, which netted me this link to the Denver Regional Council of Governments page: https://www.geoseer.net/rl.php?ql=7aa26a5ae9fe3e4c.    Not all services findable on GeoSeer are available via WFS or WCS. For example, if my trails data service was only a WMS, I could not download the data.  At the top of my search results, I had two URLs: “WMS GetCapabilities URL” and “WFS GetCapabilities URL”. The WMS version gets me to the pre-rendered map, which is what I saw displayed on the GeoSeer map screen; and the WFS allowed me to download the raw vector data.

I invite you to give GeoSeer a try!

[1] – https://spatialreserves.wordpress.com/2018/12/10/finding-data-on-arcgis-hub-open-data-portal/

[2] – https://spatialreserves.wordpress.com/?s=geoss

geoseer_mapping_interface0

Search screen for GeoSeer. 

geoseer_mapping_interface

GeoSeer Mapping Interface.

With thanks to the GeoSeer team for technical assistance with this post.

–Joseph Kerski