Welcome

April 16, 2012 5 comments

Welcome to the Spatial Reserves blog.

The GIS Guide to Public Domain Data was written to provide GIS practitioners and instructors with the essential skills to find, acquire, format, and analyze public domain spatial data. Some of the themes discussed in the book include open data access and spatial law, the importance of metadata, the fee vs. free debate, data and national security, the efficacy of spatial data infrastructures, the impact of cloud computing and the emergence of the GIS-as-a-Service (GaaS) business model. Recent technological innovations have radically altered how both data users and data providers work with spatial information to help address a diverse range of social, economic and environmental issues.

This blog was established to follow up on some of these themes, promote a discussion of the issues raised, and host a copy of the exercises that accompany the book.  This story map provides a brief description of the exercises.

pdd

Climate GIS Data from WorldClim.org

Some of the most sought-after GIS data sets are those on climate, and rightly so, given its importance.  Worldclim.org is one of my favorite sources.  WorldClim’s data sets include minimum and maximum temperature, average temperature, precipitation, solar radiation, wind speed, water vapor pressure, plus 19 bioclimate variables (including such items as minimum temperature of the coldest month).  The following link explains the variables:

https://www.worldclim.org/bioclim

The following link provides access to the data, at a variety of spatial resolutions from 30 seconds to 10 minutes, all in grid format, as zipped geoTIFF files:

http://worldclim.org/version2

WorldClim is supported by Feed the Future to the Geospatial and Farming Systems Consortium of the Sustainable Intensification Innovation Lab.  However, you will need to dig for metadata on WorldClim–the site is extremely spartan, and take note – contains some ads – but don’t let that put you off — if you want a no-nonsense, quick way of accessing specific types of climate data, this is a valuable resource.

aa_wellington

Speaking of climate, ah! – the skies above Wellington, New Zealand, on the Autumnal Equinox there, March 2019.  Photo by Joseph Kerski. 

–Joseph Kerski

Categories: Public Domain Data Tags: , , ,

A review of the Humanitarian Data Exchange

July 7, 2019 2 comments

The Humanitarian Data Exchange, https://data.humdata.org/, has as its stated goal offering users to “find, share, and use humanitarian data all in one place”.  This seems like a tall order, but the resource in my judgment does an admirable job of meeting this goal.  What constitutes “humanitarian data”?  Broadly defined, it is any data that could be used to benefit humanity–population, health, environmental, and much more.  At the time of this review, the site contained 9,673 datasets in 253 locations from 1,189 sources.  HDX is managed by OCHA’s Centre for Humanitarian Data, located in The Hague. OCHA is part of the United Nations Secretariat and is responsible for bringing together humanitarian actors to ensure a coherent response to emergencies. The HDX team includes OCHA staff and a number of consultants who are based in North America, Europe, and Africa.  The site seems to focus on geospatial data.

One unique and interesting aspect of the Humanitarian Data Exchange as befitting the name of “exchange” is that the site offers users the ability to clean and visualize their data with easy-to-use tools that work with the Humanitarian Exchange Language (HXL) standard.  The site offers assistance in tagging one’s data, spotting for potential errors, and creating charts and graphs from data. Indeed, the resource is more than a geodata portal–it is a community of data sharing and sharers, including a blog.

I recently used the Humanitarian Data Exchange to obtain data in Brazil for revising some GIS-based lessons on for our set of exercises for ArcGIS Pro.  I found the site to be straightforward and easy to use, with results that were actually incredibly pertinent to my search terms of such things as states of Brazil political boundaries, hydrography, and the Human Development Index.  I also appreciated the clear graphics that told me right away what format the data were in–zipped shapefile, XLS table, and so on.   One can filter by location, file type, license type (a very nice feature), and organization, and even follow organizations of interest.

I highly recommend using the Humanitarian Data Exchange in your day to day GIS work.

hdx

The Humanitarian Data Exchange.

–Joseph Kerski

Categories: Public Domain Data

Using GeoSeer to find geospatial data

GeoSeer (https://www.geoseer.net) is a search engine for spatial data covering (at the time of this writing) over 1.2 million distinct spatial datasets from over 180,000 public OGC services (Web Map Services (WMS), Web Feature Services (WFS), Web Coverage Services (WCS), and Web Map Tile Services (WMTS)).

There are a huge number of OGC services online but they’re largely invisible. GeoSeer is designed to solve this “discoverability problem”, similar to how regular search engines like Google, Bing, and DuckDuckGo find web pages, but GeoSeer is focused on OGC services.  In fact, I was originally drawn to GeoSeer because of their statement, “We created GeoSeer to solve a problem: it’s an absolute pain to find spatial data.”  Indeed, this was one driving force for our book and this blog!  Another thing that attracted me was its simple interface (see below), which reminded me, after years of using 37.com, Webcrawler, AltaVista, and other web search tools over the 1990s, the first time I saw the simple but powerful Google search interface.

The GeoSeer bot scrapes over 350 Open Data portals looking for OGC services to add to the index, including the ArcGIS Hub Open Data Portal [1], the Global Earth Observation System of Systems (GEOSS) Portal [2], and many others. By scraping all of these portals and combining all of the discovered services into a single search engine, GeoSeer makes it easy to find open and public spatial data.  One can search by bounding box, lat-long, and service type, as explained here.

For end users, the benefits are a much easier data discovery process, while for the data providers it improves uptake of services and data that would otherwise be invisible and unused.   GeoSeer also includes an API to allow organisations to use the search functionality in their own WebGIS or application, allowing non-expert users to easily find and use these services.  I also liked working with the map-based interface to find data (see below).

How do you obtain data once you have found it?   GeoSeer is designed to demonstrate how the API can be integrated into a webGIS. Rather than trying to be a full webGIS, it was created to demonstrate how smooth the entire search-add process can be for end-users with the API.  It is a search engine to data, but unlike some of the other resources we have reviewed here, itself does not contain data.  Data can be downloaded via WFS or WCS.  WFS are raw vector/Feature data, while WCS are raw raster/Coverage data. If data is WMS/WMTS, then what the user sees is a pre-rendered map only.  Some datasets are available via multiple services, which is why GeoSeer says “distinct” in its “1.2 million distinct spatial layers” statements.  A statistics page shows how many of each data type GeoSeer has in its index: https://www.geoseer.net/stats/

To properly interact with the resulting data, the user will need to load the data into a proper (web) GIS.  The simplest way to do that is to use the regular search, which for me was to search for trails along the Front Range of Colorado, which netted me this link to the Denver Regional Council of Governments page: https://www.geoseer.net/rl.php?ql=7aa26a5ae9fe3e4c.    Not all services findable on GeoSeer are available via WFS or WCS. For example, if my trails data service was only a WMS, I could not download the data.  At the top of my search results, I had two URLs: “WMS GetCapabilities URL” and “WFS GetCapabilities URL”. The WMS version gets me to the pre-rendered map, which is what I saw displayed on the GeoSeer map screen; and the WFS allowed me to download the raw vector data.

I invite you to give GeoSeer a try!

[1] – https://spatialreserves.wordpress.com/2018/12/10/finding-data-on-arcgis-hub-open-data-portal/

[2] – https://spatialreserves.wordpress.com/?s=geoss

geoseer_mapping_interface0

Search screen for GeoSeer. 

geoseer_mapping_interface

GeoSeer Mapping Interface.

With thanks to the GeoSeer team for technical assistance with this post.

–Joseph Kerski

 

 

Make scientific data FAIR – article review

In a new article in Nature, author Shelley Stall and her colleagues argue that all disciplines should follow the geosciences and demand best practice for publishing and sharing data.  These authors make bold statements that I believe are long overdue, and the statements touch on many of the themes of this blog and our book, including the following:

(1) Although the amount of scientific data generated are enormous, and growing each year, these data are “not being used widely enough to realize their potential” and that “most researchers come up against obstacles when they try to get their hands on data sets.”  The authors show evidence that that only 1/5 of published papers typically post the supporting data in scientific repositories.  While I do not have the figures at hand, this seems to be even more of an acute need in the area of research that makes use of GIS and remote sensing–how often are the links to the data sets provided?  Very seldom. The authors give several key reasons why authors do not share data.

(2)  The authors state that “Too much valuable, hard-won information is gathering dust on computers, disks and tapes.”  I spent much of my career in federal data gathering agencies, and while much data has been digitized, not all of it has, and then — what happens when technology changes, including media (such as specific types of physical storage–see our post “Tossing the Floppies” for example) and means of access (such as the demise of most FTP sites where data was stored).  On a related theme, we have documented the demise of useful geospatial data portals in these essays that may sound good in that the new and improved portals perform better, but often, data are not ported over, or are done so in a way where researchers cannot find them.  Two of the many examples include the National Atlas of the USA and the Global Land Cover Facility.  I have spent many hours this year alone trying to obtain data that were on both of these sites.

On a positive note, the central theme of the article is to encourage disciplines to follow the leadership of Earth Sciences and adopt the “Enabling FAIR Data Project’s Commitment Statement in the Earth, Space, and Environmental Sciences for depositing and sharing data” principles.  This helps ensure that data be “findable, accessible, interoperable and reusable”.   Perhaps we will see the day when the majority of articles will provide, for the reader, access to the data behind them.  I do hope, however, that if authors cannot share the data for reasons of confidentiality or safety or for another valid reason, that provisions will be made for the research to still be published if it is deemed by peer review to be of value to the scientific community.

spatialagent2

A new article in Nature discusses why data is seldom shared in published scholarly research, and what might be done about the situation.

Nebraska: New Digital Hub and Geospatial Data Portal

May 26, 2019 2 comments

I recently gave the new Nebraska digital hub and geospatial data portal a test run.  As some of the GIS-based lessons I teach are based in Nebraska, some of which are in the Exercises section of this blog (including the fire tower site selection exercise), I have been using GIS portals in the state for at least 20 years.  As I have stated in reviews of other data portals, my preference is to have searching tools as well as the ability to browse categories of data, and this site provides both.

As detailed in a recent article in Enterprise Digitalization, the unique characteristic of the Nebraska portal is that it includes data from every single state agency, served in one location.  Crime, childcare, gas stations, and other information is all supposed to be here.  My searches for fuel or gas came up with nothing, as was my search for robberies, crime, and soils.   My search on land cover netted me unrelated items.   Admittedly, these were my first experiences with the site, and no doubt the site as well as my skills in using it will improve over time.  I did find some interesting data sets that I had not been able to find in the Nebraska data portals I had been using up to now.  I was also pleased to see that many of the data sets are offered as downloads and also as  streaming data services.  Plus, metadata is easily found and understandable.

I know it is a challenge to pull together a state portal, and I salute the efforts of those behind this one.  Admittedly, all I am interested in is geospatial data, and not the other data on the site.  For now, I will use this portal, but because I could not find all of the data sets I am interested in, I will continue to use the Nebraska DNR site, the state’s open data portal, and the sites I have found useful in the past at the University of Nebraska.   I only hope that if those other sites are removed at some point, that the developers do not do so before verifying that every single GIS dataset is moved to the new portal (please!).

Nebraska data portal.

–Joseph Kerski

Verifying location data with blockchain cryptography

May 20, 2019 1 comment

Following on from Joseph’s recent post on some of the issues associated with the plethora of image resources we now have access to, another interesting aspect of verifying those data sources relates to the basic premise of proof. How can a data provider, whether that’s an individual or global company, prove the data they collect and publish are an authoritative and accurate representation of the locations they seek to record? The problems associated with Geolocation and GPS Spoofing are not new, with many protocols and procedures now in place to help prevent this type of deception. Conversely, GPS simulators are generally available, making it relatively easy for location hackers to interfere with GPS signals.

So how do data providers prove entities, in both the physical and human-made environments, really do exist at a particular location? One company, XYO, has been working on an alternative to satellite networks as a source of verified location information – the XYO Network. By augmenting our increasingly interconnected network of digital devices with location tracking technologies that incorporate blockchain cryptography, these co-opted devices (acting as sentinels or bridges) can be configured to recognise, validate and confirm the location of each other. As each device acts as a witness to the location of other devices; the more witnesses there are confirming a device’s location, the less chance there is that location is incorrect. The end result is a decentralised location data network that is arguably at less risk of being compromised.

Bound witnesses (sentinel and bridge devices) in San Francisco – https://matrix.xyo.network/map

Using device networks in this manner is an interesting new development in evolution of geospatial data and an emerging technology to watch.

 

Imagery: It is what it is: Well, not always.

May 12, 2019 2 comments

Being critical of all data, including geospatial data, is a chief theme of this blog and our book:  Use it wisely.  Know where it came from.  Know what has been done to it. Imagery has always been an important component of geospatial data, and today, it takes many forms, ranging from images taken on the ground with a phone or camera to those taken by webcams, UAVs, aircraft, satellites, and more.  Doesn’t this imagery “tell it like it is” and is it therefore exempt from us casting a critical eye on it?   I contend that no, images must be viewed critically so that you can determine if they are fit for your use in whatever project you are working on.

As one example in a larger story, consider the example of the removal of individuals from 20th Century photographs in the USSR. But lest we think such image doctoring was a thing of the past, there are dozens of articles that arise in current web searches highlighting modern day image manipulation, from fashion magazines to science journals, some inadvertent, some on purpose, some for purposes to benefit society and some unfortunately to mislead.  In the geospatial world, consider the image offsets on Google Maps in China that we wrote about not long ago.

Let’s discuss another aspect of image manipulation.  Consider the satellite images below.  They cover many different dates and are derived from many different sources.  But pay attention to what’s on–or off-the highways in the images.   These images all cover the same location—Interstate 30 – 35E interchange on the southwest side of downtown Dallas Texas USA, known locally as the Mixmaster.  Constructed between 1958 and 1962, you can spot, in some images, recent much-needed improvements due to the substantial increase in traffic volume in the past 60 years.

arcgis_online_dallas..PNGmapquest_dallas.png

Study area in ArcGIS Online (first image) and Mapquest (second image).

yahoo_maps_dallas.PNGbing_dallas.PNG

Study area in Yahoo maps (first image) and Bing maps (second image).

gmaps_web_dallas

Study area in Google Maps (it appears the same way in Google Earth). 

Notice anything missing on the Google Maps image?  No traffic. This would be unheard of, day or night, in this busy part of one of the largest cities in the USA.  Obviously, a vehicle-removal algorithm has been applied to the image. It is actually quite impressive, because the algorithm seems to only be applied to moving vehicles–cars, buses, trains–and not to stationary vehicles.  No doubt it is applied to make the process of viewing and interpreting the now-clearer imagery easier.  Interestingly, the moving-vehicle removal seems to only, at the present time, be present in running Google Maps on the web on a tablet or laptop:  Running the Google Maps app on a smartphone shows moving vehicles (below).

gmaps_dallas_app_on_phone.png

Study area in Google Maps app on smartphone. 

Key takeaway points and teachable moments here include:  (1) Images that can be obtained in web mapping services, in geodatabases, and portals should not be considered the same view that would be seen if someone were to be peering over the side of a UAV, airplane, spacecraft, or standing next to a webcam when the image was captured.  They can and are manipulated for a variety of purposes.   They need to be viewed critically.  At the same time, we can marvel and appreciate the many choices we have nowadays in imagery for use in geospatial analysis.   (2)  The machine learning and artificial intelligence techniques applied to imagery are further evidence for the enterprise nature of geotechnologies:  GIS is increasingly connected to mainstream IT trends and innovations.

–Joseph Kerski