Archive

Posts Tagged ‘Open Data’

A review of the Oak Hill West Virginia Open Data Site

September 1, 2019 Leave a comment

Recently at the Esri User Conference, I met the amazing and innovative GIS coordinator for the city of Oak Hill West Virginia.  The open data portal that this coordinator created represents an excellent example of what we have been describing in this blog–the open data movement combining with tools that enable GIS administrators to create and maintain the resources that will serve their internal and external data users.

Oak Hill Open Data works alongside its main website to provide information and enhance transparency to constituents through the power of GIS. Oak Hill Open Data is the repository for data, maps, and apps being generated by the City of Oak Hill. It features the city’s zoning map viewer, “Oak Hill CPR” for submitting citizen complaints directly to the city, an Operations Dashboard for dilapidated structures, and Story Maps about Needleseye Park and the city’s monthly city council meetings.

The site is easy to use, and includes web applications, pages, and even social media feeds and videos.  A search in the top search box on zoning, structures, transportation, and other words and phrases immediately netted me exactly what I was looking for, as streaming data services or as downloadable files in a variety of formats.  I have already used it to create several GIS-based lessons, such as investigating traffic accidents, and intend to do so in the future.  I salute those involved with putting this resource together and I encourage other governmental, nonprofit, academic, and private organizations to make use of the tools such as ArcGIS Hub to do something similar.  Fortunately, many are doing just that!

oak_hill_open_dataA section of the open Data Portal for Oak Hill, West Virginia (https://gis-cityoh.opendata.arcgis.com/).  

–Joseph Kerski

List of Open Geospatial Data Portals from Recent GIS MOOC Discussions

April 29, 2019 Leave a comment

In our recent offerings of Do-It-Yourself GeoApps, an Esri MOOC, we’ve had some wonderfully rich discussions about open data portals.  I thought that the readers of this blog might appreciate viewing the listing of these portals from local to global that my colleague compiled, here:  diy_geoapps_data_sources_m19_combined 

As always, investigate each of these before using them, but this extensive list is a further reflection of the open data movement that we have been discussing in this blog, and it is exciting to see.

data_door

–Joseph Kerski

New open location data services for Europe

April 22, 2019 1 comment

New open location data services developed by the Open European Location Services (ELS) project were recently demonstrated at the Geospatial World Forum earlier this month. Included in the test data services on show were EuroGlobalMap—a 1:1 million topographic dataset covering 45 countries and territories across Europe—and a geographical names gazetteer.

EuroGlobalMap

EuroGlobalMap

 

The Open ELS project is a two-year programme (started May 2017) aimed at improving access to location-based information captured and maintained by various public bodies across Europe. Among the participating organisations are national mapping authorities from Great Britain, Poland, Germany, Finland, Norway, Spain, Sweden and the Netherlands. Although there is no universally accepted definition of open data, the Open ELS project has defined open data as … Date from nation sources available free of charge under an open licence that are free to access and made available to the public without any restriction that impedes reuse.

With the exception of one EuroGlobalMap web feature service, which is licensed separately, the remaining data services developed for the ELS project are available under the Open ELS licence. Any one using the Open ELS data services is free to:

  • Copy, publish, share and re-use the data
  • Adapt the data and services
  • Use the data for both commercial and non-commercial applications

There no indication yet when the data services will be generally available; a demonstration site is available at https://demo.locationframework.eu/.

 

Finding Data on ArcGIS Hub Open Data Portal

December 10, 2018 2 comments

The ArcGIS Hub Open Data Portal  has in a short period of time become a very useful means by which geospatial data can be searched, found, and used.   I believe that there are two main reasons why:  The ArcGIS Hub (1) allows organizations to easily host their own data, and (2) provides an easy to use but powerful set of tools for users to find data.  At the time of this writing, nearly 111,000 data sets were linked to the ArcGIS Hub Open Data Portal from nearly 6,000 organizations worldwide.

In keeping with the theme of our book and this blog, pay close attention to each of the data sets listed here that you are interested in using, and make sure you understand the usage restrictions, if any.  Not all data sets listed are necessarily “open” for any conceivable use, so again, understand the licensing and usage for your desired data set.

One advantage to using the ArcGIS Hub Open Data Portal from the user’s perspective is its simple layout (Figure 1):  The user is presented with a search category box along with a location box; i.e. “near <location x>.”  This surprisingly straightforward interface reminds me of how simple I found Google search to be nearly 20 years ago after years of using WebCrawler, AltaVista, and other search engines.

My education outreach team recently used the ArcGIS Hub in an educational context, in our Esri MOOC entitled “Do It Yourself Geo-Apps”.  In the MOOC, we had participants leverage open data to build web apps using Washington D.C.’s Vision Zero Safety data to help people learn more about pedestrian and bicyclist safety within the community. Specifically, students in the course searched and found data on commuting in Washington DC, downloaded the data as a shapefile, and uploaded it to their ArcGIS Online account (Figure 2) and began analyzing it.

An alternative workflow becoming rapidly adopted, as we have documented in this blog, rather than download and upload, is to obtain the link for the data as a REST endpoint and add it directly into a working session in ArcGIS Online, from which analysis tools can be run.  To do this using the ArcGIS Hub, use the APIs link on the right side after you find your desired data set, with one modification:  The GeoService full dataset are often tagged with a query statement.  For example, the Michigan hydrography polygons are listed as:  https://gisago.mcgi.state.mi.us/arcgis/rest/services/OpenData/hydro/MapServer/17/query?outFields=*&where=1%3D1.  To view the data in ArcGIS Online, remove everything after MapServer/, as shown in Figure 3.

Another fascinating feature in that same right-hand zone on the metadata results page is “create story map”, which, as the name implies gets you started right away creating and displaying the data in a story map (Figure 4) – in my case, a map series story map.  From this point, you could add additional layers, audio, video, photographs, and narrative to this same story map.

It is understandable with any open portal such as this, with contributions from a wide variety of organizations, that some challenges will exist.  From the perspective of the data user, one of those current challenges is finding results to searches on medium sized polygon areas, such as “Colorado” or “Platte River drainage”.   However, in the above Washington DC example, even if you did not know the term “Vision Zero”, a search on bicycle safety near Washington DC would provide you the result you are seeking.  The data extent for the Washington DC Vision Zero covered the entire North Atlantic Ocean, but that’s no doubt the result of an improperly encoded data point.

There is much more to ArcGIS Hub than this open data portal.  ArcGIS Hub includes community engagement tools such as event management, comment management, engagement dashboards, and initiatives.  One of the most appealing things about the ArcGIS Hub is that if you have an ArcGIS Online subscription, you can share your own authoritative open data with ArcGIS Hub.  By using your existing ArcGIS Online groups to identify data to share, you can set up public-facing websites for people to easily find and download your data in a variety of open formats. Your open datasets are connected to the source and are automatically updated.   I highly recommend spending time with the ArcGIS Hub, beginning with the open data portal.

openhub.JPG

Figure 1.  ArcGIS Hub Open Data interface, a very useful tool for finding geospatial data. 

hub1

Figure 2.  Vision Zero safety data for Washington DC from the ArcGIS Hub streamed into ArcGIS Online. hub2

Figure 3.  Michigan hydrography data from the ArcGIS Hub streamed into ArcGIS Online.

hub4

Figure 4.  Story map from Michigan hydrography polygons. 

A review of Google’s search engine for Open Data

September 17, 2018 4 comments

An article in Nature described Google’s new search engine for open data, and since geospatial data is a fundamental part of open data, and after all these years, still challenging at times to find, I was immediately interested in testing it.

The tool, called Google Dataset Search, is accessible on this link.  Like Google Scholar and Google Books both of which I make heavy use of, this is a “specialized search engine.”

The utility of this tool will depend on metadata tagging. Indeed, as the article points out, “those who own the data sets should ‘tag’ them, using a standardized vocabulary called Schema.org, an initiative founded by Google and three other search-engine giants (Microsoft, Yahoo and Yandex)”. The schema.org dataset markup is the standard used here, but others are supported, such as CSVs, imagery, and proprietary formats.  I had to laugh at the open-ness of the last line in the list of what could qualify as a dataset, “Anything that looks like a dataset to you.”

Many data search engines and portals have a vast amount of data but very little geospatial data.  But with this tool, in my test searches, I found many useful geospatial data sets, some of which I knew and some that were new to me.  I have had challenges finding stream gauging data services for Australia recently, and with this tool found some new leads to investigate.  Being Google based, the searches were rapidly returned, with what I considered enough information to decide whether or not to investigate more fully (see screenshot below).  The data format was featured prominently, as was the coverage, both of which I appreciated.  NOAA was an early adopter of the indexing, and so it makes sense that I could find many NOAA data sets using this search engine.

I wonder if data in Github, or in Esri’s Living Atlas, or on state, national, and international portals will be findable.  I also wonder how the sheer importance of Google will influence how organizations tag their data in the future, and the influence this will have on agencies that perhaps did not put as much time on metadata as they perhaps should have.  Time will tell, but if Google Scholar and Google Books are any indication, the Google Dataset Search could indeed prove to be extremely useful for many of us in GIS research and education.

1111

Result of stream gauge search in the new Google data search engine. 

Ordnance Survey GB to provide OS MasterMap data for free.

June 19, 2018 1 comment

The UK government announced last week that key parts of the OS MasterMap dataset (OSMM) are to be made available free of charge (see full announcement from OS). The following two datasets are due to be released under the Open Government licence (OGL) agreement:

  • OS MasterMap Topography Layer property extents
  • OSMM Topography Layer TOIDs (TOpographic IDentifiers), built into the features in the OS OpenMap-Local dataset.

In addition, a number of datasets will be made available (through an API) for free, up to a threshold, including:

  • OS MasterMap Topography Layer, including building heights and functional sites
  • OS MasterMap Greenspace Layer
  • OS MasterMap Highways Network
  • OS MasterMap Water Network Layer
  • OS Detailed Path Network

The announcement didn’t included any information on what the threshold for free access was, but no doubt details will start to filter out shortly as organisations start making use of these new data assets.

An update on the World Bank’s Spatial Agent

May 28, 2018 5 comments
It sounds like a modern detective novel, but the Spatial Agent is actually a new, free app from the World Bank that offers one-stop access to interactive maps and charts of national, regional, and global datasets.  Jill Clark reviewed this site on our blog here. As we have written about data sources on this blog for nearly six years, and covered this topic in our book, the phrase “one-stop access” naturally caught my attention.  Could the Spatial Agent truly be all that it claims to be?

 

To find out, I began by watching a webinar that Mr Harshadeep recently conducted, which is as of this writing, still available online, here, after a short registration process.  In the webinar, and after my subsequent investigations, I was amazed at how the Spatial Agent as an app could bring together on-demand thousands of free, public-domain spatial data and analytical services (from in-situ and earth observation sources and also live cloud computing services).  It represents the data from sources such as the UN, NASA, NOAA, ESA, World Bank, many universities, and thousands of other sources, covering themes such as social (poverty, water supply), environmental (land use, biodiversity), economic (GDP, energy), and climate (snow cover, precipitation, for example).

The goal of the Spatial Agent is to offer solutions to many of the development challenges faced across the globe, which are often hampered by the poor availability of spatial data. For example, the app can be used to determine the areas in Madagascar that are susceptible to cyclones, or the areas in India that have high child malnutrition, or discovering the major exports of Vietnam, or determining how fast the population in Lagos is rising.  As these examples show, the Spatial Agent’s data cross boundaries, disciplines, and cover many different scales.  The Spatial Agent is the creation of Nagaraja Harshadeep, the lead environmental specialist and global lead for watersheds at the World Bank.  Mr Harshadeep has decades of experience working with spatial data and the application reflects his knowledge and passion.  There is much more than maps and imagery here, but rich tabular databases and other services, and the metadata for each of the data sets is quite robust.

I have been a long time fan of the spatial data from the World Bank, and use their data in several systems, including many layers available in ArcGIS Online.  The major limitation with the Spatial Agent app at this point that I can see is that it is just that — an app.  Therefore it only works on mobile phones and tablets.  I understand in part why it is focused on these devices–these are what many people are using day to day in their work.  Still, to bring the data sets into a GIS and more fully use them, I would love to see its capabilities inside of a series of user-driven interfaces that could be run in a standard web browser on a computer where I also have GIS and statistics tools available to me.  But I was glad to see this note about this very thing on the project’s site:  “The web version is being developed with the Bank’s Global Reach effort for launch later this year.”  Since the data and documentation are so rich on this site, I look forward to finding out how we will be able to use the services in a GIS.  Even without a GIS, the Spatial Agent is already very useful, because it is helping to bring data-driven decisions to daily decision making.

 

Two views of the hundreds of data layers and statistics available via the Spatial Agent.

For more information, including the links to access the apps, and the tutorials, see this page.