Archive

Author Archive

Data Quality on Live Web Maps

June 19, 2017 3 comments

Modern web maps and the cloud-based GIS tools and services upon which they are built continue to improve in richness of content and in data quality.  But as we have focused on many times in this blog and in our book, maps are representations of reality.  They are extremely useful representations, to be sure, particularly so in the cloud, but still are representations.   These representations are dependent upon the data sources, accuracy standards, map projections, completeness, processing and rendering procedures used, regulations and policies in place, and much more.  A case in point are offsets between street data and the satellite image data that I noticed in mid-2017 in Chengdu in south-central China.  The streets are about 369 meters southeast of where they appear on the satellite image (below):

china-google-maps

Puzzled, I panned the map to other locations in China.  The offsets varied, but they appeared everywhere in the country; for example, note the offset of 557 meters where a highway crosses the river at Dongyang, again to the southeast:

china-google-maps2

As of this writing, the offset appears in the same cardinal direction and only in China; indeed; After examining border towns with North Korea, Vietnam, and other countries, the offset appears to stop along those borders.  No offsets exist in Hong Kong nor in Macao.  Yahoo Maps Bing Maps both show the same types of offsets in China (Bing maps example, below):

china_bing

MapQuest, which uses an OpenStreetMap base, showed no offset.  I then tested ArcGIS Online with a satellite image base and the OpenStreetMap base, and there was no offset there, either (below).  This offset is a datum issue related to national security that is documented in this Wikipedia article.  The same data restriction issues that we discuss in our book and in our blog touch on other aspects of geospatial data, such as fines for unauthorized surveys, lack of geotagging information on many cameras when the GPS chip detects a location within China, and seeming unlawfulness of crowdsourced mapping efforts such as OpenStreetMap.

But furthermore, as we have noted, the satellite images are processed tiled and data sets, and like other data sets, they need to be critically scrutinized as well.  They should not be considered “reality” despite their appearance of being the “actual” Earth’s surface.  They too contain error, may have been taken on different dates or seasons, may be reprojected on a different datum, and other data quality aspects need to be considered.

china-agol

Another difference between these maps is the wide variation in the amount of detail in terms of the streets data in China.  The OpenStreetMap was the most complete; the other web mapping platforms offered a varying level of detail; some of which were seriously lacking, surprisingly especially in the year 2017, in almost every type of street except major freeways.  The streets content was much more complete in other countries.

It all comes back to identifying your end goals in using any sort of GIS or mapping package.  Being critical of the data can and should be part of the decision making process that you use and the choice of tools and maps to use.  By the time you read this, the image offset problem could have been resolved.  Great!  But are there now new issues of concern? Data sources, methods, and quality vary considerably among different countries. Furthermore, the tools and data change frequently, along with the processing methods, and being critical of the data is not just something to practice one time, but rather, fundamental to everyday work with GIS.

IndianaMap: Data and Visualization for Indiana

June 4, 2017 3 comments

IndianaMap is a resource for visualizing and accessing spatial data for the US state of Indiana.   The data source contains elements that other provinces, regional, and state governments might wish to adopt because as I see it, they are incredibly useful to the data user.  In addition, I keep talking with people who state that such adoption has helped them build internal support for their organization’s mission, and recommend examining IndianaMap for that reason as well as a model for how it could work.

One of my favorite things about IndianaMap is that it contains a map viewer, a map gallery, and a layer gallery, linked right at the top of the user interface.   At least 75 layers exist in this resource at the time of this writing.  Two that I was particularly glad to see were the geology layers and the historical 1990s Digital Orthophotoquads.  New imagery at 1 foot spatial resolution is also available.  Yes, 1 foot!  Each of the layers can be examined in more detail, previewed, its metadata viewed, or downloaded for use in desktop GIS software. Layers as map services can also be examined in a web based client, or one can choose to add the layer to the interface’s own Map View. Once you have explored layers of interest, you can use the “Add Content” tool on IndianaMap to quickly add, remove, and manage each layer. Each layer can be examined and saved as a favorite.  Sure enough, after I had used the tool, revisiting the site showed me the layers I had favorited in “My layers” so I could resume my work from a few days ago, right away.

On the Spatial Reserves blog, we often feature sites with a particularly useful user interface.  IndianaMap definitely achieves high quality marks in this regard.  Give it a try!

 

indianamap

User Interface for the Indiana Map showing bedrock geology, karst springs, and selecting one of my favorite parts of Indiana’s geology–its limestone.

Accessing Landsat 8 Data through DevelopmentSeed’s Libra Portal

May 21, 2017 1 comment

Libra is a very useful browser for open Landsat 8 satellite imagery. You can use it to browse, filter, sort, and download imagery for the entire planet.  Libra was developed by DevelopmentSeed, an engineering team solving complex problems with open software and open data, and AstroDigital, a company focused on providing imagery analyzed in real time and streamed to applications via their API.

Libra’s map interface is one of the most straightforward and useful that I’ve ever seen for imagery as evidenced in the screenshot below.  On the interface, each circle on the map represents the number of available images at that location. Filters at the top of the map can be used to select a date range, cloud cover percentage, and sun azimuth angle. Bundled downloads are available within one week of the image being taken. Individual bands are available for all 2015 scenes.

Once your download is complete, the site organizers recommend Landsat-util for processing the raw image and getting it ready for publication and analysis.  Search results are powered by Landsat 8 Metadata API and images from USGS Earth Explorer. Downloads are provided via Google Earth Engine and Amazon Web Services.

More information and the official announcement about this resource can be found here.   Give it a try and let us know what you think.

Interface for the Libra Development Seed Landsat Site.

Interface for the Libra Development Seed Landsat Site.

New LandViewer Tool for Quickly Finding and Analyzing Satellite Imagery

May 7, 2017 2 comments

The LandViewer tool and data portal quickly and painlessly allows you to browse and access satellite imagery for the planet.  The tool, developed by the Earth Observing System Inc.’s Max Polyakov, currently features Landsat 8 and Sentinel 2 imagery with more image sets soon to arrive.  Landsat 8 carries two instruments: The Operational Land Imager (OLI) sensor includes refined heritage bands, along with three new bands: a deep blue band for coastal/aerosol studies, a shortwave infrared band for cirrus detection, and a Quality Assessment band. The Thermal Infrared Sensor (TIRS) provides two thermal bands. Sentinel 2 is an Earth observation mission developed by the ESA as part of the Copernicus Programme to perform terrestrial observations in support of services such as forest monitoring, land cover changes detection, and natural disaster management.

Using the LandViewer tool, you can quickly zoom on an interactive web map to your area of interest.  You can filter on geography and time, including cloudiness, sun angle, and other parameters. At the time of this writing, 18 filters such as Atmospheric Removal, Panchromatic, NDVI, Thermal Infrared, False Color, and more, are available so that you can obtain the band combinations most suitable to your analysis in the areas of agriculture, geology, or other applications. A very helpful image interpretation screen is available to help you choose the combination that are best for your analysis goals.  You can do some contrast stretching in the web tool itself.  Then after signing in to the site, you can download the images in GeoTIF for further analysis using your favorite GIS tools.

The tool was also reviewed on the Geoawesomeness web site, and I wholeheartedly agree with their sentiments expressed–this is one of the most useful and fastest satellite image portals I have used. It is useful for research but also, given its ease of use, can even be used effectively to teach concepts of remote sensing.  Give it a try and let us know in the comments section what you think.

landsat_viewer.JPG

Landsat scenes with band combinations possible for an area on the southwest side of Costa Rica.

 

A review of the Los Angeles GeoHub

April 23, 2017 2 comments

The Los Angeles GeoHub represents, in many ways, the next generation GIS data portal. It is in my view what a data portal should be, and given the population and areal size of Los Angeles, that the portal is robust makes it even more impressive.  The data user can search the city’s open data site, and also do something that not all sites allow:  “Explore all data”.  At the time of this writing, “exploring all data” resulted in 554 results, which one can then add to “my favorites” for later investigation and download.  One can also explore the data by category, including business, boundaries, health, infrastructure, planning, recreation and parks, safety, schools, and transportation.  Most data sets can be downloaded as a spreadsheet, as a KML file, or a shapefile.  These layers include grasslands, fire stations, cell phone towers, road work projects, traffic, parcels, and dozens and dozens more–even bus stop benches and other treasures.  Each download is quick and painless.

A unique and very useful characteristic of the GeoHub is that each layer lists the number of attributes, which are easily displayed on the site.  Another wonderful feature is that each layer is displayed above its metadata listing as a web service inside ArcGIS Online, which can be opened immediately in ArcMap or ArcGIS Pro or streamed as a GeoJSON or GeoService as a full or filtered data set. Applications based on the data can also be accessed on the site, such as the CleanStat clean streets index and the social determinants of health app.  And yet there is even more–charts can be generated straight from the data, and a whole set of ArcGIS Online mapping applications that the city has generated are displayed in a gallery here.  Because of these applications, the site can be used effectively even by someone who is not familiar with how to run a GIS to understand Los Angeles better and to make smarter decisions.

If you are a data user, explore the data on the GeoHub today.  If you are a data administrator, consider using the GeoHub as a model for what you might develop and serve for your own data users in your own location.

la_geohub

Los Angeles GeoHub results from examining cell phone towers.  Note the many data-user-friendly items and choices to stream and download.

An Open Letter to the Open Data Community: Reaction

April 9, 2017 3 comments

A group of people at the Civic Analytics Network recently wrote “An Open Letter to the Open Data Community” that focuses on topics central to this blog and to our book. The Civics Analytics Network, is “a consortium of Chief Data Officers and analytics principals in large cities and counties throughout the United States.”  They state that their purpose is to “work together to advance how local governments use data to be more efficient, innovative, and in particular, transparent.”

The letter contained 8 guidelines the group believed that if followed, would “advance the capabilities of government data portals across the board and help deliver upon the promise of a transparent government.”  The guidelines included the following:

  1.  Improve accessibility and usability to engage a wider audience. 
  2. Move away from a single dataset centric view.
  3. Treat geospatial data as a first class data type.
  4. Improve management and usability of metadata. 
  5. Decrease the cost and work required to publish data. 
  6. Introduce revision history.
  7. Improve management of large datasets.
  8. Set clear transparent pricing based on memory, not number of datasets.

It is difficult to imagine a letter that is more germane to what we have been advocating on the Spatial Reserves blog.  We have been open about our praise of data portals that are user friendly–and critical of those that miss the mark–over the past five years.  We have noted the impact that the open data movement has had on the data portals themselves–becoming in many cases more user friendly and encouraging adoption of GIS beyond its traditional departmental boundaries.  The principles we have adhered to are also mentioned in this letter, such as being intuitive, data-driven, and with metrics.  The letter highlights a continued need, the ability to tie together and compare related data sets, which is at times challenging given “data silos.”

One of my favorite points in the letter is the authors’ admonition to “treat geospatial data as a first class data type.”  The authors claim that geospatial data is an underdeveloped and undervalued asset; and it “needs to be an integral part of any open data program”, citing examples from Chicago’s OpenGrid and Los Angeles’ GeoHub as forward-thinking models.

On the topic of metadata, the authors call for portals and managers to allow “custom metadata schemes, API methods to define and update the schema and content, and user interfaces that surface and support end-user use of the metadata.”  Hear, hear!  Equally welcome is the authors’ call to decrease the cost and work required to publish data. Through their point #6 about revision history, they advocate that these data sets need to be curated and updated but also allow historical versions to be accessed.

What are your reactions to this letter?  What do we need to do as the geospatial community to realize these aims?

Categories: Public Domain Data Tags:

New Developments in Open Data Portal services from the state of Utah

March 26, 2017 1 comment

In this blog, we have reviewed many international, national, regional, and local data portals over the past 5 years, those that are useful and those that still need “some work.”  One of the oldest USA state data portals is from Utah’s Automated Geographic Reference Center (AGRC).  I remember as a young US Census Bureau geographer, working with the AGRC on building the TIGER system back in the late 1980s, and they were thinking about data distribution even back then.  As GIS has evolved, so has the AGRC, and they remain one of the organizations I respect most in GIS.  Besides the wide variety of raster and vector data sets they offer for download, the AGRC also provides geocoding and point-in-polygon map queries via their own APIs, from api.mapserv.utah.gov.  In addition, the AGRC provides access to Utah’s TURN high resolution GPS base station network.

Utah has also established an open data site with a wide variety of data sets, in a multitude of formats, with extensive metadata, for download and also in GeoJSON and GeoService formats.  In short, the Utah portal is everything a geodata portal should be, modern and responsive, with links to web based GIS services, designed with the data user in mind.  I am not surprised by this, as I have long had a high regard for the way that those in academia, nonprofit organizations, government agencies, private companies, and even primary and secondary schools work closely together in the Utah GIS community, as I document in this video on one of my trips there.

You can also find Utah data in ArcGIS Online by querying on the data set owner name UtahAGRC here, or via its group here.

This Utah story map created by my colleague at Esri shows how some of these data sources can be used to tell the story of demographic change and the natural resources of Utah. Scroll down to the links at the end of the map to explore the data sources behind the map.  I encourage you to give the Utah AGRC data portal a try.

ut_gis.JPG

Utah AGRC State Data Portal after running a query for Trails.