Archive

Author Archive

Imagery: It is what it is: Well, not always.

Being critical of all data, including geospatial data, is a chief theme of this blog and our book:  Use it wisely.  Know where it came from.  Know what has been done to it. Imagery has always been an important component of geospatial data, and today, it takes many forms, ranging from images taken on the ground with a phone or camera to those taken by webcams, UAVs, aircraft, satellites, and more.  Doesn’t this imagery “tell it like it is” and is it therefore exempt from us casting a critical eye on it?   I contend that no, images must be viewed critically so that you can determine if they are fit for your use in whatever project you are working on.

As one example in a larger story, consider the example of the removal of individuals from 20th Century photographs in the USSR. But lest we think such image doctoring was a thing of the past, there are dozens of articles that arise in current web searches highlighting modern day image manipulation, from fashion magazines to science journals, some inadvertent, some on purpose, some for purposes to benefit society and some unfortunately to mislead.  In the geospatial world, consider the image offsets on Google Maps in China that we wrote about not long ago.

Let’s discuss another aspect of image manipulation.  Consider the satellite images below.  They cover many different dates and are derived from many different sources.  But pay attention to what’s on–or off-the highways in the images.   These images all cover the same location—Interstate 30 – 35E interchange on the southwest side of downtown Dallas Texas USA, known locally as the Mixmaster.  Constructed between 1958 and 1962, you can spot, in some images, recent much-needed improvements due to the substantial increase in traffic volume in the past 60 years.

arcgis_online_dallas..PNGmapquest_dallas.png

Study area in ArcGIS Online (first image) and Mapquest (second image).

yahoo_maps_dallas.PNGbing_dallas.PNG

Study area in Yahoo maps (first image) and Bing maps (second image).

gmaps_web_dallas

Study area in Google Maps (it appears the same way in Google Earth). 

Notice anything missing on the Google Maps image?  No traffic. This would be unheard of, day or night, in this busy part of one of the largest cities in the USA.  Obviously, a vehicle-removal algorithm has been applied to the image. It is actually quite impressive, because the algorithm seems to only be applied to moving vehicles–cars, buses, trains–and not to stationary vehicles.  No doubt it is applied to make the process of viewing and interpreting the now-clearer imagery easier.  Interestingly, the moving-vehicle removal seems to only, at the present time, be present in running Google Maps on the web on a tablet or laptop:  Running the Google Maps app on a smartphone shows moving vehicles (below).

gmaps_dallas_app_on_phone.png

Study area in Google Maps app on smartphone. 

Key takeaway points and teachable moments here include:  (1) Images that can be obtained in web mapping services, in geodatabases, and portals should not be considered the same view that would be seen if someone were to be peering over the side of a UAV, airplane, spacecraft, or standing next to a webcam when the image was captured.  They can and are manipulated for a variety of purposes.   They need to be viewed critically.  At the same time, we can marvel and appreciate the many choices we have nowadays in imagery for use in geospatial analysis.   (2)  The machine learning and artificial intelligence techniques applied to imagery are further evidence for the enterprise nature of geotechnologies:  GIS is increasingly connected to mainstream IT trends and innovations.

–Joseph Kerski

 

List of Open Geospatial Data Portals from Recent GIS MOOC Discussions

April 29, 2019 Leave a comment

In our recent offerings of Do-It-Yourself GeoApps, an Esri MOOC, we’ve had some wonderfully rich discussions about open data portals.  I thought that the readers of this blog might appreciate viewing the listing of these portals from local to global that my colleague compiled, here:  diy_geoapps_data_sources_m19_combined 

As always, investigate each of these before using them, but this extensive list is a further reflection of the open data movement that we have been discussing in this blog, and it is exciting to see.

data_door

–Joseph Kerski

Testing positional accuracy underwater

April 15, 2019 2 comments

I recently had the opportunity to test positional accuracy while underwater.  And I did not even have to get wet!  While I was doing some GIS work with the excellent faculty at the University of Hamburg, I walked through the St. Pauli Elbe Tunnel while collecting a track.  As I did so, I reflected on the fascinating cultural and physical geography of this 1911 engineering masterpiece that is still in use:  The tunnel is 426 m (1,398 ft) long; it was a technical sensation when constructed; photos at the entrance show Kaiser Wilhelm II dedicating it.  It connected central Hamburg on the north side of the river with the docks and shipyards on the south side of the river Elbe.  The most amazing part was the four massive elevators, capable of carrying bikes and whole vehicles, and of course, 100 years ago, carriages and horses.  These elevators and tunnel are still functional and being used today!

While pondering these thoughts, I collected a track in the Runkeeper app, and mapped it as a GPX file in ArcGIS Online as a 2D webmap and as a shapefile in a 3D scene.  I wanted to test how spatially accurate a track underwater would be, in the x and y dimensions, but also in the z dimension.  First, let’s consider the x and y:  As I walked through the tunnel 24 m (80 ft) beneath the surface through one of the two 6 m (20 ft) diameter tubes, I expected the my app to lose sight of the GPS, Wi-Fi, and cell phone towers, but I did not know how far off my position would be.   My recent experiments on an above-ground track gave me a ray of hope that perhaps my position would be recorded as somewhere in Germany rather than in the North Sea or the Atlantic Ocean.

I was told by a local source who said that the tunnels are 8 m below the bottom of the river, making the water 16 m deep here (this depth here allowed Hamburg to become of the largest container ports in the world).  Thus, above me was 8 m of sediment (glacial, in this area), and 16 m of water for a total of 24 meters above me.  The elevation at the water surface here is approximately 5 m above sea level.  Thus, my elevation in the tunnel should be 5 – 24 = -19 meters, minus 3 more meters because I was standing on the bottom of the tunnel rather than the top, so -22 meters.

My results as a 2D webmap and as a 3D scene are shown below.  As is evident, the recorded elevations are all above sea level, at around 4 or 5 meters, so they were 22+5=27 meters off of my tunnel elevation.

elbe_tunnel_experiment_screen

A 2D map in ArcGIS Online showing the results of my experiment, with elevations in meters above sea level shown as labels. 

Feel free to open and interact with the data!  For example, to test the X and Y:  Using the measure tool, measure the distance between the tunnel as shown on the OpenStreetMap basemap and the position recorded by my track.  As I left the train station on the north side, my position was fairly accurately recorded, but once I descended the stairs into the tunnel, my position was off to the east by about 140 meters, and then shifted to the west and was off by about 240 meters.  But as I continued walking south, for the last 1/3 of my trek through the tunnel, my XY positional accuracy was only off by 50 meters.   I ascended the stairs and circled the parking lot on the south side, and was only 1 to 2 meters off once more.  I descended into the tunnel and walked north.  This time, my position was about 100 meters off, becoming worse as I kept walking.  My position overcorrected 80 meters to the north as I ascended the stairs, and “settled back” to being a few meters off as I walked to the train station.

To test the Z position:  The elevations were, as I suspected, not displaying their correct number below sea level; that is, 19 meters below sea level. However, you can see that the elevations are actually quite close to the elevation of the surface of the river in this area; at about 4.5 meters.

elbe_tunnel_gps_3D_scene.JPG

A 3D scene in ArcGIS Online showing the results of my experiment, with elevations in meters above sea level shown as labels and symbolized as cylinders.   Feel free to open and interact with this 3D scene!

Overall, with only a smartphone and a fitness app, displaying the data in ArcGIS Online, I was rather pleased with the fact that my positions all around were usually only in the tens or a few dozen meters off of true. This aligns with my recent reports of above-ground experiments and is further evidence of the improvements in spatial accuracy with all location based services.

Interested in further exploration?  See the evidence of my field trip in the photographs below.

tunnel1.jpgThe enormous elevators that carry pedestrians, bicycles, and even vehicles from the street level to the level of the tunnels.  This one is at the north side of the river with a photo of the opening ceremony with Kaiser Wilhelm II dedicating it.

tunnel2Standing at the entrance to the tunnel; photo also shows one of the glazed terra cotta art sculptures.

Now, go conduct your own accuracy experiments!

–Joseph Kerski

 

Accessing data with the WRI Open Data Portal

April 1, 2019 1 comment

The good folks at Blue Raster recently announced the creation of the World Resources Institute’s (WRI) Open Data Portal.  As their article explains, this portal is built with the open source platform CKAN, and provides a centralized, searchable catalog of all data provided by WRI.  We have written about specific WRI resources in this blog in the past, such as here, and one of our exercises makes use of WRI data for Kenya.  WRI has long been one of my favorite organizations and I have made extensive use of their data over the past 15 years.  Before this new portal, WRI researchers would publish data to their own WRI website, but this site only contained data from only a fraction of their projects.  The advantage of the new data portal is that it provides for the first time a full catalog of WRI datasets. Internally, the portal  also provides a more streamlined process for uploading and hosting data, which benefits WRI staff and data users, alike.

I tested the WRI Open Data Portal and was impressed at its clean interface and its ability to filter by file format and region.  I was puzzled, though, because I did not find that many additional data sets beyond what I have seen on WRI in the past.  But perhaps it is because of the search terms I used.  I also could not discover how to stream in data from ArcGIS Online as the article above mentions.  Nevertheless, the portal holds great promise; I highly encourage you to try it.

wri

WRI Open Data Portal interface.

–Joseph Kerski

A Report Card on the U.S. National Spatial Data Infrastructure (NSDI)

The Coalition of Geospatial Organizations (COGO) recently released its 2018 Report Card on the U.S. National Spatial Data Infrastructure (NSDI). The report card utilizes a letter grading system to depict the status and condition of the USA’s geospatial infrastructure.  COGO commissioned 24 content area experts to develop this second Report Card for the NSDI. These experts, drawn from the 12 member organizations of COGO, focused on the NSDI Framework to grade national efforts, and also candidly point to some of the shortcomings of those efforts.

The national assessment of the NSDI’s ability to meet future geospatial data, based on address, cadastral, elevation, geodetic control, government units, hydrography, orthoimagery, and transportation themes rose from a C in the 2015 Report Card, to a B- in the 2018 Report Card.   Grades improved across all themes; cadastral and transportation scoring a C- and a C, respectively.

The report also contains updated statements about the Federal Geographic Data Committee and the NSDI, which should be useful for anyone immersed in using geospatial data as well as to anyone teaching these concepts.  For example, on page 11 is a concise statement about what the NSDI should be, namely:
• A geographic resource for both the present and the future.
• A foundation for helping the public and private sectors use geospatial data for better decision making.
• A resource for many people and organizations working together towards common
goals.
• A collection of current and accurate geospatial data available for local, state,
national, and global use.
• An infrastructure for geospatial applications and services.
• A flexible resource that changes as technology, business requirements, and user needs change.

This 100-page document provides some excellent information about the history of data development and about the major data sets available for each theme.  In that sense, outside of the recommendations, the document is helpful as a short of “Data 101” document.  Plus, in some ways similar to the reviews that we have done on this blog, the authors review the major ways to access geospatial data.  The document provides insightful recommendations on how access can be improved, and how the data sets themselves can be improved, and so in the interests of all of us in the GIS profession, it is my fervent hope that these recommendations will be read and acted upon by those in the organizations responsible for each data set.

reportcard

Report card on the NSDI–a detailed and helpful document.

–Joseph Kerski

Categories: Public Domain Data Tags: , ,

The Top 10 Most Useful Geospatial Data Portals, Revisited

February 18, 2019 7 comments

We have been writing this geospatial data column for 7 years now, beginning when our book The GIS Guide to Public Domain Data, was published.  Over those years, in addition to keeping issues such as data quality, copyright, privacy, and fee vs. free at the forefront of the conversation, we have tested and reviewed many geospatial data portals.  Some of these portals promise more than they deliver, some have been frustrating, but many have been extremely valuable in GIS work.  Back in 2017 we listed 10 of those that we have found most useful, rich with content, easy to use, and with metadata that is available and understandable.  A few are no longer functioning, and a few have emerged that merit inclusion in the top 10 list.  In creating such a list, we realize that “most useful” really depends on the application that one is using GIS for, but the list below should be useful for GIS users across many disciplines. Some allow for data to be streamed from web servers into your GIS software, and all allow data to be downloaded.

  1. The Open Data portal based on ArcGIS Hub technology.   This portal’s simple “what” and “where” interface is the entry point to a vast, curated, and growing list of valuable open data sites, along with a helpful story map described here.
  2. The Esri Living Atlas of the World is an expanding, curated set of data and maps on thousands of topics that can be used and also contributed to by the GIS community.
  3. The European Space Agency’s Sentinel Online data portal includes a wide variety of image-related data sets on the five themes of land, marine, atmosphere, emergency, and security.
  4. CIESIN at Columbia University has been serving data for over 20 years on climate, population, soil, econonics, land use, biodiversity, and other themes, including its Socioeconomic Data and Applications Center (SEDAC).
  5. The Atlas of the Biosphere serves global data, largely in grid format, of human impact, land use, ecosystems, and water resources themes.
  6. Natural Earth is a public domain dataset at small scale (1:10,000,000, 1:50,000,000, and 1:110,000,000) for the globe, in vector and raster formats that are easily ingestible in GIS software.
  7. The World Resources Institute hosts a variety of data geospatial data sets for specific areas of the world, such as Kenya and Uganda.
  8. The FAO GeoNetwork.  This portal contains global to regional scale data from administrative  boundaries and agriculture to soils, population, land use, and water resources.
  9. OpenTopography.   This NSF data facility from UC San Diego focuses on “Earth science-related, research-grade, topography and bathymetry data”, including a mountain-load of Lidar data.
  10. Many “lists of data sites” have appeared over the years.  Most are not kept up to date and end up being less useful over time.  However, those that are still quite helpful that we have reviewed are Dr Karen Payne’s list from the University of Georgia, and Robin Wilson’s list of free spatial data.  A few others that are useful are this list from the USGS that I started back when I worked at that organization, and this list from Stanford University.

A few others “almost make the top 10” :  The National Map from the USGS, data.gov from the US Government (though I am still frustrated that they removed the zebra mussels data that I used to access all the time), environmental and population data from TerraPopulusDiva-GIS’s data layers for each country, the UNEP Environmental Data Explorer, the NEO site at NASA Earth Observations, and OpenStreetMap (which besides roads, also includes buildings, land use, railroads, and, waterways)

For more details on any of these resources, search the Spatial Reserves blog for our reviews, remain diligent about being critical of the data you are considering using, and as always, we welcome your feedback.

lidar

Working with Lidar data obtained from the USGS National Map data portal. 

–Joseph Kerski