Archive

Posts Tagged ‘ArcGIS Online’

Track on Track, Revisited: Spatial Accuracy of Field Data

February 4, 2019 3 comments

Back in 2014, I tested the accuracy of smartphone positional accuracy in a small tight area by walking around a track.  During a recent visit to teach GIS workshops at Carnegie Mellon University, I decided to re-test, again on a running track.  My hypothesis was that triangulation off of wi-fi hotspots, cell phone towers, and the improved GPS constellation would have improved the spatial accuracy of my resulting track over those intervening years.

After an hour of walking, and collecting the track on my smartphone with a fitness app (Runkeeper), I uploaded my track as a GPX file and created a web map showing it in ArcGIS Online.  Open this map > use bookmarks > navigate to the Atlanta and Pittsburgh (Carnegie Mellon University) locations (also shown on the graphic below on the left and right, respectively).   Once I mapped my data, my hypothesis was confirmed:  I kept to the same lane on the running track, and the width of the resulting lines averaged about 5 meters, as opposed to 15 meters on the track from four years ago.  True, the 2014 track variability was no doubt in part because I was surrounded by tall buildings on three sides (as you can see in my video that I recorded at the same time) , while the building heights on the Carnegie Mellon campus were much lower.  However, you can measure for yourself on the ArcGIS Online map linked above and see the improvement over those two tracks taken just 4 years apart.

I did another test while at Carnegie Mellon University–during my last lap on the track, I moved to the inside lane.   This was 5 meters inside the next-to-outer lane where I completed my other laps.  I wanted to see whether this shift would be visible on the resulting map.  It is!  The lane is clearly visible on the map and on the right side of the graphic below, which I labeled as “inside lane.”

To explore further, on the map above, go to > Contents, to the left of the map, and turn on the World Imagery Clarity layer.   Then use the Measure tool to determine how close the track is to the satellite imagery (which isn’t perfect either, but see teachable moments link below).  You will find that at times the track was 0.5 meters from the image underneath Lane 1, and at other times 3.5 meters away.

Both tracks featured “zingers” – lines stretching away from the actual walking tracks, resulting from points dropped as I exited the nearby buildings and walked outside, as my location based service first got its bearing.  But again, an improvement was seen:  The initial point was 114 meters off in 2014, but in 2018, only 21.5 meters.  In both cases, as I remained outside, the points became more accurate.  When you collect data, the more time you spend on the point you are collecting, typically the more spatially accurate that point is.

tracks_comparison

Comparison of tracks taken with the same application (RunKeeper) on a smartphone four years apart illustrate the improvements in positional accuracy over that time. 

To dig deeper into issues of GPS track accuracy and precision, see my related essay on errors and teachable moments in collecting data, and on comparing the accuracy of GPS receivers and smartphones and mapping field collected data in ArcGIS Online here.

Location based services on the smartphone still do not yet deliver the spatial accuracy for laying fiber optic cable or determining differences in closely-spaced headstones in cemeteries (unless a device such as Bad Elf or a survey-grade GPS is used).  Articles are appearing that predict spatial accuracy improvements in smartphones.  Even today, though, I was quite pleased with my track’s spatial accuracy, particularly in 2018.  I was even more pleased considering that I had the phone in my pocket most of the time I was walking.  I did this in part because it was cold, and cold temperatures tend to rapidly deplete my cell phone’s battery (which is unfortunate, and the subject of other posts, many of which sport numerous adds, so they are not listed here).   Happy field data collection and mapping!

–Joseph Kerski

Finding Data on ArcGIS Hub Open Data Portal

December 10, 2018 2 comments

The ArcGIS Hub Open Data Portal  has in a short period of time become a very useful means by which geospatial data can be searched, found, and used.   I believe that there are two main reasons why:  The ArcGIS Hub (1) allows organizations to easily host their own data, and (2) provides an easy to use but powerful set of tools for users to find data.  At the time of this writing, nearly 111,000 data sets were linked to the ArcGIS Hub Open Data Portal from nearly 6,000 organizations worldwide.

In keeping with the theme of our book and this blog, pay close attention to each of the data sets listed here that you are interested in using, and make sure you understand the usage restrictions, if any.  Not all data sets listed are necessarily “open” for any conceivable use, so again, understand the licensing and usage for your desired data set.

One advantage to using the ArcGIS Hub Open Data Portal from the user’s perspective is its simple layout (Figure 1):  The user is presented with a search category box along with a location box; i.e. “near <location x>.”  This surprisingly straightforward interface reminds me of how simple I found Google search to be nearly 20 years ago after years of using WebCrawler, AltaVista, and other search engines.

My education outreach team recently used the ArcGIS Hub in an educational context, in our Esri MOOC entitled “Do It Yourself Geo-Apps”.  In the MOOC, we had participants leverage open data to build web apps using Washington D.C.’s Vision Zero Safety data to help people learn more about pedestrian and bicyclist safety within the community. Specifically, students in the course searched and found data on commuting in Washington DC, downloaded the data as a shapefile, and uploaded it to their ArcGIS Online account (Figure 2) and began analyzing it.

An alternative workflow becoming rapidly adopted, as we have documented in this blog, rather than download and upload, is to obtain the link for the data as a REST endpoint and add it directly into a working session in ArcGIS Online, from which analysis tools can be run.  To do this using the ArcGIS Hub, use the APIs link on the right side after you find your desired data set, with one modification:  The GeoService full dataset are often tagged with a query statement.  For example, the Michigan hydrography polygons are listed as:  https://gisago.mcgi.state.mi.us/arcgis/rest/services/OpenData/hydro/MapServer/17/query?outFields=*&where=1%3D1.  To view the data in ArcGIS Online, remove everything after MapServer/, as shown in Figure 3.

Another fascinating feature in that same right-hand zone on the metadata results page is “create story map”, which, as the name implies gets you started right away creating and displaying the data in a story map (Figure 4) – in my case, a map series story map.  From this point, you could add additional layers, audio, video, photographs, and narrative to this same story map.

It is understandable with any open portal such as this, with contributions from a wide variety of organizations, that some challenges will exist.  From the perspective of the data user, one of those current challenges is finding results to searches on medium sized polygon areas, such as “Colorado” or “Platte River drainage”.   However, in the above Washington DC example, even if you did not know the term “Vision Zero”, a search on bicycle safety near Washington DC would provide you the result you are seeking.  The data extent for the Washington DC Vision Zero covered the entire North Atlantic Ocean, but that’s no doubt the result of an improperly encoded data point.

There is much more to ArcGIS Hub than this open data portal.  ArcGIS Hub includes community engagement tools such as event management, comment management, engagement dashboards, and initiatives.  One of the most appealing things about the ArcGIS Hub is that if you have an ArcGIS Online subscription, you can share your own authoritative open data with ArcGIS Hub.  By using your existing ArcGIS Online groups to identify data to share, you can set up public-facing websites for people to easily find and download your data in a variety of open formats. Your open datasets are connected to the source and are automatically updated.   I highly recommend spending time with the ArcGIS Hub, beginning with the open data portal.

openhub.JPG

Figure 1.  ArcGIS Hub Open Data interface, a very useful tool for finding geospatial data. 

hub1

Figure 2.  Vision Zero safety data for Washington DC from the ArcGIS Hub streamed into ArcGIS Online. hub2

Figure 3.  Michigan hydrography data from the ArcGIS Hub streamed into ArcGIS Online.

hub4

Figure 4.  Story map from Michigan hydrography polygons. 

Data Quality on Live Web Maps

June 19, 2017 6 comments

Modern web maps and the cloud-based GIS tools and services upon which they are built continue to improve in richness of content and in data quality.  But as we have focused on many times in this blog and in our book, maps are representations of reality.  They are extremely useful representations, to be sure, particularly so in the cloud, but still are representations.   These representations are dependent upon the data sources, accuracy standards, map projections, completeness, processing and rendering procedures used, regulations and policies in place, and much more.  A case in point are offsets between street data and the satellite image data that I noticed in mid-2017 in Chengdu in south-central China.  The streets are about 369 meters southeast of where they appear on the satellite image (below):

china-google-maps

Puzzled, I panned the map to other locations in China.  The offsets varied, but they appeared everywhere in the country; for example, note the offset of 557 meters where a highway crosses the river at Dongyang, again to the southeast:

china-google-maps2

As of this writing, the offset appears in the same cardinal direction and only in China; indeed; After examining border towns with North Korea, Vietnam, and other countries, the offset appears to stop along those borders.  No offsets exist in Hong Kong nor in Macao.  Yahoo Maps Bing Maps both show the same types of offsets in China (Bing maps example, below):

china_bing

MapQuest, which uses an OpenStreetMap base, showed no offset.  I then tested ArcGIS Online with a satellite image base and the OpenStreetMap base, and there was no offset there, either (below).  This offset is a datum issue related to national security that is documented in this Wikipedia article.  The same data restriction issues that we discuss in our book and in our blog touch on other aspects of geospatial data, such as fines for unauthorized surveys, lack of geotagging information on many cameras when the GPS chip detects a location within China, and seeming unlawfulness of crowdsourced mapping efforts such as OpenStreetMap.

But furthermore, as we have noted, the satellite images are processed tiled and data sets, and like other data sets, they need to be critically scrutinized as well.  They should not be considered “reality” despite their appearance of being the “actual” Earth’s surface.  They too contain error, may have been taken on different dates or seasons, may be reprojected on a different datum, and other data quality aspects need to be considered.

china-agol

Another difference between these maps is the wide variation in the amount of detail in terms of the streets data in China.  The OpenStreetMap was the most complete; the other web mapping platforms offered a varying level of detail; some of which were seriously lacking, surprisingly especially in the year 2017, in almost every type of street except major freeways.  The streets content was much more complete in other countries.

It all comes back to identifying your end goals in using any sort of GIS or mapping package.  Being critical of the data can and should be part of the decision making process that you use and the choice of tools and maps to use.  By the time you read this, the image offset problem could have been resolved.  Great!  But are there now new issues of concern? Data sources, methods, and quality vary considerably among different countries. Furthermore, the tools and data change frequently, along with the processing methods, and being critical of the data is not just something to practice one time, but rather, fundamental to everyday work with GIS.

US Congress: Maps and Data Portals

Did you know that the US Congress is increasing its use of web mapping tools in GIS?  In one of these map collections from the US Congress, you can discover organizations who host maps and data.  The themes of these maps include business locations, wildfires, hurricanes and current weather, public transportation, education, and many others.  For education, these maps could be useful for discussions in political science, geography, biology, business, and in other disciplines, as well as access to data for courses that use GIS.  To find out why the US Congress finds value in web mapping and GIS, read the excellent article that my colleague Lauren Lipovic recently wrote about how GIS is increasingly being used to inform and disseminate public policy.   Also, see the government policy page from Esri for more case studies, maps, and data.

The Mapping for Congress Featured Set contains some maps on the above themes; the most compelling one in my opinion is the set for Senator Risch from Idaho, on wildland fire potential, brownfields development success stories, and other themes of interest to citizens of that state and to those living elsewhere.  Another notable example from the Featured Set collection is this new map of Virginia state parks.

The Federal Agency Map Collection assembled by my colleague provides another excellent way of accessing more of these maps and data sets.  This collection includes the Mapping for Congress Expanded Gallery.  While some of the maps are applications that do not allow for streaming or downloading the data, GIS users with a bit of digging will find plenty of layers that they can use for analysis.  And equally importantly, these data sets represent a trend that we noted in our book–with increasing use of web mapping tools such as Esri story maps and the ability to assemble these map galleries, decision makers are increasing their use of GIS and spatial analysis in their decision making, which is good news for all of us.

id_map_gallery.JPG

The Idaho Map Gallery.

Landsat Thematic Bands Web Mapping Application in ArcGIS Online

December 20, 2015 2 comments

Teaching remote sensing?  Or just want to understand remotely sensed imagery better?  The Landsat Thematic Bands web mapping application can serve as a very useful teaching, learning, and research tool.  It covers the entire planet and the map is updated daily with new Landsat 8 scenes.

You can access many band combinations and indices by hovering over the tools to the left of the map image and selecting among the following:

  • Agriculture: Highlights agriculture in bright green. Bands 6,5,2
  • Natural Color: Sharpened with 25m panchromatic band. Bands 4,3,2+8
  • Color Infrared: Healthy vegetation is bright red. Bands 5,4,3
  • SWIR (Short Wave Infrared): Highlights rock formations. Bands 7,6,4
  • Geology: Highlights geologic features. Bands 7,4,2
  • Bathymetric: Highlights underwater features. Bands 4,3,1
  • Panchromatic: Panchromatic image at 15m. Band 8
  • Vegetation Index: Normalized Difference Vegetation Index (NDVI). (Band5-Band4)/(Band5+Band4)
  • Moisture Index: Normalized Difference Moisture Index (NDMI). (Band5-Band6)/(Band5+Band6)

The Time tool for different indices at larger scales based on a user-selected location enables examination of changes over years or over seasons.  It also provides temporal profiles for NDVI (Normalized Difference Vegetation Index), NDMI (Normalized Difference Moisture Index) and an Urban Index, dating back to 1973.  The Identify tool enables access to information on the date, cloud cover, and a spectral profile about each scene.  The Bookmark tool allows access to interesting locations such as the “Eye of the Sahara” in Mauritania.

The application is written using Web AppBuilder for ArcGIS accessing image services using the ArcGIS API for JavaScript, with access to the following Image Services:

  • Landsat Multispectral on AWS – 8-band multispectral 30m resolution image services and functions that provide different band combinations and indices.
  • Landsat Pan-sharpened on AWS – Panchromatic-sharpened imagery; 4-band (Red, Green, Blue and NIR); 30m resolution.
  • Landsat Panchromatic on AWS – Panchromatic imagery; 15m resolution.

These services can also be accessed through the public Landsat on AWS group on ArcGIS Online.  Because you can add these services as layers to your own maps or are adding to maps made by others, or if you are simply using the above web mapping application as a standalone map, you truly have “the world at your fingertips” with these maps and apps.  But there is a third option: Use the Unlock Earth’s Secrets page, also useful for instruction, with the above application embedded in it, but also with explanatory text and featured places around the planet as they have changed through time.

Think of the above as solid introductory segments to help your students, customers, or stakeholders see the value in remote sensing.  These maps and applications require very little geospatial technology skills to use, but allow you to focus on building remote sensing concepts and principles while exploring some truly engaging content and places.

To dig deeper, delve into the many powerful remote sensing functions available in ArcGIS Desktop.  One source for engaging, hands-on activities, is Kathryn Keranen and Bob Kolvoord’s book Making Spatial Decisions Using GIS and Remote Sensing:  A Workbook.

Give these resources a try!

Landsat web application

Landsat web application in ArcGIS Online.