Search Results

Keyword: ‘tools’

Access data and analytical tools online from EOS Data Analytics

A new platform from EOS Data Analytics (https://eos.com/platform/) allows data users not only to access data, but even perform online image processing in a web browser.  It is a set of mutually integrated cloud products for searching, analyzing, storing, and visualizing geospatial data.  This is a representation of what we have been discussing on this blog, namely, the increasing adoption of Software as a Service, and in addition, the combination of SaaS with data services and analytical services.  Thus, GIS professionals can search for, analyze, store, and visualize large amounts of geospatial data in one platform.  Doing all this in one system and also in a browser is, quite frankly, quite amazing.

With the EOS Platform, GIS users have access to an ecosystem of four mutually integrated EOS products, which together provide a powerful toolset for geospatial analysts. Image data is stored in cloud-based storage and is available for image processing or remote sensing analysis at any time; this can be a raw user file, an imagery obtained from their LandViewer data portal, or an output file from their online EOS Processing tools.  The EOS Platform is currently available for free during an open Beta.   The LandViewer tool has been freely available for some time and will continue to be.

eos1

There are at least two reasons why image processing is the platform’s major asset: the processing of large data amounts runs online and offers as many as 16 workflows with even more coming soon. On top of that, users can get the best cartographic features of EOS Vision for vector data visualization and soon to come, analysis.

eos2

A list of EOS Processing workflows that can be filtered by industry and input data type.

I found the LandViewer tool easy to use, with a wide variety of data sets to choose from, including Landsat, MODIS, NAIP, and others.

eos3.png

The LandViewer tool interface.

eos4

Choosing one’s own area of interest via the LandViewer interface.

Advertisements

Sharing Geoprocessing Tools on the Web

January 15, 2017 3 comments

An article co-authored by Benjamin Pross, Christoph Stasch, and Albert Remke, of the 52°North Initiative for Geospatial Open Source Software GmbH; and Satish Sankaran and Marten Hogeweg of Esri describes a development that should interest anyone who uses geospatial data.  The 52°North Initiative for Geospatial Open Source Software has developed an open-source extension to ArcGIS for Desktop that enables access to Open Geospatial Consortium, Inc. (OGC), Web Processing Services (WPS).  The result?  This initiative makes it possible for these services to be used in the same manner as native ArcGIS geoprocessing tools.  In other words, they appear in the list of tools just as a standard buffer or overlay tool would appear.  Yes, it could be just that easy.

The article explains that “while ArcGIS allows geoprocessing tools to be published as a WPS, [ArcGIS] does not offer a WPS client interface. Consequently, it is not easy to access external non-ArcGIS geoprocessing tools such as simulation models, rich data interfaces, or processing capabilities from any other legacy software that supports the WPS interface.”  This points to the reason why this initiative offers such promise:  “The 52°North Extensible WPS Client for ArcMap was implemented as an open-source extension to ArcGIS that fully integrates into the ArcGIS for Desktop environment. It enables OGC WPS to be accessed and used in the same manner as native ArcGIS geoprocessing tools. This makes it easy to run WPS-based processes and integrate the results of that processing into ArcMap for use with other applications.”

In plain language, because the complex issues grappled with by GIS analysts often require major investments of time to generate models, services, and customized workflows and code, why should each analyst have to create all of this from scratch?  An enormous time savings could be realized if there was an easy way to share these things. This article both explains recent progress in this area but also encourages the community to think creatively about how to pursue further collaborative methods.

52north_initiative

ArcGIS Web Processing Service client architecture.

 

Hangar: A new on-demand UAV Data Service

October 28, 2018 Leave a comment

We have written about the rapid evolution of imagery platforms and portals many times in these blog essays over the years.  One major recent advancement is the UAV service from Hangar, an Esri business partner.  This service allows data users to order UAV imagery and receive it according to their project specifications:
https://www.spar3d.com/news/uav-uas/hangar-esri-reality-data-arcgis/

In my way of thinking, it is sort of like an” Uber for Drones”.  Let’s say you don’t have a pilot’s license, or time, or equipment, or expertise to fly your own UAV imagery.  Hangar is a new UAV service covering all areas that may be of interest to a client requiring imagery.  For this service, Esri partners with Hangar, a company that holds hundreds of waivers to fly almost anywhere and the expertise and equipment to serve clients from just about any discipline and with any need.   For more information, read the article “Hangar Joins Esri Startup Program to Add ‘Task & Receive’ Aerial Insights ArcGIS:”   https://www.prweb.com/releases/2018/05/prweb15514744.htm. 

And two of the best examples of some of the Hangar imagery is this story map of the devastation from the Carr fire in California and this story map showing some of their imagery for Kilauea, shown below.   Be sure to zoom and pan the 360-degree UAV imagery shown in these story maps.  Warning!  They are highly addictive and fascinating.  And for those of us in education, they make for an attention-getting teaching tools which I have already used numerous times from primary school to university and beyond to teach about wildfires and volcanic hazards.

uav_hangar.JPG

-Joseph Kerski

Categories: Public Domain Data Tags: , ,

An update on the World Bank’s Spatial Agent

May 28, 2018 5 comments
It sounds like a modern detective novel, but the Spatial Agent is actually a new, free app from the World Bank that offers one-stop access to interactive maps and charts of national, regional, and global datasets.  Jill Clark reviewed this site on our blog here. As we have written about data sources on this blog for nearly six years, and covered this topic in our book, the phrase “one-stop access” naturally caught my attention.  Could the Spatial Agent truly be all that it claims to be?

 

To find out, I began by watching a webinar that Mr Harshadeep recently conducted, which is as of this writing, still available online, here, after a short registration process.  In the webinar, and after my subsequent investigations, I was amazed at how the Spatial Agent as an app could bring together on-demand thousands of free, public-domain spatial data and analytical services (from in-situ and earth observation sources and also live cloud computing services).  It represents the data from sources such as the UN, NASA, NOAA, ESA, World Bank, many universities, and thousands of other sources, covering themes such as social (poverty, water supply), environmental (land use, biodiversity), economic (GDP, energy), and climate (snow cover, precipitation, for example).

The goal of the Spatial Agent is to offer solutions to many of the development challenges faced across the globe, which are often hampered by the poor availability of spatial data. For example, the app can be used to determine the areas in Madagascar that are susceptible to cyclones, or the areas in India that have high child malnutrition, or discovering the major exports of Vietnam, or determining how fast the population in Lagos is rising.  As these examples show, the Spatial Agent’s data cross boundaries, disciplines, and cover many different scales.  The Spatial Agent is the creation of Nagaraja Harshadeep, the lead environmental specialist and global lead for watersheds at the World Bank.  Mr Harshadeep has decades of experience working with spatial data and the application reflects his knowledge and passion.  There is much more than maps and imagery here, but rich tabular databases and other services, and the metadata for each of the data sets is quite robust.

I have been a long time fan of the spatial data from the World Bank, and use their data in several systems, including many layers available in ArcGIS Online.  The major limitation with the Spatial Agent app at this point that I can see is that it is just that — an app.  Therefore it only works on mobile phones and tablets.  I understand in part why it is focused on these devices–these are what many people are using day to day in their work.  Still, to bring the data sets into a GIS and more fully use them, I would love to see its capabilities inside of a series of user-driven interfaces that could be run in a standard web browser on a computer where I also have GIS and statistics tools available to me.  But I was glad to see this note about this very thing on the project’s site:  “The web version is being developed with the Bank’s Global Reach effort for launch later this year.”  Since the data and documentation are so rich on this site, I look forward to finding out how we will be able to use the services in a GIS.  Even without a GIS, the Spatial Agent is already very useful, because it is helping to bring data-driven decisions to daily decision making.

 

Two views of the hundreds of data layers and statistics available via the Spatial Agent.

For more information, including the links to access the apps, and the tutorials, see this page.

Need access to thousands of historical aerials and topographic maps at your fingertips? Try www.historicaerials.com

April 30, 2018 3 comments

Imagine having instant access to thousands upon thousands of historic aerial photographs and topographic maps to be able to examine change over time.  Thanks to a resource called Historic Aerials, you do have this wealth of information at your fingertips.  These aerials and maps, which go back 50, 60, and even 70 years or more, can be used for research, for instruction, for planning, and for other purposes.  Being in the field of geography and GIS education, I can think of many disciplines in which this can be used — urban and rural geography most certainly, but also biology, environmental science, city planning, history, agriculture, and also in GIS courses.  These resources foster spatial thinking about changes in time and space, from natural causes, such as volcanic eruptions or changes in river meanders, or from human causes, such as urbanization or the construction of reservoirs.  And given the connection that often exists between human and natural changes, sometimes these causes are intertwined–the construction of jetties along barrier islands often influences the naturally occurring longshore sediment transport and the migration of the islands themselves, as can be seen by comparing the historic to modern aerials of Ocean City, Maryland, for example.

The interface for the Historic Aerials is intuitive and provides tools that allow the user to compare USGS topographic maps of various years as well as the aerials themselves.  In the example below, I compare a 1958 with a 2009 aerial of a section of Grand Junction Colorado, before and after Interstate 70 and some surrounding housing was constructed.  You can also use the spotlight tool to “see back in time” for wherever you pan your cursor, and you can turn on the streets to see where  streets would one day be constructed on top of historical imagery.

The site comes from the Nationwide Environmental Title Research group, which has spent over 20 years collecting the worlds largest database of historical aerial images and topographical maps of the USA.  Their sources include USGS and USDA imagery, several private collections, and they are continually acquiring more. All the imagery they collect is orthorectified to provide the data in a searchable and precise geo-locatable format.  A print or digital image (for GIS users, GeoTIFF will be especially useful, delivered in lat-long coordinates, but JPG and PNG are also offered) of any of the maps or aerials is available.  In addition, a subscription service allows anyone to access the site with the following advantages: Full screen viewer, no advertising, PDF builder, quick JPEG downloads, and multiple user accounts.   I had a pleasant chat with the good people behind the site and found that their prices are quite reasonable.  Their FAQ, forum, and tutorials make it clear that they are committed to user success with these resources, and there are human beings behind this site to help as well.

Quite frankly, during my years working at the USGS, I always dreamed that my agency would create something like this.  Kudos to the HistoricAerials staff for making this a reality!

historic_aerials

The interface for HistoricAerials.com is quite intuitive and allows for fascinating investigations back in time for lands across the USA. 

Using the Data Interoperability Extension to import SDTS DLG files into ArcGIS Pro

April 16, 2018 1 comment

One of the themes of this blog and our book has been the wide variety of spatial data formats in existence.  Some of these spatial data formats have remained challenging to import into a GIS right up to the present day.  To meet this challenge, Esri’s Data Interoperability Extension has been a longstanding and useful set of tools that enables a wide variety of spatial data formats to be imported for use in a GIS.  It is an integrated spatial ETL (Extract, Transform, and Load) toolset that runs within the geoprocessing framework using Safe Software’s FME technology. It enables you to integrate data from multiple sources and formats, use that data with geoprocessing tools, and even publish it with ArcGIS Server.

I recently tested the Data Interoperability Extension in ArcGIS Pro and was thrilled with the results.  Read about how to install and authorize the extension here.  The extension does many things, but one that is particularly useful is that the extension creates a toolbox directly in ArcGIS Pro (graphic below).  I used this toolbox’s Quick Import tool to import a SDTS Format DLG (USGS Digital Line Graph) file directly to a file geodatabase.  The tool, like other ArcGIS Pro geoprocessing tools, walked me right through the process:  Data Interoperability > Quick Import > I then pointed to my DLG files in SDTS format > I named the resulting gdb (file geodatabase).  Once imported, I was then able to work with my hydrography, hypsography, roads, boundaries, and other data.

DLG files have existed since the early 1990s.  Why are we still working with them?  The reasons are that (1) They are dated but still useful vector data sets; (2) Many geospatial data portals still host data only in this format, such as the USGS Earth Explorer.  Another way to import these DLG files into ArcGIS Pro or ArcMap is to use the DLG2SHP tools that I wrote about in this set of guidelines using a standalone program.  See below for step-by-step instructions with the Data Interoperability Extension with screen shots.

data_interoperability

1. Use Toolboxes > Data Interoperability Tools > Quick Import, as shown above.data_interoperability_use_for_dlg_screen1

2.  Using QuickImport pulls up a “specify data source” dialog box, as shown above.

data_interoperability_use_for_dlg_screen3

3.  In the specify data source dialog box, use “find other source” and then specify SDTS format.

data_interoperability_use_for_dlg_screen2

4.  Selecting SDTS format.

data_interoperability_use_for_dlg_screen4

5.  Pointing to the SDTS file (after it has been unzipped and un-TAR’d) and saving it into a geodatabase.

data_interoperability_use_for_dlg_screen5

6. Once the file has been imported into a geodatabase, it can be added to a new map in ArcGIS Pro.  The data is now ready for use, as shown for this hydrography example, above. 

 

The Coastal Atlas from the Maryland Department of Natural Resources

February 5, 2018 1 comment

The Maryland Coastal Atlas serves up ocean use and resource data, coastal hazard and shoreline data, and near-shore and estuarine data.  The purpose of the atlas is to make coastal related geospatial datasets available to agencies, researchers, and the general public for viewing and for performing basic overlays.  Tools are being added to make the atlas more versatile for users to do analysis and to help simplify or select data important for different users’ needs. The list of layers is extensive; at least 100 items are included.  But equally impressive is its ability to add dozens more layers from the MDiMapD database on such themes as agriculture, housing, demographics, hydrology, and much more.

The Atlas uses the Esri Web App Builder for its interactive map capabilities.  One of my favorite things about the atlas is the user’s ability to add data to the web interface from ArcGIS Online, a URL, or a file of the user’s own creation.  The site features unexpected helpful touches such as palette of drawing tools that makes the atlas a rich teaching tool, and transects that can be drawn in the map to analyze such things as erosion rates.

A few enhancements on the site could be done to make it more useful, such as an expansion of the fairly limited query tool and an explanation of how it can be used.  I was puzzled how to close the transect results once I had created one, but this and other user interface questions were small; overall, the interface was intuitive.  The Maryland Coastal Atlas provides an excellent addition to the other portals we have written about in this region, such as the Maryland iMap Data Catalog We wrote about the state of Maryland’s GIS portal in the past, and the selected other data portals for the Chesapeake Bay.

The atlas uses the map services available from the Maryland GIS Portal and the iMap Open Data Catalog that we reviewed above.  To obtain the data, go to the Maryland Data Catalog to download the data or get the API to use in an online mapping application.  All of the Maryland Coastal Hazard datasets on the atlas are available through the data catalog but not all are downloadable.  Here is an example of a dataset on the atlas shown in the iMap Data Catalog with the Download and API function available on the listing.  Every layer is a REST service hosted by Maryland iMap, managed by the Geographical Information Office (GIO) and the state IT group (DOIT).

mdcoastal

The Coastal Atlas from the Maryland Department of Natural Resources.

Privacy concerns from fitness maps and apps

January 31, 2018 2 comments

We frequently write about the need to teach about and be aware of location privacy with the rapid advancement and web-enablement of GIS.  Thus it wasn’t a surprise when recent concerns arose over an amazing map from Strava Labs.  Maps generated from GPS-enabled fitness devices and other recreational uses of GPS such as GPS Drawing, as well as those from the fitness tracker market such as Fitbit and Garmin, have for several years been sharable and viewable.  Strava has been one of the leaders in helping people stay motivated to meet their fitness goals by providing tools such as apps and maps.  But perhaps the Strava map attracted more attention than others because it contains an amazing “over 1 billion activities and 13 trillion data points”, or perhaps because the map is so responsive and contains some stunning cartography that the web map user can customize.

Whatever the reason, as reported in USA TodayPopular MechanicsWired, and elsewhere, location privacy concerns have arisen recently over the new Strava map.  Specifically, “Security experts over the weekend questioned whether the user-generated map could not only show the locations of military bases, but specific routes most heavily traveled as military personnel unintentionally shared their jogging paths and other routes.”  Some of the posts have reported that it may even be possible to scrape the data to discover the person behind each of the tracks, and the Strava CEO has responded to these and other concerns.  Any GIS user knows that much can be discovered through mapped layers and satellite imagery these days, shedding new light on what is really “secret” in our 21st Century world, but maps aimed at the recreational user are bringing these discussions to the general public.  The particular concern with the Strava data is not so much just the location information, but the temporal data tied to the location, and potential identification of individuals.

Much of it comes down to what we have been saying in this blog–understand the defaults for whatever you are doing in GIS, whether it is the projection of your geospatial data or the location-based app on your phone.  Ask yourself, “What is the default–is my data public by default? Is my projection Web Mercator by default?  Can I override the default, and if so, how?  What is the best way to represent this spatial information?  Do I need to share this information?  If I need to share the information, how should I do it?”  and then act accordingly.   For more on this topic, I encourage you to read some of our short essays, such as Why Does a Calculator App need to know my location?, Making the Most of Our Personal Location Dataposting cat pictures and The Invasion of the Data Snatchers.

stravamap

A section of the Strava heat map, showing the results of people who have recorded and shared their fitness walks and runs.  As one might expect, city park and a high school track stand out as places where more people conduct these activities.  As with other maps showing locations where people are now or where they have been, location privacy concerns have been raised. 

Accessing and Using Lidar Data from The National Map

January 8, 2018 1 comment

We have written about the USGS data portal NationalMap numerous times in this blog and in our book, but since the site keeps getting enhanced, a re-examination of the site is warranted.  One of the enhancements over the past few years is the addition of Lidar data to the site.  I did some recent testing of searching for and downloading Lidar data on the site and wanted to report on my findings.  For videos of some of these procedures, go to the YouTube Channel geographyuberalles and search on Lidar.

From a user perspective, in my view the site is still a bit challenging, where the user encounters moments in the access and download process where it is not clear how to proceed.  However, (1) the site is slowly improving; (2) the site is worth investigating chiefly because of its wealth of data holdings:  It is simply too rich of a resource to ignore.  One challenging thing about using NationalMap is, like many other data portals, how to effectively narrow the search from the thousands of search results.  This in part reflects the open data movement that we have been writing about, so it is a good problem to have, albeit still cumbersome in this portal.  Here are the procedures to access and download the Lidar data from the site:

  1. To begin:  Visit the National Map:  https://nationalmap.gov/ > Select “Elevation” from this page.
  2. Select “Get Elevation Data” from the bottom of the Elevation page.  This is one of several quirks about the site – why isn’t this link in a more prominent position or in a bolder font?
  3. From the Data Elevation Products page left hand column:   Select “1 meter DEM.”
  4. Select the desired format.  Select “Show Availability”.   Zoom to the desired area using a variety of tools to do so.  In my example, I was interested in Lidar data for Grand Junction, in western Colorado.
  5. Note that the list of  available products will appear in the left hand column.  Lidar is provided in 10000 x 10000 meter tiles.  In my example, 108 products exist for the Grand Junction Lidar dataset.  Use “Footprint” to help you identify areas in which you need data–the footprints appear as helpful polygon outlines.  At this point, you could save your results as text or CSV, which I found to be quite handy.
  6. You can select the tiles needed one by one to add to your cart or select “Page” to select all items.  Select the Cart where you can download the tiles manually or select the “uGet Instructions” for details about downloading multiple files.  Your data will be delivered in a zip format right away, though Lidar files are large and may require some time to download.

 

lidar_results.JPG

The National Map interface as it appeared when I was selecting my desired area for Lidar data.

Unzip the LAS data for use in your chosen GIS package.  To bring the data into ArcGIS Pro, create a new blank project and name it.  Then, Go to Analysis > Tools > Create LAS dataset from your unzipped .las file, noting the projection (in this case, UTM) and other metadata.  Sometimes you can bring .las files directly into Pro without creating a LAS dataset, but with this NationalMap Lidar data, I found that I needed to create a LAS dataset first.

Then > Insert:  New Map > add your LAS dataset to the new map. Zoom in to see the lidar points.  View your Lidar data in different ways using the Appearance tab to see it as elevation, slope, aspect (shown below), and contours.  Use LAS dataset to raster to convert the Lidar data to a raster.  In a similar way, I added the World Hydro layer so I could see the watersheds in this area, and USA detailed streams for the rivers.

lidar_results2.JPG

Aspect view generated from Lidar data in ArcGIS Pro.

There are many things you can do with your newly downloaded Lidar data:  Let’s explore just a few of those.  First, create a Digital Elevation Model (DEM) and a Digital Surface Model (DSM).  To do this, in your .lasd LAS dataset > LAS Filters > Filter to ground, and visualize the results, and then use LAS Dataset to Raster, using the Elevation as the value field.  Your resulting raster is your digital elevation model (DEM).  Next, Filter to first return, and then convert this to a raster:  This is your digital surface model (DSM).  After clicking on sections of each raster to compare them visually, go one step further and use the Raster Calculator to create a comparison raster:  Use the formula:  1streturn_raster – (subtract) the ground_raster.  The first return result is essentially showing the objects or features on the surface of the Earth–the difference between “bare earth” elevation and the “first return”–in other words, the buildings, trees, shrubs, and other things human built and natural.  Symbolize and classify this comparison surface to more fully understand your vegetation and structures.  In my study area, the difference between the DEM and the DSM was much more pronounced on the north (northeast, actually) facing slope, which is where the pinon and juniper trees are growing, as opposed to the barren south (southwest) facing slope which is underlain by Mancos Shale (shown below).

lidar_veght

Comparison of DEM and DSM as a “ground cover” raster in ArcGIS Pro.

My photograph of the ridgeline, from just east of the study area, looking northwest.  Note the pinon and juniper ground cover on the northeast-facing slopes as opposed to the barren southwest facing slope.

Next, create a Hillshade from your ground raster (DEM) using the hillshade tool.   Next, create a slope map and an aspect map using tools of these respective names.  The easiest way to find the tools is just to perform a search.  The hillshade, slope, and aspect are all raster files.  Once the tools are run, these are now saved as datasets inside your geodatabase as opposed to earlier—when you were simply visualizing your Lidar data as slope and aspect, you were not making separate data files.

Next, create contours, a vector file, from your ground raster (DEM), using the create contours tool.  Change the basemap to imagery to visualize the contours against a satellite image.  To create index contours, use the Contour with Barriers tool.  To do this, do not actually indicate a “barriers” layer but rather use the contour with barrier tool to achieve an “index” contour, as I did, shown below.  I used 5 for the contour interval and 25 (every fifth contour) for the index contour interval.  This results in a polyline feature class with a field called “type”.  This field receives the value of 2 for the index contours and 1 for all other contours.  Now, simply symbolize the lines as unique value on the type field, specifying a thicker line for the index contours (type 2) and a thinner line for all the other contours.

lidar_results4

Next, convert your 2D map to a 3D scene using the Catalog pane.  If you wish, undock the 3D scene and drag it to the right side of your 2D map so that your 2D map and 3D scene are side by side.  Use View > Link Views to synchronize the two.  Experiment with changing the base map to topographic or terrain with labels.  Or, if your area is in the USA like mine is, use the Add Data > USA topographic > add the USGS topographic maps as another layer.  The topographic maps are at 1:24,000 scale in the most detailed view, and then 1:100,000 and 1:250,000 for smaller scales.

 

lidar_results3.JPG

2D and 3D synced views of the contours symbolized with the Contours with Barriers tool in ArcGIS Pro. 

At this point, the sky’s the limit for you to conduct any other type of raster-based analysis, or combine it with vector analysis.  For example, you could run the profile tool to generate a profile graph of a drawn line (as I did, shown below) or an imported shapefile or line feature class, create a viewshed from your specified point(s), trace downstream from specific points, determine which areas in your study site have slopes over a certain degree, or use the Lidar and derived products in conjunction with vector layers to determine the optimal site for a wildfire observation tower or cache for firefighters.

Profile graph of the cyan polyline that I created from the Lidar data from the National Map in ArcGIS Pro.

lidartrace

Tracing downstream using the rasters derived from the lidar data in ArcGIS Pro.

lidar_over40

Slopes over 40 degrees using the slope raster derived from the lidar data in ArcGIS Pro.

I hope these procedures will be helpful to you.