Archive

Posts Tagged ‘USGS’

Accessing USGS Topographic Maps on The Internet Archive – Archive.org

September 30, 2018 1 comment

For years, I have used the Internet Archive (https://archive.org/about/) for many things, from archiving multimedia that I created for my story maps to looking up information on historical web pages through their Wayback Machine, (as well as for listening to some old wonderful sound recordings) and through those efforts became aware of the wealth of information on the site.   And when I say wealth, I truly mean enormous – 279 billion web pages, 11 million books and texts, 4 million audio recordings (including 160,000 live concerts), 3 million videos (including 1 million Television News programs), 1 million images, and100,000 software programs. But did you know that The Internet Archive also houses some geospatial data?  The Internet Archive, a 501(c)(3) non-profit organization that has existed since 1996, states that its mission is to “provide Universal Access to All Knowledge,” so it makes sense that some geospatial data for the public good is there.

Let’s focus here on the USGS topographic map data on The Internet Archive, also known as Digital Raster Graphics (DRGs).  Start here for a list of these maps by state, and then underneath each state, a variety of search options are available.  It isn’t the most intuitive unless you know the specific map name that you are looking for, so a topographic map index may still come in handy; a scanned version of these is not easy to come by, but one such archive is here.  Formats include GeoTIFF, essential for use in a GIS.

archive-org.JPG

Interface on The Internet Archive for USGS Digital Raster Graphics. 

While I still find the interface on the other main DRG archive, LibreMap, to be a bit easier to use, LibreMap is not maintained any longer, and is starting to return some errors during certain searches.  The Esri USGS Historical Map Explorer, and the USGS TopoView, reviewed here, is more modern approach to obtaining topographic maps, with the added benefit of historical editions.  USGS topographic maps are part of the set of basemaps available inside ArcGIS Online as data services, which is increasingly part of modern GIS workflows, rather than downloading the data and using it locally.  Still another archive is that from Historical Aerials, which I reviewed here. 

orleansin.JPG

A section of my all-time favorite USGS topographic map, for Mitchell Indiana, simply because of the intricacies of the depression contours and disappearing streams in the magnificent karst landscape depicted. 

Advertisements

A Data Converter for DLG Vector GIS Files

April 2, 2018 2 comments

My colleague Dr Bruce Ralston, a geography and GIS professor from the University of Tennessee, wrote a very useful program some years back to convert SDTS format (Spatial Data Transfer Standard) DLG (Digital Line Graph) vector files from the USGS into shapefiles.   With Dr Ralston’s permission, I have placed the program on the following site as a zip file:  https://esri.box.com/v/sdtsdlg2shp.  The reasons why this program and this format are useful touches on a key theme of this blog and our book:  Some data formats remain cumbersome (to put it mildly) to use.  SDTS is one of those formats.  Conversion programs like this one enable data in SDTS format to be read directly by a GIS program, such as ArcGIS.  Futhermore, the site that these programs were archived on is now blocked for non university users.  Changing sites and changing access is another theme of this blog!

To use, access the link above and download the file to your local (Windows) device.  Unzip the file, which will result in the following files:  setupdlg.exe, entity.zip, entity.dbf, and dlgmanual3.pdf.   The PDF is the well-written manual from Dr Ralston.  To start, access the setup program, setupdlg.exe.  This will install the program DLG2SHP, which is a Windows application for converting USGS Digital Line Graphs-3 (DLG3) in Optional Format or SDTS Format to Esri shapefiles.  For more about SDTS, read the Library of Congress information here and at the USGS, here.  This was a format widely promoted in the late 1990s and early 2000s, but was not widely adopted.  However, web sites such as The USGS National Map continue to serve very useful and detailed data in this format.  Fortunately, once the DLG files are downloadded, DLG2SHP makes converting these files easy.

The program automatically decompresses zipped DLG files, can batch process them, and allows the user to specify the output folder.  Hypsography shape features are automatically assigned elevations (lines and points).  There is no need for joining an elevation table to the shapes—DLG2SHP does this automatically.  In addition, the program performs complete SDTS Format Attribute Coding.  The SDTS format DLGs contain many tables. DLG2SHP converts these tables to dbf files with key fields for easy joining and linking to the geographic entities to which they refer. The shapes and attribute tables have descriptive names for ease in linking.

The program works with point, line, and polygon layers for a particular feature type.
The major feature types that DLG3 files cover are:   Hypsography, Hydrography, Vegetative Surface Cover, Non-vegetative Features, Boundaries, Survey Control and Markers, Manmade Features, the US Public Land Survey System, and Transportation.  In SDTS format, all transportation features are in a single DLG. In Optional format, they are broken into 3 groups: Roads and Trails, Railroads and Pipelines, Transmission Lines, and Miscellaneous Transportation Features.

For each type of feature in a DLG, the program will generate shapes for points,
lines, and areas. Each of these topological structures contains certain basic attributes.  For nodes, the basic attribute is the node ID. For Lines, the basic attributes are the line ID, the from node ID, the to node ID, the polygon on the left of the line, the polygon on the right of the line, and the length of the line.  For Areas, the basic attribute is the area id. For Optional Format DLGs, the X coordinate of the polygon centroid, and the Y coordinate of the polygon centroid also are included. For SDTS format DLGs, the centroids are stored as a separate point shape.  More attributes are stored in tables that can be joined or linked to each map layer.

One of our exercises makes use of these types of files for wildfire analysis.  The program looks like the graphic below when it is accessed.  For more details, see the SDTS2DLG manual.

 

sdtsdlg2shp

Running the SDTS2DLG program.

dlgroads

Digital Line Graph (DLG) road and hydro files for a single 1:24,000-scale area.  The DLG2SHP program enables the DLG files to be used inside a GIS, such as ArcGIS Pro, as shown here.  The lines marked by the arrows are cartographic neatlines that mark the edge of the area covered by the 1:24,000 cell.  They are not features, so they need to be selected by an attribute query and filtered out of any subsequent analysis.

 

 

 

Accessing and Using Lidar Data from The National Map

January 8, 2018 1 comment

We have written about the USGS data portal NationalMap numerous times in this blog and in our book, but since the site keeps getting enhanced, a re-examination of the site is warranted.  One of the enhancements over the past few years is the addition of Lidar data to the site.  I did some recent testing of searching for and downloading Lidar data on the site and wanted to report on my findings.  For videos of some of these procedures, go to the YouTube Channel geographyuberalles and search on Lidar.

From a user perspective, in my view the site is still a bit challenging, where the user encounters moments in the access and download process where it is not clear how to proceed.  However, (1) the site is slowly improving; (2) the site is worth investigating chiefly because of its wealth of data holdings:  It is simply too rich of a resource to ignore.  One challenging thing about using NationalMap is, like many other data portals, how to effectively narrow the search from the thousands of search results.  This in part reflects the open data movement that we have been writing about, so it is a good problem to have, albeit still cumbersome in this portal.  Here are the procedures to access and download the Lidar data from the site:

  1. To begin:  Visit the National Map:  https://nationalmap.gov/ > Select “Elevation” from this page.
  2. Select “Get Elevation Data” from the bottom of the Elevation page.  This is one of several quirks about the site – why isn’t this link in a more prominent position or in a bolder font?
  3. From the Data Elevation Products page left hand column:   Select “1 meter DEM.”
  4. Select the desired format.  Select “Show Availability”.   Zoom to the desired area using a variety of tools to do so.  In my example, I was interested in Lidar data for Grand Junction, in western Colorado.
  5. Note that the list of  available products will appear in the left hand column.  Lidar is provided in 10000 x 10000 meter tiles.  In my example, 108 products exist for the Grand Junction Lidar dataset.  Use “Footprint” to help you identify areas in which you need data–the footprints appear as helpful polygon outlines.  At this point, you could save your results as text or CSV, which I found to be quite handy.
  6. You can select the tiles needed one by one to add to your cart or select “Page” to select all items.  Select the Cart where you can download the tiles manually or select the “uGet Instructions” for details about downloading multiple files.  Your data will be delivered in a zip format right away, though Lidar files are large and may require some time to download.

 

lidar_results.JPG

The National Map interface as it appeared when I was selecting my desired area for Lidar data.

Unzip the LAS data for use in your chosen GIS package.  To bring the data into ArcGIS Pro, create a new blank project and name it.  Then, Go to Analysis > Tools > Create LAS dataset from your unzipped .las file, noting the projection (in this case, UTM) and other metadata.  Sometimes you can bring .las files directly into Pro without creating a LAS dataset, but with this NationalMap Lidar data, I found that I needed to create a LAS dataset first.

Then > Insert:  New Map > add your LAS dataset to the new map. Zoom in to see the lidar points.  View your Lidar data in different ways using the Appearance tab to see it as elevation, slope, aspect (shown below), and contours.  Use LAS dataset to raster to convert the Lidar data to a raster.  In a similar way, I added the World Hydro layer so I could see the watersheds in this area, and USA detailed streams for the rivers.

lidar_results2.JPG

Aspect view generated from Lidar data in ArcGIS Pro.

There are many things you can do with your newly downloaded Lidar data:  Let’s explore just a few of those.  First, create a Digital Elevation Model (DEM) and a Digital Surface Model (DSM).  To do this, in your .lasd LAS dataset > LAS Filters > Filter to ground, and visualize the results, and then use LAS Dataset to Raster, using the Elevation as the value field.  Your resulting raster is your digital elevation model (DEM).  Next, Filter to first return, and then convert this to a raster:  This is your digital surface model (DSM).  After clicking on sections of each raster to compare them visually, go one step further and use the Raster Calculator to create a comparison raster:  Use the formula:  1streturn_raster – (subtract) the ground_raster.  The first return result is essentially showing the objects or features on the surface of the Earth–the difference between “bare earth” elevation and the “first return”–in other words, the buildings, trees, shrubs, and other things human built and natural.  Symbolize and classify this comparison surface to more fully understand your vegetation and structures.  In my study area, the difference between the DEM and the DSM was much more pronounced on the north (northeast, actually) facing slope, which is where the pinon and juniper trees are growing, as opposed to the barren south (southwest) facing slope which is underlain by Mancos Shale (shown below).

lidar_veght

Comparison of DEM and DSM as a “ground cover” raster in ArcGIS Pro.

My photograph of the ridgeline, from just east of the study area, looking northwest.  Note the pinon and juniper ground cover on the northeast-facing slopes as opposed to the barren southwest facing slope.

Next, create a Hillshade from your ground raster (DEM) using the hillshade tool.   Next, create a slope map and an aspect map using tools of these respective names.  The easiest way to find the tools is just to perform a search.  The hillshade, slope, and aspect are all raster files.  Once the tools are run, these are now saved as datasets inside your geodatabase as opposed to earlier—when you were simply visualizing your Lidar data as slope and aspect, you were not making separate data files.

Next, create contours, a vector file, from your ground raster (DEM), using the create contours tool.  Change the basemap to imagery to visualize the contours against a satellite image.  To create index contours, use the Contour with Barriers tool.  To do this, do not actually indicate a “barriers” layer but rather use the contour with barrier tool to achieve an “index” contour, as I did, shown below.  I used 5 for the contour interval and 25 (every fifth contour) for the index contour interval.  This results in a polyline feature class with a field called “type”.  This field receives the value of 2 for the index contours and 1 for all other contours.  Now, simply symbolize the lines as unique value on the type field, specifying a thicker line for the index contours (type 2) and a thinner line for all the other contours.

lidar_results4

Next, convert your 2D map to a 3D scene using the Catalog pane.  If you wish, undock the 3D scene and drag it to the right side of your 2D map so that your 2D map and 3D scene are side by side.  Use View > Link Views to synchronize the two.  Experiment with changing the base map to topographic or terrain with labels.  Or, if your area is in the USA like mine is, use the Add Data > USA topographic > add the USGS topographic maps as another layer.  The topographic maps are at 1:24,000 scale in the most detailed view, and then 1:100,000 and 1:250,000 for smaller scales.

 

lidar_results3.JPG

2D and 3D synced views of the contours symbolized with the Contours with Barriers tool in ArcGIS Pro. 

At this point, the sky’s the limit for you to conduct any other type of raster-based analysis, or combine it with vector analysis.  For example, you could run the profile tool to generate a profile graph of a drawn line (as I did, shown below) or an imported shapefile or line feature class, create a viewshed from your specified point(s), trace downstream from specific points, determine which areas in your study site have slopes over a certain degree, or use the Lidar and derived products in conjunction with vector layers to determine the optimal site for a wildfire observation tower or cache for firefighters.

Profile graph of the cyan polyline that I created from the Lidar data from the National Map in ArcGIS Pro.

lidartrace

Tracing downstream using the rasters derived from the lidar data in ArcGIS Pro.

lidar_over40

Slopes over 40 degrees using the slope raster derived from the lidar data in ArcGIS Pro.

I hope these procedures will be helpful to you.

 

 

 

A Review of the Gap Analysis Program’s Protected Areas Data Portal

March 19, 2017 Leave a comment

Today’s guest blog essay comes from Linda Zellmer, Government Information & Data Services Librarian, Western Illinois University.  Linda can be contacted at LR-Zellmer @ wiu.edu.

Several years ago, I worked with a class in our Recreation, Parks and Tourism Administration department. The students in the class were getting their first exposure to GIS, and used it to analyze the populations served by a park to develop a plan for managing and expanding its services. At the time, students had to obtain geospatial data on park locations and boundaries from local or state government agencies or download Federal lands data from the National Atlas of the United States. Then they combined the park boundary data with data from the Census Bureau to learn about the population characteristics of the people in the area. Finally, they visited the park of interest to get information on park usage and amenities. A new data set, the Protected Areas Database of the United States (PAD-US) will make this class and related research much easier, because it provides data on all types of protected areas for either the entire United States, a U.S. Region, by landscape region, or by US State or Territory.  PAD-US data is available for downloading, viewing and as a web map service from the PAD-US website.

The PAD-US data was developed as part of the Gap Analysis Program of the U.S. Geological Survey. The Gap program collects data on land cover, species distribution and stewardship to determine whether a given species’ habitat is protected, so that plans for further protection (if needed) can be developed. According to the PAD-US Standards and Methods Manual for Data Stewards, the data set contains geospatial data on “marine and terrestrial protected areas” that are “dedicated to the preservation of biological diversity and to other natural, recreation and cultural uses.” The data set contains geospatial data showing the extent and location of Federal, State, Local and private lands set aside for recreation and conservation. It also provides information on the owner name and type, whether the site is publicly accessible, and information on whether the site is being managed for conservation.

 

padus_usgs

The Gap Analysis Program’s Protected Areas of the US Data Portal.

USGS National Elevation Dataset (NED) moving to Lidar-based elevation model

February 8, 2015 1 comment

The USGS National Elevation Dataset (NED) is transitioning to a Lidar-based elevation model. This transition is part of the 3D Elevation Program (3DEP) initiative, whose goal is to systematically collect enhanced elevation data in the form of Lidar data over the conterminous United States, Hawaii, and the U.S. territories, with data acquired over an 8-year period. Interferometric synthetic aperture radar (IFSAR) data will be collected over Alaska, where cloud cover and remote locations preclude the use of Lidar over much of the state (yes, physical geography still matters!).

This initiative was born in response to a study funded by the USGS named “The National Enhanced Elevation Assessment.”  The study documented business uses for elevation needs across 34 federal agencies, agencies from all 50 States, selected local government and Tribal offices, and private and not-for profit organizations. Each need was characterized by the following:

  • Data accuracy.
  • A refresh cycle for the data.
  • Coverage for geographic areas of interest.

Conservative annual benefits for flood risk management alone are $295 million; for infrastructure and construction management, $206 million; and for natural resources conservation, $159 million.  Results are detailed in the Dewberry report on the National Enhanced Elevation Assessment, which details costs and benefits, how the data will be collected, standards and specifications, and organizations involved in the effort.  An additional report details how the data could help in terms of taking action for climate change.

How will this affect us in the geospatial data community?  The NED activities and website will continue until a full transition to 3DEP is completed. 3DEP planning and research is underway at the USGS to transition to a unified service that will provide both gridded bare earth data products and point cloud data, along with capabilities to produce other derived elevation surfaces and products from 3D data.  When the data does appear, data users should notice the difference in resolution and quality.  In our book, we detailed the rise of Lidar data, and since its publication, these data sets have greatly expanded in quality and availability.

High-resolution lidar image of Mount St. Helens, Washington

High-resolution lidar image of Mount St. Helens, Washington.

Categories: Public Domain Data Tags: , , ,

European Atlas of the Seas

June 2, 2014 1 comment

The European Atlas of the Seas, launched in 2011, provides open access to a variety of global and European maritime and geographical datasets covering eight main themes:

  • Geography
  • Nature – bathing water quality, protected areas
  • Tourism – museums, aquariums
  • Security and safety – major oil spills, accident density
  • People and employment – coastal population, employment in the fishing industry
  • Transport and energy – shipping for goods and passenger transport.
  • Governance and European policies – fisheries local action groups (FLAGs), regional advisory councils (RACs)
  • Fisheries and aquaculture – fishing quotas, state of fish stocks, fish farms

EU_AtlasOfSeas

The Atlas is continually updated with revised and additional datasets provided by the contributing departments, agencies and international organisations including UNESCO, FAO, USDA FSA, USGS, NOAA, Esri and IHQ. Some of the datasets are available to download in shapefile and KML format, and the accompanying metadata provide details on the data sources referenced.

Licensing and the public domain

April 21, 2014 1 comment

A central theme in the GIS Guide to Public Domain Data is data licensing; the emergence of licensing frameworks for spatial data, the types of licenses that are available for data producers and users, and what is means to place data in the public domain. Despite much attention there is as yet no universally accepted definition of the term ‘public domain’. A number of organisations have posted their own interpretations, including:

US Copyright Office: The public domain is not a place. A work of authorship is in the ‘public domain’ if it is no longer under copyright protection or if it failed to meet the requirements for copyright protection. Works in the public domain may be used freely without the permission of the former copyright owner.

UK’s Intellectual Property Office: The body of works not or no longer protected by Intellectual Property rights which are available for the public to use without seeking permission or paying royalties.

Creative CommonsWhen a work is in the public domain, it is free for use by anyone for any purpose without restriction under copyright law. Public domain is the purest form of open/free, since no one owns or controls the material in any way.

Common to all these definitions are the freedom from royalty payments and the absence of intellectual property rights and copyright restrictions on the use and reuse of the data. During the recent State of the Map US conference in Washington DC, some of the lingering issues regarding data licensing for spatial data were raised again. In his presentation on OpenStreetMap (OSM) Alex Barth of Mapbox discussed some of the current licensing challenges facing the current and future use of OSM data.

OSM data is open data licensed under the Open Data Commons Open Database License (ODbL), and the cartography in the map tiles and the documentation are licensed under a Creative Commons Attribution-ShareAlike 2.0 license (CC BY-SA). Common to both licensing frameworks is the share-alike clause that means any OSM data that is updated and improved, or third party data remixed with OSM data, must be shared under the same licensing terms.

For some organisations integrating OSM data with their own private data, or organisations who are mandated to make their data available in the public domain (for example the US Geological Survey), wider use of this data resource is not an option and the benefits of crowd-sourced, free and open datasets like OSM will never be fully realised. For many observers, the only sensible long-term option is dropping the share-alike clause from the OSM licensing arrangements. For others, designation as a public domain data set is the solution. It remains to be seen which licensing path the OpenStreetMap community will choose.