Archive

Posts Tagged ‘vector’

Using the Data Interoperability Extension to import SDTS DLG files into ArcGIS Pro

April 16, 2018 Leave a comment

One of the themes of this blog and our book has been the wide variety of spatial data formats in existence.  Some of these spatial data formats have remained challenging to import into a GIS right up to the present day.  To meet this challenge, Esri’s Data Interoperability Extension has been a longstanding and useful set of tools that enables a wide variety of spatial data formats to be imported for use in a GIS.  It is an integrated spatial ETL (Extract, Transform, and Load) toolset that runs within the geoprocessing framework using Safe Software’s FME technology. It enables you to integrate data from multiple sources and formats, use that data with geoprocessing tools, and even publish it with ArcGIS Server.

I recently tested the Data Interoperability Extension in ArcGIS Pro and was thrilled with the results.  Read about how to install and authorize the extension here.  The extension does many things, but one that is particularly useful is that the extension creates a toolbox directly in ArcGIS Pro (graphic below).  I used this toolbox’s Quick Import tool to import a SDTS Format DLG (USGS Digital Line Graph) file directly to a file geodatabase.  The tool, like other ArcGIS Pro geoprocessing tools, walked me right through the process:  Data Interoperability > Quick Import > I then pointed to my DLG files in SDTS format > I named the resulting gdb (file geodatabase).  Once imported, I was then able to work with my hydrography, hypsography, roads, boundaries, and other data.

DLG files have existed since the early 1990s.  Why are we still working with them?  The reasons are that (1) They are dated but still useful vector data sets; (2) Many geospatial data portals still host data only in this format, such as the USGS Earth Explorer.  Another way to import these DLG files into ArcGIS Pro or ArcMap is to use the DLG2SHP tools that I wrote about in this set of guidelines using a standalone program.  See below for step-by-step instructions with the Data Interoperability Extension with screen shots.

data_interoperability

1. Use Toolboxes > Data Interoperability Tools > Quick Import, as shown above.data_interoperability_use_for_dlg_screen1

2.  Using QuickImport pulls up a “specify data source” dialog box, as shown above.

data_interoperability_use_for_dlg_screen3

3.  In the specify data source dialog box, use “find other source” and then specify SDTS format.

data_interoperability_use_for_dlg_screen2

4.  Selecting SDTS format.

data_interoperability_use_for_dlg_screen4

5.  Pointing to the SDTS file (after it has been unzipped and un-TAR’d) and saving it into a geodatabase.

data_interoperability_use_for_dlg_screen5

6. Once the file has been imported into a geodatabase, it can be added to a new map in ArcGIS Pro.  The data is now ready for use, as shown for this hydrography example, above. 

 

Advertisements

A Data Converter for DLG Vector GIS Files

April 2, 2018 2 comments

My colleague Dr Bruce Ralston, a geography and GIS professor from the University of Tennessee, wrote a very useful program some years back to convert SDTS format (Spatial Data Transfer Standard) DLG (Digital Line Graph) vector files from the USGS into shapefiles.   With Dr Ralston’s permission, I have placed the program on the following site as a zip file:  https://esri.box.com/v/sdtsdlg2shp.  The reasons why this program and this format are useful touches on a key theme of this blog and our book:  Some data formats remain cumbersome (to put it mildly) to use.  SDTS is one of those formats.  Conversion programs like this one enable data in SDTS format to be read directly by a GIS program, such as ArcGIS.  Futhermore, the site that these programs were archived on is now blocked for non university users.  Changing sites and changing access is another theme of this blog!

To use, access the link above and download the file to your local (Windows) device.  Unzip the file, which will result in the following files:  setupdlg.exe, entity.zip, entity.dbf, and dlgmanual3.pdf.   The PDF is the well-written manual from Dr Ralston.  To start, access the setup program, setupdlg.exe.  This will install the program DLG2SHP, which is a Windows application for converting USGS Digital Line Graphs-3 (DLG3) in Optional Format or SDTS Format to Esri shapefiles.  For more about SDTS, read the Library of Congress information here and at the USGS, here.  This was a format widely promoted in the late 1990s and early 2000s, but was not widely adopted.  However, web sites such as The USGS National Map continue to serve very useful and detailed data in this format.  Fortunately, once the DLG files are downloadded, DLG2SHP makes converting these files easy.

The program automatically decompresses zipped DLG files, can batch process them, and allows the user to specify the output folder.  Hypsography shape features are automatically assigned elevations (lines and points).  There is no need for joining an elevation table to the shapes—DLG2SHP does this automatically.  In addition, the program performs complete SDTS Format Attribute Coding.  The SDTS format DLGs contain many tables. DLG2SHP converts these tables to dbf files with key fields for easy joining and linking to the geographic entities to which they refer. The shapes and attribute tables have descriptive names for ease in linking.

The program works with point, line, and polygon layers for a particular feature type.
The major feature types that DLG3 files cover are:   Hypsography, Hydrography, Vegetative Surface Cover, Non-vegetative Features, Boundaries, Survey Control and Markers, Manmade Features, the US Public Land Survey System, and Transportation.  In SDTS format, all transportation features are in a single DLG. In Optional format, they are broken into 3 groups: Roads and Trails, Railroads and Pipelines, Transmission Lines, and Miscellaneous Transportation Features.

For each type of feature in a DLG, the program will generate shapes for points,
lines, and areas. Each of these topological structures contains certain basic attributes.  For nodes, the basic attribute is the node ID. For Lines, the basic attributes are the line ID, the from node ID, the to node ID, the polygon on the left of the line, the polygon on the right of the line, and the length of the line.  For Areas, the basic attribute is the area id. For Optional Format DLGs, the X coordinate of the polygon centroid, and the Y coordinate of the polygon centroid also are included. For SDTS format DLGs, the centroids are stored as a separate point shape.  More attributes are stored in tables that can be joined or linked to each map layer.

One of our exercises makes use of these types of files for wildfire analysis.  The program looks like the graphic below when it is accessed.  For more details, see the SDTS2DLG manual.

 

sdtsdlg2shp

Running the SDTS2DLG program.

dlgroads

Digital Line Graph (DLG) road and hydro files for a single 1:24,000-scale area.  The DLG2SHP program enables the DLG files to be used inside a GIS, such as ArcGIS Pro, as shown here.  The lines marked by the arrows are cartographic neatlines that mark the edge of the area covered by the 1:24,000 cell.  They are not features, so they need to be selected by an attribute query and filtered out of any subsequent analysis.

 

 

 

Maps as representations of reality: The deciduous-coniferous tree “line”

November 5, 2012 3 comments

One of the themes running through our book The GIS Guide to Public Domain Data is that maps are representations of reality.  While almost everyone reading this statement is likely to agree with it, in the fast-paced world that GIS analysis and creating maps has become, it is easy to lose sight of this fact when staring at tables, maps, and imagery.  In a recent video, I discuss just one place where care needs to be made in making decisions based on spatial data.  In the video, observe my surroundings as I stand near the traditional “line” that divides the deciduous forest to the south from the coniferous forest to the north in North America. Is the “line” really a line at all, or is it better described as a gradual change from deciduous to coniferous as one travels north?  Is that vector line then better symbolized as a “zone”, or is vegetation better mapped as a raster data set, with each cell representing the percentage of deciduous and coniferous trees?

How many other data sets do we tend to see as having firm boundaries, when the boundaries are not really firm at all in reality?  How does that affect the decisions we make with them?  Even the boundary between wetlands and open water were originally interpreted based on land cover data or a satellite or aerial image.   As we state in the book, even contour lines were often interpreted originally from aerial stereo pairs.  And each data set was collected at a specific scale, with certain equipment and software, at a specific date, and within certain margins of error that the organization established.  Maps are representations of reality.  They are incredibly useful representations to be sure, but care needs to be taken when using this or any abstracted data.