Archive

Archive for April, 2018

Need access to thousands of historical aerials and topographic maps at your fingertips? Try www.historicaerials.com

April 30, 2018 2 comments

Imagine having instant access to thousands upon thousands of historic aerial photographs and topographic maps to be able to examine change over time.  Thanks to a resource called Historic Aerials, you do have this wealth of information at your fingertips.  These aerials and maps, which go back 50, 60, and even 70 years or more, can be used for research, for instruction, for planning, and for other purposes.  Being in the field of geography and GIS education, I can think of many disciplines in which this can be used — urban and rural geography most certainly, but also biology, environmental science, city planning, history, agriculture, and also in GIS courses.  These resources foster spatial thinking about changes in time and space, from natural causes, such as volcanic eruptions or changes in river meanders, or from human causes, such as urbanization or the construction of reservoirs.  And given the connection that often exists between human and natural changes, sometimes these causes are intertwined–the construction of jetties along barrier islands often influences the naturally occurring longshore sediment transport and the migration of the islands themselves, as can be seen by comparing the historic to modern aerials of Ocean City, Maryland, for example.

The interface for the Historic Aerials is intuitive and provides tools that allow the user to compare USGS topographic maps of various years as well as the aerials themselves.  In the example below, I compare a 1958 with a 2009 aerial of a section of Grand Junction Colorado, before and after Interstate 70 and some surrounding housing was constructed.  You can also use the spotlight tool to “see back in time” for wherever you pan your cursor, and you can turn on the streets to see where  streets would one day be constructed on top of historical imagery.

The site comes from the Nationwide Environmental Title Research group, which has spent over 20 years collecting the worlds largest database of historical aerial images and topographical maps of the USA.  Their sources include USGS and USDA imagery, several private collections, and they are continually acquiring more. All the imagery they collect is orthorectified to provide the data in a searchable and precise geo-locatable format.  A print or digital image (for GIS users, GeoTIFF will be especially useful, delivered in lat-long coordinates, but JPG and PNG are also offered) of any of the maps or aerials is available.  In addition, a subscription service allows anyone to access the site with the following advantages: Full screen viewer, no advertising, PDF builder, quick JPEG downloads, and multiple user accounts.   I had a pleasant chat with the good people behind the site and found that their prices are quite reasonable.  Their FAQ, forum, and tutorials make it clear that they are committed to user success with these resources, and there are human beings behind this site to help as well.

Quite frankly, during my years working at the USGS, I always dreamed that my agency would create something like this.  Kudos to the HistoricAerials staff for making this a reality!

historic_aerials

The interface for HistoricAerials.com is quite intuitive and allows for fascinating investigations back in time for lands across the USA. 

Advertisements

General Data Protection Regulation comes to the EU … and it covers location.

April 23, 2018 1 comment

Legislation protection typically lags behind technological innovation until a clearer picture of how a particular technology is used and abused emerges. The issues associated with personal location data privacy and the often unintentional disclosure of personal and sensitive information, are subjects we have discussed many times before on Spatial Reserves (Always On: The analysts are watchingPrivacy concerns from fitness maps and apps).  However, things are set to change in the European Union (EU) with the introduction of new legislation covering how personal location information is collected, used and stored.

On 25th May 2018, General Data Protection Regulation (GDPR) will come into effect in the EU, replacing an earlier data privacy directive introduced in 1995. Aimed at protecting the personal data of all EU citizens, GDPR establishes new safeguards to minimise data breaches and misuse. It introduces a number of Data Subject Rights including right of access, right to be forgotten and privacy by design. Although such regulatory initiatives are often associated with the data collecting activities of larger companies and organisations and the type of data they collect, such as those in the financial and marketing sectors, GDPR extends the definition of personal data in Article 4 (1) to include:

‘… any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person

This legislative acknowledgement recognises location as an equally important component of personal information and an indicator of what GDPR defines as sensitive traits (political affiliation, religious beliefs and so on). Companies, organisations and institutions in the EU will be required to ask permission to use personal location information, be transparent about the use of this information and delete it if requested to do so.

 

 

Using the Data Interoperability Extension to import SDTS DLG files into ArcGIS Pro

April 16, 2018 Leave a comment

One of the themes of this blog and our book has been the wide variety of spatial data formats in existence.  Some of these spatial data formats have remained challenging to import into a GIS right up to the present day.  To meet this challenge, Esri’s Data Interoperability Extension has been a longstanding and useful set of tools that enables a wide variety of spatial data formats to be imported for use in a GIS.  It is an integrated spatial ETL (Extract, Transform, and Load) toolset that runs within the geoprocessing framework using Safe Software’s FME technology. It enables you to integrate data from multiple sources and formats, use that data with geoprocessing tools, and even publish it with ArcGIS Server.

I recently tested the Data Interoperability Extension in ArcGIS Pro and was thrilled with the results.  Read about how to install and authorize the extension here.  The extension does many things, but one that is particularly useful is that the extension creates a toolbox directly in ArcGIS Pro (graphic below).  I used this toolbox’s Quick Import tool to import a SDTS Format DLG (USGS Digital Line Graph) file directly to a file geodatabase.  The tool, like other ArcGIS Pro geoprocessing tools, walked me right through the process:  Data Interoperability > Quick Import > I then pointed to my DLG files in SDTS format > I named the resulting gdb (file geodatabase).  Once imported, I was then able to work with my hydrography, hypsography, roads, boundaries, and other data.

DLG files have existed since the early 1990s.  Why are we still working with them?  The reasons are that (1) They are dated but still useful vector data sets; (2) Many geospatial data portals still host data only in this format, such as the USGS Earth Explorer.  Another way to import these DLG files into ArcGIS Pro or ArcMap is to use the DLG2SHP tools that I wrote about in this set of guidelines using a standalone program.  See below for step-by-step instructions with the Data Interoperability Extension with screen shots.

data_interoperability

1. Use Toolboxes > Data Interoperability Tools > Quick Import, as shown above.data_interoperability_use_for_dlg_screen1

2.  Using QuickImport pulls up a “specify data source” dialog box, as shown above.

data_interoperability_use_for_dlg_screen3

3.  In the specify data source dialog box, use “find other source” and then specify SDTS format.

data_interoperability_use_for_dlg_screen2

4.  Selecting SDTS format.

data_interoperability_use_for_dlg_screen4

5.  Pointing to the SDTS file (after it has been unzipped and un-TAR’d) and saving it into a geodatabase.

data_interoperability_use_for_dlg_screen5

6. Once the file has been imported into a geodatabase, it can be added to a new map in ArcGIS Pro.  The data is now ready for use, as shown for this hydrography example, above. 

 

A Data Converter for DLG Vector GIS Files

April 2, 2018 2 comments

My colleague Dr Bruce Ralston, a geography and GIS professor from the University of Tennessee, wrote a very useful program some years back to convert SDTS format (Spatial Data Transfer Standard) DLG (Digital Line Graph) vector files from the USGS into shapefiles.   With Dr Ralston’s permission, I have placed the program on the following site as a zip file:  https://esri.box.com/v/sdtsdlg2shp.  The reasons why this program and this format are useful touches on a key theme of this blog and our book:  Some data formats remain cumbersome (to put it mildly) to use.  SDTS is one of those formats.  Conversion programs like this one enable data in SDTS format to be read directly by a GIS program, such as ArcGIS.  Futhermore, the site that these programs were archived on is now blocked for non university users.  Changing sites and changing access is another theme of this blog!

To use, access the link above and download the file to your local (Windows) device.  Unzip the file, which will result in the following files:  setupdlg.exe, entity.zip, entity.dbf, and dlgmanual3.pdf.   The PDF is the well-written manual from Dr Ralston.  To start, access the setup program, setupdlg.exe.  This will install the program DLG2SHP, which is a Windows application for converting USGS Digital Line Graphs-3 (DLG3) in Optional Format or SDTS Format to Esri shapefiles.  For more about SDTS, read the Library of Congress information here and at the USGS, here.  This was a format widely promoted in the late 1990s and early 2000s, but was not widely adopted.  However, web sites such as The USGS National Map continue to serve very useful and detailed data in this format.  Fortunately, once the DLG files are downloadded, DLG2SHP makes converting these files easy.

The program automatically decompresses zipped DLG files, can batch process them, and allows the user to specify the output folder.  Hypsography shape features are automatically assigned elevations (lines and points).  There is no need for joining an elevation table to the shapes—DLG2SHP does this automatically.  In addition, the program performs complete SDTS Format Attribute Coding.  The SDTS format DLGs contain many tables. DLG2SHP converts these tables to dbf files with key fields for easy joining and linking to the geographic entities to which they refer. The shapes and attribute tables have descriptive names for ease in linking.

The program works with point, line, and polygon layers for a particular feature type.
The major feature types that DLG3 files cover are:   Hypsography, Hydrography, Vegetative Surface Cover, Non-vegetative Features, Boundaries, Survey Control and Markers, Manmade Features, the US Public Land Survey System, and Transportation.  In SDTS format, all transportation features are in a single DLG. In Optional format, they are broken into 3 groups: Roads and Trails, Railroads and Pipelines, Transmission Lines, and Miscellaneous Transportation Features.

For each type of feature in a DLG, the program will generate shapes for points,
lines, and areas. Each of these topological structures contains certain basic attributes.  For nodes, the basic attribute is the node ID. For Lines, the basic attributes are the line ID, the from node ID, the to node ID, the polygon on the left of the line, the polygon on the right of the line, and the length of the line.  For Areas, the basic attribute is the area id. For Optional Format DLGs, the X coordinate of the polygon centroid, and the Y coordinate of the polygon centroid also are included. For SDTS format DLGs, the centroids are stored as a separate point shape.  More attributes are stored in tables that can be joined or linked to each map layer.

One of our exercises makes use of these types of files for wildfire analysis.  The program looks like the graphic below when it is accessed.  For more details, see the SDTS2DLG manual.

 

sdtsdlg2shp

Running the SDTS2DLG program.

dlgroads

Digital Line Graph (DLG) road and hydro files for a single 1:24,000-scale area.  The DLG2SHP program enables the DLG files to be used inside a GIS, such as ArcGIS Pro, as shown here.  The lines marked by the arrows are cartographic neatlines that mark the edge of the area covered by the 1:24,000 cell.  They are not features, so they need to be selected by an attribute query and filtered out of any subsequent analysis.