Archive

Archive for the ‘Public Domain Data’ Category

An update on the World Bank’s Spatial Agent

May 28, 2018 5 comments
It sounds like a modern detective novel, but the Spatial Agent is actually a new, free app from the World Bank that offers one-stop access to interactive maps and charts of national, regional, and global datasets.  Jill Clark reviewed this site on our blog here. As we have written about data sources on this blog for nearly six years, and covered this topic in our book, the phrase “one-stop access” naturally caught my attention.  Could the Spatial Agent truly be all that it claims to be?

 

To find out, I began by watching a webinar that Mr Harshadeep recently conducted, which is as of this writing, still available online, here, after a short registration process.  In the webinar, and after my subsequent investigations, I was amazed at how the Spatial Agent as an app could bring together on-demand thousands of free, public-domain spatial data and analytical services (from in-situ and earth observation sources and also live cloud computing services).  It represents the data from sources such as the UN, NASA, NOAA, ESA, World Bank, many universities, and thousands of other sources, covering themes such as social (poverty, water supply), environmental (land use, biodiversity), economic (GDP, energy), and climate (snow cover, precipitation, for example).

The goal of the Spatial Agent is to offer solutions to many of the development challenges faced across the globe, which are often hampered by the poor availability of spatial data. For example, the app can be used to determine the areas in Madagascar that are susceptible to cyclones, or the areas in India that have high child malnutrition, or discovering the major exports of Vietnam, or determining how fast the population in Lagos is rising.  As these examples show, the Spatial Agent’s data cross boundaries, disciplines, and cover many different scales.  The Spatial Agent is the creation of Nagaraja Harshadeep, the lead environmental specialist and global lead for watersheds at the World Bank.  Mr Harshadeep has decades of experience working with spatial data and the application reflects his knowledge and passion.  There is much more than maps and imagery here, but rich tabular databases and other services, and the metadata for each of the data sets is quite robust.

I have been a long time fan of the spatial data from the World Bank, and use their data in several systems, including many layers available in ArcGIS Online.  The major limitation with the Spatial Agent app at this point that I can see is that it is just that — an app.  Therefore it only works on mobile phones and tablets.  I understand in part why it is focused on these devices–these are what many people are using day to day in their work.  Still, to bring the data sets into a GIS and more fully use them, I would love to see its capabilities inside of a series of user-driven interfaces that could be run in a standard web browser on a computer where I also have GIS and statistics tools available to me.  But I was glad to see this note about this very thing on the project’s site:  “The web version is being developed with the Bank’s Global Reach effort for launch later this year.”  Since the data and documentation are so rich on this site, I look forward to finding out how we will be able to use the services in a GIS.  Even without a GIS, the Spatial Agent is already very useful, because it is helping to bring data-driven decisions to daily decision making.

 

Two views of the hundreds of data layers and statistics available via the Spatial Agent.

For more information, including the links to access the apps, and the tutorials, see this page.

Advertisements

A review of the ArcGIS Pro Cookbook

May 14, 2018 4 comments

GIS Professional Tripp Corbin’s book, the “ArcGIS Pro 2.x Cookbook” (2018, Packt Publishing) is new but I believe will quickly become a valued and oft-used resource. Mr Corbin’s goal in writing this extensive (694 pages) resource is to help GIS professionals “create, manage, and share geographic maps, data, and analytical models using ArcGIS Pro.” The audience for this book includes all who are learning GIS, or learning Pro, as well as those migrating from ArcMap to Pro.

Tripp’s “cookbook” theme is evident throughout the book’s format, where in each section and problem to be solved, he shows how to get ready, how to do it, how it works, and … “there’s more” (additional resources). That the book is from Packt is excellent, because Packt (www.packtpub.com) offers eBook versions of every one of its books, and also offers newsletters and tech articles. That Tripp is a full time trainer and instructor is evident–he understands the challenges in learning a rapidly-changing and complex technology inherent in GIS with just enough tips to keep the reader engaged. He also encourages the reader to think about how to apply each tool and method to his or her own work. He offers the reader the ability to download the sample data for the book, and the data bundle is also on GitHub. He also includes PDFs of all images of screen shots and diagrams.

I like Tripp’s approach because, similar to my own instruction, and central to the theme of this blog, he starts with data. He’s not hesitant to discuss the benefits but also the limitations of each data format such as shp, gdb, and CAD files. He spends quality time in the book helping the reader understand how to convert data to the format that best fits his or her needs. His sections on linking tables from outside sources to existing data, on editing (in particular, a focus on topologies to improve data accuracy and increasing editing efficiency), and on 2D and 3D analysis are very helpful. I was pleased to see much attention to what I consider to be a chief advantage of Pro–the ability to more easily share content from Pro to ArcGIS Online and hence the wider community. Another wonderful new function in ArcGIS Pro is also included in the book–writing and using Arcade scripts, applied to symbology, classification, and analysis.

As a GIS book author, I know the challenges faced in writing such a book–what should be included, and what should be left out? Tripp does a nice job here as well, including the fundamentals that most users will touch. The book’s chapters include: 1: Capabilities and terminology. 2: Creating and storing data. 3: Linking data together. 4: Editing spatial and tabular data. 5: Validating and editing data with topologies. 6: Projections and coordinate systems. 7: Converting data from one format to another. 8: Proximity analysis. 9: Spatial statistics and hot spots. 10: 3D maps and 3D analyst. 11: Arcade, labeling and symbology expressions. 12: ArcGIS Online, 13: Publishing your own content to ArcGIS Online. 14: Creating web apps using ArcGIS Online.

These chapters cover a great deal of ground. In the editing chapter, for example (Chapter 4), configuring editing options, reshaping existing, splitting, merging, aligning, creating new point line polygon features, creating new polygon feature using autocomplete, and editing attributes using attribute pane and in the table view, are all examined. The examples in the book are interesting and relevant, and not without some humor (Trippville is a community that is often studied). In my view, the book contains just the right amount of graphics. Tripp provides answers to the questions he poses, and then gives the explanation for each answer. Despite the “recipes” provided in the cookbook, not all of them require the previous recipe to be used, which is excellent for all of us in GIS who have limited time and want to select sections in a non-sequential order.

I highly recommend using this book in conjunction with Tripp’s other book on this topic, “Learning ArcGIS Pro.” The Learning book focuses on installing, assigning licenses, navigating the interface, creating and managing projecrts, creating 2D and 3D maps, authoring map layouts, importing existing projects, creating standardized workflows using tasks, and automating analysis and processes using modelbuilder and python. The Learning ArcGIS Pro book ideally should be used first, before the ArcGIS Pro 2.x Cookbook, but if you are pressed for time, these two books could be used in tandem. Keep both of them handy–they will be very useful to you.

tripp4

Covers of Tripp Corbin’s ArcGIS Pro books. 

tripp3

An example of the detail provided in Tripp Corbin’s ArcGIS Pro 2.x Cookbook. 

Need access to thousands of historical aerials and topographic maps at your fingertips? Try www.historicaerials.com

April 30, 2018 2 comments

Imagine having instant access to thousands upon thousands of historic aerial photographs and topographic maps to be able to examine change over time.  Thanks to a resource called Historic Aerials, you do have this wealth of information at your fingertips.  These aerials and maps, which go back 50, 60, and even 70 years or more, can be used for research, for instruction, for planning, and for other purposes.  Being in the field of geography and GIS education, I can think of many disciplines in which this can be used — urban and rural geography most certainly, but also biology, environmental science, city planning, history, agriculture, and also in GIS courses.  These resources foster spatial thinking about changes in time and space, from natural causes, such as volcanic eruptions or changes in river meanders, or from human causes, such as urbanization or the construction of reservoirs.  And given the connection that often exists between human and natural changes, sometimes these causes are intertwined–the construction of jetties along barrier islands often influences the naturally occurring longshore sediment transport and the migration of the islands themselves, as can be seen by comparing the historic to modern aerials of Ocean City, Maryland, for example.

The interface for the Historic Aerials is intuitive and provides tools that allow the user to compare USGS topographic maps of various years as well as the aerials themselves.  In the example below, I compare a 1958 with a 2009 aerial of a section of Grand Junction Colorado, before and after Interstate 70 and some surrounding housing was constructed.  You can also use the spotlight tool to “see back in time” for wherever you pan your cursor, and you can turn on the streets to see where  streets would one day be constructed on top of historical imagery.

The site comes from the Nationwide Environmental Title Research group, which has spent over 20 years collecting the worlds largest database of historical aerial images and topographical maps of the USA.  Their sources include USGS and USDA imagery, several private collections, and they are continually acquiring more. All the imagery they collect is orthorectified to provide the data in a searchable and precise geo-locatable format.  A print or digital image (for GIS users, GeoTIFF will be especially useful, delivered in lat-long coordinates, but JPG and PNG are also offered) of any of the maps or aerials is available.  In addition, a subscription service allows anyone to access the site with the following advantages: Full screen viewer, no advertising, PDF builder, quick JPEG downloads, and multiple user accounts.   I had a pleasant chat with the good people behind the site and found that their prices are quite reasonable.  Their FAQ, forum, and tutorials make it clear that they are committed to user success with these resources, and there are human beings behind this site to help as well.

Quite frankly, during my years working at the USGS, I always dreamed that my agency would create something like this.  Kudos to the HistoricAerials staff for making this a reality!

historic_aerials

The interface for HistoricAerials.com is quite intuitive and allows for fascinating investigations back in time for lands across the USA. 

General Data Protection Regulation comes to the EU … and it covers location.

April 23, 2018 1 comment

Legislation protection typically lags behind technological innovation until a clearer picture of how a particular technology is used and abused emerges. The issues associated with personal location data privacy and the often unintentional disclosure of personal and sensitive information, are subjects we have discussed many times before on Spatial Reserves (Always On: The analysts are watchingPrivacy concerns from fitness maps and apps).  However, things are set to change in the European Union (EU) with the introduction of new legislation covering how personal location information is collected, used and stored.

On 25th May 2018, General Data Protection Regulation (GDPR) will come into effect in the EU, replacing an earlier data privacy directive introduced in 1995. Aimed at protecting the personal data of all EU citizens, GDPR establishes new safeguards to minimise data breaches and misuse. It introduces a number of Data Subject Rights including right of access, right to be forgotten and privacy by design. Although such regulatory initiatives are often associated with the data collecting activities of larger companies and organisations and the type of data they collect, such as those in the financial and marketing sectors, GDPR extends the definition of personal data in Article 4 (1) to include:

‘… any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person

This legislative acknowledgement recognises location as an equally important component of personal information and an indicator of what GDPR defines as sensitive traits (political affiliation, religious beliefs and so on). Companies, organisations and institutions in the EU will be required to ask permission to use personal location information, be transparent about the use of this information and delete it if requested to do so.

 

 

Using the Data Interoperability Extension to import SDTS DLG files into ArcGIS Pro

April 16, 2018 1 comment

One of the themes of this blog and our book has been the wide variety of spatial data formats in existence.  Some of these spatial data formats have remained challenging to import into a GIS right up to the present day.  To meet this challenge, Esri’s Data Interoperability Extension has been a longstanding and useful set of tools that enables a wide variety of spatial data formats to be imported for use in a GIS.  It is an integrated spatial ETL (Extract, Transform, and Load) toolset that runs within the geoprocessing framework using Safe Software’s FME technology. It enables you to integrate data from multiple sources and formats, use that data with geoprocessing tools, and even publish it with ArcGIS Server.

I recently tested the Data Interoperability Extension in ArcGIS Pro and was thrilled with the results.  Read about how to install and authorize the extension here.  The extension does many things, but one that is particularly useful is that the extension creates a toolbox directly in ArcGIS Pro (graphic below).  I used this toolbox’s Quick Import tool to import a SDTS Format DLG (USGS Digital Line Graph) file directly to a file geodatabase.  The tool, like other ArcGIS Pro geoprocessing tools, walked me right through the process:  Data Interoperability > Quick Import > I then pointed to my DLG files in SDTS format > I named the resulting gdb (file geodatabase).  Once imported, I was then able to work with my hydrography, hypsography, roads, boundaries, and other data.

DLG files have existed since the early 1990s.  Why are we still working with them?  The reasons are that (1) They are dated but still useful vector data sets; (2) Many geospatial data portals still host data only in this format, such as the USGS Earth Explorer.  Another way to import these DLG files into ArcGIS Pro or ArcMap is to use the DLG2SHP tools that I wrote about in this set of guidelines using a standalone program.  See below for step-by-step instructions with the Data Interoperability Extension with screen shots.

data_interoperability

1. Use Toolboxes > Data Interoperability Tools > Quick Import, as shown above.data_interoperability_use_for_dlg_screen1

2.  Using QuickImport pulls up a “specify data source” dialog box, as shown above.

data_interoperability_use_for_dlg_screen3

3.  In the specify data source dialog box, use “find other source” and then specify SDTS format.

data_interoperability_use_for_dlg_screen2

4.  Selecting SDTS format.

data_interoperability_use_for_dlg_screen4

5.  Pointing to the SDTS file (after it has been unzipped and un-TAR’d) and saving it into a geodatabase.

data_interoperability_use_for_dlg_screen5

6. Once the file has been imported into a geodatabase, it can be added to a new map in ArcGIS Pro.  The data is now ready for use, as shown for this hydrography example, above. 

 

A Data Converter for DLG Vector GIS Files

April 2, 2018 2 comments

My colleague Dr Bruce Ralston, a geography and GIS professor from the University of Tennessee, wrote a very useful program some years back to convert SDTS format (Spatial Data Transfer Standard) DLG (Digital Line Graph) vector files from the USGS into shapefiles.   With Dr Ralston’s permission, I have placed the program on the following site as a zip file:  https://esri.box.com/v/sdtsdlg2shp.  The reasons why this program and this format are useful touches on a key theme of this blog and our book:  Some data formats remain cumbersome (to put it mildly) to use.  SDTS is one of those formats.  Conversion programs like this one enable data in SDTS format to be read directly by a GIS program, such as ArcGIS.  Futhermore, the site that these programs were archived on is now blocked for non university users.  Changing sites and changing access is another theme of this blog!

To use, access the link above and download the file to your local (Windows) device.  Unzip the file, which will result in the following files:  setupdlg.exe, entity.zip, entity.dbf, and dlgmanual3.pdf.   The PDF is the well-written manual from Dr Ralston.  To start, access the setup program, setupdlg.exe.  This will install the program DLG2SHP, which is a Windows application for converting USGS Digital Line Graphs-3 (DLG3) in Optional Format or SDTS Format to Esri shapefiles.  For more about SDTS, read the Library of Congress information here and at the USGS, here.  This was a format widely promoted in the late 1990s and early 2000s, but was not widely adopted.  However, web sites such as The USGS National Map continue to serve very useful and detailed data in this format.  Fortunately, once the DLG files are downloadded, DLG2SHP makes converting these files easy.

The program automatically decompresses zipped DLG files, can batch process them, and allows the user to specify the output folder.  Hypsography shape features are automatically assigned elevations (lines and points).  There is no need for joining an elevation table to the shapes—DLG2SHP does this automatically.  In addition, the program performs complete SDTS Format Attribute Coding.  The SDTS format DLGs contain many tables. DLG2SHP converts these tables to dbf files with key fields for easy joining and linking to the geographic entities to which they refer. The shapes and attribute tables have descriptive names for ease in linking.

The program works with point, line, and polygon layers for a particular feature type.
The major feature types that DLG3 files cover are:   Hypsography, Hydrography, Vegetative Surface Cover, Non-vegetative Features, Boundaries, Survey Control and Markers, Manmade Features, the US Public Land Survey System, and Transportation.  In SDTS format, all transportation features are in a single DLG. In Optional format, they are broken into 3 groups: Roads and Trails, Railroads and Pipelines, Transmission Lines, and Miscellaneous Transportation Features.

For each type of feature in a DLG, the program will generate shapes for points,
lines, and areas. Each of these topological structures contains certain basic attributes.  For nodes, the basic attribute is the node ID. For Lines, the basic attributes are the line ID, the from node ID, the to node ID, the polygon on the left of the line, the polygon on the right of the line, and the length of the line.  For Areas, the basic attribute is the area id. For Optional Format DLGs, the X coordinate of the polygon centroid, and the Y coordinate of the polygon centroid also are included. For SDTS format DLGs, the centroids are stored as a separate point shape.  More attributes are stored in tables that can be joined or linked to each map layer.

One of our exercises makes use of these types of files for wildfire analysis.  The program looks like the graphic below when it is accessed.  For more details, see the SDTS2DLG manual.

 

sdtsdlg2shp

Running the SDTS2DLG program.

dlgroads

Digital Line Graph (DLG) road and hydro files for a single 1:24,000-scale area.  The DLG2SHP program enables the DLG files to be used inside a GIS, such as ArcGIS Pro, as shown here.  The lines marked by the arrows are cartographic neatlines that mark the edge of the area covered by the 1:24,000 cell.  They are not features, so they need to be selected by an attribute query and filtered out of any subsequent analysis.

 

 

 

A Review of the Geoportal

March 19, 2018 1 comment

The GEOSS is a portal run by the European Space Agency (ESA) Group and the Group on Earth Observations (GEO) that provides one way to access earth observation data from around the world.   The site focuses on satellite imagery–Sentinel and Landsat data.  One helpful feature about the site is the ability to send search results to social media or via email.  A list of Popular Searches is a good place to start with the site.  The site is definitely worth investigating as it features a wealth of data.  I found myself wishing that there were more predefined searches listed there (currently 4).  The site also offers a login option and the ability to save your “workspace” which is an intriguing idea; using this feature, you could come back to the site and continue searching and downloading with the knowledge of what you have done previously.  There are different formats available, although at many points in my work with the site, I was confused as to how to proceed, or what format my file would be in, and if I was truly downloading the extent shown in the interface.

geoportal2

Interface for the Geoportal.

Like other portals, this one allows search terms, but without knowing what is there, the user is left with some confusion knowing what search terms to use.  I found myself really wanting a tutorial and a list of data sets I could browse through.  Searching is good but the users also need to know what the possibilities are.  I am intrigued by the data offerings on the site but had trouble navigating and discovering resources; I frequently encountered this message below and even had trouble drawing the bounding box for my desired search area.

geoportal1

Given the data holdings that are available via this portal, I think it is worth investigating further.  The about page on Geoportal lists enhancements to the site that are continually being made, and the mission of the page states that one goal is to make the site intuitive and easy to use.  I therefore have high hopes that it will be moving in this direction.  Give it a try!