This BBC article on datums and plate tectonics in Australia raises several themes central to our book and to this blog. The article discusses Australia’s plans to change its latitude and longitude coordinates to address a gap between local coordinates and global ones that are derived from GNSS (Global Navigation Satellite Systems). The Geocentric Datum of Australia was last updated in 1994, and so on 1 January 2017, this datum will be adjusted by 1.8 meters to the north.
And this story is a nice reminder that physical geography matters, as well. Indeed, the whole reason for the adjustment is because the plate that Australia is riding on is moving at an annual rate of 7 cm to the north. The adjustment is therefore an overcorrection so that the local coordinates will be aligned with the global in 2020. It is expected that by that date, a system that can take changes into account can be implemented, eliminating the need for this periodic adjustment and that of the North American Datum change from 1927 to 1983 as described in our book.
With increasing numbers of people using GIS for driverless cars, precision agriculture, laying cable, and other applications that require a great degree of precision and accuracy comes the need to adjust the coordinates. It comes down to our continuing message to be critical of your data. Know the sources of your data, including the datum itself.
Survey station. Photograph by Joseph Kerski.
Geospatial metadata has been described as the language that enables information sharing and discovery. The International Standards Organization (ISO) uses this wording in its “scope” section: “This International Standard provides information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of digital geographic data.” The US Federal Geographic Data Committee (FGDC) defines it as, “A metadata record is a file of information, usually presented as an XML document, which captures the basic characteristics of a data or information resource. It represents the who, what, when, where, why and how of the resource. Geospatial metadata are used to document geographic digital resources such as Geographic Information System (GIS) files, geospatial databases, and earth imagery. A geospatial metadata record includes core library catalog elements such as Title, Abstract, and Publication Data; geographic elements such as Geographic Extent and Projection Information; and database elements such as Attribute Label Definitions and Attribute Domain Values.” In Chapter 5 of our book, we discuss these standards and the importance of metadata in more detail.
ArcGIS desktop software from Esri has for many years supported formats for organizations to attach metadata in a variety of accepted formats. Recently, ArcGIS Online’s support for metadata was enhanced. Metadata in ArcGIS Online has always been created, edited, and viewed in the item details page, but up to now, this has required a manual process, reliant on the person or organization creating the data to input the contents. ArcGIS Online now allows organizations to “enable metadata”, which means that they can include additional standards-based metadata for all item types using a built-in metadata editor. It can be used for web maps, scenes, and web apps. Once enabled, a “metadata” button appears on the detail pages for the owner of that data set and for anyone with access to view the data, they too see a metadata button to view that as well.
Any of the following styles can be chosen: FGDC CSDGM Metadata (Content Standard for Digital Geospatial Metadata), the INSPIRE Metadata Directive, according to ISO 19139 standards, ISO 19139 Metadata Implementation Specification GML3.2, identical to the one above, except the exported files use the GML 3.2 namespace, ISO 19139 Metadata Implementation Specification, allowing to view and edit a complete metadata document that complies with ISO standard 19139, and the North American Profile of ISO 19115 2003.
How can you port metadata from ArcGIS desktop to ArcGIS Online? Standards-based metadata for shapefile are stored in an accompanying metadata XML file where the shapefile is stored. To extract the metadata from a file geodatabase, use the geoprocessing XSLT Transformation tool with the exact copy of. the xslt stylesheet to export the metadata content to a stand-alone metadata XML file in the ArcGIS metadata format.
Support for standard metadata content welcome news, and we expect that metadata from ArcGIS Pro will one day soon be supported as well.
Over the last three years we’ve written many times about the problems associated with using some of the data portals that are available today. After a particularly frustrating week-end some time ago I reached the following conclusion .. ‘I will rarely find exactly what I’m after, working with data portals can be very frustrating, adding and maintaining metadata is a frequently neglected chore, and all too often data are published but not adequately promoted.‘ Although new portals continue to emerge, many fail to provide a comprehensive and user-centric search facility that helps locate the right dataset for the task in hand.
The problems that face those who are challenged to locate data are well documented – data duplication, restrictions on access, insufficiently rigorous search criteria, incomplete or out of date metadata or having to sift though the ubiquitous remnants of various ‘test’ datasets – all issues that continue to defy the stated objectives of many organisations to make the most of their spatial data resources and make them available for others to use.
Enter Voyager Search, a ‘spatially enabled, enterprise search solution‘ and a fresh take on the data portal and spatial data infrastructures (SDIs). In a recent article in CIO Review, Voyager Search was described as the Google of the GIS World and merited inclusion in their top 20 Most Promising GIS Solutions Providers in 2015. Voyager Search offers to index and catalog an organisation’s geospatial datasets, document types, database and web server content and if available, provide information on related content.
In addition, the team at Voyager recognised that simply having metadata doesn’t guarantee data quality (Embrace the Chaos), and argue that metrics such as how many times a data source has been used, where the data have been used, how the data are rated and the existence of associated non-spatial data sources such as Word documents, can provide a useful measure of the relative value of the data and potential fit for a given set of requirements. That seems like a useful approach to adopt.
Nathan Heazlewood, programme manager at the Auckland City Council in New Zealand, has written a wonderful essay about “garbage in, garbage out” in relation to geospatial data. In it, he not only ties this oft-heard phrase to the importance of GIS data quality, but he also details the checks that GIS analysts should go through when they are assessing a data set. This list is also valuable for data producers as they document their own work. And of course, these days, everyone in GIS is a potential data producer.
The list of items (at present, there are 30) is grouped under checks for positional accuracy, topological logic, geometric considerations, projections and coordinate systems, attribute and data structure checks, and attribute and data structure checks. Extremely helpful are Nathan’s diagrams showing tables lacking null values for non-null attribute data, values outside permitted ranges, and orphan records in related tables.
Nathan includes many considerations that are not often discussed but can lead to enormous problems, such as the different standards of date formats being used around the world, from year-month-day to day-month-year to month-day-year (which Nathan dubs the “super dumb American date format”). Another consideration is one I can identify with that gave me GIS problems during the first GIS workshop I taught in Turkey–the numbers in my data were formatted such as 100,000 for one hundred thousand, but the software in the university lab, given its location, was naturally configured for one hundred thousand to be coded as 100.000.
How might you be able to use this data error checklist in your own work? What checks would you add or expand?
Manuel Pascual has written a to-the-point article about GIS data accuracy that provides an excellent quick visual reference on several key topics that we discuss in our book. Its visuals include the differences between accuracy and precision, and sources of topological errors. Also valuable is Manuel’s discussion of GIS data error, including scale, age of data, formatting, qualittative and quantitative errors, positional accuracy, and topological errors. Another short but valuable article on data quality can be found in the GIS Lounge by Ravi Nishesh Srivastava entitled “Spatial Data Quality: An Introduction.”
Stretching way back to the 1990s to a unit in the Core Curriculum in GIScience, I still find the discussion there on data quality measurement and assessment to be quite useful. No matter how the technology changes, the 4 measures of quality are as valid today as when they were writtten: Accuracy, resolution, consistency, and completeness. We have it so easy as GIS users these days, from being able to easily pull in a variety of data sources to our projects from the cloud, to projecting data on the fly, to quickly being able to geocode spreadsheet data, and be able to publish our data for the world with a few mouse clicks. However, perhaps because it is so easy to access a variety of disparate data sets, is more critical than ever before that we pay close attention to these data quality issues.
Yes, it does get hot in Texas, but I’m not sure it gets this hot:
If the heat doesn’t do you in, the rainfall deluge, impossibly high humidity, and the ferocious winds will!
The above errant screen shot was sent to me recently by a GIS colleague. While this website’s data was obviously full of errors at the time of this posting, it touches on a much larger issue: A major theme of this blog and our book is to increase awareness of the data quality issues that are all around us. The troublesome thing is that obvious errors like the one above are few and far between: It is usually far more difficult to determine what is missing, inaccurate either in position or in attributes, or just plain wrong — about data, and in particular, geospatial data.
I believe the reasons why it is difficult to understand and deal with geospatial data quality is in part because we are still so accustomed to viewing maps as “correct” without questioning them, and also in part for another reason: The advent of easy-to-use cloud-based geospatial tools with data at our fingertips makes it so easy to quickly gather data from a myriad of sources. As we gather data, not only do we often pay little attention to the metadata, but also, the providers of these cloud based services may not provide complete metadata. And, some important metadata may not be captured in today’s metadata fields, as we explain in this example about mapping Lyme Disease. On this point, I was encouraged at this year’s Esri User Conference announcement that metadata in ArcGIS Online will soon get a much-needed boost; more on that in a future blog essay.
Therefore, be critical of the data! And not just when the data are obviously wrong. And stay cool!
As we discuss in our book, the advent of web-based GIS presents tremendous advantages in functionality and sharing over the era when the only way to use GIS was on a desktop computer, or before that, on a minicomputer. Yet web-based GIS and its dependency upon online mapping services also presents a challenge when those services move or change. When they do move or change, they usually render the map and the analyses that depend on them to be nonfunctional. Metadata about those services may not include a name, phone number, or email to contact when those services move or change. Compounding the problem is that metadata is sometimes not accessible when the features will not draw in an online GIS environment, so the data user may not have a starting point from which to track down its current location.
A recent example I encountered is when a data layer disappeared from ArcGIS Online, that of the USGS stream gauges. While I needed it for instructional purposes, I reasoned that since this is such an important data set for flood alerts, water quality measures, and other needs, that the layer would soon return or else I would be able to find out the location to which the layer had migrated.
Not so. It took me several weeks to find someone who could tell me where the data service had been moved. But once I was notified of the new service address, I added it to a new web map and I was once again up and running, and so is everyone who depends on this service. The service links to real-time streamflow and water quality information (where available) for thousands of gauging stations across the USA. The service also includes wind conditions, tide stations, and a few other useful layers. I added real time weather radar and real time temperature to this web map, as shown below.
The bottom line is that these web mapping services can and do move, and you as the data user at times are required to take matters into your own hands to track it down. I highly recommend that the GIS community create a better system for dealing with this issue than the labor intensive method I used above. Until that better method arrives, tenacity will remain an important skill in the rapidly changing environment of web based GIS.
- A Review of the Texas Natural Resources Information Systems Data Portal – Geospatial Now on New Exercise Using Open Data Portals from Local Governments
- A Review of the Texas Natural Resources Information Systems Data Portal – Geospatial Now on Sharing Geoprocessing Tools on the Web
- A Review of the Texas Natural Resources Information Systems Data Portal – Geospatial Now on A Review of the Texas Natural Resources Information Systems Data Portal
- josephkerski on A Top 10 List of Useful Geospatial Data Portals
- Andrew Turner on A Top 10 List of Useful Geospatial Data Portals
- February 2017
- January 2017
- December 2016
- November 2016
- October 2016
- September 2016
- August 2016
- July 2016
- June 2016
- May 2016
- April 2016
- March 2016
- February 2016
- January 2016
- December 2015
- November 2015
- October 2015
- September 2015
- August 2015
- July 2015
- June 2015
- May 2015
- April 2015
- March 2015
- February 2015
- January 2015
- December 2014
- November 2014
- October 2014
- September 2014
- August 2014
- July 2014
- June 2014
- May 2014
- April 2014
- March 2014
- February 2014
- January 2014
- December 2013
- November 2013
- October 2013
- September 2013
- August 2013
- July 2013
- June 2013
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012