Metadata matters

Interesting post from Arnulf Christl recently on the lifetime of data, in which he discusses our fascination with software at the expense of data. Like it or not, the digital data we create will in all probability out-live us, yet much more time and effort is put into developing software than into understanding and developing data. Will anyone using our data in the years to come be able to trace who created it, any limitations in the data, who has subsequently used it, how they used it, and what other information products have been derived from it? In most cases, given our collective aversion to recording metadata, probably not. Keeping data current, accurate, and well documented so it will continue to be useful years from now, is too big a task and gets in the way of developing great software. Including metadata remains a laborious task and the preserve of the technically minded.

I’ve collected my fair share of data without metadata – at the time I failed to appreciate who else would be trying to use it in 20 years time and when the focus was on trying to capture a lot of data very quickly, developing new systems and trying to convince a sceptical audience that GIS was the future, perhaps the value of metadata got lost in the rush. In the GIS Guide to Public Domain Data we ask readers to consider if they should use data that has no metadata.  If  the results of any analysis based on that data are to used in reporting, policy making and planning, can you really afford not to be as certain as possible about the accuracy and provenance of the data?

We need software to help us work with data – data won’t clean itself, analyse itself, print itself, or archive itself without software (and us to press the appropriate buttons). But we also need software to help us more with metadata – make recording metadata an automatic, and easy, part of the data capture process. That’s the challenge for software developers but it’s also up to those who capture and use data to insist on it.

GIS Data on urban areas, housing, population, and more, from The Lincoln Institute of Land Policy

Back in 2013, we first wrote about the spatial data available from The Lincoln Institute of Land Policy‘s Atlas of Urban Expansion:  https://www.lincolninst.edu/publications/books/atlas-urban-expansion

Since then, the researchers have updated their data sets in conjunction with the NYU Urban Expansion Program at the Marron Institute of Urban Management, the Stern School of Business of New York University, UN-Habitat.  I believe the new ones are even more useful for teaching and research.   Tables and GIS files for cities of over 100,000 in population are available here:  http://www.atlasofurbanexpansion.org/data.  These include areas, densities, blocks, roads, and even historical data.   A set of animations depicting urban growth in 30 cities around the world is available here:  http://www.atlasofurbanexpansion.org/historical-data.

One of the most wonderful tools that the Lincoln Institute created is the Place Database:  https://placedatabase.policymap.com/.  This is a map of the United States that uses the PolicyMap platform to visualize the latest available data for dozens of indicators, ranging from housing affordability to brownfield sites to federal government spending. Many datasets in this data visualization tool include several years’ worth of data, so the user can see how things have changed over time.  The user can also run a “match” procedure to identify locations that match – up-to-three criteria, on a wide variety of data such as environment, hazards, housing, or population.

Atlas of Urban Expansion

Animations of urban growth around the world.

The Place Database – interactive map with variables the map user can select. 

These data sets are excellent tools for research and teaching.  Associated metrics, papers, and books are explained on the site as well.  The other resources at the Lincoln Institute are worthwhile to investigate, on the topics of planning and urban form, valuation and taxation, and international studies.

Abu Dhabi Environment Agency launches Enviro-Portal

As we state in our book The GIS Guide to Public Domain Data, areas around the world where not long ago it was very difficult for data users to obtain data are now opening up some of their data archives.  One example is in Abu Dhabi, whose government is an extensive user of GIS technology.  The Environment Agency Abu Dhabi recently launched an “Enviro-Portal”, announced during the Esri Middle East and Africa User Conference on GIS Technology solutions at the Abu Dhabi National Exhibition.  This portal (http://enviroportal.ead.ae) offers access to environmental data such as geology, soils, flora, fauna, marine habitats, and protected zones within Abu Dhabi.

Some of the links offer historical data back to the 1990s.  The site is available both in Arabic and in English.   The “browse” function offers a straightforward way of offering the data, similar to a view of folders on one’s own computer.   The metadata on the site is quite useful, and the data areavailable in a variety of formats, depending on the data type, from design files to shapefiles and geodatabases.  On the “Download” tab, data users can download data by providing an email address.  The site also offers a map gallery, from which the data user can search by theme but it also offers very helpful searching by polygon, line, or point buffer, shown in the image below.

Abu Dhabi Environment Agency Map Gallery and Data Portal

As with any data portal, unique features of this portal require the data user to spend some time getting comfortable with the features and options.  However, the wealth of data themes here makes it worth spending time with, and for a fascinating and rapidly changing part of the planet.

The Internet of Things

Interesting article published by the BBC on the next big frontier for the Internet – the Internet of Things. This next stage in the evolution of the Internet allows us to access and control an increasingly diverse network of devices and sensors, such as personal fitness monitors and many household items. There are applications available now for remotely controlling central heating systems, recording TV and video when we’re not at home, and keeping friends and families informed of our whereabouts. The Ford Motor Company recently announced a new initiative using their in-car connectivity system and an interface to a mobile tracking application, allowing drivers to share their location with friends and family directly from their cars using their smartphones and voice commands.

The early days of the Internet were all about people exchanging  information. Now the technology has evolved to integrate many physical devices, allowing us to use the information collected by these devices to manage our lives more effectively. Almost inevitably, with many such innovations the attendant concerns of privacy and location tracking are raised. If I use my smartphone to adjust the central heating in my home to come on/switch off at certain times, and that information is stored on a network and accessed by others or my phone is stolen, that information could potentially be used by someone trying to gain access to the house when no one is in.

How do we control who has access to our personal information and how can we protect ourselves from individuals and organisations who would seek to use that information without our consent? Many companies and organisations prefer to focus on what these new technologies can do for us and how much more efficient managing our personal location data has become, rather than discussing concerns about the misuse of this information. However, it’s our information so perhaps we should take more responsibility for managing it by securing our smartphones and insisting on a privacy policy based on “Nothing-is-shared-unless-I-say-so”.

New Digital Globe Imagery in ArcGIS Online

In chapter 3 of our book The GIS Guide to Public Domain Data, we discuss the increasing resolution, types, and ways to access sources of imagery and the  implications of these changes for data users.  Last week, imagery and for the continental United States and 60cm imagery for large parts of Western Europe arrived on the ArcGIS Online platform.

According to the announcement, http://blogs.esri.com/esri/arcgis/2012/12/20/digitalglobe-imagery-for-the-united-states-and-western-europe-added-to-the-world-imagery-map/, this is the first of several planned releases of new imagery from DigitalGlobe that will include Europe and many other parts of the world.

Another very useful feature is that with the Identify tool in ArcMap or the ArcGIS Online Content Viewer, or even inside the ArcGIS Online map viewer, the resolution, collection date, and source of the imagery can be obtained at the location on which you click.  The metadata applies only to the best available imagery at that location. You may need to zoom in to view the best available imagery.   The service was updated on the following servers: services.arcgisonline.com and server.arcgisonline.com.  If you have previously used the World Imagery map, you may need to clear your cache in order to see the updates.  For example, in the area in which I am examining in San Francisco, the popup indicates that the imagery is from 26 October 2010 and is at 30 cm resolution.

30 cm Digital Globe imagery in ArcGIS Online