Three Lessons for Improving Data Access

 

This week’s guest post is courtesy of Brian Goldin, CEO of Voyager Search.

The Needle in the Haystack

Every subculture of the GIS industry is preaching the gospel of open data initiatives. Open data promises to result in operational efficiencies and new innovation. In fact, the depth and breadth of geo-based content available rivals snowflakes in a blizzard. There are open data portals and FTP sites to deliver content from the public sector. There are proprietary solutions with fancy mapping and charting applications from the private sector. There are open source and crowd sourced offerings that grow daily in terms of volume of data and effectiveness of their solutions. There are standards for metadata. There are laws to enforce that it all be made available. Even security stalwarts in the US and global intelligence communities are making the transition. It should be easier than ever to lay your hands on the content you need. But now, we struggle to find the needle in a zillion proverbial haystacks.

Ironically, GIS users and data consumers need to be explorers and researchers to find what they need. We remain fractured about how to reach the nirvana where not only is the data open, but also it is accurate, well documented, and available in any form. We can do better, and perhaps we learn some lessons from consumer applications that changed the way we find songs, buy a book, or discover any piece of information on the web.

Lesson one: Spotify for data.

In 1999, Napster landed a punch, knocking the wind out of the mighty music publishing industry. When the dust settled, the music industry prevailed, but it did so in a weakened state with their market fundamentally changed. Consumers’ appetite for listening to whatever they wanted for free made going back to business as usual impossible. Spotify ultimately translated that demand into an all-you-can-eat music model. The result is that in 2014 The New Yorker reported that Spotify’s user base was more than 50 million worldwide with 12.5 million subscribers. By June 2015, it was reportedly 20 million subscribers. Instead of gutting the music publishers, Spotify helped them to rebound.

Commercial geospatial and satellite data providers should take heed. Content may well be king, but expensive, complicated pricing models are targets for disruption. It is not sustainable to charge a handful of customer exorbitant fees for content or parking vast libraries of historical data on the sidelines while smaller players like Skybox, gather more than 1 terabyte of data a day and open source projects gather road maps of the world. Ultimately, we need a business model that gives users an all-you-can-eat price that is reasonable rather than a complex model based on how much the publisher thinks you can pay.

Lesson two: Google for GIS.

We have many options for finding the data, which means that we have a zillion stovepipes to search. What we need is unification across those stovepipes so that we can compare and contrast their resources to find the best content available.

This does not mean that we need one solution for storing the data and content. It just means we need one place for searching and finding all of the content no matter where it exists, what it is, what software created it or how it is stored. Google does not house every bit of data in a proprietary solution, nor does it insist on a specific standard of complex metadata in order for a page to be found. It if did, Internet search would resemble the balkanised GIS search experience we have today. But when I want GIS content, I have to look through many different potential sources to discover what might be the right one.

What is required is the ability to crawl all of the data, content, services and return a search page that shows the content on a readable, well formatted page with some normalised presentation of metadata that includes the location, the author, a brief description and perhaps the date it was created, no matter where it this resides. We need to enable people to compare content with a quick scan and then dig deeper into whatever repository houses it. We need to use their search results to inform the next round of relevancy and even to anticipate the answers to their questions. We need to enable sharing and commenting and rating on those pages to show where and how user’s feel about that content. This path is well-worn in the consumer space, but for the GIS industry these developments lag years behind as limited initiatives sputter and burn out.

Lesson 3. Amazon for geospatial.

I can find anything I want to buy on Amazon, but it doesn’t all come from an Amazon warehouse nor does Amazon manufacture it. All of the content doesn’t need to be in one place, one solution or one format; so long as it is discoverable in and deliverable from one place. Magically, anything I buy can be delivered through a handy one-click delivery mechanism! Sure, sometimes it costs money to deliver it, other times it’s free, but consumers aren’t challenged to learn a new checkout system each and every time they buy from a new vendor. They don’t have to call a help desk for assistance with delivery.

Today, getting your hands on content frequently requires a visit an overburdened GIS government staffer who will deliver the content to you. Since you might not be able to see exactly what they have, you almost always ask for more than you need. You’ll have no way of knowing when or how that data was updated. What should be as easy as clip-zip-and-ship delivery — the equivalent of gift-wrapping a package on Amazon — seems a distant dream. But why is this?

While agency leadership extols the virtues of open government initiatives, if their content is essentially inaccessible, the risk of being punished for causing frustration is minimal compared with that of exposing bad data or classified tidbits. So why bother when your agency’s first mandate is to accomplish some other goal entirely and your budget is limited? Government’s heart is certainly behind this initiative, but is easily outweighed by legitimate short-term risks and the real world constraints on human and financial resources.

The work of making public content discoverable in an open data site as bullet proof as Amazon’s limitless store seems can and should be done by industry with the support of the government so that everyone may benefit. In the private sector, we will find a business model to support this important work. But here’s the catch. This task will never be perceived as being truly open if it is done by a company that builds GIS software. The dream of making all GIS content discoverable and open, requires that it everyone’s products are equally discoverable. That’s a huge marketing challenge all by itself. Consider that Amazon’s vision of being the world’s largest store does not include making all of the stuff sold there. There really is a place for a company to play this neutral role between the vendors, the creators of the content and the public that needs it.

On the horizon

We have come so far in terms of making content open and available. The data are out there in a fractured world. What’s needed now isn’t another proprietary system or another set of standards from an open source committee. What’s really needed is a network of networks that makes single search across all of this content, data and services possible whether it’s free or for a fee. We should stop concerning ourselves with standards for this or that, and let the market drive us toward those inevitable best practices that help our content to be found. I have no doubt that the brilliant and creative minds in this space will conquer this challenge.

Brian Goldin, CEO of Voyager Search.

Landsat 8 Data Now Available on Amazon AWS

Last month Amazon announced the release of Landsat 8 data on its AWS S3 platform. The data are freely available in GeoTiff format and are not subject to any restrictions on use. The imagery is updated on a 16 day cycle and is available on AWS within hours of reception by USGS.

All of the scenes from 2015 are available, along with a selection of scenes from 2013 and 2014. For those interested in downloading the data (no Amazon account required), each scene’s directory includes the following:

• a .TIF GeoTIFF for each of the scene’s up to 12 bands
• .TIF.ovr overview file for each .TIF
• a _MTL.txt metadata file
• a small rgb preview jpeg, 3 percent of the original size
• a larger rgb preview jpeg, 15 percent of the original size
• an index.html file including the RGB preview and links to the GeoTIFFs and metadata files

As a partner in the initiative to provide easier access to the imagery, Esri has created a set of Landsat Web Services that are available through ArcGIS Online. The services provide dynamic access to the entire collection of Landsat 8 data on AWS.  The web services show one Landsat 8 service that has a subset of the Landsat 8 imagery. If you are interested in downloading more, see the “downloading” link  here.

Cambridge Gulf in Australia

 

 

 

Drones: A new voluntary code of practice

Last year we wrote about some of the privacy concerns being raised with respect to the operation of commercial drones or UAVs (Unmanned Aerial Vehicles). Despite the many actual and potential beneficial uses of drones in search and rescue, emergency response (for example, Typhoon Haiyan), farming and in retail (Amazon’s Prime Air program), persistent fears about government surveillance and snooping neighbours continue unabated.

Drone. AP Photo/Jae C. Hong

Politico recently reported on a new voluntary code of practice being developed by the Obama administration for owners and operators of commercial drones. Although the FAA in the United States has authority over the operation of drones near airports, there is no national policy covering commercial drone operations and no formal rules governing what type of data can be collected by drones.

With the cost of drones continuing to come down, and the number of small and independent drone operators continuing to rise, technological innovation has once again raced ahead of the legislation governing its use. Given that there are so many commercial and private sector operators either preparing to use or actively using drones, it remains to be seen if a voluntary code of practice will be effective in managing drone use and addressing legitimate privacy concerns or if government regulation is the only solution.

 

 

Company ethics versus technical reputation

Over the last two years we have written a number of posts on some of the issues surrounding personal information and data privacy; from UAVs (drones) to the secret lives of phones, the collection and reuse that information continue to challenge end users and customers. How much of our personal information are we willing to trade for access to products and services?

A recent ZDNet article by Jack Schofield reported the results of a Harris poll into corporate reputation and the responses from 18,000 American adults to six categories: emotional appeal, financial performance, products and services, social responsibility, vision and leadership and workplace environment. The survey indicated that 76% of those surveyed were concerned about the amount of personal information captured by large companies, including technology giants Apple, Google, Samsung, Microsoft and Amazon, and less than half (44%) reported that they trusted companies to act responsibly with that information. In the category Social Responsibility, the only technology company to appear in the top five was Microsoft, ahead of both Google and Apple.

How much of that mistrust materialises as lost sales or changing preferences? According to the poll company business practices are an increasingly important factor for customers, with 60% of those surveyed reporting that they researched companies before they considered engaging with them. It seems that technical reputation is not the only measure by which companies are judged and company ethics, in particular personal information policies and practices, now play a major role in influencing our choices.