Finding data via ArcGIS Hubs around the world

As of 2017, over 10,000 open data sites have been published by governments all over the world using ArcGIS Hub technology, as described here.  I recently wrote about how to use the ArcGIS Hub open data portal to search for and find geospatial data.  This story map is another way to find data, because it lists and provides access to some of these hubs.  The story map shows 1,500 sites created back in 2017, so while only a fraction of the total, the map provides an excellent way to browse and find spatial data.  I used it recently to easily find transportation spatial data in Victoria, Australia, for example, for the purpose of constructing a GIS lesson around.  Update:  I do not have the exact figures for 2020, but there must be multiple tens of thousands of Hub sites in existence by now.

Equally important as accessing the data is that the story map raises the important point that the open spatial data movement is more than just about sharing data.  As we have written about extensively in this blog, the open spatial data movement is a reflection of the transformation that GIS is bringing in the way government, private, and nonprofit organizations work.  This increased collaboration between departments and engagement with citizens is a long-term and oft-difficult effort, but is resulting in increased efficiency, and yes, in data access for all.

Try this story map as a way to discover spatial data, particularly by region, and I think you will find it to be a useful tool.  For more information and background on ArcGIS Hub, see this essay by my colleague Andrew Turner.

Hubs around the world story map.  This map provides a description and access to dozens of data sets served via ArcGIS Hub portal technology.

The Montana Digital Atlas

We wrote extensive reviews of local, regional, state, provincial, national, and international government data portals in our book and from time to time do so in this blog.  One of the finest state geospatial data portals in our judgment is the Montana Digital Atlas.

We have been critical here and in our book about data portals that were obviously set up simply to satisfy some organizational mandate without regard to those who will actually use the data portal.  I have spent time with the MAGIP (Montana Association of Geographic Information Professionals) community, and most recently was honored to give the keynote at their annual conference.  I am happy to report that they have built their data portal with the end user in mind.  What’s more, the Montana State Library has been a leader in the GIS community there for years, and I have found that when library information professionals are involved–people who really understand data–their resources will be extensive, the metadata will be rich, and the services are actually going to work.

The Digital Atlas features geographic databases, aerial photos, and topographic maps of lands in Montana.  The functions begin with an interactive map, where you can select base maps, thematic map layers, and tabular data from which you can draw on the screen, generate reports from in XLS and CSV formats, and download in various GIS formats.  The site features functionality that I wish all data portals had, such as the ability to move popup boxes to locations most convenient for you, choices on datums and projections for your data, the ability to clip data to specific geographic areas, and the ability to search the state library catalogs for articles, books, and other resources for the area you are investigating.  You can even copy the map link to get a web link to the current map that you have created on the site.  Furthermore, you can load some of the layers directly from the Montana State Library to ArcGIS Online, via the services in the ‘MSDI_Framework’ and ‘MSL’ folders, on https://gisservicemt.gov/arcgis/rest/services/MSDI_Framework and https://gisservicemt.gov/arcgis/rest/services/MSL.

Lastly, the portal managers graciously say to contact them at MSLDA @ mt.gov if you experience difficulty with the application, if you see a problem with any of the data, or even–and this is wonderful–to suggest additional map layers.

 

The Montana Digital Atlas.  I have selected dams, wetlands, and riparian zones.  At this point, I can generate reports, download the data, or clip and otherwise modify my data search.

 

US Congress: Maps and Data Portals

Did you know that the US Congress is increasing its use of web mapping tools in GIS?  In one of these map collections from the US Congress, you can discover organizations who host maps and data.  The themes of these maps include business locations, wildfires, hurricanes and current weather, public transportation, education, and many others.  For education, these maps could be useful for discussions in political science, geography, biology, business, and in other disciplines, as well as access to data for courses that use GIS.  To find out why the US Congress finds value in web mapping and GIS, read the excellent article that my colleague Lauren Lipovic recently wrote about how GIS is increasingly being used to inform and disseminate public policy.   Also, see the government policy page from Esri for more case studies, maps, and data.

The Mapping for Congress Featured Set contains some maps on the above themes; the most compelling one in my opinion is the set for Senator Risch from Idaho, on wildland fire potential, brownfields development success stories, and other themes of interest to citizens of that state and to those living elsewhere.  Another notable example from the Featured Set collection is this new map of Virginia state parks.

The Federal Agency Map Collection assembled by my colleague provides another excellent way of accessing more of these maps and data sets.  This collection includes the Mapping for Congress Expanded Gallery.  While some of the maps are applications that do not allow for streaming or downloading the data, GIS users with a bit of digging will find plenty of layers that they can use for analysis.  And equally importantly, these data sets represent a trend that we noted in our book–with increasing use of web mapping tools such as Esri story maps and the ability to assemble these map galleries, decision makers are increasing their use of GIS and spatial analysis in their decision making, which is good news for all of us.

The Idaho Map Gallery.

US Census Business Builder App and the Opportunity Project

The Census Business Builder app and the Opportunity Project are two new tools from the US Census Bureau that make accessing and using data, and, we hope, making decisions from it, easier for the data analyst.  Both of these applications are good representatives of the trend we noted in our book and in this blog— the effort by government agencies to make their data more user-friendly.  While I would still like to see the Census Bureau address what I consider to be the still-cumbersome process of downloading and merging data from the American Community Survey and the Decennial Census with the TIGER GIS files, these two efforts represent a significant step in the right direction.  While GIS users may still not be fully satisfied by these tools, the tools should expand the use of demographic, community, and business data by non-GIS users, which seems to be sites’ goal.

The Opportunity Project uses open data from the Census Bureau and from communities along with a Software Development Kit (SDK) to place information in the hands of decision makers.  Because these decision makers are not likely to be familiar with how to conduct spatial analysis within a GIS, the appeal of this effort is for wiser decision making with the geographic perspective.  A variety of projects are already on the site to spark ideas, including Streetwyze, GreatSchools, and Transit Analyst.

The Census Business Builder is a set of web based mapping services that provides selected demographic and economic data from the Census Bureau.  You can use it to create customized maps and county and city level reports and charts.  A small business edition presents data for a single type of business and geography at a time, while the regional analyst edition presents data for all sectors of the economy and for one or more counties at a time.  These tools are based on Esri’s online mapping capabilities and offer some of the functionality of Esri’s Business Analyst Online.  Give them a try and we look forward to your comments below.

A map that I created using the Census Business Builder.

 

The Changing Geospatial Landscape: Report

The US National Geospatial Advisory Committee recently released The Changing Geospatial Landscape:  A Second Look.   The report follows the first report from 2009. This committee consists of 28 experts from academia, the private sector and all levels of government: Federal, Tribal, State, regional, and municipal.  The committee’s stated goals of the new report are “to contribute its perceptions of incipient technologies that we expect will guide, define or determine the development of this industry in the near and medium term. Of even greater importance, the report highlights those aspects of innovation that bear directly on public policy and on individual privacy and security. The NGAC has also prepared this report to help inform the development of the next iteration of the strategic plan for the National Spatial Data Infrastructure (NSDI).”

In the first section, several near and medium term trends are noted and briefly described, including satellite imagery, including small platforms of which we have written about in this blog, advances in GNSS, UAVs, 4G mobile telephone technology, indoor positioning, platform evolution, cloud storage, crowdsourced data, and communications.  Next, social, economic, and policy issues are noted, such as the rural-urban dichotomy in the availability of internet services, workforce development in the geospatial industry, data analytics, standards, privacy and health issues, and data access.

I believe that skimming the report would be useful for anyone wanting to know what the main geospatial issues are of concern to this committee and for the geospatial industry in general, although I admit that after the seven years following the first report, I would have hoped for some clearer recommendations.  The report seems rather disorganized, but does point to the one constant in the geospatial industry:  Change.

Your reactions?

The Changing Geospatial Landscape:  Report of the National Geospatial Advisory Committee.

Marine Cadastre Data Viewer and Portal

The Marine Cadastre Data Viewer and Portal provides direct access to authoritative marine cadastral data from U.S. federal and state sources, including information on the tracks of vessels, bathymetry, administrative boundaries, and fish, bird, and other species.  It provides baseline information needed for ocean planning efforts, particularly those that involve finding the best location for renewable energy projects. The MarineCadastre.gov National Viewer uses Esri web GIS technology and is also a helpful tool in the permit review process. Users can select the ocean geography of their choosing and quickly see the applicable jurisdictional boundaries, restricted areas, laws, critical habitat locations, and other important features. With the national viewer, potential conflicts can be identified and avoided early in the planning process.

The site offers two distinct advantages:  1)  The ability to view over 75 data layers from a variety of sources in a single live ArcGIS Online-based web map viewer; 2)  The ability to download those same layers from the map interface for additional analysis.  We have been critical in this blog about sites that get the user tantalizingly close to downloading the data but never quite allow it.  This one delivers. The only thing I have not been able to get to work during my review of the resource is the buffer tool.

See this site for additional information about the data layers and services.  In addition, you can explore the set of layers on ArcGIS Online in the Marine Cadastre group.

The Marine Cadastre Data Viewer and Portal.

A United Nations resolution on geospatial data

During their 69th General Assembly, the United Nations passed a resolution on a “global geodetic reference frame for sustainable development“.   This resolution reaffirmed earlier endorsements of resolutions (1) concerning transport, search, and rescue operations that depend on GNSS, (2) concerning coordination among global observing systems and programs, including remote sensing efforts, and (3) concerning geospatial information for sustainable development policymaking, programming, and project operations.

Specifically with regard to work with geodetic reference frames, the resolution “recognized the economic and scientific importance of and the growing demand for an accurate and stable global geodetic reference frame for the Earth that allows the interrelationship of measurements taken anywhere on the Earth and in space, combining geometric positioning and gravity field-related observations, as the basis and reference in location and height for geospatial information, which is used in many Earth science and societal applications, including sea -level and climate change monitoring, natural hazard and disaster management and a whole series of industrial applications (including mining, agriculture, transport, navigation and construction) in which precise positioning introduces efficiencies.”  The resolution also recognizes “the extraordinary achievements made by [many organizations] in measuring and monitoring changes in the Earth’s system on a best-effort basis, including the development of the now adopted International Terrestrial Reference Frame.”

It recognizes “the investments of Member States in developing satellite missions for positioning and remote sensing of the Earth, supporting a range of scientific endeavours that improve our understanding of the “Earth system” and underpin decision-making, and recognizing that the full societal benefits of these investments are realized only if they are referenced to a common global geodetic reference frame at the national, regional and global levels.”  It acknowledges that “the global geodetic reference frame depends upon the participation of countries all around the globe, and the need to take action to strengthen international cooperation.”  The UN mentioned a Committee of Experts on Global Geospatial Information Management to develop a global geodetic road map that addresses key elements relating to the development and sustainability of the global geodetic reference frame.

The resolution encourages UN Member States and other relevant international organizations to enhance global cooperation in providing technical assistance, especially for capacity development in geodesy for developing countries, with the aim of ensuring the development, sustainability and advancement of a global geodetic reference frame.  It urges Member States to implement open sharing of geodetic data, standards and conventions, on a voluntary basis, (which we discuss in this blog frequently) to contribute to the global reference frame.”  As an educator, I was especially glad to see that the resolution “invites Member States to develop outreach programmes that make the global geodetic reference frame more visible and understandable to society.”

To move the GIS field forward, and to ensure that geospatial data is available and is shared, we need grass roots efforts, but also recognition from high level organizations.  These statements from the UN are welcomed, and it is our hope that the Member States will take these resolutions seriously and translate them into actionable items in their own countries and across countries to grapple with the 21st Centuries all around us — almost all of which transcend borders.

A global geodetic reference frame for sustainable development from the United Nations.

Reflections on Geospatial Information and Consolidation Options for the US Federal Government

GIS analyst Nathan Lowry has written a paper on geospatial information coordination and consolidation options for the US Federal government.   The paper examines recent reports by the US Government Accountability Office that criticize federal departments and agencies for their lack of progress on supporting, managing, and coordinating geospatial information, and analyzes these criticisms and the effectiveness of these and other proposed solutions.

Mr. Lowry begins the paper by providing an extensive background of US federal government involvement in mapping, through the formation of the Federal Geographic Data Committee and the National Spatial Data Infrastructure, with each section of the paper, provides thorough references. As such, this section is an excellent supplement to Chapter 5 of our book, where we discuss national and state data portals and metadata standards.

Mr. Lowry cuts right to the heart of the matter in statements such as “centralization never seems to work out the way it’s planned” and the focus by the GAO on three large government data producing agencies “may indicate some systematic issues, but will never redirect the redundant efforts of the federal government as a whole because the scope is too small”.  He recommends that “If the problem is that the US federal government wastes money by duplicating expenditure for geospatial data(and hardware, software and personnel) required across many agencies and underfunds activities that could most effectively and efficiently meet those needs (my assertion), then more time needs to be spent understanding the problem culturally, procedurally, organizationally and comprehensively in order to craft the right solution”.

One of the most interesting sections of the paper is an analysis of several models for geospatial portfolio management, beginning with the concept of the Geographic Information Officer for the state of California, and other states, federal agencies, and local governments, continuing with a consideration of how the US intelligence community funds and staffs its geointelligence activities, and discussion of principles of intergovernmental relations as a way to implement the NSDI.

As a fellow resident of the same state in which Mr. Lowry resides, I can attest that he is a strong proponent of geospatial technology and a valued member of our GIS community, and his comments and reflections about the problems surrounding geospatial data coordination are thoughtful and insightful.  In other words, he knows what he is talking about, he cares deeply about the value that geospatial technologies brings to decision making, and he’s not afraid to make strong recommendations.  I hope that many decision makers read his paper… and act upon his recommendations.

Nathan Lowry’s paper on federal geospatial information coordination and consolidation. Photograph by Joseph Kerski.

Fee vs. Free Geospatial Data: Like a Snow Shovel?

One of my students recently shared something that I considered to be a thought-provoking analogy in the “fee vs. free” geospatial data debate that we included in our book and discuss on this blog.  The debate, in sum, revolves around the issue, “Should government data providers charge a fee for their geospatial data, or should they provide the data for free?”

The student commented, “I tend toward the “at cost” position of the debate for local governments and free side of the debate for federal data. For me, the “tax dollars are used to create the data so it has already been paid for argument” does not hold water. Taxpayers have no expectation (or shouldn’t have) of walking into the local parks department to borrow a shovel that in theory their tax dollars paid for.  The same logic could be applied to spatial assets.”  The student went on to say that the above argument should be applied to local and regional government data, because “federal level data […] tends to be more directly reflective of the population and the federal government more directly benefits from the economic opportunities created by free data.”

While I have tended to advocate on the side that geospatial data should be freely available, I believe that the student’s snow shovel analogy for local governments has merit.  Following this argument, a small fee for data requested that is over and above what that government agency provides on its website seems reasonable.  But I still am firmly on the side of that government providing at least some geospatial data for free on its website, citing the numerous benefits as documented in case studies in this blog and in our book.  These benefits range from positive public relations, saving lives and property in emergency situations, and saving time in processing requests from data users. Consider what one person can do with the snow shovel versus what one person could do with a geospatial data such as a flood dataset.  The shovel might help dredge a small section to help a few neighbors get out of their houses, but the flood dataset could help identify hundreds of houses at risk and provide a permanent, effectively managed solution.  There is an order of magnitude difference in the benefit to be gained from making geospatial data easily and freely available.

What are your thoughts on this important issue?  We invite you to share your thoughts below.

City of Boulder GIS Resources.

Adoption and Advantages of ArcGIS Open Data

According to Esri’s 2014 Open Data  year in review,  over 763 organizations around the world have joined ArcGIS Open Data, publishing 391 public sites, resulting in 15,848 open data sets shared.  These organizations include over 99 cities, 43 countries, and 35 US states.  At the beginning of 2015, the organizations represent 390 from North America, 157 from Europe, 121 from Africa, 39 from Asia, and 22 from Oceania.  Over 42,000 shapefiles, KML files, and CSV files were downloaded from these sites since July 2014.  Recently, we wrote about one of these sites, the Maryland Open Data Portal, in this blog.  Another is the set of layers from the city of Launceton, in Tasmania, Australia.

While these initiatives are specifically using one set of methods and tools to share, that of the ArcGIS Open Data, the implications on the data user community are profound:  First, the adoption of ArcGIS Open Data increases availability for the entire user community, not just Esri users.  This is because of the increased number of portals that result, and also because the data sets shared, such as raster and vector data services, KMLs, shapefiles, and CSVs, are the types of formats that can be consumed by many types of GIS online and desktop tools.  Second, as we have expressed in our book and in this blog, while there were noble attempts for 30 years on behalf of regional, national, and international government organizations to establish standards, to share data, and to encourage a climate of sharing, and while many of those attempts were and will continue to be successful, the involvement of private industry (in this case, Esri), nonprofit organizations, and academia will lend an enormous boost to government efforts.

Third, the advent of cloud-based GIS enables these portals to be fairly easily established, curated, and improved.  Using the ArcGIS Open Data platform, organizations can leave their data where it is–whether on ArcGIS for Server or in ArcGIS Online–and simply share it as Open Data. Esri uses Koop to transform data into different formats, to access APIs, and to get data ready for discovery and exploration. Organizations add their nodes to the Open Data list and their data can then be accessed, explored, and downloaded in multiple formats without “extraneous exports or transformations.”  Specifically, organizations using ArcGIS Open Data first enable the open data capabilities, then specify the groups for open data, then configure their open data site, and then make the site public.

I see one of the chief ways tools like ArcGIS Open Data will advance the open data movement is through the use of tools that are easy to use, and also that will evolve over time.  Nobody has an infinite amount of time trying to figure out how to best serve their organization’s data, and then to construct the tools in which to do so.  The ability for data-producing organizations to use these common tools and methods represents, I believe, an enormous advantage in the time savings it represents.  As more organizations realize and adopt this, all of us in the GIS community, and beyond, will benefit.

ArcGIS Open Data users and information.