Popular Datasets from MfE

For a few weeks now, we’ve been showing off some of the most popular datasets on the Koordinates platforms. First, we looked at some of the ridiculously popular datasets on Koordinates.com. Next, we looked at the top three layers the LINZ Data Service. Last week, we checked out the LRIS Portal from Landcare Research.

As we wrote these posts, we realised how many other datasets — and data suppliers — we wanted to show off. So, this week, we’re looking at the Ministry for the Environment. They’ve uploaded over sixty layers to Koordinates.com, ranging from REC catchments to locations of disposal facilities. Below, we’ve outlined three of the most popular MfE datasets; you can check out the other fifty-eight at Koordinates.com.

REC New Zealand

REC stands for River Environment Classification. The REC New Zealand dataset gives information about the physical characteristics — such as climate, land cover and source — of over 425,000 kilometres of New Zealand river. This dataset can be used to determine river quality and can, in turn, inform public policy discussions.

LUCAS New Zealand Land Use Map 1990-2008 (v011)

Another acronym, LUCAS stands for Land Usage Change Analysis System, which means that this map tracks changes in how New Zealand land is being used. Specifically, the LUCAS Land Use Map tracks changes in land use classifications from 1990 to 2008. Because this includes sectors like agriculture and forestry, this map allows for calculations to be made about New Zealand’s carbon emissions. This, in turn, allows the country to meet its reporting obligations under the Kyoto Protocol and the United Nations

Screenshot of 'LUCAS New Zealand Land Use Map 1990-2008 (v011),' via Koordinates.

Screenshot of ‘LUCAS New Zealand Land Use Map 1990-2008 (v011)’ from the Ministry for the Environment, via Koordinates.com

 Erosion Susceptibility 4 Classes

Acronym-free, this dataset outlines the likelihood a specific unit of land will suffer erosion. The dataset looks at the predisposition of land to erosion, the partial causes of erosion and the likely consequences of an “erosion event.” The dataset provides an ESC — or Erosion Susceptibility Class — for each polygon, as you can see below.

Screenshot of Erosion Susceptibility 4 Classes Layer from the Ministry for the Environment, via Koordinates.com

Screenshot of ‘Erosion Susceptibility 4 Classes’ from the Ministry for the Environment, via Koordinates.com

The Most Popular Datasets in the LRIS Portal

The LRIS Portal is a Koordinates Enterprise platform that hosts datasets produced by Landcare Research, a Crown Research Institute that focuses on New Zealand’s land environment. They produce datasets on everything from soil quality to terrestrial biodiversity.

These sorts of datasets are extremely important, partly because they can be used to inform public discussion about sustainability and land use — two pretty contentious issues in New Zealand. Given the importance of research and public discussion on sustainability, it’s essential that these datasets can be easily viewed, downloaded and reused, by experts and non-experts alike.

This ease of use is precisely what the LRIS portal allows, and it’s no surprise to see that some of their datasets have become extremely popular. Here are their top three most popular layers:

1. FDL New Zealand Soil Classification

This dataset contains the Fundamental Data Layers (FDLs) for soil classification, containing spatial information for 16 key soil attributes. This layer has received over 5000 views and over 700 downloads. For more information about the dataset, check out the wonderfully detailed description on LRIS.

FSL

Screenshot of FLS Soil Classification, Auckland region, via LRIS

2. NZDEM North Island 25 Metre

NZDEM refers to the North Island NZ National Digital Elevation Model, which provides an elevation grid of the North Island using data pulled from a LINZ Topographic layer. This NZDEM layer has received over 3500 views, and over 1100 downloads.

NZDEM

Screenshot of NZDEM North Island 25 Metre, via LRIS

3. NZLRI Land Use Capability

NZLRI refers to the NZ Land Resource Inventory, a complex and highly detailed national database of physical land resource information. This dataset comes in two parts. First, it gives an inventory of ‘physical factors’ like rocks and vegetation. Second, it gives each polygon a ‘Land Use Capability’ rating, based on the capacity of each polygon to sustain agricultural production. This layer has received almost 4000 views and just under 600 downloads.

NZLRI

Screenshot of NZLRI Land Use Capability, Lower South Island, via LRIS.

The Case for WMTS

As any geospatial engineer will tell you, ensuring quick and easy access to geospatial data within web browsers and GIS software isn’t easy. Some of the datasets on koordinates.com and various Koordinates Enterprise sites are truly massive. Given that we want to get this data out to thousands of users, the way we choose to provide the data as maps to users — i.e. the technical standard we adopt — becomes incredibly important.

Better technical standards mean a better user experience. Simply put, we need a technical standard that enables rapid access across many users without demanding a ludicrous amount of server power, to ensure that data users can get the maps they need, when they need it. Ludicrous demands on server power are also ludicrously expensive for the data publisher (as you might imagine, this is something that we at Koordinates are looking to avoid!).

So, what are our options? While there are a range of technical standards used across the industry, there are only two open geospatial standards for displaying maps in applications, websites and GIS software, each with their own special acronym: WMS and WMTS.

Let’s start with WMS. This standard was developed way back in 1999. Without going into technical detail, WMS works by giving users a ‘screenshot’ of spatial data. This ‘screenshot’ has to be regenerated and reloaded every time a user repositions the map. When we’re talking about big maps with lots of layers, this adds up to a lot of reloading.

Why does this matter? First, the IT infrastructure costs to support a small number of users are very high. When we’re talking about delivering spatial data to all New Zealanders, this becomes uneconomic pretty quickly. Given that it can be hard to predict usage, planning future infrastructure costs is almost impossible. If an agency using WMS gets more users than it was expecting, then the quality of its service is going to drop dramatically.

Second, WMS is as awkward as hell. It doesn’t work with consumer mapping sites like Google Maps, and is slow and difficult for non-specialists to use. If you’re looking to get visualisations of your datasets seen by a lot of users, WMS isn’t the best choice. It will mean that people will find it harder to get and reuse the data; data release will also be a heap more expensive.

WMS was good for its day – but its day is rapidly passing. Currently, Koordinates deploys WMS only for small raster layers on LDS. If Koordinates used WMS for the large, popular layers on Koordinates.com, LRIS or LDS, we would never be able to cater for tens of thousands of users.

Luckily, we’ve come a long way since 1999. Google Maps, OpenStreetMap, Koordinates and other sites have since developed a better approach known as ’tiled maps’.

Tiled maps introduce two simple innovations. First, map loading in web browsers and GIS software happens much faster. Instead of sending users a big ‘screenshot’, tiled maps send a bunch of little map images. Viewed together, these images give users the map view they are after. This means that when you pan across the map – going, say, from Dunedin to Port Chalmers — you don’t need to wait for the entire map to reload, as you would under WMS. Instead, a tiled map simply adds new images to the section of the map already loaded. You can see this at work on Koordinates.com or Google Maps.

Second, tiled maps dramatically reduce IT infrastructure costs. Tiled maps pre-generate map images at a bunch of different zoom levels, which means you no longer have live servers generating ‘screenshots’ on demand.

To replace WMS in the future, we added support for the open tiled map standard WMTS, an OGC standard developed in 2011 that’s built on the success of sites like Google Maps.

If you check out the New Zealand SDI Cookbook, which provides guidance for the development of a national Spatial Data Infrastructure, you’ll note that it recommends WMS. This is because WMTS was in its infancy when the STI cookbook was being developed; as a result, it didn’t make the cut into the first version.

Since 2011, though, WMTS has found wider support, including in the latest releases of Esri’s ArcGIS. Unlike Google Maps, WMTS can be published in different map projections, including the standard NZ Transverse Mercator. This makes it a practical solution for the publication of online maps aimed at both professionals and general users.

So, in summary: WMTS is faster, it’s cheaper and it scales – which is precisely what we need for wider scale map release. 

Top Three LDS Layers

Last week, we wrote about the most popular layers on Koordinates.com, which came from DOC, LINZ and the Wellington City Council. This included aerial photography, tracks and building footprints. But this was only one part of the story: two Koordinates Enterprise sites–the LINZ Data Service and the LRIS Portal from Landcare Research–have also had some great success.

For those who don’t know, Koordinates Enterprise sites have the software provided by Koordinates and are maintained by Koordinates engineers; Koordinates Entreprise sites run in customer data centres, and are branded and populated with data by our customers. (That’s why the look and feel of LDS and LRIS is so similar to that of Koordinates.com). Koordinates Enterprise allows an organisation to have its own expertly maintained and continuously upgraded versions of the Koordinates platform, while the organisation itself maintains control over the site’s content and branding.

LDS is a stellar example of Koordinates Enterprise at work. As we crowed about a month ago, LDS has been winning both international geospatial awards and ministerial plaudits. Judging by their user numbers, it seems like the site has also been a success with the public.

So, what are the top three LDS layers?

1. NZ Property Titles

This layer is self-explanatory: excluding ownership information–for obvious privacy reasons–this layer provides title information for NZ property. It’s had over thirty three thousand views, and over ten thousand downloads.

NZ Property Titles Screenshot

New Zealand Property Titles, lower South Island, via LINZ Data Service.

2. NZ Mainland Contours (Topo, 1:50k)

Coming in second, we have NZ Mainland Contours, which give contour lines across the entire country. Views for this layer are just over twenty-one thousand; downloads are over eleven thousand, making this the most downloaded layer on LDS.

NZ_Mainland_Contours_Screenshot

New Zealand Mainland Contours, Taranaki, via LINZ Data Service.

 3. NZ Mainland Topo 50 Map

At three, we have the NZ Mainland Topo 50, which is a topographic representation of New Zealand, including the Chathams and other offshore islands. This layer has received over twenty thousand views and over eight thousand downloads.

NZ_Mainland_Topo50_Screenshot

New Zealand Mainland Contours, greater Wellington region, via LINZ Data Service.

Job: DevOps at Koordinates

Who is Koordinates?

Koordinates is an innovative platform for publishing and accessing geospatial data via the Web – we deliver datasets every day to professional mapmakers and data wranglers, and our clients range from end users through to national mapping agencies. Our team is small, relaxed, friendly, and hard-working; and we have some of New Zealand’s best Geo and Python developers.

What’s the job?

As a devops gal or guy at Koordinates, you’ll be working to keep our sites fast and available – designing, developing, implementing, and improving our technical infrastructure. Your responsibilities will include:

  • Configuration management, deployments, builds.
  • Improving processes and automation tools to support deployment and development.
  • Monitoring and optimisation of systems and applications.
  • Getting the most out of cloud-based environments, particularly AWS.
  • Systems and network security and responses.
  • Writing internal documentation for our systems and processes.
  • Debugging performance issues and outages, including on-call response.
  • Selecting and working with infrastructure and service suppliers.

Koordinates is a modern stack, using a wide variety of technologies in our backend, including AWS, Python, Django, Linux, Puppet, PostgreSQL, Memcached, GlusterFS. The data we deal with is complex, we solve hard problems, and our applications are evolving quickly.

If you were here in the last couple of weeks you might have:

  • Built out an EC2-based application cluster, including PostgreSQL and GlusterFS replication/failover; network and application configurations.
  • Planned and implemented an improved database backup process.
  • Reviewed the implementation plans for a client’s infrastructure upgrades.
  • Rebuilt and upgraded some packages after security releases.
  • Troubleshooted a client outage due to network connectivity issues.
  • Debugged a replication problem between two file nodes.
  • Replicated terabytes of data across the Internet.
  • Improved our in-house monitoring dashboard with some new metrics.
  • Booked tickets to the next AWS summit.
  • Setup the configuration and deployment of a new application.
  • Been on-call for a rotation.

So what are we looking for?

  • 2-3 years of experience in a devops, developer, or system administration role.
  • A Comp Sci or Software Engineering degree, or be bright enough to get one.
  • Fluent in Python, Ruby, Java, or PHP. Understand databases and SQL. Keen to learn new frameworks and languages.
  • Comfortable at the Linux command line; love or loathe bash scripts; understand TCP/IP; know how to use compilers, linkers, debuggers.
  • Great troubleshooting and debugging skills, backing yourself to pick up new concepts and code-bases quickly.
  • Like to document things, almost as much as you like to automate them.
  • Independent and self-motivated, comfortable balancing multiple priorities and projects.
  • Great English and interpersonal skills (we want all your personality and talents).

What else?

This is a full-time, permanent position based in downtown Auckland, New Zealand. You may need to go through a police check for our clients. For the right candidate we can investigate work visas for New Zealand.

Tell us why you would like to join us and why you think you are a good fit. We’re very interested in links to previous projects that you’ve played a significant part in (please include a description of what you did). Feel free to give us links to code snippets, design sketches, etc as well.

If you’d like to chat about this role, call Rob on 09-966 0433 or email robert.coup@koordinates.com.

Listed 13 May 2013

The Most Popular Koordinates Layers

For a company that hasn’t done a lot of press (Ed’s recent appearance with Kim Hill aside), map layers on Koordinates tend to get a whole lot of use. For some layers, we’re talking views and downloads in the tens of thousands.

Let’s put this in perspective: Prior to using Koordinates, the organisations mentioned below would have usually sent out a few harddrives a week. If an organisation sent out an average of 100 harddrives per year, then it would take over one hundred years for ten thousand users to get its data. 

We can assume that hard-core data users, the ninjas and professionals, have always known the right channels to get public data. Koordinates, though, has shown that there is a much larger community of people — from curious newbies to bedroom programmers to architects — who want access to this data, often for interesting and innovative projects of their own. These are the users who are viewing the Wellington City Council Building Footprints or the Chorus Fibre from the New Zealand Broadband Map in their thousands. 

But what about the most popular layers on Koordinates? Let’s check out the top five (so far!).

1. Christchurch Post-Earthquake Aerial Photos (24 Feb 2011)

With almost ninety thousand views and over ten thousand downloads, the most popular Koordinates layer has been the aerial photography released by LINZ after the February 2011 Christchurch earthquake.

2. DOC Public Conservation Areas

Next we have Department of Conservation Public Conversation areas. With over forty-three thousand views, this layer shows the conversation units managed by DOC.

3. DOC Tracks

At number three, we have DOC’s layer of public walking tracks have also had a whole lot views: fifteen thousand and growing.


DOC Tracks on Koordinates

4. Wellington City Council Parks and Gardens Tracks and Walkways

This layer represents tracks and walkways maintained by Wellington City Council. It has over fifteen thousand views.

5.Wellington City Building Footprints

Another layer from Wellington City Council, Wellington City Building Footprints has also been popular, with over nine thousand views and over eight thousand downloads.


Wellington City Building Footprints on Koordinates

What’s Happening With Koordinates Support?

As you might expect, most Koordinates users spend their time on the front page, finding data and putting it to use. What you might not realise is that Koordinates has a heap of knowledge tucked away in the Support section of our site.

Recently, we’ve done a bit of Autumn cleaning, removing some of the clutter and adding some more community space. As we were cleaning, we re-discovered a bunch of good content–including guides, demos, tips and tricks–covering everything from how to use the cropping tool to changing passwords to data licensing.

One of the best posts I discovered is also one of the shortest: ‘Embedding Koordinates In Your Own Website‘ points readers to the HTML code they can use to–surprise–embed Koordinates maps. If you’re a blogger or have a website of your own, this is one way to share some of the great layers hosted by Koordinates. You can either cut and paste the HTML code we provide, or add ”?embed=1″ to the end of the URL and add it to your ‘iframe’ code.


NZTA State Highway 2010-2011 Aerial Imagery, 0.15m on Koordinates

We’ve also added a Community section, for Koordinates users to suggest new features, request data and ask questions of other Koordinates users. As you’ll be able to see, there are already a few feature requests in place–feel free to check them out and add any requests of your own. There’s a tonne of expertise in the Koordinates user community, and we want you to make use of it.

A Brief History of the Koordinates Interface

As anyone who’s checked out the Wayback Machine will know, web design changes all the time: today’s innovative ideas inevitably become tomorrow’s embarrassing relics. The home of New Zealand’s state broadcaster, circa 2003, is one example; the Ministry of Education is another.

Web design catches up with everybody, sooner or later — and Koordinates is no exception. This is why over the last six years we’ve made a bunch of tweaks and updates to our user interface. While most of these changes — like removing buttons and tabs — can seem rather minor on their own, taken together you can see a general trend towards simplifying the user experience. As always, our goal is to ensure that there are as few barriers as possible to users getting their data.

Future changes to the user interface will continue down this road: we want to make it ever easier for users to find, view and download data. At the same time, we’re going to make some cosmetic changes, so that the site is a bit nicer to look at.

In the meantime, we thought we’d take a quick tour of some of the former incarnations of the Koordinates user interface, to see how out platforms have changed over the years: 

Koordinates interface DATE

Koordinates Interface 2007

Koordinates Interface 2008

20090310-search-results-koordinates

Koordinates Interface 2009

Koordinates Interface Jan 2010 – present.

Our CEO chats with Kim Hill

On April 3, Koordinates CEO Ed Corkery was invited by the Royal Society of New Zealand and the National Library to talk about open data with Radio New Zealand’s Kim Hill.

Ed was joined by Julian Carver, spokesperson for Open New Zealand, and Peter Griffin, Manager of the Science Media Centre. Covering a lot of ground, the panel talked about their experiences with open government data, data journalism and data reuse.

There were also — as is obligatory in all public discussions about technology — references to Facebook’s privacy settings.

The session was the latest in a series of five panel discussions around the theme of ‘Big Data,’ (very) broadly defined. The other fours sessions looked at globalisation, urban disruption, the sentient planet and ‘digital natives,’ and can be downloaded here.

Koordinates and the LDS Win

LDS Win

LINZ staff celebrating their win at the Asia Pacific Spatial Excellence Awards, November 2012.

In case you missed it, we got some good news last year: the LINZ Data Servicebuilt using Koordinates Enterprise and supported by Koordinates engineers, won both the Spatial Enablement category and the JK Barrie Award for Overall Excellence at the Asia Pacific Spatial Excellence Awards.

It’s always nice to get awards, of course; but it’s particularly nice to get awards when what you’re doing is substantially different from the old model.

What was the old model? If you wanted to get data before LDS, LINZ staff would have manually extracted the data and sent it to your physical address. It could take more than a week to get the data you needed.

With LDS, it can take less than a minute. This obviously saves geospatial professionals a bunch of time (and money). What is also does is allow the rest of us to access and reuse data that we wouldn’t have otherwise known about.

NZ Mainland Topo50 Map, via the LINZ Data Service. Made available under a CC-BY licence

NZ Mainland Topo50 Map, via the LINZ Data Service. Made available under a CC-BY licence

And it seems like we really do want to access this data. Under the old model, LINZ staff would process a handful data requests a week. At present, LINZ’s NZ Mainland Topo50 Map - a single LDS layer – has had tens of thousands of views and downloads.

Why is it so popular? Because its easy: Even newbies like me, with decidedly un-GIS backgrounds, can layer a Topographic Map of New Zealand on, say, Property Titles or River Centrelines.

And honestly, if newbies like me can layer geospatial data, it must have become really easy for all the other data users. And the easier you make it for them, the more likely it is that you’ll see the data reused in surprising, innovative ways.

Hence the awards. As LINZ Chief Executive Peter Mersi pointed out, the awards are ”recognition – of the highest order – of the value of developing and implementing an easy-to-use geospatial data sharing service.”

It’s this ease-of-use that led Land Information Minister Maurice Williamson to suggest that the LDS ”could set the international benchmark for how government agencies approach the release of public data for reuse, innovation and enterprise.”

We agree and with upcoming developments we’re looking to make it even easier. With more and more data getting released, we’ll be seeing even more examples of innovative reuse.