31 January 2020

A trip down Memory Lane

This month we’ve upgraded to Elasticsearch 7.5.1 🎉

At Geocode Earth, we’re routinely re-creating our entire infrastructure behind the scenes, with no visible user impact, ensuring you’re always getting the most up-to-date features and data available. Our Geocoding API customers are already benefiting from increased performance and reduced latency.

To celebrate the new release, I’d like to take you on a trip down ⬅ Memory Lane to recount our previous Elasticsearch upgrades as both open-source projects evolved independently over the last six years.

When we first started on Pelias in 2014 the cutting edge of Elasticsearch was version 0.9! Our initial release targeted the newly available version 1.2.1. With the 1.x series came some brand new features such as the cat API, snapshot/restore and doc values which today feel so fundamental it’s hard to imagine Elasticsearch without them.

During this time, we were using a fairly new Elasticsearch feature called Completion Suggesters to provide autocomplete functionality. Based on an FST structure, this was separate from, and not easy to combine with, queries targeting the inverted-index.

Suggesters offered lightning fast autocomplete results but unfortunately, they weren’t quite right for us. Despite being involved in the Completion Suggesters V2 discussion, we switched to using an Ngram index for autocomplete in May 2015.

Through 2016 we spent five long months upgrading Pelias to support version 2.4.x. At this time, we were beginning to better understand how Elasticsearch applied to geocoding specifically, and how best to structure the data and generate the best queries.

Not long after, Elasticsearch made changes to how they version their products, and the next release skipped over versions three and four and went directly to 5.0.0.

Along with this change in policy came an accelerated release schedule from Elasticsearch as their product offering and head-count grew, in anticipation of their IPO in mid-2018.

At this time, we were primarily focused on providing free geocoding APIs through Mapzen Search. Released in 2015, this was a significant shift for us as the open-source project had now evolved into a hosted API which grew in popularity through 2016 and 2017 as other organizations began integrating with Mapzen.

With this change came some maturity in how we handled releases; a hosted service can’t have any downtime. We learned a lot about continuous delivery, how best to do it in a live user-facing environment and most importantly how to handle major upgrades to dependencies, such as Elasticsearch, with zero downtime.

And then the unthinkable happened. Mapzen shut down. In January 2018 the teams split up and went different directions and with that went all the free hosted services. The future of Pelias and other Mapzen projects was unclear.

During this difficult time, we received a huge amount of encouragement and support from our community and it was clear that Pelias development must continue. What wasn’t clear, however, was exactly how the project would be funded in order to ensure continuity.

In early 2018 Geocode Earth was formed as an independent organization, wholly owned by the core developers and without external funding. Not a startup, but a small independent business focused on the sustainable development of Pelias.

During 2018 and 2019, we completely overhauled both the development process and the deployment process to use Docker containers, making Pelias easier than ever to deploy and develop. We also introduced new features such as improved support for alternative names of places, our open source free-form address parser and numerous performance and quality improvements.

In early 2019 we moved away from the 2.4 series which had served us well during the early stages of the project and shifted focus to the 5.6.x series. Lucene 6 came with a new datastructure for geo-point fields called Block K-D trees which reduced latency for all geo-point queries.

With the end of a decade, came an upgrade to Elasticsearch 6.8.5, a much-anticipated release which came with a lot of performance improvements and a ~50% reduction of the on-disk index size.

We’re pleased to announce that Pelias is now compatible with the latest 7.5.x versions of Elasticsearch and with this update we say goodbye 👋 to version 6.8.5.

Docker images are available right now on Dockerhub, you can find the corresponding configuration files on Github.