Using and (Partially) Selfhosting Overpass

Data acquisition and initialization

Since I did not want to host the whole planet file, I have downloaded the regional data for Germany from Geofabrik. At the current time, this means around 4.5 GiB of data.

The data comes as .pbf file, which is not supported directly by Overpass. Guides say that you should use osmium to convert the file to .osm.bz2, but there is actually a more efficient way: By changing bin/init_osm3s.sh with the patch

44c44
< bunzip2 <$PLANET_FILE | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION
---
> osmium cat -f .osm -o - $PLANET_FILE | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION

we can make init_osm3s.sh ingest .pbf files directly. This has three main benefits:

• We only traverse the file once, not twice, which saves time.
• We don’t need temporary space to keep the redundant data copy.
• We get rid of pointless work (compressing data just to uncompress it right away again).

The import took around 3.5 hours, and produced a database of size 68 GiB. Memory usage was at around 6.5 GiB.

Updates

To keep the map data up-to-date, I have set up the import of updates as described in the installation guide. Of course, the update URL needs to be adapted to the region that you host. In my case, these also come from Geofabrik (https://download.geofabrik.de/europe/germany-updates). The link is at the top of a region’s download page, under Other Formats and Auxiliary Files.

The default fetch_osc.sh script polls the server for updates every 15 seconds. This is fine for minutely updates, but Geofabrik provides only daily changesets – so a 15 second poll interval is excessive. A small change to fetch_osc.sh fixes this:

163c163
<     sleep 15
---
>     sleep 3600

Now, we only poll once an hour. The script apply_osc_to_db.sh remains unchanged.

Authentication

Overpass is a “simple” CGI script that comes without authentication. Therefore, authentication is also not part of the “official” Overpass API. Many providers seem to have a system in which they put the API key into the URL, but I decided to use HTTP basic authentication instead. This is something that can be done without much additional setup, as it can be handled directly by Apache.

The downside of this approach is that it is non-standard, so Overpass clients (or client libraries) might not support it (unlike the key-in-URL approach, which works everywhere). As I need Overpass for Fietsboek, this is not a problem, as I can simply adjust Fietsboek to my needs.

Systemd services

Yes, I am aware of the blog post explaining why systemd is unfit to run Overpass (or other databases). I don’t think I fully agree with the reasoning. Systemd is capable of starting programs without deleting their sockets or interfering with their shared memory, and I like to have my services managed by systemd so I can quickly check on their status and have journalctl collect their logs.

I have set up three unit files, for the three different parts of Overpass. First, there is the dispatcher daemon, which is needed for the web interface interpreter and the update application:

# /etc/systemd/system/overpass.service
[Unit]
Description=Overpass API dispatcher

[Service]
WorkingDirectory=/home/overpass
ExecStart=/home/overpass/bin/dispatcher --osm-base --db-dir=/home/overpass/db/
User=overpass
Group=overpass
Restart=no

[Install]
WantedBy=multi-user.target

I have disabled the restart feature, as Overpass seems to leave sockets behind that require a manual check and cleanup.

Then, there is the part that fetches the changesets:

# /etc/systemd/system/overpass-fetcher.service
[Unit]
Description=Overpass changeset fetcher

[Service]
WorkingDirectory=/home/overpass
ExecStart=/home/overpass/bin/fetch_osc.sh 4912 https://download.geofabrik.de/europe/germany-updates/ diffs
User=overpass
Group=overpass
Restart=no

[Install]
WantedBy=multi-user.target

Finally, there is the changeset application script:

# /etc/systemd/system/overpass-applier.service
[Unit]
Description=Overpass changeset applier

[Service]
WorkingDirectory=/home/overpass
ExecStart=/home/overpass/apply_osc_current.sh
User=overpass
Group=overpass
Restart=no
Requires=overpass

[Install]
WantedBy=multi-user.target

This makes use of a small helper that reads the correct replication ID to avoid re-applying old changesets:

#!/bin/bash
current=$(cat db/replicate_id)
exec ./bin/apply_osc_to_db.sh diffs $current --meta=no

Note that this requires at least one changeset to have been applied, otherwise the db/replicate_id does not exist yet.

Conclusion

Setting up Overpass is very straightforward. The only time consuming part is importing the data, but that can happen in the background without user interaction.

The main resource seems to be disk space. Even the “small” 4.5 GiB of import data produced a large 68 GiB database, where most of the space is occupied by the nodes.map – I assume this is some sort of spatial index to efficiently find nodes by coordinates(?).

Memory usage during operation seems to be lower than during the import. Systemd reports a peak for the dispatcher of 23 MiB after some simple queries, so I assume that Overpass is quite efficient about which pieces of the database to load for a query. Importing changesets on the other hand requires more resources, the apply_osc_to_db.sh script is reported with a maximum use of 11.3 GiB.