Tag: AlphabeticalZurich

Fun with maps

Published on by Balise2 Comments

My AlphabeticalZürich project may not be very active when it comes to content, but it’s been an interesting source of tinkering lately. I’ve moved it out of WordPress to a statically-generated set of pages (that’s a story for another blog post, which I should write before I forget everything) and, in the past couple of days, I’ve added a progress map.

The idea of a progress map has been around since the early days of the project – I’m pretty sure Matthias was the one suggesting it in the first place, and it stayed in a corner of my brain. At the time, it felt somewhat overwhelming; I had explored stuff around the OpenStreetMap ecosystem, but had not dug that rabbit hole deep enough to get anywhere interesting.

And then, a few days ago, a few stars aligned in the form of “having a few days off”, seeing a Mastodon post about custom maps, remembering that the person in question DOES have custom maps on her website, digging around source code to see how that kind of things could possibly work, and finding the right resources at the right time.

First thing first: displaying map tiles

The data I want to display are lines representing streets of Zürich. And technically I could probably display a set of lines of different colors and be done with it, but a map is nicer with stuff like context and labels, so I needed a base map. The canonical way of displaying a map is to use tiles, so I knew this was one of the building bricks of my project.

The fact that I found Protomaps early in my “okay, how would I do this”-research was instrumental in the existence of this project, because the rest felt far more achievable on my own. Protomaps has a free tier that should be more than enough for the needs of my tiny website, and it looked easy enough to integrate. Its main feature is also to provide the tiles in a single file, so if I wanted to move that to my own storage, that’s a possibility. I went for the Leaflet integration because the doc promised me it was simple, and indeed it was.

Add centering coordinates, decide for a color scheme (I’m cheating, this came a bit later 🙂 ), and I have map tiles, which is one problem solved.

Adding progress data

To my map tiles, I wanted to add colored lines for “streets that I have published”, “streets that I have visited but not yet published” and “streets that I have planned to visit next”. I have this information in a spreadsheet, so that’s easy enough to exploit; but to be able to add lines to the map, I needed coordinates. The one format I’m vaguely familiar with (because I have written some code for Kartographer, the map extension of MediaWiki) is GeoJSON, and Leaflet supports that, so GO, GO, GO! I first started playing with the idea of making my own geometries with geojson.io and promptly decided against it (“this is going to make my publication process more complicated, how about no”) and remembered that Zürich has a lot of open data, and in particular the Strassennamenverzeichnis (“street name directory”) that does have line geometries in there.

So I wrote a small script to merge my spreadsheet (exported to CSV) and the Zürich open data source into a custom GeoJSON, and added it as a layer to my map. As a first test, I copied the whole thing in geojson.io, and for the first time I had a map of “where did I go already”, which felt pretty good!

It required some tweaking to get it to work on Leaflet, because, as it turns out, while the geometry definition is well-specified by GeoJSON, there doesn’t seem to be a standard for their display. The styles are typically defined as properties stuffed along the geometry, and these properties do not have consistent naming or schema depending on the display software. Still, eventually, I did manage what I wanted, and so at that point I had, on my local machine, a map base enriched with progress information. Wonderful.

That said, I had colors, but no legend whatsoever, and a map without a legend isn’t very useful. Thankfully, Leaflet has a way to add a “control”, which can contain arbitrary DOM – so I added a small legend in a very ugly but hopefully still vaguely reasonable way. (I’ll need to fix that at some point.)

Interlude: limiting the access to the API

So I had all my stuff still on my local machine, and the goal was still to have that map somewhere on AlphabeticalZürich. And there came something that kind of bothered me: the access to Protomaps puts the API key in the URL, and provides a way to define CORS limitations (which are client-side, not server side – although in that case there is some validation on the server side too). I am reading this as “API keys are not secret”, and the usage policy made me believe that, if my key was used by someone else that would mess up with my free quota, I could recover from that, but I took it as a challenge to try to not leak that key. Turns out, it was a bad idea, as I realized when writing that post.

Additionally, I’m trying to be a good citizen, and to not hit my wonderful tiles API more than I should. In particular, if I can avoid accessing tiles from any other area than Zürich, it feels like a good idea.

Some reverse proxying fun (and learning some lessons)

Now for the “let’s avoid leaking the key” part. It was pretty obvious that anything client-side would leak, so my goal was to send requests to my own stuff, inject the key there, transfer the request and get the result back. That’s the job of a reverse proxy, so I played with my Apache config until it worked (and I didn’t mess up Apache restart once in the process, proud of myself there).

Now, obviously, I do have an open URL on my website (because client-side Javascript needs to be able to access it), which doesn’t have an API key, that gets transformed behind the scenes to an url with said API key. Which means that anything can use my public URL to hit the Protomaps API without a key. Somewhat counter-productive.

The following train of thought was to add a filtering on the HTTP referrer of the URL, which does work, but which is also trivial to bypass by injecting the same header. That kind of made the whole process useless overall, but it felt “well, not worse than having an API key on the page, because the potential abuse mechanism I can see also basically is “add a HTTP header and be happy”.

Except, it actually *is* worse, which I realized when writing this blog post and feeling uncomfortable writing this down. It is actually worse for two reasons:

  • All the requests in the reverse proxy abuse scenario are eventually made from *my* machine – I’m basically running an open proxy for which I’d be responsible to shut down bad traffic (oops)
  • More importantly: it makes “changing the API key in case something goes wrong” COMPLETELY useless (large oops).

So all in all, I was feeling very smart when I made Apache do what I wanted to do, and very stupid when I realized that what I wanted to do was utterly counterproductive and actually actively harmful. Lesson learnt: if your client is supposed to access the key, so be it, and don’t try to outsmart documentation to deal with imaginary dangers. And yes, I suppose I could have gone the route of making a proper back-end and running things server-side and be happy, but I really don’t want to have a back-end on this website. This thing is made to be integrated to a web page, this is the way.

I’m probably still going to want to avoid putting that key on a public git repository, because there’s a difference between “it’s in a JS somewhere on a low-traffic website” and “it’s on GitHub open to anyone searching for ‘key='”, but that’s a problem for future me, probably (and actually an easy enough problem, since I’m already adding menus to that page programmatically.)

Handling map boundaries

I still wanted to handle map boundaries correctly, because that just felt nicer. It was an interesting problem, because for a while I thought it just wasn’t working – but, in fact, it wasn’t working *as I expected*. What ended up working was a combination of three settings on Leaflet.

  • Setting maxBounds to “area around Zürich” – this is what I expected to need to do, so far, so good.
  • Setting maxBoundViscosity to 1 – that’s a setting on Leaflet that defines how much the maxBounds are actually enforced; by default it’s 0; 1 bounces the display back into the bounds if the user pans out of the map
  • Setting minZoom to 12 – that’s the thing that required me to think most. I was very confused at the beginning, because I could zoom out to the world and then zoom back in to any place in the world outside of maxBounds, and I wasn’t sure why – until I noticed that the maxBounds documentation was explicitly talking about panning. Hence, setting a minZoom to “some value that will allow to see the whole map but would not allow to zoom in to something wildly outside of the chosen bounds” seems to work decently enough. I was happy to have a tiny bit of a sense of how tiles are structured, because it made me connect a few dots in my head quicker than it would have otherwise.

Bells and whistles: Zürich city boundary

For the finishing touch, I also wanted to add the Zürich city boundary to the map. It was somewhat more annoying to get the correct data – I didn’t find it on the Zürich-city level (because everything I had was defining multiple areas, for which I would have needed to get the outer polygon – feasible, but annoying), and finally found it on the Zürich-canton level. Note to self, as it took me a while to find how to do this (and a while to find AGAIN how to do this): click on the “Datenbezug” download arrow, and then on the first question instead of “OGD Produkte” choose “WFS-Datenquelle”, and then the rest is relatively straightforward.

RELATIVELY, because there’s a final trap: the default coordinate system is in the Swiss coordinate system, and it took me a bit of time to understand why I wouldn’t get a polygon on my map. Once that was fixed, I fought a bit with the styling definition, but I finally got the map I wanted to have.

Conclusion

I’m happy that I started with “okay, how would I do this” and managed to get through the whole project, which was not that large, but on which I had given up previously, and that connected quite a few points and a couple of rabbit holes. I’ve learnt stuff and I have something to show for it, so all in all that was very satisfying 🙂

Now introducing: AlphabeticalZürich

Published on by Balise1 Comment

I recently ran into a Mastodon post of someone who started photographing all the streets of Paris, in alphabetical order: MonParisAlphabétique. I thought that it was a GREAT idea, worth pursuing for other cities, and, since I live in Zürich, I recently started AlphabeticalZurich, which I’m hosting on WordPress and Pixelfed. So if you’re only interested in the pictures and the photography side of things, you can stop reading here and go there (WordPress) or there (Pixelfed; sorted by collections/streets here).

But, there’s a few gritty details that belong to this blog rather than the other one 😉 I prefer to start projects with a tiny bit of logistics, and in particular establishing the list of streets and how to traverse it sounded like a reasonable idea. Here comes the rambling blog post about what I tried, what I played with, and what’s the current status of said logistics.

To get the list of streets of Zürich, I turned to the Swiss OpenData data, and got a link to the CSV of all the streets of Switzerland on geo.admin.ch (other formats are available and documented in the metadata).

The lines look mostly like this:

10006621;Bahnhofstrasse;8001 Zürich;261;Zürich;ZH;Street;existing;true;12.08.2023;2683111;1247210

in which I’m interested in the second and third field. There’s a bit more subtlety, the third field is sometimes a multi-valued field separated by commas (when a street spans several zipcodes or even several cities).

Let’s clean this up a bit:

$ cut -d ";" -f 2,3 pure_str.csv | grep -E "8[0-9]{3} Zürich" | sort

The first few lines are kind of “meh” because they’re highways (and that sounds kind of dangerous), so let’s drop the A\d streets:

$ cut -d ";" -f 2,3 pure_str.csv | grep -E "8[0-9]{3} Zürich" | grep -v "^A[0-9]" | sort

and we have a list. Now, that list contains 2425 entries, so I’m going to need to do a bit more than one street a week if I want to have a chance of finishing this. Since I know myself, I need to optimize “a bit, but not too much”. So if I’m in a street starting with A, and there’s another one in the vicinity, I may want to go shoot it while I’m at it, even if it’s technically not the next one on the list. My first idea was to use zip codes as a heuristic for “places that are in the same vicinity”. So the algorithm would look a bit like this:

pick the first non-photographed street on the alphabetical list
pick all the streets starting with the same letter in the same zipcode, in order
take pictures of all these streets, mark them as photographed

Okay, this starts to be too complicated for my one-liner Bash (I tried. I really did.), so let’s get some Python instead (I did wonder if I wanted to write some ugly PHP or some ugly Python, so you’re getting some ugly Python.) I also added a bit of output to get a query for overpass turbo. 

f = open('zuri_sorted.txt', 'r' )
currLett = '0'
zipStr = {}
zipList = []

for line in f.readlines():
    line = line.strip()
    if (not line.startswith(currLett)):
        for zip in zipList:
            print(zip)
            print(', '.join(zipStr[zip]))
            print('====')
            print( '(' )
            for street in zipStr[zip]:
                print( 'way["name" = "', street, '"]({{bbox}});', sep='')
            print(');', "out body;", ">;", "out qt;", sep="\n")
            print('====')

        currLett = line[0]
        zipStr = {}
        zipList = []
    
    toks = line.split(';')
    street = toks[0]
    zips = toks[1].split(',')

    visited = False
    for zip in zips:
        if (zip in zipStr):
            zipStr[zip].append(street)
            visited = True
            break
    
    if not visited:
        zipStr[zips[0]] = [street]
        zipList.append(zips[0])

My output for a given letter and a given zip code now looks like this:

8001 Zürich
Cäcilienstrasse, Caroline-Farner-Weg, Chorgasse
====
(
way["name" = "Cäcilienstrasse"]({{bbox}});
way["name" = "Caroline-Farner-Weg"]({{bbox}});
way["name" = "Chorgasse"]({{bbox}});
);
out body;
>;
out qt;
====

I can send the part between the ==== lines to overpass turbo to see where these are on a map:

Section of a map of the center of Zürich with Cäcilienstrasse, Caroline-Farner-Weg and Chorgasse highlighted in blue
Credit: OpenStreetMap

I quickly realized that this was not necessarily the best approach because a/ zipcode areas are actually quite large b/ stopping at the boundary of zipcodes is actually fairly arbitrary. But, playing with these did bring overpass turbo to my attention, including the fact that it has an API that looked useful: Overpass API. I consequently modified my query to get “streets starting with the same letter within a radius of 500m1 of a starting point”, with the starting point defined as “the first street that I haven’t processed yet”. I actually have two overpass queries now. The first one displays the map:

[out:json];
(
  area[name="Zürich"][place="city"];
  way(area)["name"="Aargauerstrasse"]->.a;
  way(area)(around.a:500)["name" ~ "^A"][highway]({{bbox}});
);
out body;
>;
out skel qt;

The second one tells me exactly which streets I’m visiting that day:

[out:csv("name";false)];
(
	area[name="Zürich"][place="city"];
	way(area)["name"="Aargauerstrasse"]->.a;
	way(area)(around.a:500)["name" ~ "^A"][highway]({{bbox}});
	for (t["name"])
	(
  		make x name=_.val;
  		out; 
	);
);

Ideally I’d be able to get both in one query somehow, and in a way that doesn’t require editing both the name of the street and its first letter in two places; for now, let’s call that good enough. Compared to the zipcode approach, I’ll also have to manually track streets and feed the next one to the query; maybe I’ll do something fancier at some point, but for now, again, let’s start things and see where the pain points are before prematurely optimizing.

There – I am now READY to go exploring the streets of Zürich! I expect the process there to be: for each street, take a picture of the street sign (to have an idea of where pictures are taken!), take a general view picture of the street, try to find a few fun details, go home, process pictures. And then, publish pictures on Wikimedia Commons, on the blog, and on Pixelfed, rinse and repeat. Oh, and update the spreadsheet, too.

Screenshot of a spreadsheet with the lines as "street names" and the columns as "Shot / Processed / Commons / Written blog / Published blog / Pixelfed". The cells are red and green no/yes, where the two first lines are all "yes", the second line has "yes" on "shot" and "processed", and the rest is "no".

Let’s go!

  1. Value decided by taking the first street and see what looks reasonable from there. Very scientific approach. Also, this seems to yield 2-3km paths for photo walks, which is pretty good, actually. ↩︎