Perf considerations

laem

I've had a good call with one of the NLnet experts about performance.

We've discussed the Web performance first, client-side. It appears from our talk and our live test on a Fairphone 3 (old and low-end phone) that the map and page load performance isn't that worse considering the one of other applications the user is used to. Of course, things can be optimzed further, like tile weight, discussed in large in another issue.

Performance on lower devices with Firefox could be an issue though, as both webgl and js are slower than on chromium browsers.

We've then talked about the challenge of doing Web-based navigation ("GPS mode"). It appears difficult, but we won't know until we try.

Then we talked about server performance. The main conclusion was that we should be able to scale horizontally, spawning new servers to handle load. Of course, optimisations should still be done, but being able to load balance between a few servers in a pool is key.

Knowing what load a single VM can take would also be very useful, through load tests that would reproduce (at best) a typical user navigation : tiles (pmtiles), search (photon), osm place info (overpass).

Doing this should let us know how many users we should be able to handle on a single server, and thus also estimate how much we could bill this service to entreprises or paid users.

1. laem

   New elements since then :

   • we've gained a boost of server credits, which lets us experiment with multiple servers on top of what we already have !
   • we've launched the paid subscription plan : we can now build a logic that directs paid users to less overcrowded servers
2. laem

   Create a virtual data set that reproduces a user with pseudo random sequences.

   Record an interaction with tiles ? Or use a library to create navigation patterns. Then launch an itinerary. Then check opening times. Again. Search for places through photon, then overpass. Etc.

   Investigate "test cafe" ?

   Use Deno Astral like in https://github.com/laem/politix ?

3. laem

   For Web crawlers : I didn't know about the "Crawl-delay: 1" command in robots.txt cc @etienneJr

4. laem

   Important : the deployment of a new server (2) shouldn't depend on the production server (1).

   We should probably upload the important files to S3 instead of keeping them on the server 1.

   E.g. the planet.pmtiles file. But uploading to S3 could be way slower. But serving tiles directly from s3 seems to be a good solution, according to https://docs.protomaps.com/deploy/

   

   S3 = scaleway S3 in our case, so we have to check this is possible, without relying on Amazon.

   Edit : I've checked, the guide is for S3/Cloudflare, doing it for Scaleway would take a bit of time for a not-so-obvious gain compared to simply downloading the 90 go file to the server with a high-bandwidth cloud.

5. laem

   Maybe try haproxy load balancing between our current server and the new one.

6. laem

   Here is the first test https://codeberg.org/cartes/web/pulls/2567

7. laem

   • make more realistics search terms by logging our own real search terms (beware RGPD)
8. laem

   Update.

   We've been struggling against bots regularly this summer. Bots from botnets that are impossible to simply block because they rotate their IP and UA.

   We're now sure that our regular userbase does not saturate our infrastructure anymore. Bots do.

   We've installed Crowdsec, logging tools for Overpass and Photon. Then corrected the Crowdsec installation.

   Yesterday our Web server got 1,7 million requests, most in the late hours. Crowdsec was overcrowded, taking 7 CPUS.

   We've analysed the logs, the pattern is clear : no-JS bots take all the links of the pages that they probably obtained through our registry of places or through our indexed codeberg issues.

   We've built this registry https://cartes.app/lieux precisely to index the Website. So we're stuck fighting bad bots but keeping good indexing bots.

   Next steps :

   • cache all the places of our registry. Overpass requests should be done monthly at most. Don't serve non-cached places
   • remove all the irrelevant links from the bot's view : less queryParam combinatory, nofollow on all the user-reserved interactions (e.g. favorite page)
   • implement the StupidAntibot pattern to be activated when we get heavily hit. It should not be activated all the time, because it forces the user to a static page that sets a cookie. Automate that in case there's more than 10 request / second
   • alternative 2 : implement a reverse-proxy level redirect to another "safe" server (VPS) to handle returning users (they have a cookie set), and keep both bots and new users (that we cannot differentiate) on the overloaded one
   • alternative 3 : setup docker swarm to have 2 or 3 servers handling requests. We need a registry.

   Apart from the bot problem, we should also prepare for a real spike of trafic

   • shut down the server1, legacy, not based on ansible, still using nginx and not caddy
   • spawn a new server1
   • do subscription based routing and / or load balancing