Has relation handling changed recently?

[StyXman]

I have two rendering DBs imported with osm2pgsql the way osm-carto still does (pgsql backend, not flex). One is updated daily, the other is smaller just for faster rendering tests, never updated. But the render is not updated every day, I just do it in batches from time to time.

One of the things I render is subways with a thick semi transparent line using the color from the DB or red if not available. A few weeks back I rendered Paris with subways and noticed that all subway lines were all rendered with red. Today I decided to look into it. Using a smaller sample, I chose Marseille because it's in the smaller static DB.

For instance, M1 has a bluish color:

https://www.openstreetmap.org/relation/6274531

colour: #0075be

And yes, I take in account the different spellings of the tag :) Also, notice osm-carto's DB style does not import color as a column, so it's in the tags hstore.

Meanwhile, in my updated DB:

europe=# select tags from planet_osm_line where osm_id = -6274531;
 tags 
------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
(19 rows)

All empty. In my static DB:

paca=# select tags->'colour' from planet_osm_line where osm_id = -6274531;
 ?column? 
----------
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
 #0075be
(19 rows)

So it seems all tags have been removed? Any ideas?

[lonvia]

What version of osm2pgsql do you use (before and after)? What is the full command line of your import?

[StyXman]

mdione@ioniq:~/src/projects/elevation/data/osm$ osm2pgsql --version
osm2pgsql version 2.0.0
Build: None
Compiled using the following library versions:
Libosmium 2.20.0
Proj 9.5.0
Lua 5.3.6

but the version has changed over time, since I use Debian Unstable and update every Sunday. Command might have been something like:

osm2pgsql --create \
    --port $port --database "$db" --cache 0 --number-processes 8 --verbose \
    --slim --flat-nodes europe.cache --hstore \
    --multi-geometry --style osm_carto/openstreetmap-carto.style \
    --tag-transform-script osm_carto/openstreetmap-carto.lua \
     "europe-latest.pbf"

I say "something like" because I have this scripted and the opts might have slightly changed over time too. I checked y git history and nothing seems to be relevant.

The style files have not been changed from osm-carto's.

[joto]

I don't know of any changes related to --hstore or otherwise that could explain this.

I am not sure I understand correctly. Both databases have been updated over time but with different intervals? What happens if you import a fresh database, does it have any tags in it?

[StyXman]

No, the smaller one was imported once a some months ago.

I tried importing one just for France PACA, but currently I'm running out of disk space. I'm working on that.

Maybe there is another smaller region with subways? Berlin?

[StyXman]

I don't understand, I can't import Berlin (83MiB) with 43GiB free:

osm2pgsql --create --port 5432 --database berlin --cache 0 --number-processes 8 --verbose --slim --flat-nodes /home/mdione/src/projects/elevation/data/osm/berlin.cache --hstore --multi-geometry --style /home/mdione/src/projects/elevation/osm-carto/openstreetmap-carto.style --tag-transform-script /home/mdione/src/projects/elevation/osm-carto/openstreetmap-carto.lua --drop berlin-latest.osm.pbf
2024-09-30 20:04:26  osm2pgsql version 2.0.0
2024-09-30 20:04:26  [00] Database version: 16.4 (Debian 16.4-1)
2024-09-30 20:04:26  [00] PostGIS version: 3.4
[...]
Processing: Node(4700k 63.5k/s) Way(0k 0.00k/s) Relation(0 0.0/s)
2024-09-30 20:05:41  [00] Removing persistent node cache at '/home/mdione/src/projects/elevation/data/osm/berlin.cache'.
2024-09-30 20:05:44  [00] ERROR: Could not resize file: Not enough space on filesystem: No space left on device

mdione@ioniq:~/src/projects/elevation/data/osm$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  1,8T  1,7T   43G  98% /

[lonvia]

A flatnode file needs close to 100GB disk space. Try without.

[StyXman]

I thought those were mostly sparse, specially with such few data. I compressed the one I have for Europe an tried again and yes, colors are there. No matter how much I hate this, I will reimport Europe, unless you think we can learn something from this first.

[StyXman]

I can also keep updating Berlin instead, just to make sure it doesn't go away again.