Duplicate entries in download

[rs1729]

When I download e.g.

sondehub --download U1440085 > U1440085.json

I get duplicate entries, only the order of the fields may differ, e.g. the first two entries

{"alt": 192.99898, "batt": 2.9, "datetime": "2023-01-20T22:45:33.000Z", "frame": 1553, "frequency": 404.1, "heading": 201.29372, "lat": 53.71182, "lon": 7.15211, "manufacturer": "Vaisala", "position": "53.71182,7.15211", "rssi": -80.5, "sats": 8, "serial": "U1440085", "software_name": "rdzTTGOsonde", "software_version": "devel20221113", "time_received": "2023-01-20T22:45:16.000Z", "type": "RS41", "upload_time_delta": -19.649, "uploader_alt": 11, "uploader_antenna": "3x5/8", "uploader_callsign": "DJ2DS", "uploader_position": "53.546,7.58", "user-agent": "Amazon CloudFront", "vel_h": 4.35333, "vel_v": 5.85647, "launch_site": "10113", "launch_site_range_estimate": 56.704276734652034}
{"alt": 192.99898, "batt": 2.9, "datetime": "2023-01-20T22:45:33.000Z", "frame": 1553, "frequency": 404.1, "heading": 201.29372, "lat": 53.71182, "launch_site": "10113", "launch_site_range_estimate": 56.704276734652034, "lon": 7.15211, "manufacturer": "Vaisala", "position": "53.71182,7.15211", "rssi": -80.5, "sats": 8, "serial": "U1440085", "software_name": "rdzTTGOsonde", "software_version": "devel20221113", "time_received": "2023-01-20T22:45:16.000Z", "type": "RS41", "upload_time_delta": -19.649, "uploader_alt": 11, "uploader_antenna": "3x5/8", "uploader_callsign": "DJ2DS", "uploader_position": "53.546,7.58", "user-agent": "Amazon CloudFront", "vel_h": 4.35333, "vel_v": 5.85647}

only

, "launch_site": "10113", "launch_site_range_estimate": 56.704276734652034

is in a different place, the data is the same. Sometimes there are other frames between two matching frames.
Don't know if it is already in the database or when fetched for the download, but it inflates the amount of data considerably.

[darksidelemm]

You're right, this shouldn't be happening. There's a possibility that there's 2 stations running with the same callsign, but it's difficult to be sure that's whats happening.

[darksidelemm]

I can confirm this with data from other flights. As an example, callsign V3831020.
Grabbed json data with:
sondehub --download V3831020 > V3831020.json

I wrote a very short python script to read in each line and print out the datetime, station callsign, and software version:

2024-04-04T23:18:37.000000Z,    BARC-RRR,               1.7.2
2024-04-04T23:18:37.000000Z,    VK5ARG,                 1.7.3-beta9
2024-04-04T23:18:37.000000Z,    BARC-RRR,               1.7.2
2024-04-04T23:18:37.000000Z,    VK5ALG_AUTO_RX,                 1.7.1
2024-04-04T23:18:37.000000Z,    VK5ALG_AUTO_RX,                 1.7.1
2024-04-04T23:18:37.000000Z,    VK5ARG,                 1.7.3-beta9

2024-04-04T23:18:38.000000Z,    VK5ARG,                 1.7.3-beta9
2024-04-04T23:18:38.000000Z,    VK5ZBI_AUTO_RX,                 1.7.2
2024-04-04T23:18:38.000000Z,    BARC-RRR,               1.7.2
2024-04-04T23:18:38.000000Z,    VK5ARG,                 1.7.3-beta9
2024-04-04T23:18:38.000000Z,    VK5ALG_AUTO_RX,                 1.7.1
2024-04-04T23:18:38.000000Z,    VK5ALG_AUTO_RX,                 1.7.1
2024-04-04T23:18:38.000000Z,    VK5ZBI_AUTO_RX,                 1.7.2
2024-04-04T23:18:38.000000Z,    BARC-RRR,               1.7.2

2024-04-04T23:18:39.000000Z,    VK5ALG_AUTO_RX,                 1.7.1
2024-04-04T23:18:39.000000Z,    BARC-RRR,               1.7.2
2024-04-04T23:18:39.000000Z,    VK5ARG,                 1.7.3-beta9
2024-04-04T23:18:39.000000Z,    VK5ALG_AUTO_RX,                 1.7.1
2024-04-04T23:18:39.000000Z,    BARC-RRR,               1.7.2
2024-04-04T23:18:39.000000Z,    VK5ARG,                 1.7.3-beta9

We are clearly seeing duplicate packets.

[rs1729]

Yes, I believe the duplicate entries are for every frame number. Only the field order may be different.

[xssfox]

Not sure if its the only issue but it appears to be related to launchsite enrichment