Upgrade GeoIP Lite to GeoLite2

[PeterDaveHello]

GeoLite v1 is eol now, we'll need to move to GeoLite v2
https://dev.maxmind.com/geoip/legacy/geolite/

As of April 1, 2018, updated versions of the GeoLite Legacy databases will be available to redistribution license customers only, although anyone can continue to download the March 2018 GeoLite Legacy builds. On January 2, 2019, the last build will be removed from our website. GeoLite Legacy database users will need to switch to the GeoLite2 or commercial GeoIP databases and update their integrations by January 2, 2019.

If you are a redistribution license customer, please note that you must download via the account portal or geoipupdate in order to get an updated database.

For more information, please visit our Support Center.

In addition, in 2019, latitude and longitude coordinates in the GeoLite2 databases will be removed.*

* GeoLite2 databases are free IP geolocation databases comparable to, but less accurate than, MaxMind’s GeoIP2 databases. Latitude and longitude coordinates will continue to be provided in GeoIP2 databases. Please check back for updates.

cc #977 @transmission-remote-gui/development

[antekgla]

That would be very difficult because dont exist a class API done for the new format of the database.
Now this is done with GeoIP.pas writen by W. Tracz/Yvan Schwab dated in 2005.
That is for the old legacy format GeoIP.dat.
Now the GeoLite2 v2 format is different the file is: GeoLite2-Country.mmdb .New extension and I assume new internal organization.
Without a Pascal class made for someone who know the internals of that database format would be hard to addapt TransGUI.
Sadly Maxmind provides API in several languages but not in Delphi/Pascal.

Maybe some experienced coder with knowing of C++ or Java could adapt that code to Pascal.

[PeterDaveHello]

Or maybe package the legacy version in the release, need to confirm about its license and ours.

[PeterDaveHello]

Looks like there is a v1 example file here: https://www.maxmind.com/download/geoip/api/pascal/Sample.pas, not sure if they just haven't v2 sample yet.

[PeterDaveHello]

@antekgla I found another choice maybe:
https://lite.ip2location.com/database/ip-country

it's easy to deal with csv file :)

[PeterDaveHello]

Another choices:

• https://db-ip.com/db/

Maybe this code can help us understand GeoLite2 :)
https://github.com/fiorix/freegeoip/blob/master/db.go

[antekgla]

Looks like there is a v1 example file here: https://www.maxmind.com/download/geoip/api/pascal/Sample.pas, not sure if they just haven't v2 sample yet.

The problem with that is the use geoip.pas

implementation

uses GeoIP;

that unit is the main unit because is the one what deals directly with the Maxmind database and is for v1.

[rzcat]

Or perhaps we could use a web service(get method)/3rd party api to get geo result instead of an offline database?

Pros:

1. more up to date, no need to update db manually
2. accuracy increased, may be more accurate than the free version of geoip

Cons:

1. the web services must be specified in .ini file, for users in different countries different web services shall be used due to local internet connection quality and speed
2. transgui would have to make more connections when displaying a lot of peers, local cache file required, with expire time
3. more complicated

[PeterDaveHello]

@rzcat I prefer not to, the operation will be slow, IP locations don't change that fast and frequently, I found other free DBs above maybe we could use.

[rzcat]

@PeterDaveHello
thats why i mention the web service must be specified in .ini file

[PeterDaveHello]

@rzcat that's not a good idea, we don't need to make it so complex

[cfpp2p]

Just compile the C library from https://github.com/maxmind/libmaxminddb
Then a little work pseudo call and utilize:
mmdblookup --file [FILE PATH] --ip-file [FILE PATH]
https://raw.githubusercontent.com/maxmind/libmaxminddb/master/bin/mmdblookup.c
Just a little tweaking of lookup_from_file() should return what we want

Or just compile and use the library (i.e. libmaxminddb.dll for windows) directly https://github.com/maxmind/libmaxminddb/blob/master/doc/libmaxminddb.md.

[rzcat]

@PeterDaveHello
how about this, we use a local/offline sqlite datebase, which comes empty at first, along with some additional features allowing user to import data from external file, say a .CSV file or GeoIP2(eventually
the csv file is easy to read but may not be that efficient as other format to query frequently as it is plain text, i prefer sqlite😐
the additional function may locate in menu like "Update IP database from..." and each database could be added into submenu item for different structure
i am reading GeoIP.pas, see if i could make some help😴

[PeterDaveHello]

If we can import from GeoIP2, I suppose we could do the same thing just as the version 1(instead of another import step)? If we are allowed to "import" data, then we should deal with "conflicts" in the dataset/database, or every import will clean up the old data. What do you think?

[PeterDaveHello]

Let's just keep the legacy one working first:
089f319

Don't have enough time to dig or rewrite GeoIP.pas yet.

[angel228282]

GeoIP Data download 404 error (v5.16.0)

[PeterDaveHello]

@angel228282 you should try v5.17 which includes the fix

[TonyRL]

Starting December 30, 2019, downloads will no longer be served from our public GeoLite2 page, from geolite.maxmind.com/download/geoip/database/*, or from any other public URL. See the section below for steps on how to migrate to the new download mechanism.

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

[mS0gxokXEsg5ysJ7J79kzWiqO1fHgFA6ZR8eEls]

Let's just keep the legacy one working first:
089f319

Don't have enough time to dig or rewrite GeoIP.pas yet.

Can I add a manual import feature?

[PeterDaveHello]

@mS0gxokXEsg5ysJ7J79kzWiqO1fHgFA6ZR8eEls The workaround is working right now, not sure why if you need it? Actually, you can manually do that without a feature added.

[u2n]

Been missing this feature! Can't get flags, legacy db or other. Tried with fresh install and by doing manually. On picking Show Country Flags, alert shows: "Flag images archive is needed... Download this archive now?" But clicking Yes brings up a second alert: "SSL/TLS support is not compiled!" Can this currently be made to work? Many thanks for great UI.