Stages not triggered when importing to Postgres

[cheetah90]

All other stages in org.wikibrain.download.loader are not triggered when selecting Postgres as the data source in the GUILoader. (Not sure if this is solely related to GUILoader).

[cheetah90]

No worries. I figured it out. Looks like if the Postgres' tables such as "local_articles" etc., cannot distinguish between different language editions so once the tables are created for one language, those stages are skipped when importing other languages? Did I miss something or is it really a issue?

[cheetah90]

Still waiting for the answer on this.... I realize that after I imported one language into the Postgres, importing the second language edition will halt since it thinks that the language has already been parsed.

[bjhecht]

We don't need spatial for this project, so we can probably skip this.

The intended use though is that you import them all at the same time.

Sent from Mobile Device
(please excuse brevity and any autocorrect-induced errors)

On Dec 8, 2015, at 16:45, Allen Lin [email protected] wrote:

Still waiting for the answer on this.... I realize that after I imported one language into the Postgres, importing the second language edition will halt since it thinks that the language has already been parsed.

—
Reply to this email directly or view it on GitHub.

[cheetah90]

Oh okay. I just want to leverage the speed of Postgres to handle the non-spatial data. H2 is slow on dataset over 1M entires. I'll try the H2 then.

[shilad]

Sorry for the delayed response! The only way to add languages right now is to reinstall from scratch with all the languages you want. This is the number one feature on our summer to-do list, but it isn't easy to do efficiently with the current setup. I would recommend that you stick to postgres. It will definitely be faster for es.

[bjhecht]

Ah, read the e-mail too quickly. Using postgres makes sense.

The key thing here is that they have to be imported at the same time, I
believe.

On 12/8/2015 5:14 PM, Allen Lin wrote:

Oh okay. I just want to leverage the speed of Postgres to handle the
non-spatial data. H2 is slow on dataset over 1M entires. I'll try the
H2 then.

—
Reply to this email directly or view it on GitHub
#254 (comment).

[cheetah90]

@shilad I currently manually run the stages to import data of different language editions. This works for me as I successfully imported EN, ES into the database. However, if I use the Loader class, it seems that some tables get dropped.