Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User migration via export file #3054

Merged
merged 65 commits into from
Jan 2, 2024
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
6889783
Implement self-contained archives to import and export entire users b…
CSDUMMI Sep 5, 2023
a4bfcb3
fix tests and clean up
hughrun Oct 15, 2023
f07d730
Merge pull request #2980 from CSDUMMI/upstream2
hughrun Oct 15, 2023
781b01a
add error handling and status for user exports
hughrun Oct 21, 2023
20114b0
add notifs and error handling for user export/import
hughrun Oct 21, 2023
836127f
cooldown period for user exports
hughrun Oct 21, 2023
a27c652
admin view for user imports
hughrun Oct 22, 2023
b34a491
run black
hughrun Oct 22, 2023
c0a5e55
Merge branch 'user-migration' into user-migrate
hughrun Oct 22, 2023
11a726b
Merge pull request #3037 from hughrun/user-migrate
hughrun Oct 22, 2023
fd1ebf5
formatting and pylint fixes
hughrun Oct 22, 2023
6222088
Merge branch 'user-migrate' of github.com:hughrun/bookwyrm into user-…
hughrun Oct 22, 2023
0a2efeb
Merge branch 'user-migration' into user-migrate
hughrun Oct 22, 2023
0c846ca
Merge pull request #3055 from hughrun/user-migrate
hughrun Oct 22, 2023
07ef12c
fix tests and linting
hughrun Oct 22, 2023
c5e536a
Merge pull request #3056 from hughrun/user-migrate
hughrun Oct 22, 2023
b6b55b2
once more into the linting breach!
hughrun Oct 22, 2023
d05cf8e
Merge pull request #3057 from hughrun/user-migrate
hughrun Oct 22, 2023
2b6852e
oops import Any
hughrun Oct 22, 2023
afb5c01
Merge pull request #3058 from hughrun/user-migrate
hughrun Oct 22, 2023
8477d0b
Merge branch 'main' into user-migration
hughrun Oct 22, 2023
b8fc5c9
fix tests
hughrun Oct 23, 2023
ddec2db
fix tar types notification docstring
hughrun Oct 23, 2023
e29c93a
complete jobs more sensibly
hughrun Oct 23, 2023
f662e4e
Merge pull request #3060 from hughrun/user-migrate
hughrun Oct 23, 2023
f30555b
minor pylint and mypy fixes
hughrun Oct 23, 2023
5592a8e
Merge pull request #3061 from hughrun/user-migrate
hughrun Oct 23, 2023
25a2615
stop pylint constantly whining
hughrun Oct 27, 2023
935779b
Merge pull request #3064 from hughrun/user-migrate
hughrun Oct 27, 2023
67822d3
Merge branch 'main' into user-migration
mouse-reeve Nov 5, 2023
89b87db
Adds merge migration
Nov 5, 2023
ff2bb51
Adds migration for notification types
Nov 5, 2023
9e9e9a9
Uses explicit imports to avoid circular import in migrations code
Nov 5, 2023
d2f06e8
update references to bookwyrm models in export job
hughrun Nov 6, 2023
7a6b607
Merge pull request #3087 from hughrun/migration-explicit-imports
hughrun Nov 6, 2023
8d3c2d9
Merge pull request #3085 from bookwyrm-social/migration-explicit-imports
hughrun Nov 6, 2023
93a32f4
update import/export user templates
hughrun Nov 6, 2023
2d185df
remove unnecessary test data files
hughrun Nov 6, 2023
d34b70c
remove pointless viewer_aware
hughrun Nov 6, 2023
15b7b7e
Merge pull request #3088 from hughrun/user-migration
hughrun Nov 6, 2023
716e64d
Changes notification links for user import/export
Nov 6, 2023
85d1760
Changes recent exports table
Nov 6, 2023
06d822d
Alternative format for user import guide
Nov 6, 2023
3f038b4
Moves if to the right place
Nov 6, 2023
e152b62
Merge pull request #3090 from bookwyrm-social/user-migration-instruct…
mouse-reeve Nov 6, 2023
282f7dd
show filesize on user downloads page
hughrun Nov 7, 2023
1b958a9
Merge pull request #3091 from hughrun/notification-and-download-links
mouse-reeve Nov 7, 2023
9ddd631
Merge pull request #3089 from bookwyrm-social/notification-and-downlo…
mouse-reeve Nov 7, 2023
0a5e104
Add more info to user export page (#3093)
hughrun Nov 7, 2023
891b72c
update user export file to use ActivityPub objects where possible. (#…
hughrun Nov 13, 2023
d5762f1
Merge branch 'main' into user-migration
hughrun Nov 13, 2023
bbc78f0
fix DB migrations (#3111)
hughrun Nov 13, 2023
54ec5e2
Fix migrations properly (#3116)
hughrun Nov 15, 2023
62f985e
Merge branch 'main' into user-migration
Nov 16, 2023
1a7a843
Re-creates migrations and removes failing test
Nov 16, 2023
d94b27b
Merge branch 'main' into user-migration
mouse-reeve Nov 20, 2023
0276c15
Merge branch 'main' into user-migration
hughrun Nov 22, 2023
72c1c6e
merge migrations and lint
hughrun Nov 22, 2023
065e15e
Merge pull request #3131 from hughrun/user-migration
hughrun Nov 22, 2023
99a9a64
notification type migration after merge
hughrun Nov 23, 2023
48904fc
Merge pull request #3132 from hughrun/user-migration
hughrun Nov 23, 2023
dd72013
Small fixes for notifications
Dec 9, 2023
029b438
Clarify import/export of book vs user
Dec 9, 2023
7fcadb1
fix upsert_statuses
hughrun Dec 13, 2023
1649457
Merge pull request #3156 from hughrun/user-migration
hughrun Dec 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions bookwyrm/forms/forms.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ class ImportForm(forms.Form):
csv_file = forms.FileField()


class ImportUserForm(forms.Form):
archive_file = forms.FileField()


class ShelfForm(CustomForm):
class Meta:
model = models.Shelf
Expand Down
1 change: 1 addition & 0 deletions bookwyrm/importers/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
""" import classes """

from .importer import Importer
from .bookwyrm_import import BookwyrmImporter
from .calibre_import import CalibreImporter
from .goodreads_import import GoodreadsImporter
from .librarything_import import LibrarythingImporter
Expand Down
20 changes: 20 additions & 0 deletions bookwyrm/importers/bookwyrm_import.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
"""Import data from Bookwyrm export files"""
from bookwyrm.models.bookwyrm_import_job import BookwyrmImportJob


class BookwyrmImporter:
"""Import a Bookwyrm User export file.
This is kind of a combination of an importer and a connector.
"""

def process_import(
self, user, archive_file, settings
): # pylint: disable=no-self-use
"""import user data from a Bookwyrm export file"""

required = [k for k in settings if settings.get(k) == "on"]

job = BookwyrmImportJob.objects.create(
user=user, archive_file=archive_file, required=required
)
return job
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# Generated by Django 3.2.19 on 2023-08-31 22:57

from django.conf import settings
import django.contrib.postgres.fields
from django.db import migrations, models
import django.db.models.deletion
import django.utils.timezone


class Migration(migrations.Migration):

dependencies = [
("bookwyrm", "0178_auto_20230328_2132"),
]

operations = [
migrations.CreateModel(
name="ParentJob",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("task_id", models.UUIDField(blank=True, null=True, unique=True)),
(
"created_date",
models.DateTimeField(default=django.utils.timezone.now),
),
(
"updated_date",
models.DateTimeField(default=django.utils.timezone.now),
),
("complete", models.BooleanField(default=False)),
(
"status",
models.CharField(
choices=[
("pending", "Pending"),
("active", "Active"),
("complete", "Complete"),
("stopped", "Stopped"),
],
default="pending",
max_length=50,
null=True,
),
),
(
"user",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
to=settings.AUTH_USER_MODEL,
),
),
],
options={
"abstract": False,
},
),
migrations.CreateModel(
name="BookwyrmExportJob",
fields=[
(
"parentjob_ptr",
models.OneToOneField(
auto_created=True,
on_delete=django.db.models.deletion.CASCADE,
parent_link=True,
primary_key=True,
serialize=False,
to="bookwyrm.parentjob",
),
),
("export_data", models.FileField(null=True, upload_to="")),
],
options={
"abstract": False,
},
bases=("bookwyrm.parentjob",),
),
migrations.CreateModel(
name="BookwyrmImportJob",
fields=[
(
"parentjob_ptr",
models.OneToOneField(
auto_created=True,
on_delete=django.db.models.deletion.CASCADE,
parent_link=True,
primary_key=True,
serialize=False,
to="bookwyrm.parentjob",
),
),
("archive_file", models.FileField(blank=True, null=True, upload_to="")),
("import_data", models.JSONField(null=True)),
(
"required",
django.contrib.postgres.fields.ArrayField(
base_field=models.CharField(blank=True, max_length=50),
blank=True,
size=None,
),
),
],
options={
"abstract": False,
},
bases=("bookwyrm.parentjob",),
),
migrations.CreateModel(
name="ChildJob",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("task_id", models.UUIDField(blank=True, null=True, unique=True)),
(
"created_date",
models.DateTimeField(default=django.utils.timezone.now),
),
(
"updated_date",
models.DateTimeField(default=django.utils.timezone.now),
),
("complete", models.BooleanField(default=False)),
(
"status",
models.CharField(
choices=[
("pending", "Pending"),
("active", "Active"),
("complete", "Complete"),
("stopped", "Stopped"),
],
default="pending",
max_length=50,
null=True,
),
),
(
"parent_job",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="child_jobs",
to="bookwyrm.parentjob",
),
),
],
options={
"abstract": False,
},
),
]
13 changes: 13 additions & 0 deletions bookwyrm/migrations/0182_merge_20230905_2240.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Generated by Django 3.2.19 on 2023-09-05 22:40

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
("bookwyrm", "0179_bookwyrmexportjob_bookwyrmimportjob_childjob_parentjob"),
("bookwyrm", "0181_merge_20230806_2302"),
]

operations = []
85 changes: 85 additions & 0 deletions bookwyrm/migrations/0183_auto_20231021_2050.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Generated by Django 3.2.20 on 2023-10-21 20:50

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

dependencies = [
("bookwyrm", "0182_merge_20230905_2240"),
]

operations = [
migrations.AddField(
model_name="notification",
name="related_user_export",
field=models.ForeignKey(
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="bookwyrm.bookwyrmexportjob",
),
),
migrations.AlterField(
model_name="childjob",
name="status",
field=models.CharField(
choices=[
("pending", "Pending"),
("active", "Active"),
("complete", "Complete"),
("stopped", "Stopped"),
("failed", "Failed"),
],
default="pending",
max_length=50,
null=True,
),
),
migrations.AlterField(
model_name="notification",
name="notification_type",
field=models.CharField(
choices=[
("FAVORITE", "Favorite"),
("REPLY", "Reply"),
("MENTION", "Mention"),
("TAG", "Tag"),
("FOLLOW", "Follow"),
("FOLLOW_REQUEST", "Follow Request"),
("BOOST", "Boost"),
("IMPORT", "Import"),
("USER_IMPORT", "User Import"),
("USER_EXPORT", "User Export"),
("ADD", "Add"),
("REPORT", "Report"),
("LINK_DOMAIN", "Link Domain"),
("INVITE", "Invite"),
("ACCEPT", "Accept"),
("JOIN", "Join"),
("LEAVE", "Leave"),
("REMOVE", "Remove"),
("GROUP_PRIVACY", "Group Privacy"),
("GROUP_NAME", "Group Name"),
("GROUP_DESCRIPTION", "Group Description"),
],
max_length=255,
),
),
migrations.AlterField(
model_name="parentjob",
name="status",
field=models.CharField(
choices=[
("pending", "Pending"),
("active", "Active"),
("complete", "Complete"),
("stopped", "Stopped"),
("failed", "Failed"),
],
default="pending",
max_length=50,
null=True,
),
),
]
18 changes: 18 additions & 0 deletions bookwyrm/migrations/0184_sitesettings_user_import_time_limit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Generated by Django 3.2.20 on 2023-10-22 02:27

from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
("bookwyrm", "0183_auto_20231021_2050"),
]

operations = [
migrations.AddField(
model_name="sitesettings",
name="user_import_time_limit",
field=models.IntegerField(default=48),
),
]
1 change: 1 addition & 0 deletions bookwyrm/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
from .group import Group, GroupMember, GroupMemberInvitation

from .import_job import ImportJob, ImportItem
from .bookwyrm_import_job import BookwyrmImportJob
hughrun marked this conversation as resolved.
Show resolved Hide resolved

from .site import SiteSettings, Theme, SiteInvite
from .site import PasswordReset, InviteRequest
Expand Down
Loading
Loading