Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RemoteProgress update broken while fetching #1969

Open
fvalette-ledger opened this issue Oct 14, 2024 · 6 comments
Open

RemoteProgress update broken while fetching #1969

fvalette-ledger opened this issue Oct 14, 2024 · 6 comments

Comments

@fvalette-ledger
Copy link
Contributor

fvalette-ledger commented Oct 14, 2024

Used version

Git: 2.34.1
GitPython: 3.1.43
Python: 3.10.12
FYI, progress bar rendered with rich (13.7.1)

Issue

RemoteProgress update method is called with ill-formed parameters when use with remote.fetch and thus leading to broken progress bar. This is working well with clone and/or clone_from. Update is called only once per git operation (e.g. COUNTING, COMPRESSING) with BEGIN and END opcode flag set and all messages concatenated.
It seems that remote progress lines are not parsed correctly while fetching remote.

Correct behavior with clone

Cloning git repository mypackage (revision=branch1)...
  Counting    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% eta 0:00:00                       
  Compressing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% eta 0:00:00                       
  Receiving   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% eta 0:00:00 6.45 MiB | 12.85 MiB/s
  Resolving   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% eta 0:00:00                       

Broken progress bar with fetch

Updating mypackage (revision=main)...
Count… remote: Counting objects:   1% (6/592)        remote: Counting objects:   2% (12/592)        remote: Counting objects:   3% (18/592)        remote: Counting objects:   4% (24/59…
Compr… remote: Compressing objects:   1% (3/240)        remote: Compressing objects:   2% (5/240)        remote: Compressing objects:   3% (8/240)        remote: Compressing objects:  …
Recei… Receiving objects:  77% (456/592)Receiving objects:  78% (462/592)Receiving objects:  79% (468/592)Receiving objects:  80% (474/592)Receiving objects:  81% (480/592)Receiving ob…
Resol… Resolving deltas:   1% (5/406)Resolving deltas:   2% (9/406)Resolving deltas:   3% (13/406)Resolving deltas:   4% (17/406)Resolving deltas:   5% (21/406)Resolving deltas:   6% (…
@Byron
Copy link
Member

Byron commented Oct 14, 2024

Thanks for reporting.

Parsing should happen in real-time even though this looks like it's not able to separate the lines, or it sees all lines at once for some reason. Maybe it's related to synchronization also in the relation to how the program that uses GitPython is written, even though it seems to work with clone and clone_from.

So maybe it would be possible to find a difference between the fetch and clone, which to my knowledge use the same underlying engine.

@fvalette-ledger
Copy link
Contributor Author

With GitPython 3.1.40, it works well, progress bar is not broken on fetch sub command.
I'll try the two next patch level release.

@fvalette-ledger
Copy link
Contributor Author

This is broken from 3.1.41.

@fvalette-ledger
Copy link
Contributor Author

fvalette-ledger commented Oct 14, 2024

Here dumps of handle process output while fetching:

b'POST git-upload-pack (294 bytes)\n'


b'POST git-upload-pack (958 bytes)\n'


b'remote: Enumerating objects: 592, done.        \n'


b'remote: Counting objects:   0% (1/592)        \rremote: Counting objects:   1% (6/592)        \rremote: Counting objects:   2% (12/592)        \rremote: Counting objects:   3% (18/592)        \rremote: Counting objects:   4% (24/592)        \rremote: Counting objects:   5% (30/592)        \rremote: Counting objects:   6% (36/592)        \rremote: Counting objects:   7% (42/592)        \rremote: Counting objects:   8% (48/592)        \rremote: Counting objects:   9% (54/592)        \rremote: Counting objects:  10% (60/592)        \rremote: Counting objects:  11% (66/592)        \rremote: Counting objects:  12% (72/592)        \rremote: Counting objects:  13% (77/592)        \rremote: Counting objects:  14% (83/592)        \rremote: Counting objects:  15% (89/592)        \rremote: Counting objects:  16% (95/592)        \rremote: Counting objects:  17% (101/592)        \rremote: Counting objects:  18% (107/592)        \rremote: Counting objects:  19% (113/592)        \rremote: Counting objects:  20% (119/592)        \rremote: Counting objects:  21% (125/592)        \rremote: Counting objects:  22% (131/592)        \rremote: Counting objects:  23% (137/592)        \rremote: Counting objects:  24% (143/592)        \rremote: Counting objects:  25% (148/592)        \rremote: Counting objects:  26% (154/592)        \rremote: Counting objects:  27% (160/592)        \rremote: Counting objects:  28% (166/592)        \rremote: Counting objects:  29% (172/592)        \rremote: Counting objects:  30% (178/592)        \rremote: Counting objects:  31% (184/592)        \rremote: Counting objects:  32% (190/592)        \rremote: Counting objects:  33% (196/592)        \rremote: Counting objects:  34% (202/592)        \rremote: Counting objects:  35% (208/592)        \rremote: Counting objects:  36% (214/592)        \rremote: Counting objects:  37% (220/592)        \rremote: Counting objects:  38% (225/592)        \rremote: Counting objects:  39% (231/592)        \rremote: Counting objects:  40% (237/592)        \rremote: Counting objects:  41% (243/592)        \rremote: Counting objects:  42% (249/592)        \rremote: Counting objects:  43% (255/592)        \rremote: Counting objects:  44% (261/592)        \rremote: Counting objects:  45% (267/592)        \rremote: Counting objects:  46% (273/592)        \rremote: Counting objects:  47% (279/592)        \rremote: Counting objects:  48% (285/592)        \rremote: Counting objects:  49% (291/592)        \rremote: Counting objects:  50% (296/592)        \rremote: Counting objects:  51% (302/592)        \rremote: Counting objects:  52% (308/592)        \rremote: Counting objects:  53% (314/592)        \rremote: Counting objects:  54% (320/592)        \rremote: Counting objects:  55% (326/592)        \rremote: Counting objects:  56% (332/592)        \rremote: Counting objects:  57% (338/592)        \rremote: Counting objects:  58% (344/592)        \rremote: Counting objects:  59% (350/592)        \rremote: Counting objects:  60% (356/592)        \rremote: Counting objects:  61% (362/592)        \rremote: Counting objects:  62% (368/592)        \rremote: Counting objects:  63% (373/592)        \rremote: Counting objects:  64% (379/592)        \rremote: Counting objects:  65% (385/592)        \rremote: Counting objects:  66% (391/592)        \rremote: Counting objects:  67% (397/592)        \rremote: Counting objects:  68% (403/592)        \rremote: Counting objects:  69% (409/592)        \rremote: Counting objects:  70% (415/592)        \rremote: Counting objects:  71% (421/592)        \rremote: Counting objects:  72% (427/592)        \rremote: Counting objects:  73% (433/592)        \rremote: Counting objects:  74% (439/592)        \rremote: Counting objects:  75% (444/592)        \rremote: Counting objects:  76% (450/592)        \rremote: Counting objects:  77% (456/592)        \rremote: Counting objects:  78% (462/592)        \rremote: Counting objects:  79% (468/592)        \rremote: Counting objects:  80% (474/592)        \rremote: Counting objects:  81% (480/592)        \rremote: Counting objects:  82% (486/592)        \rremote: Counting objects:  83% (492/592)        \rremote: Counting objects:  84% (498/592)        \rremote: Counting objects:  85% (504/592)        \rremote: Counting objects:  86% (510/592)        \rremote: Counting objects:  87% (516/592)        \rremote: Counting objects:  88% (521/592)        \rremote: Counting objects:  89% (527/592)        \rremote: Counting objects:  90% (533/592)        \rremote: Counting objects:  91% (539/592)        \rremote: Counting objects:  92% (545/592)        \rremote: Counting objects:  93% (551/592)        \rremote: Counting objects:  94% (557/592)        \rremote: Counting objects:  95% (563/592)        \rremote: Counting objects:  96% (569/592)        \rremote: Counting objects:  97% (575/592)        \rremote: Counting objects:  98% (581/592)        \rremote: Counting objects:  99% (587/592)        \rremote: Counting objects: 100% (592/592)        \rremote: Counting objects: 100% (592/592), done.        \n'

Output with working progress bar (v3.1.40):

remote: Enumerating objects: 150, done.        



remote: Counting objects:   0% (1/150)        



remote: Counting objects:   1% (2/150)        



remote: Counting objects:   2% (3/150)        



remote: Counting objects:   3% (5/150)        



remote: Counting objects:   4% (6/150)

@fvalette-ledger
Copy link
Contributor Author

which to my knowledge use the same underlying engine

underlying engine is the same, but input stream parsed is not.

@fvalette-ledger
Copy link
Contributor Author

Okay, seems that i found something interresting. I'll push a pull request (w/ working progress bar)

fvalette-ledger added a commit to fvalette-ledger/GitPython that referenced this issue Oct 14, 2024
See gitpython-developers#1969

stderr parser call RemoteProgress update on each line received.
With universal_newlines set to False, there is a mixup between
line feed and carriage return.
In the `handle_process_output` thread, this is thus seen as a single
line for the whole output on each steps.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants