feat: Increase file size limit from 25GB to 50.1GB #396
Conversation
nikita-smetanin
left a comment
nikita-smetanin
left a comment
There was a problem hiding this comment.
Hi Soroush, PR looks nice, I left a few suggestions :)
| chunk_size = 8192 | ||
| with file.open(encoding="utf-8") as f: | ||
| f.read() | ||
| for chunk in iter(lambda: f.read(chunk_size), ""): |
There was a problem hiding this comment.
I think just do for line in f: pass would be simpler and you won't have to specified chunk size manually (it'll use line buffering but you can override it if you think we should use larger buffers). I'd also comment we just do dry-run to let file reader decode file in utf8
|
|
||
| # Submit next part if available | ||
| if part_index < len(parts): | ||
| part_info = parts[part_index] |
There was a problem hiding this comment.
Would be great to rewrite it to deduplicate this code piece with the one above. I think you can either make a for loop to submit tasks and wait on result if we have enough already, or use executor.map with buffersize to limit concurrent tasks.
| TARGET_PART_SIZE_MB = 100 # Target part size for optimal performance | ||
| MAX_MULTIPART_PARTS = 250 # Maximum parts per upload (S3 limit) | ||
| TARGET_PART_SIZE_MB = 250 # Target part size | ||
| MAX_MULTIPART_PARTS = 250 # Maximum parts per upload |
There was a problem hiding this comment.
I see you dropped "S3 limit" mention, is it still the case?
Have you read the Contributing Guidelines?
Issue #
Describe your changes
Clearly and concisely describe what's in this pull request. Include screenshots, if necessary.
Note
Increases max file size to 50.1GB, switches multipart target part size to 250MB with sliding-window concurrency, adds progress bars to validation, and sets a download request timeout.
TARGET_PART_SIZE_MBto250; keepMAX_MULTIPART_PARTSat250.MultipartUploadManager._upload_parts_concurrentto respectmax_concurrent_partswhile continuously feeding new parts.MAX_FILE_SIZE_GB = 50.1.request_timeout=3600to raw GET inDownloadManager.download.tqdmfor progress; importtqdm.as_completedmocking to align with sliding-window behavior.Written by Cursor Bugbot for commit 7ac3d44. This will update automatically on new commits. Configure here.