Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: replace StreamReader/Writer with FileReader/Writer #1193

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open

feat: replace StreamReader/Writer with FileReader/Writer #1193

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
7a40dbb
feat: replace `StreamReader`/`StreamWriter` with `FileReader`/`FileWr…
de-sh Feb 17, 2025
4b3f3df
refactor: can't err
de-sh Feb 17, 2025
d860819
Merge branch 'main' into filewriter
de-sh Feb 19, 2025
fc6d769
Merge remote-tracking branch 'origin/main' into filewriter
de-sh Feb 20, 2025
e65d4e6
fix: query logs from on-disk arrow files
de-sh Feb 20, 2025
a829b9e
fix: `create_dir_all`
de-sh Feb 20, 2025
261fd66
fix: convert only finished arrow files
de-sh Feb 20, 2025
01abc11
fix: limit to 1684 rows per arrow file
de-sh Feb 20, 2025
ceb0659
test: no `manual_write` and fix read order
de-sh Feb 20, 2025
3398a11
Merge branch 'main' into filewriter
de-sh Feb 20, 2025
0e29387
log: don't log "string files"
de-sh Feb 21, 2025
920dc40
Merge branch 'main' into filewriter
nikhilsinhaparseable Feb 21, 2025
6fc59d5
fix: increment row count
de-sh Feb 21, 2025
240eb99
fix: don't drop in-memory records
de-sh Feb 21, 2025
76c7d58
fix: `.arrows` naming convention
de-sh Feb 21, 2025
acb5ede
fix: actually replace with new file
de-sh Feb 21, 2025
06524b6
ci: clippy suggestion
de-sh Feb 21, 2025
224574a
fix: properly group arrows into parquet
de-sh Feb 21, 2025
7b8dca1
file type is known
de-sh Feb 21, 2025
c9ff7a7
refactor: simplify in-mem threshold handling
de-sh Feb 21, 2025
8c2407d
don't panic!
de-sh Feb 21, 2025
487655e
fix: panic on unexpected filename structure
de-sh Feb 21, 2025
e957173
fix: consider custom partitions
de-sh Feb 21, 2025
76b8d2c
doc: custom parition limits
de-sh Feb 21, 2025
1a75118
doc: improve explainers
de-sh Feb 21, 2025
9c10041
doc: `arrow_path_to_parquet`
de-sh Feb 21, 2025
2bb93ec
revert: all writes into a single file
de-sh Feb 22, 2025
913979a
limit record_batch row count
de-sh Feb 22, 2025
a4d5e3d
Merge remote-tracking branch 'origin/main' into filewriter
de-sh Feb 28, 2025
78a141e
fix: don't count `part.arrows`
de-sh Feb 28, 2025
b3037ff
match with regex
de-sh Feb 28, 2025
76fda8b
Merge remote-tracking branch 'origin/main' into filewriter
de-sh Mar 14, 2025
3cd83ef
fix: reverse ordering of rbs
de-sh Mar 14, 2025
c48499e
clippy suggestion
de-sh Mar 14, 2025
d0159f1
style: coderabbit suggestion
de-sh Mar 14, 2025
e456d67
Merge remote-tracking branch 'origin/main' into filewriter
de-sh Mar 16, 2025
9622430
Merge branch 'main' into filewriter
de-sh Mar 16, 2025
ec31cfb
doc+test: improvements
de-sh Mar 16, 2025
51eabca
refactor: declutter by separating out into #1239
de-sh Mar 16, 2025
db5c90d
Merge remote-tracking branch 'origin/main' into filewriter
de-sh Mar 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/parseable/staging/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,6 @@ pub enum StagingError {
ObjectStorage(#[from] std::io::Error),
#[error("Could not generate parquet file")]
Create,
// #[error("Metadata Error: {0}")]
// Metadata(#[from] MetadataError),
#[error("Too many rows: {0}")]
RowLimit(usize),
}
Loading
Loading