Skip to content

Commit

Permalink
Merge pull request #25 from findmyway/rever_part_of_24
Browse files Browse the repository at this point in the history
add back record_type in read
  • Loading branch information
findmyway authored Aug 30, 2021
2 parents 7bb23e0 + aed4e37 commit 6b12b4a
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 3 deletions.
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "TFRecord"
uuid = "841416d8-1a6a-485a-b0fc-1328d0f53d5e"
authors = ["Jun Tian <[email protected]> and contributors"]
version = "0.4.0"
version = "0.5.0"

[deps]
BufferedStreams = "e1450e63-4bb3-523b-b2a4-4ffa8c0fd77d"
Expand Down
7 changes: 5 additions & 2 deletions src/core.jl
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ Read tensorflow records from file(s).
- `compression=nothing`. No compression by default. Optional values are `:zlib` and `:gzip`.
- `bufsize=10*1024*1024`. Set the buffer size of internal `BufferedOutputStream`. The default value is `10M`. Suggested value is between `1M`~`100M`.
- `channel_size=1000`. The number of pre-fetched elements.
- `record_type=Example`, see https://github.com/JuliaReinforcementLearning/TFRecord.jl/pull/11
!!! note
Expand All @@ -61,15 +62,17 @@ function read(
compression = nothing,
bufsize = 10 * 1024 * 1024,
channel_size = 1_000,
record_type = Example,

)
file_itr(file::AbstractString) = [file]
file_itr(files) = files
Channel{Example}(channel_size) do ch
Channel{record_type}(channel_size) do ch
@threads for file_name in file_itr(files)
open(decompressor_stream(compression), file_name, "r") do io
buffered_io = BufferedInputStream(io, bufsize)
while !eof(buffered_io)
instance = readproto(IOBuffer(read_record(buffered_io)), Example())
instance = readproto(IOBuffer(read_record(buffered_io)), record_type())
put!(ch, instance)
end
end
Expand Down

2 comments on commit 6b12b4a

@findmyway
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JuliaRegistrator
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registration pull request created: JuliaRegistries/General/43774

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.5.0 -m "<description of version>" 6b12b4a3c1396764eb167f5f786a15db8337b328
git push origin v0.5.0

Please sign in to comment.