-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for using the ObjectStore pattern #78
Conversation
validate bucket names
Put a file in a bucket
add ability to list files
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good so far!
Get object
support deleting an object from a bucket
@@ -18,7 +18,7 @@ jobs: | |||
pair: | |||
- otp: "24.3.4" | |||
elixir: "1.12" | |||
nats: "2.2.0" | |||
nats: "2.3.0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NATS 2.2.0 doesn't support a PURGE with filter: "my subject"
which is needed to support deleting objects from a bucket. So I had to bump this to 2.3.0
@brandynbennett this is ready for a review now. A couple of things I'm wondering about:
|
Since it's still a few hours before the US team is online, I decided to go ahead and release 0.0.8-alpha1 so I could test these changes |
test/jetstream/api/object_test.exs
Outdated
# create a random 2MB binary file | ||
# re-use it on subsequent test runs if it already exists | ||
defp generate_big_file do | ||
filepath = Path.join("tmp", "big_file.bin") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we us the ExUnit tmp_dir here?
@@ -24,3 +24,6 @@ jetstream-*.tar | |||
|
|||
# ASDF tool versions | |||
.tool-versions | |||
|
|||
# tmp directory to test files | |||
tmp |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we us the ExUnit tmp_dir here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 fixed
lib/jetstream/api/stream.ex
Outdated
:ok | ||
|
||
""" | ||
@spec purge(conn :: Gnat.t(), stream_name :: binary()) :: :ok | {:error, any()} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
spec is missing method
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 fixed
@stream_prefix "OBJ_" | ||
@subject_prefix "$O." | ||
|
||
def create_bucket(conn, bucket_name, params \\ []) do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I could see the case for making an Object.Bucket
module to separate responsibilities. I think from an interface perspective I would still want to delegate so calling Object.create_bucket
is the public interface.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 I can get behind that. I'll keep create_bucket
and delete_bucket
as-is. And Then change the rest of the functions to skip the _object
since that focus of this API is to work with the files in a bucket.
I temporarily installed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea about Dialyxir pr
Coneptual documentation: https://docs.nats.io/nats-concepts/jetstream/obj_store/obj_walkthrough
ADR: https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-20.md
Similar to the Key-Value functionality, this is not actually a specific thing built into the nats-server, but a documented pattern for how to store chunked files. This allows for efficient storage/streaming of larger files. I'll be modeling my changes from the ADR notes as well as the time-honored tradition of doing
nats-server -js -DV
and reverse engineering the logs that are generated when using thenats
command-line tool for creating buckets, storing objects, retrieving objects, etc.I'm going to start a checklist of functionality I want to cover before we merge/release this API