From 6c80a138b1738bb619f3735c6ef1ab88a3a763fa Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Fri, 21 Jul 2023 04:26:55 +0000 Subject: [PATCH] build based on dda0318 --- dev/api/index.html | 12 ++++++------ dev/index.html | 2 +- dev/search/index.html | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/dev/api/index.html b/dev/api/index.html index aef02e9..6d62f27 100644 --- a/dev/api/index.html +++ b/dev/api/index.html @@ -1,5 +1,5 @@ -API · HuggingFaceDatasets.jl
Edit on GitHub

API

Index

Docs

HuggingFaceDatasets.DatasetDict — Type
DatasetDict(pydatasetdict::Py; transform = identity)

A DatasetDict is a dictionary of Datasets. It is a wrapper around a datasets.DatasetDict object.

The transform is applied to each Dataset. The py2jl transform provided by this package converts python types to julia types.

See also load_dataset and Dataset.

source
HuggingFaceDatasets.load_dataset — Method
load_dataset(args...; kws...)

Load a dataset from the HuggingFace Datasets library.

All arguments are passed to the python function datasets.load_dataset. See the documentation here.

Returns a DatasetDict or a Dataset depending on the split argument.

Use the dataset.with_format("julia") method to lazily convert the observation from the dataset to julia types.

Examples

Without a split argument, a DatasetDict is returned:

julia> d = load_dataset("glue", "sst2")
+API · HuggingFaceDatasets.jl
Edit on GitHub
API
Index
Docs
HuggingFaceDatasets.DatasetDict — Type
DatasetDict(pydatasetdict::Py; transform = identity)
A DatasetDict is a dictionary of Datasets.  It is a wrapper around a datasets.DatasetDict object.
The transform is applied to each Dataset.  The py2jl transform provided by this package converts python types to julia types.
See also load_dataset and Dataset.
source
HuggingFaceDatasets.load_dataset — Method
load_dataset(args...; kws...)
Load a dataset from the HuggingFace Datasets library.
All arguments are passed to the python function datasets.load_dataset. See the documentation here.
Returns a DatasetDict or a Dataset depending on the split argument.
Use the dataset.with_format("julia") method to lazily convert the observation from the dataset  to julia types.
Examples
Without a split argument, a DatasetDict is returned:
julia> d = load_dataset("glue", "sst2")
 DatasetDict({
     train: Dataset({
         features: ['sentence', 'label', 'idx'],
@@ -28,9 +28,9 @@
 julia> mnist[1]
 Dict{String, Any} with 2 entries:
   "label" => 5
-  "image" => Gray{N0f8}[Gray{N0f8}(0.0) Gray{N0f8}(0.0) … Gray{N0f8}(0.0) Gray{N0f8}(0.0); Gray{N0f8}(0.0) Gray{N0f8}(0.0) … Gray{N0f…
source
HuggingFaceDatasets.numpy2jl — Method
numpy2jl(x)
Convert a numpy array to a Julia array using DLPack. The conversion is copyless, and mutations to the Julia array are reflected in the numpy array.
source
HuggingFaceDatasets.with_format — Method
with_format(ds::Dataset, format)
Return a copy of ds with the format set to format. If format is "julia", the returned dataset will be transformed with py2jl and copyless conversion from python types  will be used when possible.
See also set_format!.
Examples
julia> ds = load_dataset("mnist", split="test");
+  "image" => Gray{N0f8}[Gray{N0f8}(0.0) Gray{N0f8}(0.0) … Gray{N0f8}(0.0) Gray{N0f8}(0.0); Gray{N0f8}(0.0) Gray{N0f8}(0.0) … Gray{N0f…
source
HuggingFaceDatasets.numpy2jl — Method
numpy2jl(x)
Convert a numpy array to a Julia array using DLPack. The conversion is copyless, and mutations to the Julia array are reflected in the numpy array.
source
HuggingFaceDatasets.with_format — Method
with_format(ds::Dataset, format)
Return a copy of ds with the format set to format. If format is "julia", the returned dataset will be transformed with py2jl and copyless conversion from python types  will be used when possible.
See also set_format!.
Examples
julia> ds = load_dataset("mnist", split="test");
 
 julia> ds[1]
 Python dict: {'image': <PIL.PngImagePlugin.PngImageFile image mode=L size=28x28 at 0x2B5B4C1F0>, 'label': 7}
@@ -40,5 +40,5 @@
 julia> ds[1]
 Dict{String, Any} with 2 entries:
   "label" => 7
-  "image" => UInt8[0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00; … ; 0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00]
source
HuggingFaceDatasets.with_format — Method
with_format(d::DatasetDict, format)
Return a copy of d with the format set to format. If format is "julia", the returned dataset will be transformed with py2jl and copyless conversion from python types will be used when possible.
source
HuggingFaceDatasets.with_jltransform — Method
with_jltransform(ds::Dataset, transform)
-with_jltransform(transform, ds::Dataset)
Return a copy of ds with the julia transform set to transform. The transform applies when indexing, e.g. ds[1] or ds[1:2].
The transform is always applied to a batch of data, even if the index is a single integer. That is, ds[1] is equivalent to ds[1:1] from the point of view of the transform.
The julia transform is applied after the python transform (if any).  The python transform can be set with ds.set_transform(pytransform).
If transform is nothing or identity, the returned dataset will not be transformed.
See also set_jltransform! for the mutating version.
source

+  "image" => UInt8[0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00; … ; 0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00]
source
HuggingFaceDatasets.with_format — Method
with_format(d::DatasetDict, format)

Return a copy of d with the format set to format. If format is "julia", the returned dataset will be transformed with py2jl and copyless conversion from python types will be used when possible.

source
HuggingFaceDatasets.with_jltransform — Method
with_jltransform(ds::Dataset, transform)
+with_jltransform(transform, ds::Dataset)

Return a copy of ds with the julia transform set to transform. The transform applies when indexing, e.g. ds[1] or ds[1:2].

The transform is always applied to a batch of data, even if the index is a single integer. That is, ds[1] is equivalent to ds[1:1] from the point of view of the transform.

The julia transform is applied after the python transform (if any). The python transform can be set with ds.set_transform(pytransform).

If transform is nothing or identity, the returned dataset will not be transformed.

See also set_jltransform! for the mutating version.

source
diff --git a/dev/index.html b/dev/index.html index e1aad4a..9360f5d 100644 --- a/dev/index.html +++ b/dev/index.html @@ -20,4 +20,4 @@ julia> train_data[1] Dict{String, Any} with 2 entries: "label" => 5 - "image" => UInt8[0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00; … ; 0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00] + "image" => UInt8[0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00; … ; 0x00 0x00 … 0x00 0x00; 0x00 0x00 … 0x00 0x00] diff --git a/dev/search/index.html b/dev/search/index.html index 9142360..f44e243 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · HuggingFaceDatasets.jl

Loading search...

    +Search · HuggingFaceDatasets.jl

    Loading search...