vfs provides a unified interface for working with different storage backends (local OS, Azure, S3, GCS, SFTP, etc.) in a consistent way. Rather than handling each backend independently, you interact with a core set of interfaces that abstract backend details. This allows you to:
- Seamlessly read, write, and manage files across multiple storages.
- Treat remote or cloud-based paths as if they were local (e.g. using standard
io.Reader
/io.Writer
interfaces). - Simplify environment-agnostic development by keeping most logic backend-agnostic.
- Plug in or mock storage functionality to facilitate testing and modular application design.
By focusing on FileSystem, Location, and File interfaces, you can build reusable flows for file operations without needing to deeply understand each backend’s specific APIs. Users can add or swap backends as needed, providing flexibility for hybrid or evolving storage requirements.
go get -u github.com/c2fo/vfs/v7
package main
import (
"fmt"
"strings"
"io"
"log"
"github.com/c2fo/vfs/v7"
"github.com/c2fo/vfs/v7/vfssimple"
)
func main() {
// Create local OS file from a URI
osFile, err := vfssimple.NewFile("file:///tmp/example.txt")
if err != nil {
log.Fatal(err)
}
defer osFile.Close()
// Write to the file
_, err = io.Copy(osFile, strings.NewReader("Hello from vfs!"))
if err != nil {
log.Fatal(err)
}
if err := osFile.Close(); err != nil {
log.Fatal(err)
}
fmt.Println("File created and written:", osFile.URI())
}
This snippet shows the basic setup: an osFile is created from a URI and written to using standard libraries.
-
See the GoDoc for deeper technical details, backend-specific options, and additional APIs.
-
Specialized operations like copying between S3 locations or advanced configs (e.g. Azure credentials) are documented in each backend doc.
Q: Why am I seeing an empty file when using io.Copy
on some backends if my source is empty?
A: An empty Reader
often means the backend doesn't write a file until data is actually written. Use utils.TouchCopy
if you need to ensure a zero-byte file is created.
Q: Will vfs v6 still be supported? A: Yes and no. We will continue to provide security patches and bug fixes for v6, but new features and enhancements will be added to v7.
Q: How long will v6 be supported? A: We will support v6 until the next major release, v8, is releaseed. Then v7 will be supported until v9 is released.
Please review these changes and update your code accordingly to ensure compatibility with v7.
The project now uses the aws-sdk-go-v2
library instead of the deprecated, EOL aws-sdk-go
. This update necessitated
these changes to the S3 backend:
- The S3 backend's
s3fs.Client()
function now returns ans3.Client
which is a subset of AWS's sdk v2 functionality. This change may require updates to your code if you were relying on client functionality not directly required by the s3 vfs backend. - The
Option.Retry
field is now anaws.Retryer
instead of arequest.Retry
. Ensure that your Option logic is compatible with the new type.
- Scheme for Azure has been updated from
https
toaz
. Update your code to use the new scheme. - Authority for Azure has been updated from
blob.core.windows.net
tomy-container-name
, such that the full URI isaz://my-blob-name/path/to/file.txt
rather thanhttps://my-storage-account-name.core.windows.net/my-container-name/path/to/file.txt
.
- The
Options.Retry
field, with the now deprecatedvfs.Retry
type, has been moved togs.FileSystem
as the newgs.Retyer
type. It is now set viags.NewFileSystem
function using functional optiongs.WithRetryer
. All previousvfs.Retry
usage has been replaced withgs.WithRetryer
. Update your code to use the newgs.WithRetryer
functional option.
Some methods in the Location and FileSystem interfaces have been deprecated because they use terminology that doesn't apply to all backends. They will be removed in a future release. Update your code to use the new methods. See #235.
location.Volume()
method which returns the authority as a string has been deprecated in favor of thelocation.Authority()
method which returns anauthority.Authority
struct. Update your code to use theAuthority().String()
method instead ofVolume()
.location.ChangeDir()
method ash been deprecated in favor of the existinglocation.NewLocation()
method. Update your code to use theNewLocation()
method instead ofChangeDir()
.vfs.Options
struct has been deprecated in favor of using backend-specific structs.filesystem.Retry()
method has been deprecated in favor of using backend-specific functional options, such asgs.WithRetryer
.
Additionally, we have added functional option interface, FileSystemOption
, to allow for more flexible configuration
of backends. This interface allows for more complex configuration options to be passed to the via the NewFileSystem
function.
This will replace backend-specific chainable functions that require casting the filesystem to the backend type first. See #238.
With v6
, sftp.Options struct changed to accept an array of Key Exchange algorithms rather than a string. To update,
change the syntax of the auth commands.
"keyExchanges":"diffie-hellman-group-a256"
becomes
"keyExchanges":["diffie-hellman-group-a256"]
- Fork it (
git clone https://github.com/c2fo/vfs.git
) - Create your feature branch (
git checkout -b feature/fooBar
) - Commit your changes (
git commit -am "Add some fooBar"
) - Push to the branch (
git push origin feature/fooBar
) - Create a new Pull Request
Distributed under the MIT License. See the LICENSE.md file for details.