Skip to content

SelfHosting.md

Latest commit

 

History

History
211 lines (144 loc) · 12 KB

SelfHosting.md

File metadata and controls

211 lines (144 loc) · 12 KB
Raw

🐂 Self Hosting

You can either setup an oxen-server instance yourself, or use the hosted version on OxenHub. To use the hosted OxenHub solution you can sign up here.

Setup an Oxen Server

To setup a local Oxen Server instance, first install the oxen-server binary.

Start the server:

$ oxen-server start

oxen-server does not authenticate requests itself. To require authentication, run it behind a reverse proxy or gateway that checks each request before forwarding it. If the gateway checks a bearer token, give each client its token with the config command, which keeps one token per host in ~/.config/oxen/auth_config.toml and sends it as Authorization: Bearer <TOKEN> on every request to that host:

$ oxen config --auth <HOST> <TOKEN>

The default directory that Oxen stores data is data (relative to the working directory the server was started from), which is convenient for trying things out but probably not what you want in production. To change it set the SYNC_DIR environment variable to an absolute path.

$ export SYNC_DIR=/Path/To/Data
$ oxen-server start

Running 🐂 server on 0.0.0.0:3000
Syncing to directory: /Path/To/Data
[2022-06-08T10:00:48Z INFO  actix_server::builder] Starting 8 workers
[2022-06-08T10:00:48Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime

If you want to change the default IP ADDRESS and PORT you can do so by passing them in with the -i and -p parameters.

$ oxen-server start -i 0.0.0.0 -p 4321

Storage Configuration

Oxen has two TOML config files that govern where repository data is stored:

  1. A server-wide config (optional, loaded via --config <path>) that decides which storage backends the server will host new repos on and supplies admin-only S3 settings. This is the one an admin writes.
  2. A per-repo .oxen/config.toml inside each server-hosted repository, written by the server when it creates the repo. On a server this file is state the server owns rather than a place to configure anything.

These are independent files. The server-wide config is read once at startup, and the per-repo config is read every time the server opens a repo.

Server-wide config (--config <path>)

The --config flag is optional. When omitted, the server uses a conservative default: only the local-filesystem backend is enabled.

$ oxen-server start --config /etc/oxen/server.toml

The only table a self-hosted server configures is [storage], and it is optional. A table the server does not recognize stops startup with an error rather than being skipped, so a misspelled section cannot leave the server quietly running a default you meant to replace.

[storage]
backends = ["local", "s3"]   # required; first element is the server default
s3_bucket = "my-oxen-bucket" # required iff "s3" is in backends
s3_region = "us-west-1"      # required iff "s3" is in backends
  • backends — a list of which storage kinds this server is willing to host. The valid values are "local" (local-filesystem version storage) and "s3" (S3 version storage), spelled in lowercase. The first element is the server's default: when a client creates a new repo without specifying a kind, the server uses that. Each kind must appear at most once; the list must be non-empty.
  • s3_bucket — the S3 bucket the server uses for any S3-backed repo. Required when "s3" appears in backends and rejected when it doesn't. Each repo gets the prefix repo/{repo_uuid}/ inside this bucket, named for the repo's UUID in its config, so renaming a repo or moving it to another namespace leaves its objects where they are. The prefix is not configurable per repo.
    • s3_region — the AWS region the bucket lives in (e.g. us-west-1). Required when "s3" appears in backends. The server uses it to build the S3 client directly rather than detecting it at runtime; if it's wrong, startup fails when the bucket reachability check runs.

Omitting the [storage] section entirely is equivalent to:

[storage]
backends = ["local"]

The server validates the config at startup and exits with a clear error if anything is wrong (empty backends, duplicate entries, S3 listed without s3_bucket or s3_region, etc.). When the S3 backend is enabled it also probes the configured bucket at startup (a HeadBucket call) and refuses to boot if the bucket is unreachable — wrong region, missing credentials, or no permission. It will not run with an invalid config.

A few example configurations:

# Local-only (the default; no config file needed)
[storage]
backends = ["local"]
# S3-only — new repos go to S3 by default
[storage]
backends = ["s3"]
s3_bucket = "oxen-prod-versions"
s3_region = "us-west-1"
# Both backends enabled, local is the server default; clients
# can request "s3" on the wire when they create a repo
[storage]
backends = ["local", "s3"]
s3_bucket = "oxen-prod-versions"
s3_region = "us-west-1"
# Both backends enabled, s3 is the server default
[storage]
backends = ["s3", "local"]
s3_bucket = "oxen-prod-versions"
s3_region = "us-west-1"

S3 backends require valid AWS credentials in the server's environment, picked up by the AWS SDK in the usual way (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / ~/.aws/credentials / instance role / etc.). The bucket region comes from s3_region in the config above.

Moving a repository between local storage and S3

A repository moves between the local backend and the configured bucket over HTTP, with kind set to "s3" or "local". The server refuses a kind missing from backends:

curl -X PUT "http://localhost:3000/api/repos/<namespace>/<repo_name>/storage" \
  -H "Content-Type: application/json" -d '{"kind": "s3"}'

The server copies the repository's version files to the other backend while pushes continue, then holds writes to that repository only while it copies what arrived since and switches it. It then deletes the copies the backend it left held, with pushes continuing. A move that stops part-way is finished by running it again, since a file the other backend already holds is skipped, and a move to the backend a repository is already on does nothing. A move requested while another move of the same repository is still running is refused with HTTP 409. A move that finds the repository's storage changed by another operation while it copied stops with HTTP 409, and running it again finishes it. Moving to S3 needs the repository to record a repo_uuid (see the [identity] section). A move to local storage puts the version files under <repo>/.oxen/versions/files.

Per-repo .oxen/config.toml

Every field the server keeps in <repo>/.oxen/config.toml is written by the server: min_version, vnode_size, storage, merkle_node_backend, and identity. Hand-editing them corrupts the repo. The exception is vnode_size, which no command exposes, so editing the file is the only way to change it. Moving a repository to another namespace goes through PATCH /{namespace}/{repo_name}/transfer rather than through this file.

The same file format also carries remotes, subtree paths, clone depth, and workspaces. Those are the fields that make this file look user-facing, and they appear only in a client working tree's copy, never in a server-hosted repo.

When the server creates a repository, it writes a small [storage] section recording which kind of storage that repo uses. The shape is:

[storage]
kind = "local"                          # or "s3"
versions_path = "/mnt/nfs/oxen-data"    # optional, local backend only
  • kind — "local" or "s3" (lowercase; these are the only valid values today). Set by the server at repo-creation time according to its policy and what the client requested, and changed afterwards only by a move between backends. Do not change this by hand; the server expects the backend kind to match the data on disk (for local) or in the bucket (for s3).
  • versions_path (local backend only) — overrides where the local backend stores version blobs for this repo. By default, version blobs live under <repo>/.oxen/versions/files. If versions_path starts with .oxen, it is interpreted as relative to the repo's root (and so the repo stays portable if the directory is moved); otherwise it is taken as an absolute path. Useful for pointing version storage at an NFS mount or another volume while keeping the rest of the repo's metadata in place.

Legacy [storage.settings] fallback

Repositories created by older versions of oxen-server used a different shape:

[storage]
type = "local"

[storage.settings]
path = "/mnt/nfs/oxen-data"

oxen-server continues to read the legacy form on every load, so an upgrade does not force a migration and no admin action is required. The legacy type key is ignored, leaving kind at its local default, and [storage.settings] path is promoted to versions_path.

The file is opportunistically rewritten into the new shape the next time the server has another reason to save the repo's config. In practice that means after a namespace transfer; if that never happens, the legacy keys stay on disk indefinitely and the repo continues to work normally.

The [identity] section

Repositories the server creates also carry an [identity] section:

[identity]
repo_uuid = "0c0e2a8e-93a4-4a9f-9f1e-2b9b3f0f9a11"
namespace = "my-namespace"
name = "my-repo"
  • repo_uuid — the repository's immutable identity, written when the repo is created and never changed afterwards, including when the repo moves to another namespace. Do not edit it by hand.
  • namespace / name — the human-readable names, recorded so a repository directory can be identified without consulting anything else. They track renames and carry no authority: nothing is addressed by them.

A repository the server creates lives at $SYNC_DIR/repo/{uuid[0:2]}/{uuid[2:4]}/{repo_uuid}, named for its repo_uuid, and stays there when it moves to another namespace.

oxen-server refuses to create a repository under a name another repository already holds, matching without regard to case, so my-namespace/My-Repo and my-namespace/my-repo cannot both be created.

The name in each [identity] section is indexed when the server starts, so that refusal covers every repository whose config records one. Stop the server and run oxen-server seed-name-table to index the names the table is missing after restoring or otherwise changing the sync directory behind the server's back. It only adds names, so a repository directory removed by hand keeps its name taken, and the command reports those names as unclaimed.

Pushing the Changes

Once you have committed data locally and are ready to share them with colleagues (or the world) you will have to push them to a remote.

You can either create a remote through the web UI on OxenHub or if you have setup a server your self, you will have to run the create-remote command.

$ oxen create-remote --name MyNamespace/MyRepoName --host 0.0.0.0:3001 --scheme http

Repositories that live on an Oxen Server have the idea of a namespace and a name to help you organize your repositories.

Once you know your remote repository URL you can add it as a remote. Setting a remote reads the repository from the server and records the identifier it reports, so the server has to be reachable and the repository has to exist by this point.

$ oxen config --set-remote origin http://<HOST>/MyNamespace/MyRepoName

Once a remote is set you can push

$ oxen push origin main

You can change the remote (origin) and the branch (main) to whichever remote and branch you want to push.

Clone the Changes

To clone a repository from remote server you can use the URL you provided previously, and pull the changes to a new machine.

$ oxen clone http://<HOST>/MyNamespace/MyRepoName