Skip to content

Latest commit

 

History

History
144 lines (103 loc) · 5.37 KB

README.md

File metadata and controls

144 lines (103 loc) · 5.37 KB
Raw

Fl Aggregator

Federated Learning (FL) Aggregator SyftBox API.

The fl_agregator API is featured in the tutorial Getting Started with FL on SyftBox. Read more here.

Usage

1. Install the API

git clone https://github.com/OpenMined/fl_aggregator

cp -r ./fl_aggregator <SYFTBOX_DATADIR>/apis/  # default: ~/SyftBox

Note: <SYFTBOX_DATADIR> refers to the SyftBox data directory, according to your SyftBox installation (default: $HOME/SyftBox).

2. Agree on roles in the FL flow

For example: Aggregator: <[email protected]>; Clients: <[email protected]>,<[email protected]>

  • Aggregator must decide and share the model architecture with the clients.
  • Aggregator will also provide a seed model weights which will be fine-tuned by each participant.

3. Setup the FL config

Create a fl_config.json configuration file including the following information:

  • "project_name": name assigned to the FL experiment.
  • "aggregator": designated aggregator datasite.
  • "participants": list of the designated client datasites.
  • "rounds": number of FL training rounds.
  • "model_arch": the Python module containing the ML model implementation.
  • "model_weight": model parameters file
  • "epoch": number of training epochs to run on each clients,
  • "learning_rate": learning rate of the optimizer.

Please see samples/launch_config/fl_config.json for an example configuration file:

{
    "project_name": "MNIST_FL",
    "aggregator": "[email protected]",
    "participants": ["[email protected]","[email protected]"],
    "rounds": 3,
    "model_arch": "model.py",
    "model_weight": "global_model_weight.pt",
    "epoch": 10,
    "learning_rate": 0.1
}

4. Start the FL experiment

  1. To start the FL experiment, the following files needs to be copied in <SYFTBOX_DATADIR>/datasites/<aggregator_email>/api_data/fl_aggregator/launch directory:
  • fl_config.json
  • global_model_weight.pt
  • model.py
  1. Copy the test dataset to <SYFTBOX_DATADIR>/datasites/<aggregator_email>/private/fl_aggregator

An example test dataset could be found in ./samples/test_data.

If this directory isn't available, either run the syftbox client with fl_aggregator API installed OR create it manually.

Once the files are in the launch folder, the API will create a folder named after the project_name specified in the file_config.json in the running folder.

Inside this folder the API will create a folder for each client datasite, where received updates in each round are gathered. Each of these folders will have a ._syftperm file containing the appropriate permissions granting read/write access to the folder.

Finally, the API send a request to each participant to join the FL flow.

Please see below an example of the resulting folders structure during the FL execution.

api_data
└── fl_aggregator
    ├── launch
    │   ├── config.json
    │   ├── model_arch.py
    │   ├── global_model_weights.py
    └── running
        └── my_cool_fl_proj
            ├── fl_clients 
            │   ├── [email protected]
            │   ├── [email protected]
            │   ├── [email protected]
            ├── agg_weights  # to store aggregator's weights for each round
            ├── config.json  # moved from the launch folder after the app start
            ├── global_model_weights.pt
            ├── model_arch.py  # moved from the launch folder
            └── state.json
    └── done
        └── my_cool_fl_proj
            └── aggregated_model_weights.pt

5. Monitoring the FL experiment

  • Visit the aggregator's dashboard to monitor the FL progress which is available at http://server_url/datasites/<aggregator_email>/fl/<project_name>/, with <project_name> matches the name of the FL experiment setup in the fl_config.json file.

The server_url depends on which the server the client is running on.

  • syftbox.openmined.org (for the public server)
  • localhost:5001 (when running locally, with default configuration)

Running in dev mode

⚠️ Make sure you have the latest version of the SyftBox repository cloned on your local machine:

git clone https://github.com/OpenMined/syft.git

Also, make sure you have just installed.

Note: The following commands need to be run within the syft directory (the repository root), unless otherwise indicated.

Set up:

First launch a local SyftBox server, and clients for all datasites participating in the FL flow:

  • just rs: run SyftBox local dev server on port 5001
  • just rc a: creates a SyftBox client for [email protected]. (Repeat this for all the clients)

Each of those commands need to be run in a separate terminal session, or within the same terminal using tools like tmux (recommended!).

Install the aggregator and the client APIs:

Aggregator:

  • git clone https://github.com/OpenMined/fl_aggregator
  • cp -R ./fl_aggregator ./clients/[email protected]/apis

Client(s):