Skip to content

Commit

Permalink
Cherry pick commits from PRs 1088, 1100, 1120, 1140, and 1145 for doc…
Browse files Browse the repository at this point in the history
…umentation updates (#1153)
  • Loading branch information
nvkevlu authored Nov 17, 2022
1 parent af3e20b commit 311ab02
Show file tree
Hide file tree
Showing 21 changed files with 309 additions and 597 deletions.
27 changes: 27 additions & 0 deletions docs/examples/xgboost.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
Federated XGBoost
=================


Overview
--------

NVFlare supports federated learning using popular gradient boosting library XGBoost.
It uses XGBoost library with federated plugin (xgboost version >= 1.7.0rc1) to perform the learning.

Using XGBoost with NVFlare has following benefits compared with running federated XGBoost directly,

* XGBoost instance's life-cycle is managed by NVFlare. Both XGBoost client and server
are started/stopped automatically by NVFlare workflow.
* For histogram-based XGBoost federated server can be configured automatically with auto-assigned port number.
* When mutual TLS is used, the certificates are managed by NVFlare using existing
provisioning process.
* No need to manually configure each instance. Instance specific parameters
like code:`rank` are assigned automatically by the NVFlare controller.

Examples
--------

Basic components to run XGBoost are already included with NVFlare distribution.
Most XGBoost jobs can be created without custom code.

Please refer to :code:`NVFlare/examples/xgboost` for more details.
2 changes: 1 addition & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ NVIDIA FLARE is built on a componentized architecture that gives you the flexibi

Getting Started
===============
For first-time users and FL researchers, FLARE provides the FL Simulator that allows you to build, test, and deploy applications locally. The :ref:`Getting Started guide <getting_started>` covers installation and walks through an example application using the FL Simulator.
For first-time users and FL researchers, FLARE provides the :ref:`fl_simulator` that allows you to build, test, and deploy applications locally. The :ref:`Getting Started guide <getting_started>` covers installation and walks through an example application using the FL Simulator.

When you are ready to for a secure, distributed deployment, the :ref:`Real World Federated Learning <real_world_fl>` section covers the tools and process required to deploy and operate a secure, real-world FLARE project.

Expand Down
61 changes: 61 additions & 0 deletions docs/programming_guide/provisioning_system.rst
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,67 @@ A new builder to write 'gateway.conf' can be implemented as follows (for referen
f.write(f"name = {gw.name}\n")
f.write(f"port = {port}\n")

.. _distribution_builder:

Case 4: adding a builder for enabling the creation of zip archives for the startup kits
---------------------------------------------------------------------------------------
DistributionBuilder was included in NVIDIA FLARE before version 2.2.1 but has been removed from the
default builders. You can make this builder available and add it as a builder in project.yml if you want to zip the startup kits::

import os
import shutil
import subprocess

from nvflare.lighter.spec import Builder, Project
from nvflare.lighter.utils import generate_password

class DistributionBuilder(Builder):
def __init__(self, zip_password=False):
"""Build the zip files for each folder.
Creates the zip files containing the archives for each startup kit. It will add password protection if the
argument (zip_password) is true.
Args:
zip_password: if true, will create zipped packages with passwords
"""
self.zip_password = zip_password

def build(self, project: Project, ctx: dict):
"""Create a zip for each individual folder.
Note that if zip_password is True, the zip command will be used to encrypt zip files. Users have to
install this zip utility before provisioning. In Ubuntu system, use this command to install zip utility:
sudo apt-get install zip
Args:
project (Project): project instance
ctx (dict): the provision context
"""
wip_dir = self.get_wip_dir(ctx)
dirs = [
name
for name in os.listdir(wip_dir)
if os.path.isdir(os.path.join(wip_dir, name)) and "nvflare_" not in name
]
for dir in dirs:
dest_zip_file = os.path.join(wip_dir, f"{dir}")
if self.zip_password:
pw = generate_password()
run_args = ["zip", "-rq", "-P", pw, dest_zip_file + ".zip", ".", "-i", "startup/*"]
os.chdir(dest_zip_file)
try:
subprocess.run(run_args)
print(f"Password {pw} on {dir}.zip")
except FileNotFoundError:
raise RuntimeError("Unable to zip folders with password. Maybe the zip utility is not installed.")
finally:
os.chdir(os.path.join(dest_zip_file, ".."))
else:
shutil.make_archive(dest_zip_file, "zip", root_dir=os.path.join(wip_dir, dir), base_dir="startup")

If the above code is made available at ``nvflare.lighter.impl.workspace.DistributionBuilder``, add the following to your project.yml at the bottom of the list of builders::

path: nvflare.lighter.impl.workspace.DistributionBuilder
args:
zip_password: true

Takeaways for Custom Builders
-----------------------------
From the cases shown previously, implementing your own Builders only requires the following steps:
Expand Down
4 changes: 4 additions & 0 deletions docs/publications_and_talks.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ NVIDIA FLARE related blogs and other media.

#### 2019
* **2019-12** [Federated Learning powered by NVIDIA Clara](https://developer.nvidia.com/blog/federated-learning-clara/) (NVIDIA Technical Blog)
* **2019-10** [What is federated learning - in Chinese](https://blogs.nvidia.com.tw/2019/10/13/what-is-federated-learning/) (NVIDIA Technical Blog)
* **2019-10** [NVIDIA Research: First Privacy-Preserving Federated Learning System for Medical Imaging](https://www.youtube.com/watch?v=Jy7ozgwovgg) (NVIDIA video)

## Talks
Expand All @@ -58,3 +59,6 @@ Recent talks and Webinars covering federated learning research and NVIDIA FLARE.
* **2021-09** [Federated Learning](https://www.youtube.com/watch?v=YeYO4JGTBb0&amp) ([MONAI MICCAI Bootcamp 2021](https://www.gpuhackathons.org/event/monai-miccai-bootcamp-2021))
* **2021-03** [NVIDIA FLARE: An Open Federated Learning Platform](https://www.nvidia.com/en-us/on-demand/session/gtcspring22-se1991/) ([GTC Spring 2022](https://www.nvidia.com/gtc/))
* **2021-03** [Federated Learning for Healthcare – Collaborative AI without Sharing Patient Data ](https://www.youtube.com/watch?v=xr_eJp3ctzw) ([Data Science Seminar](https://www.dkfz.de/en/datascience/seminar/Rieke.html))

#### 2020
* **2020-11** [Federated Learning for Medical Imaging - in Chinese](https://www.youtube.com/watch?v=CiPdALrNEjU) (NVIDIA Taiwan)
5 changes: 5 additions & 0 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
##########
Quickstart
##########

See :ref:`quickstart`.
7 changes: 4 additions & 3 deletions docs/real_world_fl.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ This section shows how to use NVIDIA FLARE to deploy and operate an FL system.
A reference application will be used here to show provisioning and basic operation
of the system through the admin client. You will find information here about setting up the
system with all the available components and the Admin API for
operating FL. For more details on what you can do with apps with custom components and
operating FL. For instructions on how to set up the :ref:`nvflare_dashboard_ui` added in 2.2.1 to
help gather information to provision a project and distribute startup kits, see :ref:`dashboard_api`.
For more details on what you can do with apps with custom components and
the flexibility that the Controller and Worker APIs bring, see the :ref:`programming_guide`.

You can also see some `example applications <https://github.com/NVIDIA/NVFlare/tree/main/examples>`_ integrating with
Expand All @@ -24,5 +26,4 @@ to see the capabilities of the system and how it can be operated.
real_world_fl/application
real_world_fl/job
real_world_fl/workspace
real_world_fl/authorization

user_guide/federated_authorization
9 changes: 4 additions & 5 deletions docs/real_world_fl/application.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,16 +107,15 @@ the client config should have the following in order to configure it as an Execu

Configuration of Executor Tasks is ignored here.

Please follow :ref:`quickstart:Quickstart` to learn more.
Please follow :ref:`quickstart` to learn more.

.. _troubleshooting_byoc:

Troubleshooting BYOC
====================
There is an ``enable_byoc`` flag for each participant that can be set at provisioning, and if that is disabled, even if
you have custom code in your application folder, it will not be loaded. There is also a setting for ``allow_byoc``
through the authorization rule groups. This controls whether or not apps containing BYOC code will be allowed to be
uploaded and deployed.
In 2.2.1, authorization has been redesigned and BYOC is no longer controlled through settings at provisioning, but
instead by each site's authorization.json (in the local folder of the workspace). BYOC is a right and can be restricted
to certain roles or even orgs or users. See :ref:`federated_authorization` for details.

*********
Resources
Expand Down
Loading

0 comments on commit 311ab02

Please sign in to comment.