Skip to content

Commit aadc974

Browse files
sownaksownak
authored
Merge pull request #2615 from alvaropicazo/feat_2600
feat(fabric): Support of add orderer node operation
2 parents f899b83 + 63cba5e commit aadc974

File tree

12 files changed

+288
-131
lines changed

12 files changed

+288
-131
lines changed

‎docs/source/guides/fabric/add-new-orderer-peer.md

+105-44
Original file line numberDiff line numberDiff line change
@@ -3,66 +3,127 @@
33
[//]: # (SPDX-License-Identifier: Apache-2.0)
44
[//]: # (##############################################################################################)
55

6-
<a name = "adding-new-orderer-to-existing-organization-in-a-running-fabric-network"></a>
7-
# Adding a new RAFT orderer to existing Orderer organization in Hyperledger Fabric
6+
# Add Orderer Node to an existing organization
87

9-
- [Prerequisites](#prerequisites)
10-
- [Modifying Configuration File](#modifying-configuration-file)
11-
- [Run playbook](#run-playbook)
8+
This guide explains how to add an orderer node to an existing Hyperledger Fabric network using two methods:
129

10+
1. Using the `add-orderer.yaml` playbook: This method involves running an Ansible playbook that automates the process of adding an orderer node to the network.
11+
12+
1. Using `helm install`: This method involves using the helm install command to directly install the orderer node chart.
1313

14-
<a name = "prerequisites"></a>
1514
## Prerequisites
16-
To add a new Orderer node, a fully configured Fabric network must be present already, i.e. a Fabric network which has Orderers, Peers, Channels (with all Peers already in the channels) and the organization to which the peer is being added. The corresponding crypto materials should also be present in their respective Hashicorp Vault.
1715

18-
---
19-
**NOTE**: Addition of a new Orderer node has been tested on an existing network which is created by Bevel. Networks created using other methods may be suitable but this has not been tested by Bevel team.
20-
This works only for RAFT Orderer.
16+
- A fully configured Fabric network with Orderers and Peers.
17+
- Corresponding crypto materials present in Hashicorp Vault or Kubernetes secrets.
18+
- Hyperledger Bevel configured.
19+
20+
## Method 1: Using the `add-cli.yaml` playbook
21+
22+
1. **Update Configuration File**
23+
24+
To add a new Orderer node, a fully configured Fabric network must be present already, i.e. a Fabric network which has Orderers, Peers, Channels (with all Peers already in the channels) and the organization to which the peer is being added. The corresponding crypto materials should also be present in their respective Hashicorp Vault.
25+
26+
---
27+
**NOTE**: Addition of a new Orderer node has been tested on an existing network which is created by Bevel. Networks created using other methods may be suitable but this has not been tested by Bevel team.
28+
This works only for RAFT Orderer.
29+
30+
---
31+
32+
1. **Update Configuration File**
33+
34+
A Sample configuration file for adding new orderer is available [here](https://github.com/hyperledger/bevel/blob/main/platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml). Please go through this file and all the comments there and edit accordingly.
35+
36+
For generic instructions on the Fabric configuration file, refer [this guide](../networkyaml-fabric.md).
37+
38+
While modifying the configuration file(`network.yaml`) for adding new peer, all the existing orderers should have `status` tag as `existing` and the new orderers should have `status` tag as `new` under `network.organizations` as
39+
40+
```yaml
41+
42+
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:126:135"
43+
..
44+
..
45+
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:174:174"
46+
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:185:220"
47+
48+
```
49+
and under `network.orderers` the new orderer must be added.
50+
51+
```yaml
52+
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:42:66"
53+
```
54+
55+
The `network.yaml` file should contain the specific `network.organization` details.
56+
57+
Ensure the following is considered when adding the new orderer on a different cluster:
58+
- The CA server is accessible publicly or at least from the new cluster.
59+
- The CA server public certificate is stored in a local path and that path provided in network.yaml.
60+
- There is a single Hashicorp Vault and both clusters (as well as ansible controller) can access it.
61+
- Admin User certs have been already generated and store in Vault (this is taken care of by deploy-network.yaml playbook if you are using Bevel to setup the network).
62+
- The `network.env.type` is different for different clusters.
63+
- The GitOps release directory `gitops.release_dir` is different for different clusters.
64+
65+
1. **Run playbook**
66+
67+
The [add-orderer.yaml](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/configuration/add-orderer.yaml) playbook is used to add a new peer to an existing organization in the existing network. This can be done using the following command
68+
69+
```
70+
ansible-playbook platforms/hyperledger-fabric/configuration/add-orderer.yaml --extra-vars "@path-to-network.yaml"
71+
```
72+
73+
---
74+
**NOTE:** The `orderer.status` is not required when the network is deployed for the first time but is mandatory for addition of new orderer.
75+
2176

22-
---
77+
## Method 2: Using `helm install`
2378

24-
<a name = "modifying-configuration-file"></a>
25-
## Modifying Configuration File
79+
1. **Update the orderernode values.yaml file**
2680

27-
A Sample configuration file for adding new orderer is available [here](https://github.com/hyperledger/bevel/blob/main/platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml). Please go through this file and all the comments there and edit accordingly.
81+
Following changes are must in the `values.yaml` file for a new orderer node to be added to the network:
2882

29-
For generic instructions on the Fabric configuration file, refer [this guide](../networkyaml-fabric.md).
83+
- `certs.settings.createConfigMaps: false` as the ConfigMaps for certs are already generated in the same namespace.
3084

31-
While modifying the configuration file(`network.yaml`) for adding new peer, all the existing orderers should have `status` tag as `existing` and the new orderers should have `status` tag as `new` under `network.organizations` as
85+
Refer to the [fabric-orderernode chart documentation](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/charts/fabric-orderernode) for a complete list of available configuration options.
3286

33-
```yaml
87+
1. **Install the orderernode chart**
88+
89+
Execute the following command to install the Peer chart:
90+
```bash
91+
helm dependency update ./fabric-orderernode
92+
helm install <release-name> ./fabric-orderernode --namespace <namespace> --values <values-file.yaml>
93+
```
94+
Replace the following placeholders:
3495

35-
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:126:135"
36-
..
37-
..
38-
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:174:174"
39-
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:185:220"
96+
- `<release-name>`: The desired name for the orderer node release.
97+
- `<namespace>`: The Kubernetes namespace where the Peer should be deployed.
98+
- `<values-file.yaml>`: The path to a YAML file containing the new peer configuration values.
4099

41-
```
42-
and under `network.orderers` the new orderer must be added.
100+
1. **Update the osnadmin-channel-create values.yaml file**
43101

44-
```yaml
45-
--8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:42:66"
46-
```
47-
48-
The `network.yaml` file should contain the specific `network.organization` details.
102+
Following changes are must in the `values.yaml` file for a new orderer node to be added to the network:
103+
```
104+
orderer:
105+
addOrderer: true
106+
name: orderer5
107+
localMspId: orgNameMSP
108+
ordererAddress: orderer1.orgname-net:443
109+
```
49110

50-
Ensure the following is considered when adding the new orderer on a different cluster:
51-
- The CA server is accessible publicly or at least from the new cluster.
52-
- The CA server public certificate is stored in a local path and that path provided in network.yaml.
53-
- There is a single Hashicorp Vault and both clusters (as well as ansible controller) can access it.
54-
- Admin User certs have been already generated and store in Vault (this is taken care of by deploy-network.yaml playbook if you are using Bevel to setup the network).
55-
- The `network.env.type` is different for different clusters.
56-
- The GitOps release directory `gitops.release_dir` is different for different clusters.
111+
Refer to the [fabric-osn-channel-create chart documentation](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/charts/fabric-osn-channel-create) for a complete list of available configuration options.
57112

58-
<a name = "run-playbook"></a>
59-
## Run playbook
113+
1. **Install the osnadmin-channel-create chart**
114+
115+
Execute the following command to install the fabric-osnadmin-channel-create chart:
116+
```bash
117+
cd ../..
118+
helm install <release-name> ./fabric-osnadmin-channel-create --namespace <namespace> --values <values-file.yaml>
119+
```
120+
Replace the following placeholders:
60121

61-
The [add-orderer.yaml](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/configuration/add-orderer.yaml) playbook is used to add a new peer to an existing organization in the existing network. This can be done using the following command
122+
- `<release-name>`: The desired name for the Peer release.
123+
- `<namespace>`: The Kubernetes namespace where the Peer should be deployed.
124+
- `<values-file.yaml>`: The path to a YAML file containing the new peer configuration values.
62125

63-
```
64-
ansible-playbook platforms/hyperledger-fabric/configuration/add-orderer.yaml --extra-vars "@path-to-network.yaml"
65-
```
66126

67-
---
68-
**NOTE:** The `orderer.status` is not required when the network is deployed for the first time but is mandatory for addition of new orderer.
127+
## Additional Notes
128+
- The `add-orderer.yaml playbook` and `helm install` method has been tested on networks created by Bevel. Networks created using other methods may be suitable, but this has not been tested by the Bevel team.
129+
- Ensure that the network.yaml file contains the specific network.organization details along with the orderer information.

‎platforms/hyperledger-fabric/charts/fabric-cli/templates/deployment.yaml

+44-2
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,26 @@ spec:
117117
fi
118118
}
119119

120+
function getAdminTlsSecret {
121+
KEY=$1
122+
123+
echo "Getting TLS certificates from Vault."
124+
vaultBevelFunc "readJson" "${VAULT_SECRET_ENGINE}/${VAULT_SECRET_PREFIX}/users/${KEY}"
125+
if [ "$SECRETS_AVAILABLE" == "yes" ]
126+
then
127+
OSN_TLS_CA_ROOT_CERT=$(echo ${VAULT_SECRET} | jq -r '.["ca_crt"]')
128+
ADMIN_TLS_SIGN_CERT=$(echo ${VAULT_SECRET} | jq -r '.["client_crt"]')
129+
ADMIN_TLS_PRIVATE_KEY=$(echo ${VAULT_SECRET} | jq -r '.["client_key"]')
130+
131+
echo "${OSN_TLS_CA_ROOT_CERT}" > ${OUTPUT_TLS_PATH}/tlsca.crt
132+
echo "${ADMIN_TLS_SIGN_CERT}" > ${OUTPUT_TLS_PATH}/server.crt
133+
echo "${ADMIN_TLS_PRIVATE_KEY}" > ${OUTPUT_TLS_PATH}/server.key
134+
ADMIN_TLS_SECRET=true
135+
else
136+
ADMIN_TLS_SECRET=false
137+
fi
138+
}
139+
120140
{{- else }}
121141

122142
function getAdminMspSecret {
@@ -140,21 +160,41 @@ spec:
140160
fi
141161
}
142162

163+
function getAdminTlsSecret {
164+
KEY=$1
165+
KUBENETES_SECRET=$(kubectl get secret ${KEY} --namespace {{ .Release.Namespace }} -o json)
166+
if [ "$KUBENETES_SECRET" = "" ]; then
167+
ADMIN_TLS_SECRET=false
168+
else
169+
OSN_TLS_CA_ROOT_CERT=$(echo ${KUBENETES_SECRET} | jq -r '.data.ca_crt' | base64 -d)
170+
ADMIN_TLS_SIGN_CERT=$(echo ${KUBENETES_SECRET} | jq -r '.data.client_crt' | base64 -d)
171+
ADMIN_TLS_PRIVATE_KEY=$(echo ${KUBENETES_SECRET} | jq -r '.data.client_key' | base64 -d)
172+
173+
echo "${OSN_TLS_CA_ROOT_CERT}" > ${OUTPUT_TLS_PATH}/tlsca.crt
174+
echo "${ADMIN_TLS_SIGN_CERT}" > ${OUTPUT_PATH}/cacerts/server.crt
175+
echo "${ADMIN_TLS_PRIVATE_KEY}" > ${OUTPUT_PATH}/keystore/server.key
176+
ADMIN_TLS_SECRET=true
177+
fi
178+
}
179+
143180
{{- end }}
144181

145182
COUNTER=1
146183
while [ "$COUNTER" -le {{ .Values.healthCheck.retries }} ]
147184
do
148185

149186
OUTPUT_PATH="${MOUNT_PATH}/admin/msp"
187+
OUTPUT_TLS_PATH="${MOUNT_PATH}/tls/msp"
188+
mkdir -p ${OUTPUT_TLS_PATH}
150189
mkdir -p ${OUTPUT_PATH}/admincerts
151190
mkdir -p ${OUTPUT_PATH}/cacerts
152191
mkdir -p ${OUTPUT_PATH}/keystore
153192
mkdir -p ${OUTPUT_PATH}/signcerts
154193
mkdir -p ${OUTPUT_PATH}/tlscacerts
155-
getAdminMspSecret admin-msp
194+
getAdminMspSecret admin-msp
195+
getAdminTlsSecret admin-tls
156196

157-
if [ "$ADMIN_MSP_SECRET" = "true" ]
197+
if [ "$ADMIN_MSP_SECRET" = "true" ] && [ "$ADMIN_TLS_SECRET" = "true" ]
158198
then
159199
echo "Peer certificates have been obtained correctly"
160200
break
@@ -214,6 +254,8 @@ spec:
214254
value: "{{ .Values.ordererAddress }}"
215255
- name: CORE_PEER_MSPCONFIGPATH
216256
value: /opt/gopath/src/github.com/hyperledger/fabric/crypto/admin/msp
257+
- name: CORE_PEER_TLSCONFIGPATH
258+
value: /opt/gopath/src/github.com/hyperledger/fabric/crypto/admin/tls
217259
volumeMounts:
218260
- name: certificates
219261
mountPath: /opt/gopath/src/github.com/hyperledger/fabric/crypto

‎platforms/hyperledger-fabric/configuration/add-orderer.yaml

+19-29
Original file line numberDiff line numberDiff line change
@@ -24,25 +24,6 @@
2424
path: "./build"
2525
state: absent
2626

27-
# Create CA Tools helm-value files and check-in
28-
- name: Create CA tools for each organization
29-
include_role:
30-
name: "create/ca_tools/orderer"
31-
vars:
32-
component_name: "{{ item.name | lower}}-net"
33-
component: "{{ item.name | lower}}"
34-
component_type: "{{ item.type | lower}}"
35-
component_services: "{{ item.services }}"
36-
sc_name: "{{ component }}-bevel-storageclass"
37-
kubernetes: "{{ item.k8s }}"
38-
vault: "{{ item.vault }}"
39-
ca: "{{ item.services.ca }}"
40-
docker_url: "{{ network.docker.url }}"
41-
gitops: "{{ item.gitops }}"
42-
values_dir: "{{playbook_dir}}/../../../{{item.gitops.release_dir}}/{{ item.name | lower }}"
43-
loop: "{{ network['organizations'] }}"
44-
when: item.type == 'orderer'
45-
4627
# Generate script to modify the fetched configuration block
4728
- name: "Modify the system channel with tls information of new orderer"
4829
include_role:
@@ -65,18 +46,22 @@
6546
name: "create/orderers"
6647
vars:
6748
build_path: "./build"
68-
namespace: "{{ item.name | lower}}-net"
69-
component_type: "{{ item.type | lower}}"
70-
component_services: "{{ item.services }}"
71-
vault: "{{ item.vault }}"
72-
git_protocol: "{{ item.gitops.git_protocol }}"
73-
git_url: "{{ item.gitops.git_url }}"
74-
git_branch: "{{ item.gitops.branch }}"
49+
namespace: "{{ org.name | lower}}-net"
50+
component_type: "{{ org.type | lower}}"
51+
org_name: "{{ org.name | lower }}"
52+
component_services: "{{ org.services }}"
53+
kubernetes: "{{ org.k8s }}"
54+
vault: "{{ org.vault }}"
55+
git_protocol: "{{ org.gitops.git_protocol }}"
56+
git_url: "{{ org.gitops.git_url }}"
57+
git_branch: "{{ org.gitops.branch }}"
7558
docker_url: "{{ network.docker.url }}"
76-
charts_dir: "{{ item.gitops.chart_source }}"
77-
values_dir: "{{playbook_dir}}/../../../{{item.gitops.release_dir}}/{{ item.name | lower }}"
59+
charts_dir: "{{ org.gitops.chart_source }}"
60+
values_dir: "{{playbook_dir}}/../../../{{org.gitops.release_dir}}/{{ org.name | lower }}"
7861
loop: "{{ network['organizations'] }}"
79-
when: item.type == 'orderer'
62+
loop_control:
63+
loop_var: org
64+
when: org.services.orderers is defined and org.services.orderers | length > 0
8065

8166
# Generate script to modify the fetched configuration block
8267
- name: "Modify the system channel with endpoint information of new orderer"
@@ -112,10 +97,15 @@
11297
vars:
11398
build_path: "./build"
11499
channel_name: "{{ channel.channel_name | lower }}"
100+
org_peer: "{{ channel.participants | first}}"
101+
vault: "{{ org.vault }}"
102+
existingOrdererAddress: "{{ org_peer.ordererAddress }}"
103+
peer: "{{ org_peer.peers | first }}"
115104
docker_url: "{{ network.docker.url }}"
116105
loop: "{{ network.channels }}"
117106
loop_control:
118107
loop_var: channel
108+
when: '2.5.' not in network.version
119109

120110
vars: #These variables can be overriden from the command line
121111
add_new_org: 'false' #Default for this playbook is false

‎platforms/hyperledger-fabric/configuration/roles/create/new_orderer/create_appchannel_block/tasks/nested_create_appchannel_block.yaml

+13-1
Original file line numberDiff line numberDiff line change
@@ -38,14 +38,24 @@
3838
path: "{{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls"
3939
state: touch
4040

41+
# Copy tls server.crt from vault to the build directory
42+
- name: Fetch the tls server.crt from vault
43+
shell: |
44+
vault kv get -field=server_crt {{ vault.secret_path | default('secretsv2') }}/{{ network.env.type }}{{ org.name }}/orderers/{{ orderer.name }}-tls > server.crt
45+
mkdir -p {{ build_path }}/tls-cert/{{ orderer.name }}
46+
mv server.crt {{ build_path }}/tls-cert/{{ orderer.name }}
47+
environment:
48+
VAULT_ADDR: "{{ vault.url }}"
49+
VAULT_TOKEN: "{{ vault.root_token }}"
50+
4151
# Create orderer-tls for tls certificate information
4252
- name: "adding tls certificate information"
4353
shell: |
4454
if [ {{ network.env.proxy }} == 'none' ]
4555
then
4656
echo -n "{\"client_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\",\"host\":\"{{ orderer.name|lower }}.{{ component_ns }}\",\"port\":{{ orderer.grpc.port }},\"server_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\"}" >> {{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls
4757
else
48-
echo -n "{\"client_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\",\"host\":\"{{ orderer.ordererAddress.split(":")[0] | to_json }}\",\"port\":\"{{ orderer.ordererAddress.split(":")[1] | to_json }}\",\"server_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\"}" >> {{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls
58+
echo -n "{\"client_tls_cert\":\"$(cat {{ build_path }}/tls-cert/{{ orderer.name }}/server.crt | base64 -w 0))\",\"host\":\"{{ orderer.ordererAddress.split(":")[0] | to_json }}\",\"port\":{{ orderer.ordererAddress.split(":")[1] | to_json }},\"server_tls_cert\":\"$(cat {{ build_path }}/tls-cert/{{ orderer.name }}/server.crt | base64 -w 0))\"}" >> {{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls
4959
fi
5060
5161
# Create orderer file for adding new endpoint information
@@ -71,6 +81,8 @@
7181
dest: "{{ build_path }}/update-channel-script.sh"
7282
vars:
7383
component_name: "{{ org.name | lower }}"
84+
ordererAddress: "{{ orderer.ordererAddress }}"
85+
ordererOrg: "{{ org.name | lower }}MSP"
7486
os: "{{ fabric.os }}"
7587
arch: "{{ fabric.arch }}"
7688
version: "{{ network.version }}"

0 commit comments

Comments
 (0)