Error when invoking middleware/createTradeApp.js with release-1.4

[Hull4Progress]

Hi - I am attempting to run middleware/createTradeApp.js with release-1.4, but am getting an error (please see below)

I am following the instructions in the README.md at https://github.com/HyperledgerHandsOn/trade-finance-logistics/tree/release-1.4/middleware. I believe that all of my config parameters in the network and middleware directories are consistent.

I believe that my error is not related to Issue #3 (Race condition during channel join in createTradeApp.js). (As one attempt to debug, I extended the sleep time at line 196 of create-channel.js, but still getting same error.)

Here is the console output when I invoke createTradeApp.js:
++++++++++++++
rick@mac4hull middleware % node createTradeApp.js
2020-02-07T08:05:53.600Z - info: [ClientUtils]:

---

---

• Using temp dir: ../network/client-certs

---

---

Successfully extracted the config update from the configtx envelope
Successfully enrolled user 'admin' for exporterorg
Successfully signed config as admin of exporterorg
Successfully enrolled user 'admin' for importerorg
Successfully signed config as admin of importerorg
Successfully enrolled user 'admin' for carrierorg
Successfully signed config as admin of carrierorg
Successfully enrolled user 'admin' for regulatororg
Successfully signed config as admin of regulatororg
Successfully enrolled user 'admin' for every org and obtained channel config signatures
Successfully enrolled user 'admin' for orderer
2020-02-07T08:05:53.987Z - error: [Orderer.js]: sendDeliver - rejecting - status:NOT_FOUND
Channel tradechannel does not exist yet (IGNORE ANY ORDERER ERROR MESSAGES YOU SEE ABOVE!!)
Successfully signed config update as orderer admin
{ status: 'BAD_REQUEST',
info: 'error validating channel creation transaction for new channel 'tradechannel', could not succesfully apply update to template configuration: error authorizing update: error validating DeltaSet: policy for [Group] /Channel/Application not satisfied: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Admins' sub-policies to be satisfied' }

"-------------------------"

CHANNEL CREATION FAILED: Error: Error: Failed to create the channel.
at Client.newDefaultKeyValueStore.then.then.then.then.then (/Users/rick/go/src/trade-finance-logistics/middleware/create-channel.js:201:10)
at
at process._tickCallback (internal/process/next_tick.js:189:7)
at Client.newDefaultKeyValueStore.then.then.then.then.then.then (/Users/rick/go/src/trade-finance-logistics/middleware/create-channel.js:209:9)
at
at process._tickCallback (internal/process/next_tick.js:189:7)
"-------------------------"

++++++++++++++

Any suggestions would be appreciated.

(In case it is useful, I have attached logs from 3 of the containers.)
ca_peerImporterOrg--log.txt
orderer.trade.com--log.txt
peer0.importerorg.trade.com--log.txt

Thanks

[VRamakrishna]

@Hull4Progress Just to make sure that versioning is consistent across modules, can you check what versions of the fabric-client and fabric-ca-client were installed in middleware/? Run npm list | grep fabric to view the installed versions. If they are greater than 1.4.0, can you remove the ~ from the following two lines: https://github.com/HyperledgerHandsOn/trade-finance-logistics/blob/release-1.4/middleware/package.json#L7...L8, rerun npm install, and rerun the application?

Typically there shouldn't be a breaking change across minor versions but there's no guarantee of that.

If this still doesn't work, I'll do some debugging.

[Hull4Progress]

@VRamakrishna

Thank you for your ultra-quick response. Unfortunately the same bug seems to be occurring, but let me tell you some things that I attempted.

1. Indeed I had different versions for fabric-ca-client (1.4.5) and fabric-client (1.4.5) so I edited the file in ../middleware/package.json file as you recommended

2. I was using an up-to-date version of node, and when I did "> npm install" it complained about incompatibility with the fabric-client and fabric-ca-client, so I did "> nvm install v8.9.0". It came back saying that I was now using node v8.9.0 (npm v5.5.1) so that seems good.

3. When I did "> npm install" it complained about the xcode set up, that
   "xcode-select: error: tool 'xcodebuild' requires Xcode, but active
   developer directory '/Library/Developer/CommandLineTools' is a
   command line tools instance"
   So, I reloaded the Xcode app from the Apple App store (Version 11.3.1 (11C504)) and then, following the first comment in https://stackoverflow.com/questions/17980759/xcode-select-active-developer-directory-error I invoked "> sudo xcode-select -s /Applications/Xcode.app/Contents/Developer"

4. After this the "> npm install" ran with only a warning "npm WARN [email protected] No repository field." But as I look at the log there may still be something wrong with my Xcode installation -- I appears that the npm install is attempting to find certain directories multiple times, but finally gives up.

And, when I invoke "> node createTradeApp.js" I get the same error as before.

I am attaching here a new set of logs from orderer.trade.com, peer0.importerorg.trade.com, and ca_peerImporterOrg, although these look the same as before. I am also attaching the output from when I ran "> npm install", in case that is useful.

Thanks!
Rick

ca_peerImporterOrg--log.txt
npm-install--log.txt
orderer.trade.com--log.txt
peer0.importerorg.trade.com--logs.txt

[VRamakrishna]

@Hull4Progress I tested this out and made a small change to the configtx.yaml file. Now the applications run fine. Can you pull the latest code from the release-1.4 branch and retry? No need to make any changes to package.json; the latest version will work, just revert your changes and rerun npm install.

(It seems version 1.4.5 requires these setting changes to be explicit whereas 1.4.0 didn't.)

About your Xcode issues, I can't offer any advice as I develop on Linux. Perhaps @ldesrosi can help?

[Hull4Progress]

Hi - Thank you for your comments. Here are some points

1. as for the "npm WARN [email protected] No repository field.", that appears to be a warning only and I think it is not causing any problems (according to, e.g., https://stackoverflow.com/questions/16827858/npm-warn-package-json-no-repository-field)

2. In your update to configtx.yaml I noticed that for the first change (relating to "Global: &ChannelCapabilities") that you used V1_4_3, but for the other two changes you used V1_4_2. Also, after replacing my copy of configtx.yaml with the new one, when I attempted in ../network to run "> ./trade.sh generate -c tradechannel" it turned out that the orderer.trade.com container crashed. Please see first log in the attached file. Since it seemed suspicious to have V1_4_3 for the first change and V1_4_2 for the second and third, I experimented with making the first change be V1_4_2 rather than V1_4_3. In that case, when I ran "> ./trade.sh generate -c tradechannel" the tradechannel seemed to come up and the orderer.trade.com container seemed to launch satisfactorily

3. I then reverted my copy of package.json to the original as per your suggestion

4. However, I have made a couple of runs of "> node createTradeApp.js" and continue to get the same error
   { status: 'BAD_REQUEST',
   info: 'error validating channel creation transaction for new channel 'tradechannel', could not succesfully apply update to template configuration: error authorizing update: error validating DeltaSet: policy for [Group] /Channel/Application not satisfied: implicit policy evaluation failed - 0 sub-policies were satisfied, but this policy requires 1 of the 'Admins' sub-policies to be satisfied' }
   (I tried with the latest node/npm, and with node 9.8.0/npm 5.7.1 (requested by the release-1.4 middleware readme).

Do you have some additional suggestions?

BTW, I am a former IBMer, and I am still in contact with Petr Novotny. I think you are in India, and I will be up for a while - would you possibly have some time when you start working this morning, in the next 2 or 3 hours, to have a skype or webex call to attempt to troubleshoot this in realtime? << contact info deleted >>

Thanks
Rick

logs-from-orderer.trade.com-container.txt

[VRamakrishna]

@Hull4Progress I'm in the India Research Lab blockchain team and used to attend your weekly calls back in 2016, especially the discussions on the smart contract framework.

I had meetings in the morning, so I'm just getting free time now, sorry.

What you observed in (2) is strange. I know the version numbers were inconsistent, but I followed the example in the fabric samples: https://github.com/hyperledger/fabric-samples/blob/release-1.4/first-network/configtx.yaml. I assumed there's a reason for keeping different version numbers, and this worked for me.
From your logs, I see that you got the error panic: unable to bootstrap orderer. Error reading genesis block file: read /var/hyperledger/orderer/orderer.genesis.block: is a directory. This means that generate failed to create genesis.block, which then couldn't be synced to the orderer's container. I'm not sure why. Can you rerun everything from a clean slate? Kill and remove all the running containers, and delete channel-artifacts and crypto-config folders, start from scratch? And attach the console output from the generate command?

I suspect the version difference is why you are still getting the error in (4). Can you display the container image versions? Run docker images | grep hyperledger. I can tell you that I tested from a clean slate on a new VM, and my docker images and npm packages are using version 1.4.5.

[Hull4Progress]

@VRamakrishna -- Hi - Thanks for your reply and patience.

Good news - when I started with the clean slate (removing channel-artifacts, crypto-config, and docker-compose-e2e.yaml, then it has worked, at least up to the "> node createTradeApp.js" command. (I'm not sure why it didn't work for me before, I thought I had started with a clean slate. Oh well).

BTW, here is the listing of my images. I am curious why there are 3 versions of everything, and how does trade.sh know to use the 1.4.5 versions?

rick@mac4hull network % docker images | grep hyperledger
hyperledger/fabric-ca amd64-1.4.5-snapshot-292781e 8814c7914f1b 7 weeks ago 202MB
hyperledger/fabric-ca latest 8814c7914f1b 7 weeks ago 202MB
hyperledger/fabric-tools amd64-1.4.5-snapshot-eea42cbde 01a12d46c07e 7 weeks ago 1.49GB
hyperledger/fabric-tools amd64-latest 01a12d46c07e 7 weeks ago 1.49GB
hyperledger/fabric-tools latest 01a12d46c07e 7 weeks ago 1.49GB
hyperledger/fabric-buildenv amd64-1.4.5-snapshot-eea42cbde ea876a891c71 7 weeks ago 1.4GB
hyperledger/fabric-buildenv amd64-latest ea876a891c71 7 weeks ago 1.4GB
hyperledger/fabric-buildenv latest ea876a891c71 7 weeks ago 1.4GB
hyperledger/fabric-ccenv amd64-1.4.5-snapshot-eea42cbde 999aab778003 7 weeks ago 1.37GB
hyperledger/fabric-ccenv amd64-latest 999aab778003 7 weeks ago 1.37GB
hyperledger/fabric-ccenv latest 999aab778003 7 weeks ago 1.37GB
hyperledger/fabric-orderer amd64-1.4.5-snapshot-eea42cbde 845dafabf872 7 weeks ago 120MB
hyperledger/fabric-orderer amd64-latest 845dafabf872 7 weeks ago 120MB
hyperledger/fabric-orderer latest 845dafabf872 7 weeks ago 120MB
hyperledger/fabric-peer amd64-1.4.5-snapshot-eea42cbde 4b570bf6e307 7 weeks ago 128MB
hyperledger/fabric-peer amd64-latest 4b570bf6e307 7 weeks ago 128MB
hyperledger/fabric-peer latest 4b570bf6e307 7 weeks ago 128MB
hyperledger/fabric-baseimage amd64-0.4.18 9e353eca480f 3 months ago 1.3GB
hyperledger/fabric-baseos amd64-0.4.18 c256a6aad46f 3 months ago 80.8MB
hyperledger/fabric-ca-tools latest 591e9bb05cad 16 months ago 1.55GB

I will continue with the other steps in ../middleware. Hopefully it all works now. Thanks again.

[VRamakrishna]

@Hull4Progress They aren't different versions but rather the same images with different sets of tags (see the image IDs, they are identical). If you examine the list, every image has a latest tag associated with it, which is what trade.sh uses. When you ran make docker in the Fabric source folder, it build these container images and then tagged them with latest so applications can rely on them. Tomorrow, if you want to upgrade the Fabric source and rebuild the images, the latest tag will be associated with the new images, and used by the trade application without having to modify the trade.sh script or configuration files (except as required for compatibility).

Anyway, glad you were able to get things to work. Good luck! I'll close this issue now.

[Hull4Progress]

@VRamakrishna Hi - BTW - that is nice to know that we crossed paths a few years ago.

Actually, there may be a further "bug". I was successful with createTradeApp.js, which runs through all of the steps.

However, now I am attempting the "Test Run to Add a New Organization to the Network".

I checked whether the config_upgrade.json was consistent with the specs in network folder, and that seems OK. (For one thing, config.json and config_upgrade.json agree on the 4 original orgs.)

In the network folder I invoked "> ./trade.sh createneworg" to generate the cryptographic and channel artifacts for exportingentityorg. Here is the console output from that:

================

rick@mac4hull network % ./trade.sh createneworg
Generating certs and configuration for new org with channel 'tradechannel'
Continue? [Y/n] Y
proceeding ...
/Users/rick/go/src/github.com/hyperledger/fabric/.build/bin/cryptogen

######################################################################

Generate certificates for new org using cryptogen tool

######################################################################

• cryptogen generate --config=./add_org/crypto-config.yaml
  exportingentityorg.trade.com
• res=0
• set +x

/Users/rick/go/src/github.com/hyperledger/fabric/.build/bin/configtxgen
####################################################################################
######### Generating Channel Configuration for Exporting Entity Org ##############
####################################################################################

• FABRIC_CFG_PATH=/Users/rick/go/src/trade-finance-logistics/network/add_org/
• configtxgen -printOrg ExportingEntityOrgMSP -channelID tradechannel
  2020-02-10 03:40:44.686 EST [common.tools.configtxgen] main -> INFO 001 Loading configuration
  2020-02-10 03:40:44.690 EST [common.tools.configtxgen.localconfig] LoadTopLevel -> INFO 002 Loaded configuration: /Users/rick/go/src/trade-finance-logistics/network/add_org/configtx.yaml
• res=0
• set +x

============================

However, there may have been something wrong with the outcome, because I see that in the folder ../network/channel-artifacts that there is a file exportingEntityOrg.json. Which doesn't match the pattern of the other orgs, which have files such as ImporterOrgMSPanchors.tx

Anyway, then in the ../middleware folder I invoked "> node run-upgrade-channel.js" and got the following console output

========================

rick@mac4hull middleware % node run-upgrade-channel.js
2020-02-10T09:06:00.043Z - info: [ClientUtils]:

---

---

• Using temp dir: ../network/client-certs

---

---

Successfully enrolled user 'admin' for orderer
Got configuration block
Current config version: 1
configtxlator: error: open /dev/stdout: permission denied, try --help
configtxlator: error: Error decoding: error decoding input: *common.Config: unknown field "policies" in common.Config

---

CHANNEL UPGRADE FAILED: Error: Error: Error: Command failed: configtxlator proto_encode --input ./tmp/modified_config.json --type common.Config --output ./tmp/modified_config.pb
configtxlator: error: Error decoding: error decoding input: *common.Config: unknown field "policies" in common.Config

at checkExecSyncError (child_process.js:575:11)
at Object.execSync (child_process.js:612:13)
at Client.newDefaultKeyValueStore.then.then.then (/Users/rick/go/src/trade-finance-logistics/middleware/upgrade-channel.js:134:9)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:118:7)
at Client.newDefaultKeyValueStore.then.then.then.then.then (/Users/rick/go/src/trade-finance-logistics/middleware/upgrade-channel.js:180:9)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:118:7)
at Client.newDefaultKeyValueStore.then.then.then.then.then.then (/Users/rick/go/src/trade-finance-logistics/middleware/upgrade-channel.js:185:9)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:118:7)

---

====================
Any thoughts on what is going wrong here?

( I checked that I have version 1.4.5 for both fabric-client and fabric-ca-client, as shown by this command

=====================

rick@mac4hull middleware % npm list | grep fabric
├─┬ [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]

========================

[VRamakrishna]

@Hull4Progress First: run node runTradeScenarioApp.js right after you complete node createTradeApp.js so the chaincode's ledger state gets updated (if you don't do this, you'll see errors in a later step.)

The file exportingEntityOrg.json is the expected output of createneworg, so don't worry, that step succeeded. The bootstrapping of a channel requires the creation of different configuration files than upgrade does.

I don't know why you are getting an error with configtxlator. The following looks like a permissions issue?

configtxlator: error: open /dev/stdout: permission denied, try --help

The following looks like there is an incompatibility between the structure and the expected schema:

configtxlator: error: Error decoding: error decoding input: *common.Config: unknown field "policies" in common.Config

Now, assuming that you built configtxlator using make from the Fabric source, I think this happened because the command executed in line 134 in upgrade-channel.js depends on an input file that couldn't be created because of the permissions issue above. Also, can you confirm that you left the exportingEntityOrg.json file as is, and did not move or rename it? (Look at line 131; it depends on that JSON file being in the right place.)

[Hull4Progress]

Hi - BTW, I realize that when I make a "clean start" with ../network/trade.sh up, not only do I need to delete channel-artifacts, crypto-configs and docker-compose-e2e.yaml, but also I need to delete client-certs. Otherwise things break in the middle of executing ../middleware/createTradeApp.js.

[Hull4Progress]

Also - I am still stuck on errors in ../middleware/run-upgrade-channel.js which read

---

configtxlator: error: open /dev/stdout: permission denied, try --help
configtxlator: error: Error decoding: error decoding input: *common.Config: unknown field "groups" in common.Config

---

I agree that there seems to be a read/write permission challenge here, but I haven't solved it.

Can you please tell me the specific versions of node, npm, go that you are using? And whether it is different for different directories?

Some of the versions I've been testing with include
-- for go I have tried go1.12.16.darwin-amd64 (because https://hyperledger-fabric.readthedocs.io/en/release-1.4/prereqs.html says that I should) but also tried 1.13.4 darwin/amd64 and go1.13.7.darwin-amd64

-- in the hyperledger directory I have node v10.19.0 / npm 5.6.0 , (because https://hyperledger-fabric.readthedocs.io/en/release-1.4/prereqs.html says that I should)

-- in the ../middleware directory I have node v.9.8.0 / npm 5.7.1 (because https://github.com/HyperledgerHandsOn/trade-finance-logistics/tree/release-1.4/middleware says that I should)

Does that match your set up?
Thanks
Rick

[VRamakrishna]

Yes, you need to delete client-certs, which acts as a key and certificate cache. You don't have to manually delete the folders and bring down the containers. You can run ./trade.sh down followed by ./trade.sh clean, and that will do the job for you.

[VRamakrishna]

I checked my setup, and I was running node v10.19.0. But I downgraded to 9.8.0 and ran the complete workflow successfully. I don't believe the application is very sensitive to node/npm versions. The suggestions made in the READMEs are just guidelines.

[VRamakrishna]

It looks like you deleted an earlier comment? I got it in my email though. I believe you were getting those errors while running createTradeApp.js because of the presence of client-certs. (I should have mentioned that earlier, but at that point you were still stuck in the channel create phase, so it didn't seem relevant.)

About the configtxlator, I'm not sure how to help. What the code is doing is trying to execute a shell command in the Node.js runtime (using the execSync function). Perhaps you can give global write permissions to /dev/stdout using chmod and retry?

[Hull4Progress]

Yes - I did radically edit one of my comments, because at some point I realized that part of my problem was that I wasn't cleaning out the client-certs. Sorry about that. Thanks for pointing to the './trade.sh clean' capability.

Yes, I tried to give global permissions for /dev/stdout, but that doesn't work immediately because /dev/stdout is being used as a link to other more concrete directories. I had also been experimenting with different versions of go, etc. But I will go back to trying to investigate the global permissions further.

[Hull4Progress]

Hi @VRamakrishna -- Thank you for your observation that there seemed to be an access permissions problem going on. Indeed, it appears that with the OSX version that I am using -- Catalina 10.16.3 -- there are some new security features that block programs and apps from getting at certain things.

I decided to switch to Ubuntu in VirtualBox, and I have successfully run the entire exercise for the network folder and the middleware folder. So, that is good news. (I will be trying the application folder soon.)

Can I suggest that you alert users of your website (at least for release-1.4, but probably for all releases) that parts of the HL Fabric won't work with Mac OSX Catalina (and perhaps with some of the preceding OSX versions)? Perhaps in the README.md for the page https://github.com/HyperledgerHandsOn/trade-finance-logistics/tree/release-1.4.

BTW, I did post a comment into the HL JIRA to see if they would give a response. You can see that as the second comment in https://jira.hyperledger.org/browse/FAB-16987?jql=text%20~%20%22catalina%22 . But perhaps I should have started a new thread in the JIRA - what do you think?

I think it is now appropriate to close this issue. Thanks again very much for your timely and on-going assistance.
Rick

[VRamakrishna]

@Hull4Progress Good to know you got it working, and thanks for letting me know. We will add warnings to our instructions as you suggest. (Not being a Mac developer nor having access to a Mac prevents me from even attempting to fix your issue.)

@ldesrosi FYI.

I'll close this now.