Using DeltaFS with an MPI-based RPC backend

[shashankgugnani]

I'm trying to run DeltaFS with the MPI-backend in Mercury.

I can successfully launch the server using this command:

[root@head ~]# mpirun -np 1 -env DELTAFS_RPCProto=mpi+dynamic -env DELTAFS_MetadataSrvAddrs=deltafs -env DELTAFS_MDSTracing=true /opt/deltafs/bin/deltafs-srvr                                                                                                                                                          
[W] 2019/09/16-14:50:00.834175 2b4f5c9eee00 gflags is not enabled; some cmd arguments are ignored                                                             
[W] 2019/09/16-14:50:00.834248 2b4f5c9eee00 glog is not enabled; will use stderr instead                                                                      
[W] 2019/09/16-14:50:00.834251 2b4f5c9eee00 Snappy compression is not enabled                                                                                 
[I] 2019/09/16-14:50:00.834257 2b4f5c9eee00  DeltaFS: Version 1.0.0 (dev)                                                                                     
[I] 2019/09/16-14:50:00.834267 2b4f5c9eee00     Date: Mon Sep 16 14:50:00 2019                                                                                
[I] 2019/09/16-14:50:00.834766 2b4f5c9eee00      CPU: 28 x Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz                                                          
[I] 2019/09/16-14:50:00.834771 2b4f5c9eee00 CPUCache: 35840 KB (Line: 64 Bytes)                                                                               
[I] 2019/09/16-14:50:00.834915 2b4f5c9eee00   Memory: 131761628 kB                                                                                            
[I] 2019/09/16-14:50:00.834924 2b4f5c9eee00       OS: Linux 3.10.0-693.17.1.el7.x86_64 (target)                                                               
[I] 2019/09/16-14:50:00.834928 2b4f5c9eee00       OS: Linux 3.10.0-693.17.1.el7.x86_64                                                                        
[I] 2019/09/16-14:50:00.834930 2b4f5c9eee00      CXX: GNU (gcc/g++) 4.8.5                                                                                     
[I] 2019/09/16-14:50:00.841461 2b4f5c9eee00 Deltafs is initializing ...                                                                                       
[I] 2019/09/16-14:50:00.845794 2b4f5c9eee00 Delete type=3 #3                                                                                                  
[I] 2019/09/16-14:50:00.845804 2b4f5c9eee00 Remove MANIFEST-000003                                                                                            
[I] 2019/09/16-14:50:00.848343 2b4f5c9eee00 Deltafs is starting ...                                                                                           
[I] 2019/09/16-14:50:05.848537 2b4f5c9eee00 Deltafs status: OK [FCRET: 0 | MKDIR: 0 | FSTAT: 0 | LOKUP: 0]                                                    
[I] 2019/09/16-14:50:10.848683 2b4f5c9eee00 Deltafs status: OK [FCRET: 0 | MKDIR: 0 | FSTAT: 0 | LOKUP: 0]

But, I am not able create a shell and connect to the server:

[root@head ~]# mpirun -np 1 -env DELTAFS_RPCProto=mpi+dynamic -env DELTAFS_MetadataSrvAddrs=deltafs -env DELTAFS_NumOfMetadataSrvs=1 /opt/deltafs/bin/deltafs-shell -v=1 -logtostderr
Fatal error in PMPI_Comm_connect: Invalid port, error stack:                                                                                                  
PMPI_Comm_connect(129)...........................: MPI_Comm_connect(port="dynamic://deltafs", 
MPI_INFO_NULL, root=0, comm=0x84000000, newcomm=0x221fd00) failed                                                                                                                                                             
MPID_Comm_connect(191)...........................:                                                                                                            
MPIDI_Comm_connect(552)..........................: Named port dynamic://deltafs does not exist                                                                
MPIDI_Comm_connect(372)..........................:                                                                                                            
MPIDI_Create_inter_root_communicator_connect(134):                                                                                                            
MPIDI_CH3_Connect_to_root(254)...................:                                                                                                            
MPID_nem_tcp_connect_to_root(478)................:                                                                                                            
MPID_nem_tcp_get_addr_port_from_bc(587)..........: Missing port or invalid host/port description in business card

I'm using deltafs commit 6c90380 and MPICH v 3.1.3.

Any idea what the problem could be?

[shashankgugnani]

Any help is appreciated.