NerlMonitor #236

GuyPerets106 · 2023-08-21T08:30:24Z

For your review @leondavi @halfway258 @galhilu

…nitor

inputJsonFiles/Architecture/arch_1PCSIM6WorkerMNist.json

leondavi · 2023-08-21T15:23:32Z

inputJsonFiles/experimentsFlow/exp_6Workers1SourceMNist.json

@@ -1,7 +1,7 @@
 {
   "Features": 784,
   "Labels": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"],
-   "CSV path": "mnist",


@halfway258 Please approve this change

leondavi · 2023-08-21T15:25:56Z

src_py/apiServer/apiServer.py

                disp.plot(ax=axes[i, j], colorbar=False)
                disp.ax_.set_title(f'{worker}, class #{j}\nAccuracy={round(accuracy_score(workerNeuronRes[worker][globe.TRUE_LABLE_IND][j], workerNeuronRes[worker][globe.PRED_LABLE_IND][j]), 3)}')
-                if i < len(workersList) - 1:
+                if i < len(workersList) - 1: # ? Ask David why this is needed


@halfway258 Haran can you please clarify this line
@GuyPerets106 Please add a comment with description gave by Haran

leondavi · 2023-08-21T15:27:34Z

src_erl/NerlMonitor/README.md

+
+Also, install **Pyrlang** and **Term** libraries for Python-Erlang communication (follow their instructions **carefully**):
+
+Pyrlang - https://github.com/Pyrlang/Pyrlang


@GuyPerets106 @galhilu

We should add an installation step in bash script of NerlMonitor.
The bash script should accept --set-env flag.
If called with this flag then it installs an environment with required python modules (add requirement.txt with all required modules except of pyrlang that is cloned).
This python virtualenv should be installed to /tmp/nerlmonitor/venv
Each time --set-env flag is called it removes venv directory and reinstall the virtualenv.

Of course there is an assumption that virtualenv is already installed on machine.

src_erl/NerlMonitor/LICENSE.md

leondavi · 2023-08-21T15:36:24Z

src_erl/NerlMonitor/README.md

+
+# Dependencies
+`pip install` these libraries (In addition to the src_py/requirements.txt file):
+- NetowrkX


@GuyPerets106
Create a requirements.txt file
You can use pipreqs
https://stackoverflow.com/questions/31684375/automatically-create-requirements-txt

Change instructions to:

Create a python virtualenv for NerlMonitor.

Install modules: pip3 install -r /usr/local/lib/nerlnet-lib/NErlNet/src_py/NerlMonitor/requirements.txt

leondavi · 2023-08-22T20:19:33Z

@GuyPerets106 @galhilu

Please resolve conflicts and validate with @halfway258

leondavi · 2023-08-23T15:54:47Z

src_erl/NerlGUI/src/graphScreen.erl

        Edges = lists:droplast(lists:last(Devices)),

        io:format("got graph: ~p~n", [Devices]),
        DeviceList = lists:droplast(Devices),

-        {FileName, G} = gui_tools:makeGraphIMG(DeviceList, Edges),
+        {FileName, G} = gui_tools:makeGraphIMG(22),


@GuyPerets106 @galhilu
What is this magic number?!?

Maybe it's a mistake, I don't think we meant to change anything in NerlGUI
@galhilu correct me if I'm wrong

src_erl/NerlGUI/src/graphScreen.erl

leondavi · 2023-09-02T00:11:15Z

README.md

@@ -76,4 +76,4 @@ Minimum Python version: 3.8
 4. Run Jupyter notebook with ```jupyter-notebook``` and create a new notebook in the created dir from step 3. 
 5. Follow the example: https://github.com/leondavi/NErlNet/blob/master/examples/example_run.ipynb 

-Contact Email: [email protected]
+Contact Email: [email protected]


From some reason you change the README.md of the root.
I guess it was a mistake.
We should perform rebase on master together

leondavi · 2023-09-02T00:12:01Z

src_erl/Communication_Layer/http_Nerlserver/src/Client/clientStatem.erl

  Workers = ets:lookup_element(EtsRef, workersNames, ?ETS_KV_VAL_IDX),
  cast_message_to_workers(EtsRef, {NewState}),
  {next_state, waitforWorkers, State#client_statem_state{nextState = NewState, waitforWorkers = Workers}};

+waitforWorkers(cast, In ={worker_kill , Body}, State = #client_statem_state{waitforWorkers = WaitforWorkers,etsRef = EtsRef}) ->
+  %got kill command for a worker, kiil and take off waitforWorkers list


typo - kill

leondavi · 2023-09-02T00:14:24Z

src_erl/Communication_Layer/http_Nerlserver/src/Client/clientStatem.erl

+waitforWorkers(cast, In ={worker_kill , Body}, State = #client_statem_state{waitforWorkers = WaitforWorkers,etsRef = EtsRef}) ->
+  %got kill command for a worker, kiil and take off waitforWorkers list
+  ets:update_counter(EtsRef, msgCounter, 1),
+  ets:update_counter(EtsRef, infoIn, nerl_tools:calculate_size(In)),


infoIn?
What is it? add a documentation please.
If it is a size of in message, then more informative name is required.

leondavi · 2023-09-02T00:15:13Z

src_erl/Communication_Layer/http_Nerlserver/src/Client/clientStatem.erl

  Workers = ets:lookup_element(EtsRef, workersNames, ?ETS_KV_VAL_IDX),
  cast_message_to_workers(EtsRef, {NewState}),
  {next_state, waitforWorkers, State#client_statem_state{nextState = NewState, waitforWorkers = Workers}};

+waitforWorkers(cast, In ={worker_kill , Body}, State = #client_statem_state{waitforWorkers = WaitforWorkers,etsRef = EtsRef}) ->
+  %got kill command for a worker, kiil and take off waitforWorkers list


got a kill command from who?
Please more information in documentation, think about a new student that should handle this code.

leondavi · 2023-09-02T00:16:18Z

src_erl/Communication_Layer/http_Nerlserver/src/Client/clientStatem.erl

+  ets:update_counter(EtsRef, infoIn, nerl_tools:calculate_size(EventContent)),
+  case EventContent of
+    {'DOWN',_Ref,process,Pid,_Reason}->                                     %worker down
+      [[WorkerName]] = ets:match(EtsRef,{'$1',Pid,'_','_','_'}),


What is going on in this line?

leondavi · 2023-09-02T00:28:55Z

src_erl/Communication_Layer/http_Nerlserver/src/utilities_handler.erl

+-export([init/2]).
+
+init(Req0 , [MainServerPid]) ->
+    %handler for a tool connection req


What?
Please extend your explanation with more details.

leondavi · 2023-09-02T00:30:44Z

src_erl/Communication_Layer/http_Nerlserver/src/utilities_handler.erl

+           Graph = gen_server:call(MainServerPid , getGraph),
+           WorkersMap = ets:lookup_element(nerlnet_data , workers , ?DATA_IDX),
+           WorkersClients = maps:to_list(WorkersMap),
+           Workers = lists:flatten([atom_to_list(X)++"-"++atom_to_list(Y)++"!" || {X , Y} <- WorkersClients]),


What is the role of "-" and "!" ?
"-" is very very risky it can be a part of worker or client name.
Please explain what this line does and change marks to some safer patern e.g. @#@

leondavi · 2023-09-02T00:31:23Z

src_erl/Communication_Layer/http_Nerlserver/src/utilities_handler.erl

+    {ok, Req, MainServerPid};
+
+init(Req0 , [Action , MainServerPid]) ->
+    %handler for a tool's requested action (not connection) 


More info about this method

leondavi · 2023-09-02T00:32:50Z

src_erl/NerlGUI/src/graphScreen.erl

NerlGUI directory should be deprecated.
Please move desired content to NerlMonitor and remove NerlGUI directory.

leondavi · 2023-09-02T00:33:11Z

src_erl/NerlMonitor/.gitignore

leondavi and others added 30 commits August 1, 2023 12:08

init

99b39d0

Update READme.md

b660a34

Update README.md

afe543b

[NERLMONITOR] add attacher files

054e573

[NerlMonitor] Added files

ae13017

[NerlMonitor] Handler updates

253c187

[NERLMONITOR] Added MonitorGUI.py

79a0e06

[nerlMonitor] updated files

b303eec

[NerlMonitor] Updated MonitorGUI

991b342

[NerlMonitor] Updated files for messages passing

ac2fc31

Updated NerlMonitorGUI and First Communication with MainServer

4021fcf

Added WorkersMap to NerlGraph

9ad8cde

add monitor functionality to entities

9e76992

Changed WorkersMap Format

86f32e4

added worker down cast handel for main server

6ef37b8

Updated Json and Port for exp

3aa8bab

Fixed ets:new declaration in main server

3dcf863

Update README.md

5e45ba8

Updated NerlMonitor directory as rebar3 app

afb9e94

added hadle info for client states for worker fall detection

c79f1d6

NerlGraph sent from MainServer in right format

3943ccb

[NerlMonitor] Added worker failure tolerance

a53dd31

[NerlMonitor] Fix aServer behaviour, more LOG msgs

6024cc4

[NerlMonitor] Added worker termination for tests

e2ee54b

[NerlMonitor] Changed NerlGraph colors added log-msg

2173654

[NerlMonitor] Added worker_kill option

3aa34ab

[NerlMonitor] Added statistics

d86129d

[NerlMonitor] Added worker termination from GUI

748d94b

[NerlMonitor] Updated Worker Termination NerlGraph

298dea9

[NerlMonitor] debug of worker kill

df3c017

galhilu and others added 10 commits August 13, 2023 11:02

Merge branch 'NerlMonitor' of github.com:leondavi/NErlNet into NerlMo…

5c10a93

…nitor

[NerlMonitor] Tests

874136e

[NerlMonitor] Added worker termination

431ff90

[NerlMonitor] Statistics receive

aa48eae

[NerlMonitor] Final commit before testing

594d77b

[NerlMonitor] comment editing

61c9a9a

[NerlMonitor] Final commit

eda9e96

[NerlMonitor] Router bug

102a6f2

Update README.md

aea1796

Update README.md

cb4196e

GuyPerets106 requested review from leondavi and halfway258 as code owners August 21, 2023 08:30

leondavi requested changes Aug 21, 2023

View reviewed changes

leondavi requested changes Aug 23, 2023

View reviewed changes

src_erl/NerlGUI/src/graphScreen.erl Outdated Show resolved Hide resolved

Small fixes before merge

06507ed

GuyPerets106 requested review from NoaShapira8, ohad123, Orisadek, talshva and Hugina as code owners August 31, 2023 10:43

leondavi requested changes Sep 2, 2023

View reviewed changes

leondavi requested a review from galhilu September 2, 2023 00:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NerlMonitor #236

NerlMonitor #236

GuyPerets106 commented Aug 21, 2023

leondavi Aug 21, 2023

leondavi Aug 21, 2023

leondavi Aug 21, 2023 •

edited

Loading

leondavi Aug 21, 2023

leondavi commented Aug 22, 2023

leondavi Aug 23, 2023

GuyPerets106 Aug 31, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023

leondavi Sep 2, 2023


		Also, install Pyrlang and Term libraries for Python-Erlang communication (follow their instructions carefully):

		Pyrlang - https://github.com/Pyrlang/Pyrlang

NerlMonitor #236

Are you sure you want to change the base?

NerlMonitor #236

Conversation

GuyPerets106 commented Aug 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leondavi Aug 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leondavi commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leondavi Aug 21, 2023 •

edited

Loading