Changes between Version 3 and Version 4 of FeddUsersGuide

Timestamp:

May 15, 2014 11:35:15 AM (11 years ago)

Author:

faber

Comment:

--

FeddUsersGuide

@@ -56 +56 @@
  * The assignments of projects within a testbed by the {{{SERVICE}}} directives in the comments
 
+The this layout connects 2 experiment on DETERLab.  This is because the testbeds are all prefixed with the {{{deter}}} testbed name.  The ''testbed''/''sub-testbed'' syntax allows multiple instantiations of experiments on the same testbed.  This layout creates 2 experiment on DETERLab (DETERLab's [NewFeddAbout#TheExperimentController experiment contoller] maps the {{{deter}}} testbed name to DETERLab).
+
+To create the federated experiment, execute the following command on {{{users.isi.deterlab.net}}}:
+
+{{{
+users:~$ fedd_create.py --file federation1.tcl --experiment_name fed1
+}}}
+
+The parameters are the file containing the layout ({{{federation1.tcl}}}) and the shorthand name the caller wants to use to refer to the federated experiment.  The shorthand is a request. The experiment controller defines the namespace of experiments and will resolve conflicts.  That is, if another user has already named an experiment {{{fed1}}} on this controller, the controller with pick an new name.
+
+This will run for a little while, as much as a few minutes.  It is gathering rights to access remote testbeds and starting the sub-experiments on them.  Shortly you will see something like:
+
+{{{
+localname: fed1
+fedid: 6a4f58292d572c57ef612e3e44e5d8134196e550
+status: starting
+}}}
+
+The {{{localname}}} is the name that the controller picked for the federated experiment, usually the same as the {{{experiment_name}}} parameter.  If the output is different, the user will need to use that shorthand in subsequent commands.
+
+The {{{fedid}}} is a unique identifier that refers to this experiment.  We discuss ways to use this below.
+
+Assuming that {{{status}}} is {{{starting}}} the experiment is being created.  If there has been a error, there will be an error message as well.
+
+At any time a user can poll the current status of their federated experiment using the {{{fedd_multistatus.py}} command.  Run it like this on {{{users.isi.deterlab.net}}} and you should see similar output:
+
+{{{
+users:~$ fedd_multistatus.py
+fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:starting
+}}}
+
+The output is the experiment name, the experiment fedid, and the status, comma separated.  If an experiment creation has failed the output will look more like:
+
+{{{
+users:~$ fedd_multistatus.py
+fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:starting
+bad_experiment:da83eb06712a2006abeae34308c363b0ab0faa0a:failed
+}}}
+
+The experiment called {{{bad_experiment}}} has failed to get created.  When fed1 has finished swapping in (both parts) in DETERLab, the output will look like:
+
+{{{
+users:~$ fedd_multistatus.py
+fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:active
+bad_experiment:da83eb06712a2006abeae34308c363b0ab0faa0a:failed
+}}}
+
+A user can certainly use the fedd_multistatus.py command to poll and monitor experiment creation, but output is fairly terse and polling is inefficient.  The {{{fedd_spewlog.py}}} command will output a debugging log from the experiment controller.  If the experiment is being created or terminated, the command puts out the log so far and continues updating it until the operation succeeds or fails.  A user can follow the progress of the creation in real time using this command.
+
+
+Here is sample output from the sample creation above:
+
+{{{
+users:~$ fedd_spewlog.py --experiment_name fed1
+07 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23231 
+07 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23231 
+07 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23233 
+Allocated vlan: 380807 May 14 11:16:34 fedd.experiment_control.fed1 Waiting for sub threads (it has been 1 mins)
+07 May 14 11:17:34 fedd.experiment_control.fed1 Waiting for sub threads (it has been 2 mins)
+07 May 14 11:15:37 fedd.access.fed1-exp1 State is swapped
+07 May 14 11:15:37 fedd.access.fed1-exp1 [swap_exp]: Terminating fed1-exp1
+07 May 14 11:15:41 fedd.access.fed1-exp1 [swap_exp]: Terminate succeeded
+07 May 14 11:15:41 fedd.access.fed1-exp1 [make_null_experiment]: Creating experiment
+07 May 14 11:16:02 fedd.access.fed1-exp1 [make_null_experiment]: Create succeeded
+07 May 14 11:16:02 fedd.access.fed1-exp1 [start_segment]: creating script file
+07 May 14 11:16:02 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/tmp739CnG faber@users.isi.deterlab.net:tmp739CnG
+07 May 14 11:16:02 fedd.access.fed1-exp1 [ssh_cmd]: /usr/bin/ssh -n -o 'IdentitiesOnly yes' -o 'StrictHostKeyChecking no' -o 'ForwardX11 no' -i /usr/local/etc/fedd/deter/fedd_rsa faber@users.isi.deterlab.net sh -x tmp739CnG
+07 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/fedgw_rsa.pub faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/fedgw_rsa.pub
+07 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/fedgw_rsa faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/fedgw_rsa
+07 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/hosts faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/hosts
+07 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/ca.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/ca.pem
+07 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/node.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/node.pem
+07 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/client.conf faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/client.conf
+07 May 14 11:16:05 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/experiment.tcl faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/experiment.tcl
+07 May 14 11:16:05 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/software/fedkit.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp1/fedkit.tgz
+07 May 14 11:16:06 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/software/seer-fbsd71-current.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp1/seer-fbsd71-current.tgz
+07 May 14 11:16:06 fedd.access.fed1-exp1 [modify_exp]: Modifying fed1-exp1
+07 May 14 11:16:23 fedd.access.fed1-exp1 [modify_exp]: Modify succeeded
+07 May 14 11:16:23 fedd.access.fed1-exp1 [swap_exp]: Swapping fed1-exp1 in
+07 May 14 11:17:49 fedd.access.fed1-exp1 [swap_exp]: Swap succeeded
+07 May 14 11:17:49 fedd.access.fed1-exp1 [get_mapping] Generating mapping
+07 May 14 11:17:49 fedd.access.fed1-exp1 Node mapping complete
+07 May 14 11:17:49 fedd.access.fed1-exp1 Link mapping complete07 May 14 11:15:37 fedd.access.fed1-exp2 State is swapped
+07 May 14 11:15:37 fedd.access.fed1-exp2 [swap_exp]: Terminating fed1-exp2
+07 May 14 11:15:41 fedd.access.fed1-exp2 [swap_exp]: Terminate succeeded
+07 May 14 11:15:41 fedd.access.fed1-exp2 [make_null_experiment]: Creating experiment
+07 May 14 11:16:02 fedd.access.fed1-exp2 [make_null_experiment]: Create succeeded
+07 May 14 11:16:02 fedd.access.fed1-exp2 [start_segment]: creating script file
+07 May 14 11:16:02 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/tmp6dcyg4 faber@users.isi.deterlab.net:tmp6dcyg4
+07 May 14 11:16:02 fedd.access.fed1-exp2 [ssh_cmd]: /usr/bin/ssh -n -o 'IdentitiesOnly yes' -o 'StrictHostKeyChecking no' -o 'ForwardX11 no' -i /usr/local/etc/fedd/deter/fedd_rsa faber@users.isi.deterlab.net sh -x tmp6dcyg4
+07 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/fedgw_rsa.pub faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/fedgw_rsa.pub
+07 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/fedgw_rsa faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/fedgw_rsa
+07 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/hosts faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/hosts
+07 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/ca.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/ca.pem
+07 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/node.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/node.pem
+07 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/client.conf faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/client.conf
+07 May 14 11:16:05 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/experiment.tcl faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/experiment.tcl
+07 May 14 11:16:05 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/software/fedkit.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp2/fedkit.tgz
+07 May 14 11:16:06 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/software/seer-fbsd71-current.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp2/seer-fbsd71-current.tgz
+07 May 14 11:16:06 fedd.access.fed1-exp2 [modify_exp]: Modifying fed1-exp2
+07 May 14 11:16:23 fedd.access.fed1-exp2 [modify_exp]: Modify succeeded
+07 May 14 11:16:23 fedd.access.fed1-exp2 [swap_exp]: Swapping fed1-exp2 in
+07 May 14 11:18:09 fedd.access.fed1-exp2 [swap_exp]: Swap succeeded
+07 May 14 11:18:09 fedd.access.fed1-exp2 [get_mapping] Generating mapping
+07 May 14 11:18:10 fedd.access.fed1-exp2 Node mapping complete
+07 May 14 11:18:10 fedd.access.fed1-exp2 Link mapping complete07 May 14 11:18:11 fedd.experiment_control.fed1 [start_segment]: Experiment fed1 active
+active
+}}}
+
+
+Much of that is useful only for debugging but the last line indicates the final status of the experiment creation.  The two choices are {{{active}}} and {{{failed}}}.