Changes between Version 3 and Version 4 of FeddUsersGuide


Ignore:
Timestamp:
May 15, 2014 11:35:15 AM (5 years ago)
Author:
faber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • FeddUsersGuide

    v3 v4  
    5656 * The assignments of projects within a testbed by the {{{SERVICE}}} directives in the comments
    5757
     58The this layout connects 2 experiment on DETERLab.  This is because the testbeds are all prefixed with the {{{deter}}} testbed name.  The ''testbed''/''sub-testbed'' syntax allows multiple instantiations of experiments on the same testbed.  This layout creates 2 experiment on DETERLab (DETERLab's [NewFeddAbout#TheExperimentController experiment contoller] maps the {{{deter}}} testbed name to DETERLab).
     59
     60To create the federated experiment, execute the following command on {{{users.isi.deterlab.net}}}:
     61
     62{{{
     63users:~$ fedd_create.py --file federation1.tcl --experiment_name fed1
     64}}}
     65
     66The parameters are the file containing the layout ({{{federation1.tcl}}}) and the shorthand name the caller wants to use to refer to the federated experiment.  The shorthand is a request. The experiment controller defines the namespace of experiments and will resolve conflicts.  That is, if another user has already named an experiment {{{fed1}}} on this controller, the controller with pick an new name.
     67
     68This will run for a little while, as much as a few minutes.  It is gathering rights to access remote testbeds and starting the sub-experiments on them.  Shortly you will see something like:
     69
     70{{{
     71localname: fed1
     72fedid: 6a4f58292d572c57ef612e3e44e5d8134196e550
     73status: starting
     74}}}
     75
     76The {{{localname}}} is the name that the controller picked for the federated experiment, usually the same as the {{{experiment_name}}} parameter.  If the output is different, the user will need to use that shorthand in subsequent commands.
     77
     78The {{{fedid}}} is a unique identifier that refers to this experiment.  We discuss ways to use this below.
     79
     80Assuming that {{{status}}} is {{{starting}}} the experiment is being created.  If there has been a error, there will be an error message as well.
     81
     82At any time a user can poll the current status of their federated experiment using the {{{fedd_multistatus.py}} command.  Run it like this on {{{users.isi.deterlab.net}}} and you should see similar output:
     83
     84{{{
     85users:~$ fedd_multistatus.py
     86fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:starting
     87}}}
     88
     89The output is the experiment name, the experiment fedid, and the status, comma separated.  If an experiment creation has failed the output will look more like:
     90
     91{{{
     92users:~$ fedd_multistatus.py
     93fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:starting
     94bad_experiment:da83eb06712a2006abeae34308c363b0ab0faa0a:failed
     95}}}
     96
     97The experiment called {{{bad_experiment}}} has failed to get created.  When fed1 has finished swapping in (both parts) in DETERLab, the output will look like:
     98
     99{{{
     100users:~$ fedd_multistatus.py
     101fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:active
     102bad_experiment:da83eb06712a2006abeae34308c363b0ab0faa0a:failed
     103}}}
     104
     105A user can certainly use the fedd_multistatus.py command to poll and monitor experiment creation, but output is fairly terse and polling is inefficient.  The {{{fedd_spewlog.py}}} command will output a debugging log from the experiment controller.  If the experiment is being created or terminated, the command puts out the log so far and continues updating it until the operation succeeds or fails.  A user can follow the progress of the creation in real time using this command.
     106
     107
     108Here is sample output from the sample creation above:
     109
     110{{{
     111users:~$ fedd_spewlog.py --experiment_name fed1
     11207 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23231
     11307 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23231
     11407 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23233
     115Allocated vlan: 380807 May 14 11:16:34 fedd.experiment_control.fed1 Waiting for sub threads (it has been 1 mins)
     11607 May 14 11:17:34 fedd.experiment_control.fed1 Waiting for sub threads (it has been 2 mins)
     11707 May 14 11:15:37 fedd.access.fed1-exp1 State is swapped
     11807 May 14 11:15:37 fedd.access.fed1-exp1 [swap_exp]: Terminating fed1-exp1
     11907 May 14 11:15:41 fedd.access.fed1-exp1 [swap_exp]: Terminate succeeded
     12007 May 14 11:15:41 fedd.access.fed1-exp1 [make_null_experiment]: Creating experiment
     12107 May 14 11:16:02 fedd.access.fed1-exp1 [make_null_experiment]: Create succeeded
     12207 May 14 11:16:02 fedd.access.fed1-exp1 [start_segment]: creating script file
     12307 May 14 11:16:02 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/tmp739CnG faber@users.isi.deterlab.net:tmp739CnG
     12407 May 14 11:16:02 fedd.access.fed1-exp1 [ssh_cmd]: /usr/bin/ssh -n -o 'IdentitiesOnly yes' -o 'StrictHostKeyChecking no' -o 'ForwardX11 no' -i /usr/local/etc/fedd/deter/fedd_rsa faber@users.isi.deterlab.net sh -x tmp739CnG
     12507 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/fedgw_rsa.pub faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/fedgw_rsa.pub
     12607 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/fedgw_rsa faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/fedgw_rsa
     12707 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/hosts faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/hosts
     12807 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/ca.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/ca.pem
     12907 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/node.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/node.pem
     13007 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/client.conf faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/client.conf
     13107 May 14 11:16:05 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/experiment.tcl faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/experiment.tcl
     13207 May 14 11:16:05 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/software/fedkit.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp1/fedkit.tgz
     13307 May 14 11:16:06 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/software/seer-fbsd71-current.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp1/seer-fbsd71-current.tgz
     13407 May 14 11:16:06 fedd.access.fed1-exp1 [modify_exp]: Modifying fed1-exp1
     13507 May 14 11:16:23 fedd.access.fed1-exp1 [modify_exp]: Modify succeeded
     13607 May 14 11:16:23 fedd.access.fed1-exp1 [swap_exp]: Swapping fed1-exp1 in
     13707 May 14 11:17:49 fedd.access.fed1-exp1 [swap_exp]: Swap succeeded
     13807 May 14 11:17:49 fedd.access.fed1-exp1 [get_mapping] Generating mapping
     13907 May 14 11:17:49 fedd.access.fed1-exp1 Node mapping complete
     14007 May 14 11:17:49 fedd.access.fed1-exp1 Link mapping complete07 May 14 11:15:37 fedd.access.fed1-exp2 State is swapped
     14107 May 14 11:15:37 fedd.access.fed1-exp2 [swap_exp]: Terminating fed1-exp2
     14207 May 14 11:15:41 fedd.access.fed1-exp2 [swap_exp]: Terminate succeeded
     14307 May 14 11:15:41 fedd.access.fed1-exp2 [make_null_experiment]: Creating experiment
     14407 May 14 11:16:02 fedd.access.fed1-exp2 [make_null_experiment]: Create succeeded
     14507 May 14 11:16:02 fedd.access.fed1-exp2 [start_segment]: creating script file
     14607 May 14 11:16:02 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/tmp6dcyg4 faber@users.isi.deterlab.net:tmp6dcyg4
     14707 May 14 11:16:02 fedd.access.fed1-exp2 [ssh_cmd]: /usr/bin/ssh -n -o 'IdentitiesOnly yes' -o 'StrictHostKeyChecking no' -o 'ForwardX11 no' -i /usr/local/etc/fedd/deter/fedd_rsa faber@users.isi.deterlab.net sh -x tmp6dcyg4
     14807 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/fedgw_rsa.pub faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/fedgw_rsa.pub
     14907 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/fedgw_rsa faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/fedgw_rsa
     15007 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/hosts faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/hosts
     15107 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/ca.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/ca.pem
     15207 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/node.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/node.pem
     15307 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/client.conf faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/client.conf
     15407 May 14 11:16:05 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/experiment.tcl faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/experiment.tcl
     15507 May 14 11:16:05 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/software/fedkit.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp2/fedkit.tgz
     15607 May 14 11:16:06 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/software/seer-fbsd71-current.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp2/seer-fbsd71-current.tgz
     15707 May 14 11:16:06 fedd.access.fed1-exp2 [modify_exp]: Modifying fed1-exp2
     15807 May 14 11:16:23 fedd.access.fed1-exp2 [modify_exp]: Modify succeeded
     15907 May 14 11:16:23 fedd.access.fed1-exp2 [swap_exp]: Swapping fed1-exp2 in
     16007 May 14 11:18:09 fedd.access.fed1-exp2 [swap_exp]: Swap succeeded
     16107 May 14 11:18:09 fedd.access.fed1-exp2 [get_mapping] Generating mapping
     16207 May 14 11:18:10 fedd.access.fed1-exp2 Node mapping complete
     16307 May 14 11:18:10 fedd.access.fed1-exp2 Link mapping complete07 May 14 11:18:11 fedd.experiment_control.fed1 [start_segment]: Experiment fed1 active
     164active
     165}}}
     166
     167
     168Much of that is useful only for debugging but the last line indicates the final status of the experiment creation.  The two choices are {{{active}}} and {{{failed}}}.