Federation Users Guide

This tutorial walks through using the federation command line tools to create and operate federated experiments. To simplify the turotial, most of the experiments are actually federated withing DETERLab or between DETERLab and a desktop federant. The procedures generalize to more wide open federation.

In order to create federated experiments you will need permission to open holes in DETER's containment. Being able to connect across experiments or from an experiment to outside hosts means that users can send traffic that seems to originate from inside DETERLab - in fact some examples explain how to do that fairly generally. To get this privilege contact testbed operations.

Creating A Federated Experiment

We start by creating an experiment with the layout shown here:

Each square is a computer and each line a link. The layout is a simple dumbbell with nodes a and c as the ends of the inner "bar" and b, d, e, and f as the leaves. Nodes a, b, and d are in one testbed and c, e, f and f in another.

The following ns2 file is a description for such a topology:

# Simple federated topology, all on DETER
#
# SERVICE: project_export:deter/exp1::project=TIED
# SERVICE: project_export:deter/exp2::project=TIED
#

set ns [new Simulator]
source tb_compat.tcl

set a [$ns node]
set b [$ns node]
set c [$ns node]
set d [$ns node]
set e [$ns node]
set f [$ns node]
tb-set-node-testbed $a "deter/exp1"
tb-set-node-testbed $b "deter/exp1"
tb-set-node-testbed $c "deter/exp2"
tb-set-node-testbed $d "deter/exp1"
tb-set-node-testbed $e "deter/exp2"
tb-set-node-testbed $f "deter/exp2"


set link0 [ $ns duplex-link $a $b 1Gb 0ms DropTail]
set link1 [ $ns duplex-link $a $c 1Gb 0ms DropTail]
set link2 [ $ns duplex-link $a $d 1Gb 0ms DropTail]
set link3 [ $ns duplex-link $e $c 1Gb 0ms DropTail]
set link4 [ $ns duplex-link $f $c 1Gb 0ms DropTail]

$ns rtproto Static
$ns run

A copy of that file​ is attached to this page.

Most of that file looks like any other simple DETERLab experiment layout, and one can swap it in on DETERLab directly and get a single experiment. The differences are

• The assignments of nodes to testbeds using tb-set-node-testbed
• The assignments of projects within a testbed by the SERVICE directives in the comments

The this layout connects 2 experiment on DETERLab. This is because the testbeds are all prefixed with the deter testbed name. The testbed/sub-testbed syntax allows multiple instantiations of experiments on the same testbed. This layout creates 2 experiment on DETERLab (DETERLab's experiment contoller? maps the deter testbed name to DETERLab).

To create the federated experiment, execute the following command on users.isi.deterlab.net:

users:~$ fedd_create.py --file federation1.tcl --experiment_name fed1

The parameters are the file containing the layout (federation1.tcl) and the shorthand name the caller wants to use to refer to the federated experiment. The shorthand is a request. The experiment controller defines the namespace of experiments and will resolve conflicts. That is, if another user has already named an experiment fed1 on this controller, the controller with pick an new name.

This will run for a little while, as much as a few minutes. It is gathering rights to access remote testbeds and starting the sub-experiments on them. Shortly you will see something like:

localname: fed1
fedid: 6a4f58292d572c57ef612e3e44e5d8134196e550
status: starting

The localname is the name that the controller picked for the federated experiment, usually the same as the experiment_name parameter. If the output is different, the user will need to use that shorthand in subsequent commands.

The fedid is a unique identifier that refers to this experiment. We discuss ways to use this below.

Assuming that status is starting the experiment is being created. If there has been a error, there will be an error message as well.

At any time a user can poll the current status of their federated experiment using the fedd_multistatus.py}} command. Run it like this on {{{users.isi.deterlab.net and you should see similar output:

users:~$ fedd_multistatus.py
fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:starting

The output is the experiment name, the experiment fedid, and the status, comma separated. If an experiment creation has failed the output will look more like:

users:~$ fedd_multistatus.py
fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:starting
bad_experiment:da83eb06712a2006abeae34308c363b0ab0faa0a:failed

The experiment called bad_experiment has failed to get created. When fed1 has finished swapping in (both parts) in DETERLab, the output will look like:

users:~$ fedd_multistatus.py
fed1:6a4f58292d572c57ef612e3e44e5d8134196e550:active
bad_experiment:da83eb06712a2006abeae34308c363b0ab0faa0a:failed

A user can certainly use the fedd_multistatus.py command to poll and monitor experiment creation, but output is fairly terse and polling is inefficient. The fedd_spewlog.py command will output a debugging log from the experiment controller. If the experiment is being created or terminated, the command puts out the log so far and continues updating it until the operation succeeds or fails. A user can follow the progress of the creation in real time using this command.

Here is sample output from the sample creation above:

users:~$ fedd_spewlog.py --experiment_name fed1
07 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23231 
07 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23231 
07 May 14 11:15:34 fedd.experiment_control.fed1 Calling StartSegment at https://users.isi.deterlab.net:23233 
Allocated vlan: 380807 May 14 11:16:34 fedd.experiment_control.fed1 Waiting for sub threads (it has been 1 mins)
07 May 14 11:17:34 fedd.experiment_control.fed1 Waiting for sub threads (it has been 2 mins)
07 May 14 11:15:37 fedd.access.fed1-exp1 State is swapped
07 May 14 11:15:37 fedd.access.fed1-exp1 [swap_exp]: Terminating fed1-exp1
07 May 14 11:15:41 fedd.access.fed1-exp1 [swap_exp]: Terminate succeeded
07 May 14 11:15:41 fedd.access.fed1-exp1 [make_null_experiment]: Creating experiment
07 May 14 11:16:02 fedd.access.fed1-exp1 [make_null_experiment]: Create succeeded
07 May 14 11:16:02 fedd.access.fed1-exp1 [start_segment]: creating script file
07 May 14 11:16:02 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/tmp739CnG faber@users.isi.deterlab.net:tmp739CnG
07 May 14 11:16:02 fedd.access.fed1-exp1 [ssh_cmd]: /usr/bin/ssh -n -o 'IdentitiesOnly yes' -o 'StrictHostKeyChecking no' -o 'ForwardX11 no' -i /usr/local/etc/fedd/deter/fedd_rsa faber@users.isi.deterlab.net sh -x tmp739CnG
07 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/fedgw_rsa.pub faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/fedgw_rsa.pub
07 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/fedgw_rsa faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/fedgw_rsa
07 May 14 11:16:03 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/hosts faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/hosts
07 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/ca.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/ca.pem
07 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/node.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/node.pem
07 May 14 11:16:04 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/client.conf faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/client.conf
07 May 14 11:16:05 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/experiment.tcl faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp1/tmp/experiment.tcl
07 May 14 11:16:05 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/software/fedkit.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp1/fedkit.tgz
07 May 14 11:16:06 fedd.access.fed1-exp1 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-P0TiQP/software/seer-fbsd71-current.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp1/seer-fbsd71-current.tgz
07 May 14 11:16:06 fedd.access.fed1-exp1 [modify_exp]: Modifying fed1-exp1
07 May 14 11:16:23 fedd.access.fed1-exp1 [modify_exp]: Modify succeeded
07 May 14 11:16:23 fedd.access.fed1-exp1 [swap_exp]: Swapping fed1-exp1 in
07 May 14 11:17:49 fedd.access.fed1-exp1 [swap_exp]: Swap succeeded
07 May 14 11:17:49 fedd.access.fed1-exp1 [get_mapping] Generating mapping
07 May 14 11:17:49 fedd.access.fed1-exp1 Node mapping complete
07 May 14 11:17:49 fedd.access.fed1-exp1 Link mapping complete07 May 14 11:15:37 fedd.access.fed1-exp2 State is swapped
07 May 14 11:15:37 fedd.access.fed1-exp2 [swap_exp]: Terminating fed1-exp2
07 May 14 11:15:41 fedd.access.fed1-exp2 [swap_exp]: Terminate succeeded
07 May 14 11:15:41 fedd.access.fed1-exp2 [make_null_experiment]: Creating experiment
07 May 14 11:16:02 fedd.access.fed1-exp2 [make_null_experiment]: Create succeeded
07 May 14 11:16:02 fedd.access.fed1-exp2 [start_segment]: creating script file
07 May 14 11:16:02 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/tmp6dcyg4 faber@users.isi.deterlab.net:tmp6dcyg4
07 May 14 11:16:02 fedd.access.fed1-exp2 [ssh_cmd]: /usr/bin/ssh -n -o 'IdentitiesOnly yes' -o 'StrictHostKeyChecking no' -o 'ForwardX11 no' -i /usr/local/etc/fedd/deter/fedd_rsa faber@users.isi.deterlab.net sh -x tmp6dcyg4
07 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/fedgw_rsa.pub faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/fedgw_rsa.pub
07 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/fedgw_rsa faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/fedgw_rsa
07 May 14 11:16:03 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/hosts faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/hosts
07 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/ca.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/ca.pem
07 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/node.pem faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/node.pem
07 May 14 11:16:04 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/client.conf faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/client.conf
07 May 14 11:16:05 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/experiment.tcl faber@users.isi.deterlab.net:/proj/TIED/exp/fed1-exp2/tmp/experiment.tcl
07 May 14 11:16:05 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/software/fedkit.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp2/fedkit.tgz
07 May 14 11:16:06 fedd.access.fed1-exp2 [scp_file]: /usr/bin/scp -o IdentitiesOnly yes -o StrictHostKeyChecking no -o ForwardX11 no -i /usr/local/etc/fedd/deter/fedd_rsa /tmp/access-iH3Eaj/software/seer-fbsd71-current.tgz faber@users.isi.deterlab.net:/proj/TIED/software//fed1-exp2/seer-fbsd71-current.tgz
07 May 14 11:16:06 fedd.access.fed1-exp2 [modify_exp]: Modifying fed1-exp2
07 May 14 11:16:23 fedd.access.fed1-exp2 [modify_exp]: Modify succeeded
07 May 14 11:16:23 fedd.access.fed1-exp2 [swap_exp]: Swapping fed1-exp2 in
07 May 14 11:18:09 fedd.access.fed1-exp2 [swap_exp]: Swap succeeded
07 May 14 11:18:09 fedd.access.fed1-exp2 [get_mapping] Generating mapping
07 May 14 11:18:10 fedd.access.fed1-exp2 Node mapping complete
07 May 14 11:18:10 fedd.access.fed1-exp2 Link mapping complete07 May 14 11:18:11 fedd.experiment_control.fed1 [start_segment]: Experiment fed1 active
active

Much of that is useful only for debugging but the last line indicates the final status of the experiment creation. The two choices are active and failed.