= Multi-Party Experiment Creation Example = This example walks through creating a multiparty experiment from a single experiment topology description. That description may have been created from [wiki:FeddMulti#ComposingFragments a compostion program] or created by hand. We describe the key elements of the experiment description and walk through the creation. == Experiment Description == The ns-2 description of our example is ([wiki:FeddPluginArchitecture#TopologyDescriptionLanguage topdl] version is [attachment:multi-swap-example.xml attached]) {{{ # SERVICE: project_export:deter/a::project=TIED # SERVICE: project_export:deter/b::project=SAFER # SERVICE: seer_master:deter/a:deter/a,deter/b # SERVICE: local_seer_control:deter/a # SERVICE: local_seer_control:deter/b set ns [new Simulator] source tb_compat.tcl set a [$ns node] set b [$ns node] set c [$ns node] set d [$ns node] set e [$ns node] set deter_os "Ubuntu1004-STD" set seer_tar "/users/faber/seer-deploy/seer-u10-current.tgz" tb-set-node-os $a $deter_os tb-set-node-testbed $a "deter/a" tb-set-node-tarfiles $a "/usr/local/" $seer_tar tb-set-node-startcmd $a "sudo python /usr/local/seer/experiment-setup.py Basic" tb-set-node-os $b $deter_os tb-set-node-testbed $b "deter/a" tb-set-node-tarfiles $b "/usr/local" $seer_tar tb-set-node-startcmd $b "sudo python /usr/local/seer/experiment-setup.py Basic" tb-set-node-os $c $deter_os tb-set-node-testbed $c "deter/b" tb-set-node-tarfiles $c "/usr/local" $seer_tar tb-set-node-startcmd $c "sudo python /usr/local/seer/experiment-setup.py Basic" tb-set-node-os $d $deter_os tb-set-node-testbed $d "deter/b" tb-set-node-tarfiles $d "/usr/local" $seer_tar tb-set-node-startcmd $d "sudo python /usr/local/seer/experiment-setup.py Basic" tb-set-node-os $e $deter_os tb-set-node-testbed $e "deter/b" tb-set-node-tarfiles $e "/usr/local" $seer_tar tb-set-node-startcmd $e "sudo python /usr/local/seer/experiment-setup.py Basic" set link0 [ $ns duplex-link $a $b 100Mb 0ms DropTail] set link1 [ $ns duplex-link $c $b 100Mb 0ms DropTail] set lan0 [ $ns make-lan "$c $d $e" 100Mb 0ms ] $ns rtproto Static $ns run }}} This is an experiment topology containing 5 computers connected mostly by links, but with a single LAN. The computers are named with the letters a-e, and allocated to 2 parties. The party names appear in the {{{tb-set-node-testbed}}} commands (and as {{{testbed}}} attributes in the topdl). Overall the topology looks like (generated by [wiki:FeddCommands#fedd_image.py fedd_image.py] --group testbed --file multi-swap-example.xml --out multi.png): [[Image(multi.png)]] The individual topologies are configured using the SERVICE directives in the comments of the ns-2 description. They can also be given on the command line of [FeddCommands#fedd_create.py fedd_create.py] using '''--service'''. Currently that is the only way to specify them when starting a topdl experiment. The format of the line and the '''--service''' option are the same. Services in general are described [FeddAbout#ExperimentServices elsewhere] as well. Briefly, the first two lines set the DETER project that will be used to configure each party's area. The deter/a area will be an experiment owned by the TIED project; deter/b will be configured by the SAFER project. The '''project_export''' service exports the user configuration and filesystem named by the '''project''' (project=TIED) attribute. The next three lines configure the SEER experiment control system. The '''local_seer''' service will add a node named '''control''' to each experiment that is effictively invisible outside the party's area. Computers will treat that as a SEER controller, and experimenters can connect to it directly to see and manipulate their areas. The '''seer_master''' service adds a node named seer-master to the experiment that each area's controller will also connect to. Connecting to the seer-master controller allows thw whole experiment to be viewed and manipulated. == Creating the Experiment == To create the experiment: {{{ fedd_create.py --file multi-swap-example.tcl --experiment_name=faber-multi }}} You can provide any identifier you like for the experiment name. This will return pretty shortly (though not instantly) with output like this: {{{ localname: faber-multi fedid: b4339d8a339097e87ed5624a36b584fe95b0ea0d status: starting }}} The [FeddAbout#GlobalIdentifiers:Fedids fedid] is the unique name for the experiment, and the localname is a human-readable nickname. It is derived from the suggested experiment name passes to fedd_create.py. Generally the name passed is the one used, but if there is a name collision, fedd disambiguates. Finally the status is given. "Starting" means that fedd is constructing the experiment. This is generally a process that takes 10 minutes or so, though troubles with individual nodes can make it take longer.Fedd simultaneously swaps a separate experiment in for each sub experiment. You can check on the progress of the experiment using [FeddCommands#fedd_multistatus.py fedd_multistatus.py] which prints a brief status for each federated experiment: {{{ $ fedd_multistatus.py faber-multi:b4339d8a339097e87ed5624a36b584fe95b0ea0d:starting }}} When the experiment is active: {{{ $ fedd_multistatus.py faber-multi:b4339d8a339097e87ed5624a36b584fe95b0ea0d:active }}} Experiments can also be "failed" which means it will not come up, and a couple other self-explanatory states. Because creation takes a while, users want to see what's going on, even if it is somewhat cryptic. There are several ways to do this. The [FeddCommands#fedd_spewlog.py fedd_spewlog.py] command prints the fedd swapin log to stdout and terminates when the experiment fails or activates. For example: {{{ $ fedd_spewlog.py --experiment_name faber-multi 21 Sep 11 13:57:10 fedd.experiment_control.faber-multi Calling StartSegment at https://users.isi.deterlab.net:23233 21 Sep 11 13:57:10 fedd.experiment_control.faber-multi Calling StartSegment at https://users.isi.deterlab.net:23231 21 Sep 11 13:57:10 fedd.experiment_control.faber-multi Calling StartSegment at https://users.isi.deterlab.net:23231 21 Sep 11 13:57:10 fedd.experiment_control.faber-multi Calling StartSegment at https://users.isi.deterlab.net:23233 Allocated vlan: 3808Allocated vlan: 380921 Sep 11 13:58:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 1 mins) 21 Sep 11 13:59:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 2 mins) 21 Sep 11 14:00:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 3 mins) 21 Sep 11 14:01:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 4 mins) 21 Sep 11 14:02:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 5 mins) 21 Sep 11 13:57:17 fedd.access.faber-multi-a State is swapped 21 Sep 11 13:57:17 fedd.access.faber-multi-a [cmd_with_timeout]: /bin/rm -rf /proj/TIED/exp/faber-multi-a/tmp 21 Sep 11 13:57:17 fedd.access.faber-multi-a [cmd_with_timeout]: /bin/rm -rf /proj/TIED/software/faber-multi-a/* 21 Sep 11 13:57:17 fedd.access.faber-multi-a [cmd_with_timeout]: mkdir -p /proj/TIED/exp/faber-multi-a/tmp 21 Sep 11 13:57:17 fedd.access.faber-multi-a [cmd_with_timeout]: mkdir -p /proj/TIED/software/faber-multi-a 21 Sep 11 13:57:18 fedd.access.faber-multi-a [modify_exp]: Modifying faber-multi-a 21 Sep 11 13:57:34 fedd.access.faber-multi-a [modify_exp]: Modify succeeded 21 Sep 11 13:57:34 fedd.access.faber-multi-a [swap_exp]: Swapping faber-multi-a in 21 Sep 11 14:02:43 fedd.access.faber-multi-a [swap_exp]: Swap succeeded 21 Sep 11 14:02:43 fedd.access.faber-multi-a [get_mapping] Generating mapping 21 Sep 11 14:02:43 fedd.access.faber-multi-a Mapping complete21 Sep 11 14:03:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 6 mins) 21 Sep 11 14:04:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 7 mins) 21 Sep 11 14:05:10 fedd.experiment_control.faber-multi Waiting for sub threads (it has been 8 mins) }}} Once the DETER experiments begin to swap in, you can watch the activity logs on the [http://www.isi.deterlab.net DETER web interface] to see what the individual experiments are doing. In this example, you would look at SAFER/faber-multi-b and TIED/faber-multi-a. The projects come from knowing the projects to which each sub-part was assigned; the experiment names come from the log. The fedd logs are also world-readable on users.isi.deterlab.net. To watch a multi-party swap-in, the following logs may be interesting: {{{ /var/log/fedd.log # the main experiment controller log /var/log/fedd.deter.log # the log for the DETER testbed access controller (the component swapping experiments) /var/log/fedd.internal.log # the log for the access controller handing out VLANs to interconnect on DETER }}} The easiest way to view them is: {{{ $ tail -f /var/log/fedd.log }}} == Experiment Status == Once the experiment is started you can look at and control the individual pieces using standard DETER interfaces. You can get the topology from the [FeddCommands#fedd_ftopo.py fedd_ftopo.py] command. {{{ $ fedd_ftopo.py --experiment_name faber-multi d:pc027.isi.deterlab.net:deter/b e:pc020.isi.deterlab.net:deter/b a:pc009.isi.deterlab.net:deter/a b:pc044.isi.deterlab.net:deter/a c:pc022.isi.deterlab.net:deter/b }}} You can manipulate the various parts by logging into the machines or using SEER.