Changes between Version 12 and Version 13 of FeddAbout

Mar 19, 2014 11:40:22 AM (5 years ago)



  • FeddAbout

    v12 v13  
    136136= Experiments =
    138 A federated experiment combines resources from multiple independently administered testbeds into a cohesive environment for experimentation.  Earlier versions of fedd enforced a !Master/Slave relationship with one emulab project exporting its environment to others.  While this model remains viable, current fedd implementations allow compositions of the services and views of the facilities as well as their resources.
     138A federated experiment combines resources from multiple independently administered testbeds into a cohesive environment for experimentation.  How much the independent testbeds preserve their internal configurations is a matter of policy and experiment controller requests.  Most of the existing access controllers will allow a master testbed to export user and filesystem information into all segments when that is consistent with policy.  Local access controllers can configure their resources as they noremally would as well.
    140140Currently experiments can be described in the same ns2 syntax that DETER
    141 and Emulab use, except that additional annotations for the testbed on
    142 which to place each node have been added.  (The extension is
    143 set-node-testbed, and described elsewhere.)  Most users will specify experiments in this format, initially. The testbed name here is a
    144 local string meaningful to the experimenter.  Each fedd knows some set
    145 of such strings, and experimenters can also inform fedd of the
    146 appropriate mapping between local testbed name and fedd address to
    147 contact.
     141uses. Most users will specify experiments in this format, initially.
    149142Fedd also understands experiments expressed in the [FeddPluginArchitecture#TopologyDescriptionLanguage topdl] language that it uses to communicate with facilities.
    151 An experimenter creates an experiment by presenting it to a fedd that
    152 knows him or her - usually the fedd on their local testbed - along with
    153 the master testbed name, local Emulab project to export and local
    154 mappings.  That fedd will map the local user into the global (testbed,
    155 project, user) namespace and acquire and configure resources to create
    156 the experiment.  A fedd may try different mappings of the experimenter
    157 into the global namespace in order to acquire all the resources.  For
    158 example a local experimenter may be a member of several local Emulab
    159 projects that map into different global testbed-scoped projects.
    161 Once created the user can gather information on the resources allocated to the experiment and where to access them.  (Experiments are identified by
    162 fedids).  The local name is mnemonic and can be used locally to access
    163 information about the experiment and to terminate it.  The experimenter
    164 can suggest a name when requesting experiment creation.
    166 In order to create a fedid, a key pair is created.  The public and
    167 private keys for the fedid representing the experiment are returned in
    168 an encrypted channel from fedd to the requesting experimenter.  The
    169 experimenter can use this key pair as a capability to grant access to the
    170 experiment to other users.
     144An experimenter creates an experiment by presenting it to an experiment controller that knows him or her - usually the DETER experiment contoller or one on their local testbed. That controller will map the local user into the global (testbed, project, user) namespace and acquire and configure resources to create the experiment.  An experiment controller may try different mappings of the experimenter into the global namespace in order to acquire all the resources.  For example a local experimenter may be a member of several local DETER projects that different access controllers will accept.
    172146While an experiment exists, experimenters can access the nodes directly
    173147using the credentials passed back and can acquire information about the
    174 experiment's topology from fedd.  Only the experimenter who created it
    175 or one given the capability to manipulate it can get this information.
    177 Finally, fedd will terminate an experiment on behalf of the experimenter
    178 who created it, or one who has been granted control of it.  Fedd is
    179 responsible for deallocating the running experiments and telling the
    180 testbeds that this experimenter is done with their access.
    182 The `fedd` installation includes a [ command line program] that allows users to create and delete federated experiments.  We are currently developing other interfaces, including a [ SEER ] interface.
     148experiment's topology from the experiment controller. 
    184150=== Experiment Descriptions ===
    186 An experiment description is currently given in a slightly extended version of the [ ns2 dialect that is used for an Emulab experiment description.] Specifically, the `tb-set-node-testbed` command has been added.  Its syntax is:
     152An experiment description is currently given in a slightly extended version of the [ ns2 dialect that is used for an DETER experiment description.] Specifically, the `tb-set-node-testbed` command has been added.  Its syntax is:
    196162In addition, `fedd` understands the `tb-set-default-failure-action` command.  This sets the default failure action for nodes in the experiment to one of '''fatal''', '''nonfatal''', or '''ignore'''.  Those values have the same meaning as in standard Emulabs. Nodes that do not have their failure mode reset by the `tb-set-node-failure-action` command will user the explicit default.  If no default is set, '''fatal''' is used.
    198 === Federated Experiment Creation ===
    200 TO give an intuition for how `fedd` works, we present a sample experiment description and show how it would be turned into a federated experiment.  The experiment description is:
    202 {{{
    203 # simple DETER topology for playing with SEER
    205 set ns [new Simulator]
    206 source tb_compat.tcl
    208 set a [$ns node]
    209 set b [$ns node]
    210 set c [$ns node]
    211 set d [$ns node]
    212 set e [$ns node]
    214 set deter_os "FC6-STD"
    215 set deter_hw "pc"
    216 set ucb_os "FC6-SMB"
    217 set ucb_hw "bvx2200"
    219 tb-set-node-os $a $deter_os
    220 tb-set-node-testbed $a "deter"
    221 tb-set-hardware $a $deter_hw
    224 tb-set-node-os $b $deter_os
    225 tb-set-node-testbed $b "deter"
    226 tb-set-hardware $b $deter_hw
    228 tb-set-node-os $c $ucb_os
    229 tb-set-node-testbed $c "ucb"
    230 tb-set-hardware $c $ucb_hw
    232 tb-set-node-os $d $ucb_os
    233 tb-set-node-testbed $d "ucb"
    234 tb-set-hardware $d $ucb_hw
    236 tb-set-node-os $e $ucb_os
    237 tb-set-node-testbed $e "ucb"
    238 tb-set-hardware $e $ucb_hw
    241 set link0 [ $ns duplex-link $a $b 100Mb 0ms DropTail]
    242 set link1 [ $ns duplex-link $c $b 100Mb 0ms DropTail]
    243 set lan0 [ $ns make-lan "$c $d $e" 100Mb 0ms ]
    245 $ns rtproto Static
    246 $ns run
    247 }}}
    249 That experiment file generates the following topology:
    251 [[Image(example_topology.png)]]
    253 Each of the boxes is a node and the circle is a shared network.  Considering the values of the `set-node-testbed` calls, the experiment will be split up like this:
    255 [[Image(testbeds.png)]]
    257 The assignment of nodes to testbeds implicitly creates links (or networks) that must be connected across the wide area, and one of the key jobs of `fedd` is to create these connections.  Fedd splits the experiment description into two descriptions (one for each testbed), adds extra nodes and startcmds to create the shared experiment, and swaps them in.  The sub-experiments look something like these:
    259 {{{
    260 set ns [new Simulator]
    261 source tb_compat.tcl
    263 set a [$ns node]
    264 tb-set-hardware $a pc
    265 tb-set-node-os $a FC6-STD
    266 tb-set-node-tarfiles $a /usr /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz
    267 # tb-set-node-testbed $a "deter"
    268 tb-set-node-startcmd $a "sudo -H /usr/local/federation/bin/make_hosts /proj/emulab-ops/exp/bwfed/tmp//hosts >& /tmp/federate \$USER"
    269 tb-set-node-failure-action $a "fatal"
    270 set b [$ns node]
    271 tb-set-hardware $b pc
    272 tb-set-node-os $b FC6-STD
    273 tb-set-node-tarfiles $b /usr /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz
    274 # tb-set-node-testbed $b "deter"
    275 tb-set-node-startcmd $b "sudo -H /usr/local/federation/bin/make_hosts /proj/emulab-ops/exp/bwfed/tmp//hosts >& /tmp/federate \$USER"
    276 set control [$ns node]
    277 tb-set-hardware $control pc
    278 tb-set-node-os $control FC6-STD
    279 tb-set-node-tarfiles $control /usr /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz
    280 # tb-set-node-testbed $control "deter"
    281 tb-set-node-startcmd $control "sudo -H /usr/local/federation/bin/make_hosts /proj/emulab-ops/exp/bwfed/tmp//hosts >& /tmp/federate \$USER "
    282 tb-set-node-failure-action $control "fatal"
    283 # Link establishment and parameter setting
    284 set link0 [$ns duplex-link $a $b 100000kb 0.0ms DropTail]
    286 tb-set-ip-link $a $link0
    287 tb-set-ip-link $b $link0
    288 # federation gateway
    289 set ucbtunnel0 [$ns node ]
    290 tb-set-hardware $ucbtunnel0 pc3000_tunnel
    291 tb-set-node-os $ucbtunnel0 FBSD7-TVF
    292 tb-set-node-startcmd $ucbtunnel0 "sudo -H /usr/local/federation/bin/ -f /proj/emulab-ops/exp/bwfed/tmp/`hostname`.gw.conf >& /tmp/bridge.log"
    293 tb-set-node-tarfiles $ucbtunnel0 /usr/ /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz
    294 # Link establishment and parameter setting
    295 set link1 [$ns duplex-link $ucbtunnel0 $b 100000kb 0.0ms DropTail]
    297 tb-set-ip-link $ucbtunnel0 $link1
    298 tb-set-ip-link $b $link1
    300 $ns rtproto Session
    301 $ns run
    302 }}}
    304 {{{
    305 set ns [new Simulator]
    306 source tb_compat.tcl
    308 set c [$ns node]
    309 tb-set-hardware $c pc
    310 tb-set-node-os $c FC6-SMB
    311 tb-set-node-tarfiles $c /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz
    312 # tb-set-node-testbed $c "ucb"
    313 tb-set-node-startcmd $c "sudo -H /bin/sh /usr/local/federation/bin/ >& /tmp/startup \$USER "
    314 tb-set-node-failure-action $c "fatal"
    315 set d [$ns node]
    316 tb-set-hardware $d pc
    317 tb-set-node-os $d FC6-SMB
    318 tb-set-node-tarfiles $d /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz
    319 # tb-set-node-testbed $d "ucb"
    320 tb-set-node-startcmd $d "sudo -H /bin/sh /usr/local/federation/bin/ >& /tmp/startup \$USER "
    321 tb-set-node-failure-action $d "fatal"
    322 set e [$ns node]
    323 tb-set-hardware $e pc
    324 tb-set-node-os $e FC6-SMB
    325 tb-set-node-tarfiles $e /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz
    326 # tb-set-node-testbed $e "ucb"
    327 tb-set-node-startcmd $e "sudo -H /bin/sh /usr/local/federation/bin/ >& /tmp/startup \$USER"
    328 tb-set-node-failure-action $e "fatal"
    329 set f [$ns node]
    330 tb-set-hardware $f pc
    331 tb-set-node-os $f FC6-SMB
    332 tb-set-node-tarfiles $f /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz
    333 # tb-set-node-testbed $f "ucb"
    334 tb-set-node-startcmd $f "sudo -H /bin/sh /usr/local/federation/bin/ >& /tmp/startup \$USER"
    335 tb-set-node-failure-action $f "fatal"
    336 # Create LAN
    337 set lan0 [$ns make-lan "$c $d $e " 100000kb 0.0ms ]
    339 # Set LAN/Node parameters
    340 tb-set-ip-lan $c $lan0
    341 tb-set-lan-simplex-params $lan0 $c 0.0ms 100000kb 0 0.0ms 100000kb 0
    344 # Set LAN/Node parameters
    345 tb-set-ip-lan $d $lan0
    346 tb-set-lan-simplex-params $lan0 $d 0.0ms 100000kb 0 0.0ms 100000kb 0
    349 # Set LAN/Node parameters
    350 tb-set-ip-lan $e $lan0
    351 tb-set-lan-simplex-params $lan0 $e 0.0ms 100000kb 0 0.0ms 100000kb 0
    354 # federation gateway
    355 set detertunnel0 [$ns node ]
    356 tb-set-hardware $detertunnel0 pc3000_tunnel
    357 tb-set-node-os $detertunnel0 FBSD7-TVF
    358 tb-set-node-startcmd $detertunnel0 "sudo -H /usr/local/federation/bin/ -f /proj/Deter/exp/bwfed/tmp/`hostname`.gw.conf >& /tmp/bridge.log"
    359 tb-set-node-tarfiles $detertunnel0 /usr/ /proj/Deter//tarfiles/bwfed/fedkit.tgz
    360 # Link establishment and parameter setting
    361 set link1 [$ns duplex-link $c $detertunnel0 100000kb 0.0ms DropTail]
    363 tb-set-ip-link $c $link1
    364 tb-set-ip-link $detertunnel0 $link1
    366 $ns rtproto Session
    367 $ns run
    368 }}}
    370 The resulting topology across two testbeds looks like this:
    372 [[Image(federated_testbeds.png)]]
    374 The red nodes are the connector nodes inserted by `fedd`.  They both transfer experimental traffic verbatim and tunnel services, like a shared filesystem.  Exactly what gets tunneled and how the connections are made is a function of the federation kit, the topic of the next section.
    376 = Experiment Services =
    378 An important part of the cohesive experiment framework created by the DFA is the web of services interconnecting the sub experiments.  A service, in this context, is fairly liberally defined; it is any one of several pre-defined ways of sharing information statically or dynamically between sub-experiments.  Static information includes the accounts to create on various machines or the hostnames to export to sub-experiments.  Dynamic information includes mounting filesystems or connecting user experiment tools, like [ SEER].
    380 Each service has an ''exporter'' the testbed that provides the service, and zero or more ''importers'', testbeds that use the information.  In addition, services can have attribute/value pairs as parameters.
    382 The specific services currently supported by fedd are:
    384  '''hide_hosts'''::
    385  By default, all hosts in the topology are visible across the experiment.  This allows some hosts to be completely local.  The hosts chosen are given by the hosts attribute.
    386  '''local_seer_control'''::
    387  Adds a local seer control node to sub-experiments in the exporting testbed.
    388  '''project_export'''::
    389  Exports the user configuration and filesystem named by the project attribute to the importing testbeds
    390  '''seer_master'''::
    391   Adds a second seer controller to the exporting testbed that is capable of aggregating the inputs from several local seer masters.  To create a seer instance that sees several sub experiments, create a local_seer_control in each and export a seer_master to all.
    392  '''SMB'''::
    393   Export a set of SMB/CIFS filesystems.
    394  '''userconfig'''::
    395   Export the account information from a given project or group.  Currently accepts the project attribute for that purpose.
    397 Service advertisement, provision, and specification is an ongoing area of research in federation, and we expect these specifications to become more detailed and useful as that research progresses.
    399 = The Federation Kit =
    401 The federation kit is the software used by `fedd` to connect testbeds and tunnel services from the master to the other testbeds. Currently we have a single fedkit (available from the [FeddDownload download section]) that provides SSH tunnels to connect experimental testbeds at the ethernet layer, and that tunnels the emulab event system and provides shared file service via [ SMB].  That fedkit runs on any FreeBSD or Linux image that provides the SMB file system.
    403 By splitting this function out, we intend to allow different installations of `fedd` to provide different interconnection and service tunneling function.  Currently the [FeddDownload DETER fedkit] is the only federation kit in use, and `fedd` defaults its startcmd options for use with it.
    405 The federation kit has 2 roles
    407  * Configuring experiment nodes to use [FeddAbout#ExperimentServices services], such as shared file systems.
    408  * Configuring portal nodes to connect experiments
    410 == Fedkit on Experiment Nodes ==
    412 On experiment nodes, the fedkit starts dynamic routing, and optionally configures user accounts and samba filesystems if they are in use.  The system expects the following software to be available:
    414  * quagga routing system (an old gated installation will also work)
    415  * samba-client
    416  * smbfs
    418 Those are the linux package names; equivalent FreeBSD packages will also work.  For software to be available is for it to be either installed or accessible using {{{yum}}} or {{{apt-get}}}.  DETER nodes have a local repository for that purpose.
    420 The fedkit is installed in {{{/usr/local/federation}}} and when run places a log in {{{/tmp/federate}}}.
    422 Services are initialized based on the contents of {{{/usr/local/federation/etc/client.conf}}}.  Possible values include:
    424  '''!ControlGateway'''::
    425   The DNS name (or IP address) of the node that will forward services
    426  '''Hide''':
    427   Do nat add this node to the node's view of the experiment.  Used for [FeddMulti multi-party experiments].
    428  '''!PortalAlias'''::
    429   A name that will be mapped to the same IP address as the control gateway.  SEER in particular expects nodes with certain functions to have certain names.
    430  '''!ProjectUser'''::
    431   The local user under which to mount shared project directories
    432  '''!ProjectName'''::
    433   Project name to derive shared project directories from
    434  '''Service'''::
    435   a string naming the [FeddAbout#ExperimentServices services] to initialize.
    436  '''SMBShare'''::
    437   The name of the share to mount
    441 == Fedkit on Portal Nodes ==
    443 On portal nodes the fedkit uses ssh to interconnect the segments and bridges traffic at layer 2.  If the portal node is a Linux image it needs to have the {{{bridge-tools}}} package available.  Like the fedkit on experiment nodes, it will attempt to load that software from repositories if it is not present.
    445 The fedkit configures the portal based on the contents of a configuration file containing the following parameters:
    447  '''active'''::
    448   a boolean.  If true this portal will initiate ssh connections to its peer.
    449  '''nat_partner'''::
    450   a boolean.  If true the fedkit's peer is behind a network address translator.  Not used yet.
    451  '''tunnelip'''::
    452   a boolean.  If true use the DETER system for binding external addresses.
    453  '''peer'''::
    454   a string.  A list of DNS names or IP addresses.  Usually this is one value, the DNS name of the peer, but passive ends of NATted portals may use a list of addresses to establish routing.
    455  '''ssh_pubkey'''::
    456   a string.  A file in which the access controller has placed the ssh key shared by this portal and its peer.  These are nonce keys discarded after the experiment ends.
    457  '''ssh_privkey'''::
    458   a string.  A file in which the access controller has placed the ssh key shared by this portal and its peer.  These are nonce keys discarded after the experiment ends.
    460 The passive portal node establishes routing connectivity to the active end, reconfigures the local sshd to allow link layer forwarding and to allow the active end to remotely configure it, and waits.  The active end connects through ssh, establishes a link layer forwarding tunnel and bridges that to the experimental interface.  It also forwards ports to connect experiment services.
    462 = Interfaces to fedd =
    464 The fedd interfaces are completely specified in WDSL and SOAP bindings.
    465 For simplicity and backward compatibility, fedd can export the same
    466 interfaces over XMLRPC, where the message formats are trivial
    467 translations.
    469 All fedd accesses are through HTTP over SSL.  Fedd can support
    470 traditional SSL CA chaining for access control, but the intention is to
    471 use self-signed certificates representing fedids.  The fedid semantics
    472 are always enforced.
    474 There are also several internal interfaces also specified in WSDL/SOAP
    475 and accessible through XMLRPC used to simplify the distributed
    476 implementation of fedd.  You can read about those in the [FeddDevelop development section].