| 198 | | === Federated Experiment Creation === |
| 199 | | |
| 200 | | TO give an intuition for how `fedd` works, we present a sample experiment description and show how it would be turned into a federated experiment. The experiment description is: |
| 201 | | |
| 202 | | {{{ |
| 203 | | # simple DETER topology for playing with SEER |
| 204 | | |
| 205 | | set ns [new Simulator] |
| 206 | | source tb_compat.tcl |
| 207 | | |
| 208 | | set a [$ns node] |
| 209 | | set b [$ns node] |
| 210 | | set c [$ns node] |
| 211 | | set d [$ns node] |
| 212 | | set e [$ns node] |
| 213 | | |
| 214 | | set deter_os "FC6-STD" |
| 215 | | set deter_hw "pc" |
| 216 | | set ucb_os "FC6-SMB" |
| 217 | | set ucb_hw "bvx2200" |
| 218 | | |
| 219 | | tb-set-node-os $a $deter_os |
| 220 | | tb-set-node-testbed $a "deter" |
| 221 | | tb-set-hardware $a $deter_hw |
| 222 | | |
| 223 | | |
| 224 | | tb-set-node-os $b $deter_os |
| 225 | | tb-set-node-testbed $b "deter" |
| 226 | | tb-set-hardware $b $deter_hw |
| 227 | | |
| 228 | | tb-set-node-os $c $ucb_os |
| 229 | | tb-set-node-testbed $c "ucb" |
| 230 | | tb-set-hardware $c $ucb_hw |
| 231 | | |
| 232 | | tb-set-node-os $d $ucb_os |
| 233 | | tb-set-node-testbed $d "ucb" |
| 234 | | tb-set-hardware $d $ucb_hw |
| 235 | | |
| 236 | | tb-set-node-os $e $ucb_os |
| 237 | | tb-set-node-testbed $e "ucb" |
| 238 | | tb-set-hardware $e $ucb_hw |
| 239 | | |
| 240 | | |
| 241 | | set link0 [ $ns duplex-link $a $b 100Mb 0ms DropTail] |
| 242 | | set link1 [ $ns duplex-link $c $b 100Mb 0ms DropTail] |
| 243 | | set lan0 [ $ns make-lan "$c $d $e" 100Mb 0ms ] |
| 244 | | |
| 245 | | $ns rtproto Static |
| 246 | | $ns run |
| 247 | | }}} |
| 248 | | |
| 249 | | That experiment file generates the following topology: |
| 250 | | |
| 251 | | [[Image(example_topology.png)]] |
| 252 | | |
| 253 | | Each of the boxes is a node and the circle is a shared network. Considering the values of the `set-node-testbed` calls, the experiment will be split up like this: |
| 254 | | |
| 255 | | [[Image(testbeds.png)]] |
| 256 | | |
| 257 | | The assignment of nodes to testbeds implicitly creates links (or networks) that must be connected across the wide area, and one of the key jobs of `fedd` is to create these connections. Fedd splits the experiment description into two descriptions (one for each testbed), adds extra nodes and startcmds to create the shared experiment, and swaps them in. The sub-experiments look something like these: |
| 258 | | |
| 259 | | {{{ |
| 260 | | set ns [new Simulator] |
| 261 | | source tb_compat.tcl |
| 262 | | |
| 263 | | set a [$ns node] |
| 264 | | tb-set-hardware $a pc |
| 265 | | tb-set-node-os $a FC6-STD |
| 266 | | tb-set-node-tarfiles $a /usr /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz |
| 267 | | # tb-set-node-testbed $a "deter" |
| 268 | | tb-set-node-startcmd $a "sudo -H /usr/local/federation/bin/make_hosts /proj/emulab-ops/exp/bwfed/tmp//hosts >& /tmp/federate \$USER" |
| 269 | | tb-set-node-failure-action $a "fatal" |
| 270 | | set b [$ns node] |
| 271 | | tb-set-hardware $b pc |
| 272 | | tb-set-node-os $b FC6-STD |
| 273 | | tb-set-node-tarfiles $b /usr /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz |
| 274 | | # tb-set-node-testbed $b "deter" |
| 275 | | tb-set-node-startcmd $b "sudo -H /usr/local/federation/bin/make_hosts /proj/emulab-ops/exp/bwfed/tmp//hosts >& /tmp/federate \$USER" |
| 276 | | set control [$ns node] |
| 277 | | tb-set-hardware $control pc |
| 278 | | tb-set-node-os $control FC6-STD |
| 279 | | tb-set-node-tarfiles $control /usr /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz |
| 280 | | # tb-set-node-testbed $control "deter" |
| 281 | | tb-set-node-startcmd $control "sudo -H /usr/local/federation/bin/make_hosts /proj/emulab-ops/exp/bwfed/tmp//hosts >& /tmp/federate \$USER " |
| 282 | | tb-set-node-failure-action $control "fatal" |
| 283 | | # Link establishment and parameter setting |
| 284 | | set link0 [$ns duplex-link $a $b 100000kb 0.0ms DropTail] |
| 285 | | |
| 286 | | tb-set-ip-link $a $link0 10.1.1.2 |
| 287 | | tb-set-ip-link $b $link0 10.1.1.3 |
| 288 | | # federation gateway |
| 289 | | set ucbtunnel0 [$ns node ] |
| 290 | | tb-set-hardware $ucbtunnel0 pc3000_tunnel |
| 291 | | tb-set-node-os $ucbtunnel0 FBSD7-TVF |
| 292 | | tb-set-node-startcmd $ucbtunnel0 "sudo -H /usr/local/federation/bin/fed-tun.pl -f /proj/emulab-ops/exp/bwfed/tmp/`hostname`.gw.conf >& /tmp/bridge.log" |
| 293 | | tb-set-node-tarfiles $ucbtunnel0 /usr/ /proj/emulab-ops//tarfiles/bwfed/fedkit.tgz |
| 294 | | # Link establishment and parameter setting |
| 295 | | set link1 [$ns duplex-link $ucbtunnel0 $b 100000kb 0.0ms DropTail] |
| 296 | | |
| 297 | | tb-set-ip-link $ucbtunnel0 $link1 10.1.3.2 |
| 298 | | tb-set-ip-link $b $link1 10.1.3.3 |
| 299 | | |
| 300 | | $ns rtproto Session |
| 301 | | $ns run |
| 302 | | }}} |
| 303 | | |
| 304 | | {{{ |
| 305 | | set ns [new Simulator] |
| 306 | | source tb_compat.tcl |
| 307 | | |
| 308 | | set c [$ns node] |
| 309 | | tb-set-hardware $c pc |
| 310 | | tb-set-node-os $c FC6-SMB |
| 311 | | tb-set-node-tarfiles $c /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz |
| 312 | | # tb-set-node-testbed $c "ucb" |
| 313 | | tb-set-node-startcmd $c "sudo -H /bin/sh /usr/local/federation/bin/federate.sh >& /tmp/startup \$USER " |
| 314 | | tb-set-node-failure-action $c "fatal" |
| 315 | | set d [$ns node] |
| 316 | | tb-set-hardware $d pc |
| 317 | | tb-set-node-os $d FC6-SMB |
| 318 | | tb-set-node-tarfiles $d /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz |
| 319 | | # tb-set-node-testbed $d "ucb" |
| 320 | | tb-set-node-startcmd $d "sudo -H /bin/sh /usr/local/federation/bin/federate.sh >& /tmp/startup \$USER " |
| 321 | | tb-set-node-failure-action $d "fatal" |
| 322 | | set e [$ns node] |
| 323 | | tb-set-hardware $e pc |
| 324 | | tb-set-node-os $e FC6-SMB |
| 325 | | tb-set-node-tarfiles $e /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz |
| 326 | | # tb-set-node-testbed $e "ucb" |
| 327 | | tb-set-node-startcmd $e "sudo -H /bin/sh /usr/local/federation/bin/federate.sh >& /tmp/startup \$USER" |
| 328 | | tb-set-node-failure-action $e "fatal" |
| 329 | | set f [$ns node] |
| 330 | | tb-set-hardware $f pc |
| 331 | | tb-set-node-os $f FC6-SMB |
| 332 | | tb-set-node-tarfiles $f /usr /proj/Deter//tarfiles/bwfed/fedkit.tgz |
| 333 | | # tb-set-node-testbed $f "ucb" |
| 334 | | tb-set-node-startcmd $f "sudo -H /bin/sh /usr/local/federation/bin/federate.sh >& /tmp/startup \$USER" |
| 335 | | tb-set-node-failure-action $f "fatal" |
| 336 | | # Create LAN |
| 337 | | set lan0 [$ns make-lan "$c $d $e " 100000kb 0.0ms ] |
| 338 | | |
| 339 | | # Set LAN/Node parameters |
| 340 | | tb-set-ip-lan $c $lan0 10.1.2.2 |
| 341 | | tb-set-lan-simplex-params $lan0 $c 0.0ms 100000kb 0 0.0ms 100000kb 0 |
| 342 | | |
| 343 | | |
| 344 | | # Set LAN/Node parameters |
| 345 | | tb-set-ip-lan $d $lan0 10.1.2.3 |
| 346 | | tb-set-lan-simplex-params $lan0 $d 0.0ms 100000kb 0 0.0ms 100000kb 0 |
| 347 | | |
| 348 | | |
| 349 | | # Set LAN/Node parameters |
| 350 | | tb-set-ip-lan $e $lan0 10.1.2.4 |
| 351 | | tb-set-lan-simplex-params $lan0 $e 0.0ms 100000kb 0 0.0ms 100000kb 0 |
| 352 | | |
| 353 | | |
| 354 | | # federation gateway |
| 355 | | set detertunnel0 [$ns node ] |
| 356 | | tb-set-hardware $detertunnel0 pc3000_tunnel |
| 357 | | tb-set-node-os $detertunnel0 FBSD7-TVF |
| 358 | | tb-set-node-startcmd $detertunnel0 "sudo -H /usr/local/federation/bin/fed-tun.pl -f /proj/Deter/exp/bwfed/tmp/`hostname`.gw.conf >& /tmp/bridge.log" |
| 359 | | tb-set-node-tarfiles $detertunnel0 /usr/ /proj/Deter//tarfiles/bwfed/fedkit.tgz |
| 360 | | # Link establishment and parameter setting |
| 361 | | set link1 [$ns duplex-link $c $detertunnel0 100000kb 0.0ms DropTail] |
| 362 | | |
| 363 | | tb-set-ip-link $c $link1 10.1.3.2 |
| 364 | | tb-set-ip-link $detertunnel0 $link1 10.1.3.3 |
| 365 | | |
| 366 | | $ns rtproto Session |
| 367 | | $ns run |
| 368 | | }}} |
| 369 | | |
| 370 | | The resulting topology across two testbeds looks like this: |
| 371 | | |
| 372 | | [[Image(federated_testbeds.png)]] |
| 373 | | |
| 374 | | The red nodes are the connector nodes inserted by `fedd`. They both transfer experimental traffic verbatim and tunnel services, like a shared filesystem. Exactly what gets tunneled and how the connections are made is a function of the federation kit, the topic of the next section. |
| 375 | | |
| 376 | | = Experiment Services = |
| 377 | | |
| 378 | | An important part of the cohesive experiment framework created by the DFA is the web of services interconnecting the sub experiments. A service, in this context, is fairly liberally defined; it is any one of several pre-defined ways of sharing information statically or dynamically between sub-experiments. Static information includes the accounts to create on various machines or the hostnames to export to sub-experiments. Dynamic information includes mounting filesystems or connecting user experiment tools, like [http://seer.isi.deterlab.net SEER]. |
| 379 | | |
| 380 | | Each service has an ''exporter'' the testbed that provides the service, and zero or more ''importers'', testbeds that use the information. In addition, services can have attribute/value pairs as parameters. |
| 381 | | |
| 382 | | The specific services currently supported by fedd are: |
| 383 | | |
| 384 | | '''hide_hosts''':: |
| 385 | | By default, all hosts in the topology are visible across the experiment. This allows some hosts to be completely local. The hosts chosen are given by the hosts attribute. |
| 386 | | '''local_seer_control''':: |
| 387 | | Adds a local seer control node to sub-experiments in the exporting testbed. |
| 388 | | '''project_export''':: |
| 389 | | Exports the user configuration and filesystem named by the project attribute to the importing testbeds |
| 390 | | '''seer_master''':: |
| 391 | | Adds a second seer controller to the exporting testbed that is capable of aggregating the inputs from several local seer masters. To create a seer instance that sees several sub experiments, create a local_seer_control in each and export a seer_master to all. |
| 392 | | '''SMB''':: |
| 393 | | Export a set of SMB/CIFS filesystems. |
| 394 | | '''userconfig''':: |
| 395 | | Export the account information from a given project or group. Currently accepts the project attribute for that purpose. |
| 396 | | |
| 397 | | Service advertisement, provision, and specification is an ongoing area of research in federation, and we expect these specifications to become more detailed and useful as that research progresses. |
| 398 | | |
| 399 | | = The Federation Kit = |
| 400 | | |
| 401 | | The federation kit is the software used by `fedd` to connect testbeds and tunnel services from the master to the other testbeds. Currently we have a single fedkit (available from the [FeddDownload download section]) that provides SSH tunnels to connect experimental testbeds at the ethernet layer, and that tunnels the emulab event system and provides shared file service via [http://en.wikipedia.org/wiki/Server_Message_Block SMB]. That fedkit runs on any FreeBSD or Linux image that provides the SMB file system. |
| 402 | | |
| 403 | | By splitting this function out, we intend to allow different installations of `fedd` to provide different interconnection and service tunneling function. Currently the [FeddDownload DETER fedkit] is the only federation kit in use, and `fedd` defaults its startcmd options for use with it. |
| 404 | | |
| 405 | | The federation kit has 2 roles |
| 406 | | |
| 407 | | * Configuring experiment nodes to use [FeddAbout#ExperimentServices services], such as shared file systems. |
| 408 | | * Configuring portal nodes to connect experiments |
| 409 | | |
| 410 | | == Fedkit on Experiment Nodes == |
| 411 | | |
| 412 | | On experiment nodes, the fedkit starts dynamic routing, and optionally configures user accounts and samba filesystems if they are in use. The system expects the following software to be available: |
| 413 | | |
| 414 | | * quagga routing system (an old gated installation will also work) |
| 415 | | * samba-client |
| 416 | | * smbfs |
| 417 | | |
| 418 | | Those are the linux package names; equivalent FreeBSD packages will also work. For software to be available is for it to be either installed or accessible using {{{yum}}} or {{{apt-get}}}. DETER nodes have a local repository for that purpose. |
| 419 | | |
| 420 | | The fedkit is installed in {{{/usr/local/federation}}} and when run places a log in {{{/tmp/federate}}}. |
| 421 | | |
| 422 | | Services are initialized based on the contents of {{{/usr/local/federation/etc/client.conf}}}. Possible values include: |
| 423 | | |
| 424 | | '''!ControlGateway''':: |
| 425 | | The DNS name (or IP address) of the node that will forward services |
| 426 | | '''Hide''': |
| 427 | | Do nat add this node to the node's view of the experiment. Used for [FeddMulti multi-party experiments]. |
| 428 | | '''!PortalAlias''':: |
| 429 | | A name that will be mapped to the same IP address as the control gateway. SEER in particular expects nodes with certain functions to have certain names. |
| 430 | | '''!ProjectUser''':: |
| 431 | | The local user under which to mount shared project directories |
| 432 | | '''!ProjectName''':: |
| 433 | | Project name to derive shared project directories from |
| 434 | | '''Service''':: |
| 435 | | a string naming the [FeddAbout#ExperimentServices services] to initialize. |
| 436 | | '''SMBShare''':: |
| 437 | | The name of the share to mount |
| 438 | | |
| 439 | | |
| 440 | | |
| 441 | | == Fedkit on Portal Nodes == |
| 442 | | |
| 443 | | On portal nodes the fedkit uses ssh to interconnect the segments and bridges traffic at layer 2. If the portal node is a Linux image it needs to have the {{{bridge-tools}}} package available. Like the fedkit on experiment nodes, it will attempt to load that software from repositories if it is not present. |
| 444 | | |
| 445 | | The fedkit configures the portal based on the contents of a configuration file containing the following parameters: |
| 446 | | |
| 447 | | '''active''':: |
| 448 | | a boolean. If true this portal will initiate ssh connections to its peer. |
| 449 | | '''nat_partner''':: |
| 450 | | a boolean. If true the fedkit's peer is behind a network address translator. Not used yet. |
| 451 | | '''tunnelip''':: |
| 452 | | a boolean. If true use the DETER system for binding external addresses. |
| 453 | | '''peer''':: |
| 454 | | a string. A list of DNS names or IP addresses. Usually this is one value, the DNS name of the peer, but passive ends of NATted portals may use a list of addresses to establish routing. |
| 455 | | '''ssh_pubkey''':: |
| 456 | | a string. A file in which the access controller has placed the ssh key shared by this portal and its peer. These are nonce keys discarded after the experiment ends. |
| 457 | | '''ssh_privkey''':: |
| 458 | | a string. A file in which the access controller has placed the ssh key shared by this portal and its peer. These are nonce keys discarded after the experiment ends. |
| 459 | | |
| 460 | | The passive portal node establishes routing connectivity to the active end, reconfigures the local sshd to allow link layer forwarding and to allow the active end to remotely configure it, and waits. The active end connects through ssh, establishes a link layer forwarding tunnel and bridges that to the experimental interface. It also forwards ports to connect experiment services. |
| 461 | | |
| 462 | | = Interfaces to fedd = |
| 463 | | |
| 464 | | The fedd interfaces are completely specified in WDSL and SOAP bindings. |
| 465 | | For simplicity and backward compatibility, fedd can export the same |
| 466 | | interfaces over XMLRPC, where the message formats are trivial |
| 467 | | translations. |
| 468 | | |
| 469 | | All fedd accesses are through HTTP over SSL. Fedd can support |
| 470 | | traditional SSL CA chaining for access control, but the intention is to |
| 471 | | use self-signed certificates representing fedids. The fedid semantics |
| 472 | | are always enforced. |
| 473 | | |
| 474 | | There are also several internal interfaces also specified in WSDL/SOAP |
| 475 | | and accessible through XMLRPC used to simplify the distributed |
| 476 | | implementation of fedd. You can read about those in the [FeddDevelop development section]. |
