= Federating With DETER =

This discusses how to connect a [http://deterproject.org DETER] or [http://www.emulab.net Emulab]-based testbed to [https://www.isi.deterlab.net DETERLab] using [WikiStart fedd], [http://abac.deterlab.net ABAC] and their tools.

== Introduction ==

This page takes a user through the process of adding federation components to an existing DETER and assigning user credentials that will allow users to access that DETER through the fedd interfaces, and then to use those same interfaces to access federated resources.  We will also talk about how to allow federated users to use the local resources.

This document will discuss the relevant fedd components, what they do, and what they need in order to do those jobs.  We also discuss the tools an experimenter uses to create and monitor federated experiments.  Then we briefly discuss how to get the ABAC credentials necessary to access the federated environment.

Fedd really is two entities, an '''Access Controler''' that mediates access to federated resources and provides standard interfaces for allocating them, and an '''Experiment Controller''' that acts as a credential store and coordinates requests across multiple access controllers.  The [FeddAbout discussion of the Deter Federation Architecture] has more to say about these entities, but this is enough to get started. DETERLab runs an experiment controller that can be reached at [https://users.isi.deterlab.net:23235] and new federation users can use that experiment controller, but when setting up and administering a federated testbed it can be very helpful to install and configure an experiment controller locally.

Here is a block diagram of a user making a request for a federated experiment.  We will show how to configure the credentials that users and the experiment controller pass to access controllers, how to configure the policies at the access controlers to allow access and map federated users into local users.  We will also configure the experiment controller to find the access controlers and the access controllers to allocate resources (start DETER experiments) on behalf of local users.

[[Image(fed.png)]]

Both the policy and credentials are [http://abac.deterleb.net ABAC] credentials, but we provide tools to configure these for the common cases.  Specifically we explain how to get a new testbed endorsed as a federation partner and how to allow federated partners access to local resources. Tools for finer-grained access control are in development, though ambitious users can use the raw ABAC tools to specify policy.

== Road Map ==

The rest of this document explains how to do the following:

 * Install the federation software
 * Configure an experiment controller that translates local DETER projects and user identities into ABAC
 * Configure an access controller to map those ABAC credentials back into local users and to allocate resources on their behalf
 * Required software for portal nodes and nodes in federated experiments
 * How to get credentials endorsed from DETERLab to gain access to federated resources and use them.

== Installation ==

There are detailed instructions for [FeddDownload installation of fedd and the required software].  As we prepare for the fedd 4.0 release, it is best to [FeddDownload#Gitaccess install from git].

Before downloading and installing the code, there are network connectivity requirements to consider.  The experiment controller must be reachable by user tools and must be able to reach and be reached by access controllers.  These are SSL protected TCP connections, though the port is configurable.  Additionally, the some experimental nodes in the testbeds must be able to reach each other.

== Configuring The Experiment Controller ==

The experiment controller needs a directory in which to store credentials and configuration information.  This includes SSL credentials, so it should be protected.  You can create a user to run the experiment controller if necessary.

This directory will contain the following configuration files

 * fedd.conf - the main configuration file
 * fedd.pem - a self-signed X.509 certificate identifying the controller
 * exp_access_db - a human-readable database containing the mapping from experimenter identity to ABAC credentials (this is converted into ABAC)
 * exp_map_db - a mapping of symbolic names to access controller URLs
 * fedkit.tgz - the federation software run withing the experiment.  Running make in the fedkit directory creates it.

In addition the following empty directories must be created for the controller to store experiment data in

 * abac - the compiled ABAC credentials
 * certs - experiment credentials
 * repo - software that is distributed to access controlers for inclusion in federated experiments
 * userconf - user configuration information for experiments (exported accounts...)

We recommend {{{/usr/local/etc/fedd/experiment}}} as the experiment controller's home directory.

=== fedd.conf ===

Here is an annotated {{{fedd.conf}}} file that describes the various parameters necessary to run an experiment controller:

{{{
[DEFAULT]

# This section is primarily to set internal variables for customization.  A 
# variable is inserted by %(variable)s .  NB: that s after the parens is
# literal.

# The Experiment Controllers home directory.  Used in susbtitutions.
base: /usr/local/etc/fedd
# Space in /var for the Experiment controller. Used in susbtitutions.
var: /var/db/fedd
# The DNS name the Experiment controller runs on. Used in susbtitutions.
hostname:users.isi.deterlab.net

[globals]

# Real EC configuration

# The certificate file that identifies this controller.
cert_file: %(base)s/fedd.pem

# The ports and protocols this node is supporting.  This says listen on poer 
# 23235 for SOAP connections and 23232 for XMLRPC connections.
services: 23235,23232:xmlrpc


[experiment_control]

# How to convert tcl (ns2) experiments into topdl.  Generally the safest way to
# do this is to contact DETER's EC using the ns2topdl_uri setting below.  If
# the testbed is a DETER testbed, you may be able to do the conversion locally
# using the tcl_splitter parameter below.  Pick one.

#tcl_splitter: /usr/testbed/opsdir/lib/ns2ir/parse.tcl
ns2topdl_uri: http://users.isi.deterlab.net:23235

# Staging area for software being sent to federated experiments.  This line 
# stores it under /var/db/fedd (from the var variable above.
repodir: %(var)s/repo

# Contact URL for retrieving software.
repo_url: https://%(hostname)s:23235

# Contact URL for retrieving experiment configuration data
store_url: https://%(hostname)s:23235

# File in which the dynamic experiment configuration data is stored.  You do 
# not need to create this file, nor manage it.
synch_store: %(base)s/store.dat

# ABAC configuration.  No other value for auth_type is currently valid.
auth_type = abac

# Location of the ABAC state
auth_dir = %(base)s/abac

# Time to cache status information, in seconds
info_cache = 300

# Logging level, in traditional syslog values.
log_level: debug

# Debugging parameters.  If leave_tempfiles is true, temporaries are not 
# deleted.  If create_debug is true, the local testbed is not accessed (no
# resources are allocated.)
#leave_tmpfiles: true
#create_debug: false

# Internal experiment controller state.  Completely managed internally
experiment_state: %(base)s/fedd.state

# List of testbed names to access URLs
mapdb: %(base)s/exp_map_db

# Software to install on each experiment node to form federated experiments.
# The first parameter is the directory in which to install it and the second
# the tar file to unpack.  These are the correct values for the fedkit
# distributed with fedd.
fedkit: /usr %(base)s/fedkit.tgz
}}}

The {{{fedd.conf}}} above is [attachment:experiment.conf attached] to this page for download and editing.  

We have [FeddConfig#MakingaFedidCertificate instructions] for creating a certificate suitable for use as a {{{cert_file}}}.

A {{{fedkit.tgz}}} can be created from the {{{fedkit}}} directory of the git repository.  Change to that directory and make.

The exp_map_db file is also [attachment:exp_map_db attached] and its contents are:

{{{
deter:https://users.isi.deterlab.net:23231
deter-test:https://users.isi.deterlab.net:23001
deter_internal:https://users.isi.deterlab.net:23233
}}}

These are the addresses of known access controllers.  The tools can also request different mappings.

The most complex part of the configuration is setting up the ABAC policy.

=== Setting up the Experiment Controller's ABAC ===

For new federators we provide simple tools to create a policy based on the DETER projects and user accounts present in the loacl testbed.  The [http://fedd.deterlab.net/wiki/FeddCommands#exp_access_db.py exp_access_db.py] command will read the database on the boss node and output a human-readable configuration file that tells what attributes the experiment controller will assert on behalf of the user.

In particular, the example at the link above produces a file that looks like this:

{{{
# faber
fedid:12ecc7415746281efa0ed58e180c51a5cba13a57->faber
fedid:12ecc7415746281efa0ed58e180c51a5cba13a57->(Deter,faber)
fedid:12ecc7415746281efa0ed58e180c51a5cba13a57->(emulab-ops,faber)
# lahey
fedid:d7da471ff2ba5b205d31ae379b1cf634c8dce1c6->lahey
fedid:d7da471ff2ba5b205d31ae379b1cf634c8dce1c6->(emulab-ops,lahey)
}}}

That output asserts that the holder of the X.509 identity with the hash 12ecc7415746281efa0ed58e180c51a5cba13a5 will be identified as the user {{{faber}}} and as a member of projects {{{emulab-ops}}} and {{{Deter}}}.  DETER issues all users an X.509 certificate and that certificate is extracted by the script.

To create a file that asserts all users and projects in your testbed, run {{{exp_access_db.py}}} on your boss node and save the output in a file called {{{exp_access_db}}}.

To convert those assertions to ABAC in a format that access controllers will understand:

 * copy {{{exp_access_db}}} to your experiment controller's home ({{{/usr/local/etc/fedd/experiment}}})
 * run [wiki:FeddABAC#fedd_to_abac.py fedd_to_abac.py] to create the ABAC
   * {{{fedd_to_abac.py --cert fedd.pem --dir /usr/local/etc/fedd/experiment/abac --make_dir exp_access_db}}} should do it

At this point you should be able to run your experiment controller:

{{{fedd.py --config=/usr/local/etc/fedd/experiment/fedd.conf --debug}}}

You can specify a log file with the --log option.

== Configuring the Access Controller ==

The access controller needs a similar set of files configured, but there are a couple key differences. For exposition, call the configuration file {{{/usr/local/etc/fedd/access/access.conf}}}.

The configuration file looks like:

{{{
[DEFAULT]
# The home directory of the access controller
base: /usr/local/etc/fedd/access

[globals]
# Access controller identity
cert_file: %(base)s/fedd.pem

# Services as for the experiment controller
services: 23231

[access]

# syslog level
log_level: debug

# File that holds internal state between access controller boots
access_state: %(base)s/deter_access.state

# The file that maps from ABAC credentials to local user and project name.
# Generated by tools, but human readable.
accessdb: %(base)s/deter_abac_map

# Directory where certificates identifying particular allocations are created
certdir: %(base)s/certs

# The ssh identity used to move software into experiments.  Federated users
# must allow this key access to their DETER accounts.
ssh_privkey_file: %(base)s/fedd_rsa
ssh_pubkey_file: %(base)s/fedd_rsa.pub

# Same as the experiment controller values
# create_debug: true
#leave_tmpfiles: true

# This is the correct value for an Emulab or DETER testbed
type: local_emulab

# ABAC configuration and directories
auth_type: abac
auth_dir: %(base)s/abac

# URI at which to access the Emulab XMLRPC interface.  Generally only the boss
# node's DNS name needs to be
boss: https://boss.isi.deterlab.net:3069/usr/testbed
ops: users.isi.deterlab.net

# Image to be loaded on portal nodes.  Some version of FreeBSD is best.
portal_image: FBSD7-STD

# Configurable federation setup commands.  For the standard fedkit on a DETER,
# these are correct.
portal_startcommand: sudo -H /usr/bin/perl -I/usr/local/federation/lib /usr/local/federation/bin/combo.pl --use_file >& /tmp/bridge.log
node_startcommand: sudo -H /usr/bin/perl -I/usr/local/federation/lib /usr/local/federation/bin/federate.pl >& /tmp/federate


# DNS domain of the testbed
domain: .isi.deterlab.net

# Analogous to the experiment controller
federation_software: /usr %(base)s/fedkit.tgz


}}}

A copy is [attachment:access.conf attached].

An X.509 identity and ssh identity are required, and should be protected using proper file system protections.

Again, we configure the ABAC credentials in 2 steps from the existing DETER database.  The system will map from an ABAC attribute to a user and project in which to create the federated experiment.  The access controller makes allocations as a user through two mechanisms

 * The XMLRPC actions to manipulate DETER experiments are taken with a copy of the DETER user's SSL certificate issued by DETER
   * Copies of these certificates are placed under the access controller's home directory, protected by the file system
 * Additional software and data is moved into federated experiments using the access controller's ssh identity to log in as the user in question.

The analogue {{{to exp_access.db.py}}} for DETER access controllers is {{{deter_exp_access_db.py}}}.  It takes the same selection praameters as {{{to exp_access.db.py}}} and two others:

 * --testbed takes the X.509 certificate of this experiment controller.  This is used to get the fedid of the experiment controller.
 * --cert-dir is the pathname under the access controller's home in which copies of users' X.509 identities will be stored.

This script is run on boss, so copy the experiment controller's certificate to boss as {{{cert.pem}}} and run something like:

{{{
deter_exp_access_db.py --testbed cert.pem --cert-dir deter_certs
}}}

and capture it to a file named {{{deter_access}}}.

Now we need to gather the users' X.509 identities.  The following is puts them all in deter_certs in the current directory:

{{{
db_to_certs.py --cert-dir deter_certs
}}}

Make a tar archive of that directory and copy it back to users.  The certificate copies are all read-only and only accessible to the user who ran the script.  If the access controller will be running as another user, you will have to change their owners when you install them on users.

Untar the certificates into {{{/usr/local/etc/fedd/access/deter_certs}}} and set permissions appropriately.

Now to convert all this to ABAC, run the following in {{{/usr/local/etc/fedd/access}}}:

{{{
access_to_abac.py --cert=fedd.pem --dir=/usr/local/etc/fedd/access/abac --type=emulab --mapfile=deter_abac_map deter_access
}}}

Note that the parameter to {{{--mapfile}}} must be the filename given as {{{access_db:}}} in the {{{access.conf}}} file.

The certificate used by {{{deter_exp_access_db.py}}} is the experiment controller certificate; the certificate used by {{{access_to_abac.py}}} is the access controller's certificate.

At this point the access controller can be run as:

{{{fedd.py --config=/usr/local/etc/fedd/access/access.conf --debug}}}

It will also take the {{{--logfile}}} parameter.

== Preparing Users ==

In order for users to use the federation system, two things need to be done.  Their emulab credentials need to be converted to federation credentials and they need to allow the access controller to act on their behalf by adding the access controller's ssh key to their login keys using the DETER web interface.  Both of these can be done programmatically at large institutions.

Conversion of emulab credentials is done [wiki:FeddABAC#UserCredentials like this].

== Trying it out ==

At this point the federation interface is configured to let local users make local experiments through the federation interface.  To test this, add a line of the form

{{{
mytestbed:http://MYUSERS:MYPORT
}}}

to {{{/usr/local/etc/fedd/experiment/exp_map_db}}} and restart the experiment controller.

A user should be able to get a status by:

{{{
$ fedd_multistatus.py --url=$YOUR_EXPERIMENT_CONTROLLER_URL
}}}
Where $YOUR_EXPERIMENT_CONTROLLER_URL is the URL of your experiment controller.  If that URL is reachable as https://localhost:23235 you can omit it.  You can also set the FEDD_URL environment variable to the url and omit it.  We will omit the {{{--url}}} below.

That status will be empty.  To create an experiment, get the [attachment:local.xml attached experiment file], save it as {{{local.xml}}} and do:

{{{
$ fedd_create.py --file local.xml --experiment_name test-exp
localname: test-exp
fedid: 9e863dced90c570396ed8e3d433c412132b981f0
status: starting
}}}

The output indicates that the experiment is swapping in.  You can get more information using the [FeddCommands#fedd_spewlog.py fedd_spewlog.py] command or [FeddCommands#fedd_multistatus.py fedd_multistatus.py] as well as watching the fedd log files and DETER web interface.  The [FeddCommands#fedd_terminate.py fedd_terminate.py] command will delete the experiment.  There are [FeddCommands#CommandLine many command line tools to try].

== Adding federation credentials to the experiment controller ==

While each federation agreement is different, the procedure is generally to negotiate some access with DETER and then receive a credential from the DETER experiment controller that delegates some rights to the federated system.  This will be one or more X.509 attribute certificates that need to be installed in the policy of your local experiment controller.  (Actually, you could also put them into each user's ABAC credentials directory, but the experiment controller is easier.)

To do this, use the [wiki:FeddABAC#import_abac_creds.py import_abac_creds.py] program to add these credentials to your policy.  If the experiment controller's ABAC is kept in {{{/usr/local/etc/fedd/abac}}} the command will be something like:

{{{
$ import_abac_creds.py --dir /usr/local/etc/fedd/abac cred1.pem cred2.der cred3.pem cred4.der
}}}

== Portal Nodes and Required Software ==

All nodes in federated experiments use dynamic routing to discover routes to nodes in other testbeds.  To support this, Linux requires either {{{gated}}} or {{{quagga}}} to be accessible.  If that software is not available the fedkit that runs on federated nodes will attempt to load it using either {{{yum}}} or {{{apt-get}}} if they are available and configured.  If samba filesystems are imported the {{{smbfs}}} and {{{samba-client}}} packages will be needed.

Portal nodes - the nodes inserted by the federation system will need the Linux bridge utilities, {{{bridge-utils}}} installed.  Again, if they are absent, the portal software will try to load them.

In the event that a either standard linux repositories for {{{yum}}}  and {{{apt-get}}} or local copies are unavailable, local images should be provided with the software installed.  The portal image should be set in the access controller configuration using the portal_image parameter.  Users should be informed of the local node image to use.

The standard fedkit also requires access to the global DNS in order to find peer portal nodes.

== Connectivity Requirements ==

If you are planning to put a federation instance up behind a firewall, here are the communication requirements:

 * If you run an experiment controller, it needs to be reachable on the port(s) defined in the {{{services:}}} field of the experiment controller configuration.  It acts as an information exchange point for access controllers to tell one another parameters to interconnect.  Specifically, the experiment controller's service port must be accessible to any users who wish to contact it and any access controllers known to the experiment controller.
 * Access controllers need to be accessible on their {{{services:}}} ports as well, from any experiment controller that will make requests and must be able to access any experiment controller that will make requests.
 * Portal nodes need to be able to reach other portal nodes.  This may be through a network address translator, but if so the portals can only connect to globally addressible portals.  We are working to relax this.
 * Portals use DNS to rendez-vous by default, so it is very helpful if DNS is available to them.  Bare IP addresses can be used in a pinch.