Administrator's Guide: Federating With DETER

This discusses how to connect a ​DETER or ​Emulab-based testbed to ​DETERLab using fedd, ​ABAC and their tools.

Introduction

This page takes a user through the process of adding federation components to an existing DETER and assigning user credentials that will allow users to access that DETER through the fedd interfaces, and then to use those same interfaces to access federated resources. We will also talk about how to allow federated users to use the local resources.

This document will discuss the relevant fedd components, what they do, and what they need in order to do those jobs. We also discuss the tools an experimenter uses to create and monitor federated experiments. Then we briefly discuss how to get the ABAC credentials necessary to access the federated environment.

Fedd really is two entities, an Access Controler that mediates access to federated resources and provides standard interfaces for allocating them, and an Experiment Controller that acts as a credential store and coordinates requests across multiple access controllers. The discussion of the Deter Federation Architecture has more to say about these entities, but this is enough to get started. DETERLab runs an experiment controller that can be reached at https://users.isi.deterlab.net:23235 and new federation users can use that experiment controller, but when setting up and administering a federated testbed it can be very helpful to install and configure an experiment controller locally.

Here is a block diagram of a user making a request for a federated experiment. We will show how to configure the credentials that users and the experiment controller pass to access controllers, how to configure the policies at the access controlers to allow access and map federated users into local users. We will also configure the experiment controller to find the access controlers and the access controllers to allocate resources (start DETER experiments) on behalf of local users.

Both the policy and credentials are ​ABAC credentials, but we provide tools to configure these for the common cases. Specifically we explain how to get a new testbed endorsed as a federation partner and how to allow federated partners access to local resources. Tools for finer-grained access control are in development, though ambitious users can use the raw ABAC tools to specify policy.

Road Map

The rest of this document explains how to do the following:

• Install the federation software
• Configure an experiment controller that translates local DETER projects and user identities into ABAC
• Configure an access controller to map those ABAC credentials back into local users and to allocate resources on their behalf
• Required software for portal nodes and nodes in federated experiments
• How to get credentials endorsed from DETERLab to gain access to federated resources and use them.

Installation

There are detailed instructions for installation of fedd and the required software.

Before downloading and installing the code, there are network connectivity requirements to consider. The experiment controller must be reachable by user tools and must be able to reach and be reached by access controllers. These are SSL protected TCP connections, though the port is configurable. Additionally, the some experimental nodes in the testbeds must be able to reach each other.

Configuring The Experiment Controller

The experiment controller needs a directory in which to store credentials and configuration information. This includes SSL credentials, so it should be protected. You can create a user to run the experiment controller if necessary.

There is a detailed reference for the configuration files.

This directory will contain the following configuration files

• fedd.conf - the main configuration file
• fedd.pem - a self-signed X.509 certificate identifying the controller
• exp_access_db - a human-readable database containing the mapping from experimenter identity to ABAC credentials (this is converted into ABAC)
• exp_map_db - a mapping of symbolic names to access controller URLs
• fedkit.tgz - the federation software run withing the experiment. Running make in the fedkit directory creates it.

In addition the following empty directories must be created for the controller to store experiment data in

• abac - the compiled ABAC credentials
• certs - experiment credentials
• repo - software that is distributed to access controlers for inclusion in federated experiments
• userconf - user configuration information for experiments (exported accounts...)

We recommend /usr/local/etc/fedd/experiment as the experiment controller's home directory.

fedd.conf

Here is an annotated fedd.conf file that describes the various parameters necessary to run an experiment controller:

[DEFAULT]

# This section is primarily to set internal variables for customization.  A 
# variable is inserted by %(variable)s .  NB: that s after the parens is
# literal.

# The Experiment Controllers home directory.  Used in susbtitutions.
base: /usr/local/etc/fedd
# Space in /var for the Experiment controller. Used in susbtitutions.
var: /var/db/fedd
# The DNS name the Experiment controller runs on. Used in susbtitutions.
hostname:users.isi.deterlab.net

[globals]

# Real EC configuration

# The certificate file that identifies this controller.
cert_file: %(base)s/fedd.pem

# The ports and protocols this node is supporting.  This says listen on poer 
# 23235 for SOAP connections and 23232 for XMLRPC connections.
services: 23235,23232:xmlrpc


[experiment_control]

# How to convert tcl (ns2) experiments into topdl.  Generally the safest way to
# do this is to contact DETER's EC using the ns2topdl_uri setting below.  If
# the testbed is a DETER testbed, you may be able to do the conversion locally
# using the tcl_splitter parameter below.  Pick one.

#tcl_splitter: /usr/testbed/opsdir/lib/ns2ir/parse.tcl
ns2topdl_uri: http://users.isi.deterlab.net:23235

# Staging area for software being sent to federated experiments.  This line 
# stores it under /var/db/fedd (from the var variable above.
repodir: %(var)s/repo

# Contact URL for retrieving software.
repo_url: https://%(hostname)s:23235

# Contact URL for retrieving experiment configuration data
store_url: https://%(hostname)s:23235

# File in which the dynamic experiment configuration data is stored.  You do 
# not need to create this file, nor manage it.
synch_store: %(base)s/store.dat

# ABAC configuration.  No other value for auth_type is currently valid.
auth_type = abac

# Location of the ABAC state
auth_dir = %(base)s/abac

# Time to cache status information, in seconds
info_cache = 300

# Logging level, in traditional syslog values.
log_level: debug

# Debugging parameters.  If leave_tempfiles is true, temporaries are not 
# deleted.  If create_debug is true, the local testbed is not accessed (no
# resources are allocated.)
#leave_tmpfiles: true
#create_debug: false

# Internal experiment controller state.  Completely managed internally
experiment_state: %(base)s/fedd.state

# List of testbed names to access URLs
mapdb: %(base)s/exp_map_db

# Software to install on each experiment node to form federated experiments.
# The first parameter is the directory in which to install it and the second
# the tar file to unpack.  These are the correct values for the fedkit
# distributed with fedd.
fedkit: /usr %(base)s/fedkit.tgz

The fedd.conf above is attached​ to this page for download and editing.

We have instructions for creating a certificate suitable for use as a cert_file.

A fedkit.tgz can be created from the fedkit directory of the git repository. Change to that directory and make.

The exp_map_db file is also attached​ and its contents are:

deter:https://users.isi.deterlab.net:23231
deter-test:https://users.isi.deterlab.net:23001
deter_internal:https://users.isi.deterlab.net:23233

These are the addresses of known access controllers. The tools can also request different mappings.

The most complex part of the configuration is setting up the ABAC policy.

Setting up the Experiment Controller's ABAC

For new federators we provide simple tools to create a policy based on the DETER projects and user accounts present in the loacl testbed. The ​exp_access_db.py command will read the database on the boss node and output a human-readable configuration file that tells what attributes the experiment controller will assert on behalf of the user.

In particular, the example at the link above produces a file that looks like this:

# faber
fedid:12ecc7415746281efa0ed58e180c51a5cba13a57->faber
fedid:12ecc7415746281efa0ed58e180c51a5cba13a57->(Deter,faber)
fedid:12ecc7415746281efa0ed58e180c51a5cba13a57->(emulab-ops,faber)
# lahey
fedid:d7da471ff2ba5b205d31ae379b1cf634c8dce1c6->lahey
fedid:d7da471ff2ba5b205d31ae379b1cf634c8dce1c6->(emulab-ops,lahey)

That output asserts that the holder of the X.509 identity with the hash 12ecc7415746281efa0ed58e180c51a5cba13a5 will be identified as the user faber and as a member of projects emulab-ops and Deter. DETER issues all users an X.509 certificate and that certificate is extracted by the script.

To create a file that asserts all users and projects in your testbed, run exp_access_db.py on your boss node and save the output in a file called exp_access_db.

To convert those assertions to ABAC in a format that access controllers will understand:

• copy exp_access_db to your experiment controller's home (/usr/local/etc/fedd/experiment)
• run fedd_to_abac.py to create the ABAC
  • fedd_to_abac.py --cert fedd.pem --dir /usr/local/etc/fedd/experiment/abac --make_dir exp_access_db should do it

At this point you should be able to run your experiment controller:

fedd.py --config=/usr/local/etc/fedd/experiment/fedd.conf --debug

You can specify a log file with the --log option.

Configuring the Access Controller

The access controller needs a similar set of files configured, but there are a couple key differences. For exposition, call the configuration file /usr/local/etc/fedd/access/access.conf.

The configuration file looks like (detailed documentation of the file format):

[DEFAULT]
# The home directory of the access controller
base: /usr/local/etc/fedd/access

[globals]
# Access controller identity
cert_file: %(base)s/fedd.pem

# Services as for the experiment controller
services: 23231

[access]

# syslog level
log_level: debug

# File that holds internal state between access controller boots
access_state: %(base)s/deter_access.state

# The file that maps from ABAC credentials to local user and project name.
# Generated by tools, but human readable.
accessdb: %(base)s/deter_abac_map

# Directory where certificates identifying particular allocations are created
certdir: %(base)s/certs

# The ssh identity used to move software into experiments.  Federated users
# must allow this key access to their DETER accounts.
ssh_privkey_file: %(base)s/fedd_rsa
ssh_pubkey_file: %(base)s/fedd_rsa.pub

# Same as the experiment controller values
# create_debug: true
#leave_tmpfiles: true

# This is the correct value for an Emulab or DETER testbed
type: local_emulab

# ABAC configuration and directories
auth_type: abac
auth_dir: %(base)s/abac

# URI at which to access the Emulab XMLRPC interface.  Generally only the boss
# node's DNS name needs to be
boss: https://boss.isi.deterlab.net:3069/usr/testbed
ops: users.isi.deterlab.net

# Image to be loaded on portal nodes.  Some version of FreeBSD is best.
portal_image: FBSD7-STD

# If the portals connect through a NAT, Uncomment this line and set the
# parameter to either the DNS name of the externally routable address or the IP
# address of that NAT.  This is used to establish routing between portal nodes.
# Portals behind a NAT will initiate all commmunications - so two NATted
# portals cannot talk.
# nat_portal: nat.isi.deterlab.net


# Configurable federation setup commands.  For the standard fedkit on a DETER,
# these are correct.
portal_startcommand: sudo -H /usr/bin/perl -I/usr/local/federation/lib /usr/local/federation/bin/combo.pl --use_file >& /tmp/bridge.log
node_startcommand: sudo -H /usr/bin/perl -I/usr/local/federation/lib /usr/local/federation/bin/federate.pl >& /tmp/federate


# DNS domain of the testbed
domain: .isi.deterlab.net

# Analogous to the experiment controller
federation_software: /usr %(base)s/fedkit.tgz

A copy is attached​.

An X.509 identity and ssh identity are required, and should be protected using proper file system protections.

Again, we configure the ABAC credentials in 2 steps from the existing DETER database. The system will map from an ABAC attribute to a user and project in which to create the federated experiment. The access controller makes allocations as a user through two mechanisms

• The XMLRPC actions to manipulate DETER experiments are taken with a copy of the DETER user's SSL certificate issued by DETER
  • Copies of these certificates are placed under the access controller's home directory, protected by the file system
• Additional software and data is moved into federated experiments using the access controller's ssh identity to log in as the user in question.

The analogue to exp_access.db.py for DETER access controllers is access_exp_access_db.py. It takes the same selection praameters as to exp_access.db.py and two others:

• --testbed takes the X.509 certificate of this experiment controller. This is used to get the fedid of the experiment controller.
• --cert_dir is the pathname under the access controller's home in which copies of users' X.509 identities will be stored.

This script is run on boss, so copy the experiment controller's certificate to boss as cert.pem and run something like:

access_exp_access_db.py --testbed cert.pem --cert_dir deter_certs

and capture it to a file named deter_access.

Now we need to gather the users' X.509 identities. The following is puts them all in deter_certs in the current directory:

db_to_certs.py --cert_dir deter_certs

Make a tar archive of that directory and copy it back to users. The certificate copies are all read-only and only accessible to the user who ran the script. If the access controller will be running as another user, you will have to change their owners when you install them on users.

Untar the certificates into /usr/local/etc/fedd/access/deter_certs and set permissions appropriately.

Now to convert all this to ABAC, run the following in /usr/local/etc/fedd/access:

access_to_abac.py --cert=fedd.pem --dir=/usr/local/etc/fedd/access/abac --type=emulab --mapfile=deter_abac_map deter_access

Note that the parameter to --mapfile must be the filename given as access_db: in the access.conf file.

The certificate used by deter_exp_access_db.py is the experiment controller certificate; the certificate used by access_to_abac.py is the access controller's certificate.

At this point the access controller can be run as:

fedd.py --config=/usr/local/etc/fedd/access/access.conf --debug

It will also take the --logfile parameter.

Preparing Users

In order for users to use the federation system, two things need to be done. Their emulab credentials need to be converted to federation credentials and they need to allow the access controller to act on their behalf by adding the access controller's ssh key to their login keys using the DETER web interface. Both of these can be done programmatically at large institutions.

Conversion of emulab credentials is done like this.

Trying it out

At this point the federation interface is configured to let local users make local experiments through the federation interface. To test this, add a line of the form

mytestbed:http://MYUSERS:MYPORT

to /usr/local/etc/fedd/experiment/exp_map_db and restart the experiment controller.

A user should be able to get a status by:

$ fedd_multistatus.py --url=$YOUR_EXPERIMENT_CONTROLLER_URL

Where $YOUR_EXPERIMENT_CONTROLLER_URL is the URL of your experiment controller. If that URL is reachable as ​https://localhost:23235 you can omit it. You can also set the FEDD_URL environment variable to the url and omit it. We will omit the --url below.

That status will be empty. To create an experiment, get the attached experiment file​, save it as local.xml and do:

$ fedd_create.py --file local.xml --experiment_name test-exp
localname: test-exp
fedid: 9e863dced90c570396ed8e3d433c412132b981f0
status: starting

The output indicates that the experiment is swapping in. You can get more information using the fedd_spewlog.py command or fedd_multistatus.py as well as watching the fedd log files and DETER web interface. The fedd_terminate.py command will delete the experiment. There are many command line tools to try.

Adding federation credentials to the experiment controller

While each federation agreement is different, the procedure is generally to negotiate some access with DETER and then receive a credential from the DETER experiment controller that delegates some rights to the federated system. This will be one or more X.509 attribute certificates that need to be installed in the policy of your local experiment controller. (Actually, you could also put them into each user's ABAC credentials directory, but the experiment controller is easier.)

To do this, use the import_abac_creds.py program to add these credentials to your policy. If the experiment controller's ABAC is kept in /usr/local/etc/fedd/abac the command will be something like:

$ import_abac_creds.py --dir /usr/local/etc/fedd/abac cred1.pem cred2.der cred3.pem cred4.der

Portal Nodes and Required Software

All nodes in federated experiments use dynamic routing to discover routes to nodes in other testbeds. To support this, Linux requires either gated or quagga to be accessible. If that software is not available the fedkit that runs on federated nodes will attempt to load it using either yum or apt-get if they are available and configured. If samba filesystems are imported the smbfs and samba-client packages will be needed.

Portal nodes - the nodes inserted by the federation system will need the Linux bridge utilities, bridge-utils installed. Again, if they are absent, the portal software will try to load them.

In the event that a either standard linux repositories for yum and apt-get or local copies are unavailable, local images should be provided with the software installed. The portal image should be set in the access controller configuration using the portal_image parameter. Users should be informed of the local node image to use.

The standard fedkit also requires access to the global DNS in order to find peer portal nodes.

Connectivity Requirements

If you are planning to put a federation instance up behind a firewall, here are the communication requirements:

• If you run an experiment controller, it needs to be reachable on the port(s) defined in the services: field of the experiment controller configuration. It acts as an information exchange point for access controllers to tell one another parameters to interconnect. Specifically, the experiment controller's service port must be accessible to any users who wish to contact it and any access controllers known to the experiment controller.
• Access controllers need to be accessible on their services: ports as well, from any experiment controller that will make requests and must be able to access any experiment controller that will make requests.
• Portal nodes need to be able to reach other portal nodes. This may be through a network address translator, but if so the portals can only connect to globally addressible portals. We are working to relax this.
• Portals use DNS to rendez-vous by default, so it is very helpful if DNS is available to them. Bare IP addresses can be used in a pinch.