The DETER Federation Architecture (DFA)

This is a basic overview of the DETER federation system. More details on this implementation and configuration are in later documents. You can also read these academic papers that capture the evolution of the design to this point.

The DFA is a system that allows a researcher to construct experiments that span testbeds by dynamically acquiring resources from other testbeds and configuring them into a single experiment. As closely as possible that experiment will mimic a single DETER/Emulab experiment.

Though the experiment appears to be a cohesive whole, the testbeds that loan the resources retain control of those resources. Because testbeds retain this control, each testbed may issue credentials necessary for manipulating the federated resources. For example, a testbed that has loaned nodes to an experiment may require the experimenter to present a credential issued by that testbed (e.g., an SSH key or SSL certificate) to reboot those nodes. The system acquires those credentials on behalf of experimenters and distributes them on behalf of testbeds.

Testbed administrators may use the system to establish regular policies between testbeds to share resources across many users of a testbed. Similarly, a single user with accounts on multiple testbeds can use the same interfaces to coordinate experiments that share his testbed resource, assuming sharing those resources does not violate the policy of any of the constituent testbeds.

Federated Experiments

A researcher may build a federated experiment to

• Connect resources from testbeds with different focuses - e.g., connect power system resources with networking resources
• Connect resource owned by different actors for collaboration or competition
• Connect a local workstation environment to an experiment to monitor or demonstrate the results of an experiment.

An all of these cases, a researcher specifies the resources and their interconnections to the DFA, which coordinates between the testbeds and connects the resources.

Consider a researcher who has access to several testbeds and wants to interconnect resources from those testbeds.

From disparate testbeds, the DETER Federation System will create a single unified environment.

Combining Specialized Resources

The most common reason to create a federated experiment is to build an experimental environment that combines the specialties of multiple institutions.

For example, DETER researchers have used the federation system to interconnect electrical power systems, high speed power simulation, a controlled IP network and a situational awareness display sited at three disparate research labs into a coherent whole. This example shows a federation between DETER, tyhe University of Illinois and the Pacific Northwest National Laboratory (PNNL).

The federation system handles the internetworking of those resources, allowing the researchers to combine expertise.

Collaboration and Competition

Another use of federation is to construct experiments where different reserachers have different levels of access to different parts of the overall experiment. Experimenters preparing defenses can be linked to experimenters preparing attacks and to a shared environment unknown to both of them. The result is a controlled interaction between groups of experiment elements.

DETER researchers have interconnected attackers trying to spread an distributed denial of service (DDoS) attack using a worm with a defender that deploys multiple service access points in various large networks using this technology. A simple example of a visualization of this situation appears below.

Desktop/Light Weight Federation

One of the newest and most practical uses of DFA technology is to allow a researcher to connect an existing desktop workstation to a federated experiment. This lets a researcher bring their visualization and debugging tools into an experiment while keeping their home environment intact. In addition, researchers can provide limited external connectivity to other nodes in the experiment through the desktop.

The desktop federator is a powerful way to begin using federation without much configuration and overhead. Researchers have used desktop federation to demonstrate complex experiments at their local sites using powerful tools and hardware. Others have used the lightweight federation as a prototype to test the usefulness and feasibility of heavier weight, full featured federation in new experimental environments.

The simplest way to use desktop federation is to make use of our pre-packaged virtual machine image. More features can be made available if researchers install the desktop federator directly on a host. Finally, users of the ExoGENI testbed can use our ExoGENI image directly.

The Software Architecture

The overall architecture of the DFA is shown below. Users specify their experiment environments to a system that coordinates access to multiple testbeds/resource providers. Each testbed may use different interfaces and have different policies for access.

The plug-in system simplifies integration of different testbed interfaces to the DFA. Each plug-in translates from the local testbed's interface into a standard interface.

The Attribute-based Access Control (ABAC) authorization logic and some associated DFA translation technology allows different policies to mesh. ABAC is a powerful, well-specified authentication logic that can encode a variety of policies. It allows sophisticated delegation and other access control rules to be expressed interoperably. It is in use both in the DFA and in the ​GENI testbed.

In principle, testbeds can use the full power of ABAC to describe their policies, but in practice DFA tools allow simple configuration of access control for most testbeds.

The function of the Federator in the diagram above is divided into two parts:

Experiment controller

Presents the abstraction of a federated experiment to the users. Allows allocation, configuration, and release of resources to the federated experiment.

Access Controller

Negotaties access and allocation of resources from federant testbeds.

The access controller is depicted by the regular plug-in shapes in the federator. These are standard plug-in interfaces that allow standard access to a variety of testbed types. Currently emulab style testbeds and DRAGON style transit networks are supported. The regular system interface to fedd allows new systems to be added easily.

Depending on the nature of the access controller it may be implemented on the local testbed, on the same host as the experiment controller (even in the same process) or partially in both places. This depends on the access mechanisms of the underlying testbed and the policies of the administrators. We have implemented plugins for Emulab-style testbeds that run completely remotely and access the testbed through ssh and plug-ins that run on the Emulab testbed directly and communicate outside the testbed entirely through fedd interfaces. Access controllers that run directly on the remote testbed have more discretion in executing privledged local commands than remote access controllers.

The access controller/experiment controller interface is documented.

The Experiment Controller

A researcher creating a federated experiment uses our federation tools to talk directly to an experiment controller. Commonly DETER users talk to DETER's experiment controller. While experiment controllers can run on any testbed - in fact each user is welcome to run their own controller - using DETER's controller leverages the trust other testbeds place in that controller and the information it has gathered.

The experiment controller is responsible for

• Getting permission to access to resources at the various testbeds
• Coordinating the creation of the sub-topologies that make up the federated experiment
• Requesting their interconnection

As the experiment runs, the experiment controller can monitor and perform coarse control operations on the various elements of the experiment. Controls are coarse because of the wide variety of elements that might be federated. In testbeds that require periodic renewals of resource reservations, the experiment controller handles them. Finally the experiment controller is responsible for coordinating the release of elements when the experiment ends.

These functions are accessed by users via comand line tools available on DETER and packaged with the federation system.

Each experiment controller creates a namespace of users and groups those users belong to. When the experiment controller contacts an access controller to request resources, the user and group information (as the experiment controller knows it) is passed on. The access controller makes its decisions based on that information. All of the information and policies about dealing with that information is encoded in ​ABAC. DETER provides tools for generating basic user/group mappings.

The main DETER testbed (DETERLab) runs an experiment controller that exports DETER users and the projects to which they belong. This provides a simple way for federating testbeds to add their users to the DETER namespace by registering as DETER users. In addition, it is simple to map back from DETER user names and groups (projects) into local access control.

Other documentation provides more information about running an experiment controller.

The Access Controller

An access controller runs for each testbed that makes resources available via the DFA. Researchers carrying out experiments rarely, if ever, interact directly with an access controller. It is responsible for

• Authorizing accesses based on ABAC information provided by the experiment controller (and local policy)
• Reserving, connecting and intializing local resources based on requests from the experiment controller (and local policy)
• Implementing the coarse controls requested by the experiment controller (within local policy)
• Safely releasing local resources

The access controller translates between the fairly generic experiment ontroller primitives and the testbed-specific resource controls that a local testbed requires.

There are access controllers available for several existing testbed architectures: ​DETER, ​Emulab, ​GENI AM, ​StarBed, and desktops. The existing controllers are configurable for a wide range of network connections (behind a NAT or not, etc.) and policies.

For new testbeds, the architecture and code is structured as a plug-in architecture. New code can make use of existing code to parse and handle the basic operations and act as a base for specialization.

Other documentation provides more information about running an Access Controller.

Experiment Layouts

A federated experiment combines resources from multiple independently administered testbeds into a cohesive environment for experimentation. How much the independent testbeds preserve their internal configurations is a matter of policy and experiment controller requests. Most of the existing access controllers will allow a master testbed to export user and filesystem information into all segments when that is consistent with policy. Local access controllers can configure their resources as they normally would as well.

Because federants may have very different resource models, the DFA describes experiment layouts in general terms. Resources used to compute, sense, or manipulate are described along with their communications interconnections with other resources. Connectivity is generally described in terms of which resources can send and receive messages from others.

An experimenter creates an experiment layout by presenting a description of it to an experiment controller that knows him or her - usually the DETER experiment controller or one on their local testbed. That controller will map the local user into the global (testbed, project, user) namespace and acquire and configure resources to create the layout. An experiment controller may try different mappings of the layout into the global namespace in order to acquire all the resources. For example a local experimenter may be a member of several local DETER projects that different access controllers will accept.

While an experiment exists, experimenters can access the nodes directly using the credentials passed back and can acquire information about the experiment's topology from the experiment controller.

Layout Descriptions

Currently layouts can be described in the same ​ns2 syntax that DETER uses. Fedd also understands layouts expressed in the topdl language that it uses to communicate with facilities. There are ns2 extensions that allow users to specify the various topdl constructs from within an ns2 file.

The key attribute for resources in a federated layout is the testbed attribute that determines which federant will supply the resource. In topdl this is given as an attribute on the elements. In ns2, the tb-set-node-testbed command is used. Its syntax is:

tb-set-node-testbed node_reference testbed_name

The node_reference is the tcl variable containing the node and the testbed_string is a name meaningful to the fedd creating the experiment. The experiment creation interface has a mechanism that a requester can use to supply translations from testbed names to the URIs needed to contact that testbed's fedd for access.

Multiple segments of a federerated experiment can be embedded on the same testbed specifying the testbed_name as name/subname. In that syntax, name is the identifier of the testbed on which to embed the node and subname is a unique identifier within the federated experiment. All the nodes assigned to deter/attack will be in the same sub-experiment and that will be a different sub-experiment from nodes in deter/defend.

Experiment Services

An important part of the cohesive experiment framework created by the DFA is the web of services interconnecting the sub experiments. A service, in this context, is fairly liberally defined; it is any one of several pre-defined ways of sharing information between sub-experiments. That sharing can be statically defined at experiment creation time or change dynamically throughout the course of the experiment. Static information includes the accounts to create on various machines or the hostnames to export to sub-experiments. Dynamic information includes mounting filesystems or connecting user experiment tools, like ​SEER.

Each service has an exporter the testbed that provides the service, and zero or more importers, testbeds that use the information. In addition, services can have attribute/value pairs as parameters.

The specific services currently supported by fedd are:

hide_hosts

By default, all hosts in the topology are visible across the experiment. This allows some hosts to be completely local. The hosts chosen are given by the hosts attribute to the service.

local_seer_control

Adds a local seer control node to sub-experiments in the exporting testbed.

project_export

Exports the user configuration and filesystem named by the project attribute to the service to the importing testbeds

seer_master

Adds a second seer controller to the exporting testbed that is capable of aggregating the inputs from several local seer masters. To create a seer instance that sees several sub experiments, create a local_seer_control in each and export a seer_master to all.

SMB

Export a set of SMB/CIFS filesystems.

userconfig

Export the account information from a given project or group if the testbed supports such abstractions. Currently accepts the project attribute for that purpose.

Service advertisement, provision, and specification is an ongoing area of research in federation, and we expect these specifications to become more detailed and useful as that research progresses.

Attribute-Based Access Control (ABAC)

Between the experiment controller and access controller the access control information is transferred using ​ABAC logic. The implementation uses our ABAC library, ​libabac.

This is invisible to most users

Next Steps

• The Users' Guide, that explains how to use the system in a tutorial format
• The Administrator's Guide, that explains how to install and configure the system in a tutorial format
• The Reference Guide, a detailed technical description of how to operate the components