The DETER Federation Architecture (DFA)

This is a basic overview of the DETER federation system. More details on this implementation and configuration are in later documents. You can also read these academic papers that capture the evolution of the design to this point.

The DFA is a system that allows a researcher to construct experiments that span testbeds by dynamically acquiring resources from other testbeds and configuring them into a single experiment. As closely as possible that experiment will mimic a single DETER/Emulab experiment.

Though the experiment appears to be a cohesive whole, the testbeds that loan the resources retain control of those resources. Because testbeds retain this control, each testbed may issue credentials necessary for manipulating the federated resources. For example, a testbed that has loaned nodes to an experiment may require the experimenter to present a credential issued by that testbed (e.g., an SSH key or SSL certificate) to reboot those nodes. The system acquires those credentials on behalf of experimenters and distributes them on behalf of testbeds.

Testbed administrators may use the system to establish regular policies between testbeds to share resources across many users of a testbed. Similarly, a single user with accounts on multiple testbeds can use the same interfaces to coordinate experiments that share his testbed resource, assuming sharing those resources does not violate the policy of any of the constituent testbeds.

Federated Experiments

A researcher may build a federated experiment to

• Connect resources from testbeds with different focuses - e.g., connect power system resources with networking resources
• Connect resource owned by different actors for collaboration or competition
• Connect a local workstation environment to an experiment to monitor or demonstrate the results of an experiment.

An all of these cases, a researcher specifies the resources and their interconnections to the DFA, which coordinates between the testbeds and connects the resources.

Consider a researcher who has access to several testbeds and wants to interconnect resources from those testbeds.

From disparate testbeds, the DETER Federation System will create a single unified environment.

Combining Specialized Resources

The most common reason to create a federated experiment is to build an experimental environment that combines the specialties of multiple institutions.

For example, DETER researchers have used the federation system to interconnect electrical power systems, high speed power simulation, a controlled IP network and a situational awareness display sited at three disparate research labs into a coherent whole. This example shows a federation between DETER, tyhe University of Illinois and the Pacific Northwest National Laboratory (PNNL).

The federation system handles the internetworking of those resources, allowing the researchers to combine expertise.

Collaboration and Competition

Another use of federation is to construct experiments where different reserachers have different levels of access to different parts of the overall experiment. Experimenters preparing defenses can be linked to experimenters preparing attacks and to a shared environment unknown to both of them. The result is a controlled interaction between groups of experiment elements.

DETER researchers have interconnected attackers trying to spread an distributed denial of service (DDoS) attack using a worm with a defender that deploys multiple service access points in various large networks using this technology. A simple example of a visualization of this situation appears below.

Integrating Researcher Desktops

Often a researcher will want to directly interconnect their desktop or another computer into an experimental environment. This can enable an experimenter to debug effectively by importing their tools and environment easily, or to demonstrate the experiment to others by incorporating a display engine.

This is a special case of combining specialized resources, but the lightweight system DETER uses to facilitate this is of particular interest.

The Software Architecture

The overall architecture of the DFA is shown below. Users specify their experiment environments to a system that coordinates access to multiple testbeds/resource providers. Each testbed may use different interfaces and have different policies for access.

The plug-in system simplifies integration of different testbed interfaces to the DFA. Each plug-in translates from the local testbed's interface into a standard interface.

The Attribute-based Access Control (ABAC) authorization logic and some associated DFA translation technology allows different policies to mesh. ABAC is a powerful, well-specified authentication logic that can encode a variety of policies. It allows sophisticated delegation and other access control rules to be expressed interoperably. It is in use both in the DFA and in the ​GENI testbed.

In principle, testbeds can use the full power of ABAC to describe their policies, but in practice DFA tools allow simple configuration of access control for most testbeds.

The function of the Federator in the diagram above is divided into two parts:

Experiment controller

Presents the abstraction of a federated experiment to the users. Allows allocation, configuration, and release of resources to the federated experiment.

Access Controller

Negotaties access and allocation of resources from federant testbeds.

The access controller is depicted by the regular plug-in shapes in the federator. These are standard plug-in interfaces that allow standard access to a variety of testbed types. Currently emulab style testbeds and DRAGON style transit networks are supported. The regular system interface to fedd allows new systems to be added easily.

Depending on the nature of the access controller it may be implemented on the local testbed, on the same host as the experiment controller (even in the same process) or partially in both places. This depends on the access mechanisms of the underlying testbed and the policies of the administrators. We have implemented plugins for Emulab-style testbeds that run completely remotely and access the testbed through ssh and plug-ins that run on the Emulab testbed directly and communicate outside the testbed entirely through fedd interfaces. Access controllers that run directly on the remote testbed have more discretion in executing privledged local commands than remote access controllers.

The access controller/experiment controller interface is documented.

The Experiment Controller

A researcher creating a federated experiment uses our federation tools to talk directly to an experiment controller. Commonly DETER users talk to DETER's experiment controller. While experiment controllers can run on any testbed - in fact each user is welcome to run their own controller - using DETER's controller leverages the trust other testbeds place in that controller and the information it has gathered.

The experiment controller is responsible for

• Getting permission to access to resources at the various testbeds
• Coordinating the creation of the sub-topologies that make up the federated experiment
• Requesting their interconnection

As the experiment runs, the experiment controller can monitor and perform coarse control operations on the various elements of the experiment. Controls are coarse because of the wide variety of elements that might be federated. In testbeds that require periodic renewals of resource reservations, the experiment controller handles them. Finally the experiment controller is responsible for coordinating the release of elements when the experiment ends.

These functions are accessed by users via comand line tools available on DETER and packaged with the federation system.

Each experiment controller creates a namespace of users and groups those users belong to. When the experiment controller contacts an access controller to request resources, the user and group information (as the experiment controller knows it) is passed on. The access controller makes its decisions based on that information. All of the information and policies about dealing with that information is encoded in ​ABAC. DETER provides tools for generating basic user/group mappings.

The main DETER testbed (DETERLab) runs an experiment controller that exports DETER users and the projects to which they belong. This provides a simple way for federating testbeds to add their users to the DETER namespace by registering as DETER users. In addition, it is simple to map back from DETER user names and groups (projects) into local access control.

Other documentation provides more information about running an experiment controller?.

The Access Controller

An access controller runs for each testbed that makes resources available via the DFA. Researchers carrying out experiments rarely, if ever, interact directly with an access controller. It is responsible for

• Authorizing accesses based on ABAC information provided by the experiment controller (and local policy)
• Reserving, connecting and intializing local resources based on requests from the experiment controller (and local policy)
• Implementing the coarse controls requested by the experiment controller (within local policy)
• Safely releasing local resources

The access controller translates between the fairly generic experiment ontroller primitives and the testbed-specific resource controls that a local testbed requires.

There are access controllers available for several existing testbed architectures: ​DETER, ​Emulab, ​GENI AM, ​StarBed, and desktops. The existing controllers are configurable for a wide range of network connections (behind a NAT or not, etc.) and policies.

For new testbeds, the architecture and code is structured as a plug-in architecture. New code can make use of existing code to parse and handle the basic operations and act as a base for specialization.

Access Controllers? are more deeply documented.

Experiments

A federated experiment combines resources from multiple independently administered testbeds into a cohesive environment for experimentation. How much the independent testbeds preserve their internal configurations is a matter of policy and experiment controller requests. Most of the existing access controllers will allow a master testbed to export user and filesystem information into all segments when that is consistent with policy. Local access controllers can configure their resources as they noremally would as well.

Currently experiments can be described in the same ns2 syntax that DETER uses. Most users will specify experiments in this format, initially. Fedd also understands experiments expressed in the topdl language that it uses to communicate with facilities.

An experimenter creates an experiment by presenting it to an experiment controller that knows him or her - usually the DETER experiment contoller or one on their local testbed. That controller will map the local user into the global (testbed, project, user) namespace and acquire and configure resources to create the experiment. An experiment controller may try different mappings of the experimenter into the global namespace in order to acquire all the resources. For example a local experimenter may be a member of several local DETER projects that different access controllers will accept.

While an experiment exists, experimenters can access the nodes directly using the credentials passed back and can acquire information about the experiment's topology from the experiment controller.

Experiment Descriptions

An experiment description is currently given in a slightly extended version of the ​ns2 dialect that is used for an DETER experiment description. Specifically, the tb-set-node-testbed command has been added. Its syntax is:

tb-set-node-testbed node_reference testbed_name

The node_reference is the tcl variable containing the node and the testbed_string is a name meaningful to the fedd creating the experiment. The experiment creation interface has a mechanism that a requester can use to supply translations from testbed names to the URIs needed to contact that testbed's fedd for access.

Multiple segments of a federerated experiment can be embedded on the same testbed specifying the testbed_name as name/subname. In that syntax, name is the identifier of the testbed on which to embed the node and subname is a unique identifier within the federated experiment. All the nodes assigned to deter/attack will be in the same sub-experiment and that will be a different sub-experiment from nodes in deter/defend.

In addition, fedd understands the tb-set-default-failure-action command. This sets the default failure action for nodes in the experiment to one of fatal, nonfatal, or ignore. Those values have the same meaning as in standard Emulabs. Nodes that do not have their failure mode reset by the tb-set-node-failure-action command will user the explicit default. If no default is set, fatal is used.