wiki:TopdlLibrary

Version 18 (modified by faber, 12 years ago) (diff)

--

Topdl Library

This page describes the python library for manipulating topologies that follow the topdl model and are generally output in the topdl language. It is broken up into a reference section describing the various python classes and procedures and an examples section.

Reference

Installing and Accessing

The topdl library is available as oart of the DETER module of the DETER federation system. Any system that has fedd installed will have topdl installed as well. In particular, users.isi.deterlab.net has the library installed.

To install the library separately, follow the instructions for getting fedd from git, make the distribution, and install the deter-data-version.tar.gz tar file using the same procedure for installing the fedd tar file. Installing all of fedd will work as well, of course.

Once the library is installed, python programs can access it by: 

#!/usr/bin/env python

from deter import topdl

The Classes

The class hierarchy mirrors the entities of the topdl model. Where the description of the model glosses over the details of the classes that parameterize the top-level abstractions, this reference will discuss them all. Each of these classes is a python class, and can be created and manipulated independently, but in a proper topology they are grouped together.

A Substrate object has members that are Capacity and Latency objects. This listing shows that nesting:

  • Substrate
    • Capacity (0 or 1)
    • Latency (0 or 1)
    • Attribute (0 or more)

The other common element, a Computer object, has the following nested objects:

Most classes can have attributes attached to them that contain extention attributes that different applications make use of. The presence or absence of an optional nested object does not mean the presence or absence of the thing it describes, but that the description is unbound. As we have seen a Computer without a nested CPU object does not describe a computer without a CPU, but a computer with no constraints on the CPU.

Adaptive Constructors

The constructors of the various topdl classes will accept contained classes in several formats. They will always accept an object of the relevant class. They will also take a dict that maps constructor parameters to values for the given class. Examples make this clearer than description.

The following calls to create a CPU object are equivalent: 

cpu = topdl.CPU(type='Intel', attribute=[topdl.Attribute(attribute='clock_rate', value='1Ghz')])

and 

cpu = topdl.CPU(type='Intel', attribute=[{'attribute': 'clock_rate', 'value': '1Ghz'}])

and 

cpu = topdl.CPU(**{'type': 'Intel', 'attribute' : [ {'attribute': 'clock_rate', 'value': '1Ghz'}]})

That is, inner class parameters can be specified using dicts directly, and a top level dict can user the standard operator.

All topdl class constructors take the member names as parameters, and should be called using named parameters, as above. The only exceptions to this is the Substrate object's interfaces member and the Interface object's subs member. An Interface's element member can be specified, but generally is not.

Topdl Class Features

All topdl classes implement the following methods:

set_attribute(self, key, value)
Attach an user-defined attribute with the given key and value to this object; value and key are strings. The method includes a key/value pair (encoded as an attribute object) into the attribute member of the object. If an attribute with the given key exists, it is overwritten. This method accesses the attribute member of the given object, not any named members. Objects that do not have an attribute member silently discard the operation.
get_attribute(self, key)
Return the attribute of this object with the given key. The attribute - a string - is returned. If no such attribute exists, or the object has no attribute member, return None.
remove_attribute(self, key)
If an attribute with the given key exists, remove it.
clone(self)
Returns a deep copy of this object.
to_dict(self)
Return the object as a python dict. No copies are made, so make modifications with caution.
to_xml(self)
Return the XML encoding (from the topdl language) of this object (and nested objects)

Topology Class

This class holds an enitre topology. The utility functions that read and write files operate on it, and it is generally the unit of I/O for the library.

It has the following members:

substrates
A list of Substrate objects that contain the substrates of the topology. The list may be empty indicating no communcations are possible.
elements
A list of entities make up the network. These can be of any Computer objects, Testbed objects, Segment objects, or Other objects. The list may be empty indicating no elements are in the topology. attribute::: A list of Attribute objects that are attached to the machine. The list may be empty.
version
The version of topdl that this object encodes.

As usual, the constructor takes the member names as parameter names.

One useful specialized member is: 

incorporate_elements(self)

That member sanity checks the substrates to make sure they are uniquely named, assigns names to any interfaces without them, and connects the element members of the Interface objects. If there are inconsistencies, it raises a topdl.ConsistencyError with more detail in the message.

incorporate_elements() is called by the Topology constructor as well and that constructor may throw the same exceptions.

Computer Class

A Computer object is a programmable computer in a topology. A webserver is well represented by a computer, as is a desktop. It has the following members:

name
The unique name of the computer in the topology, a string. Other names may be assigned in the localname attribute.
cpu
A list of CPU objects that describe the CPUs of the machine. The list may be empty, meaning no CPU requirements or information are present.
software
A list of Software objects that describe the software to install. The list may be empty, meaning no software is to be installed.
storage
A list of Storage objects that describe the storage present the machine. Storage objects can describe memory or persistent storage requirements. The list may be empty, meaning no storage requirements or information are present.
interface
A list of Interface objects that describe connections of the Computer to communications substrates. The list may be empty, meaning that the computer is not connected to any substrates.
attribute
A list of Attribute objects that are attached to the machine. The list may be empty.
localname
A list of strings giving alternative names for the object. When a machine is allocated to a testbed the DNS names assigned to the machine are given as localnames. The list may be empty, meaning no such alternatives exist.
status
A string indicating the current status of the Computer (if any). It may also be None, indicating no status is reported. Valid values are
  • "empty" - no allocation of resources has been attempted - generally not applied to Computers
  • "active" - Computer is functioning as part of an experiment
  • "inactive" - Computer is allocated, but not functioning
  • "starting" - Computer is transitioning from "inactive" to "active"
  • "terminating" - Computer is transitioning from "active" to "inactive"
  • "failed" - an allocation of resources has failed - generally not applied to Computers
service
A list of Service objects that describe the services this machine uses or supplies. These are generally used by the federation system, and their definition is somewhat in flux. The list may be empty.
operation
A list of strings that describe the valid operations on the machine. These are generally used by the federation system, and their definition is somewhat in flux.The list may be empty.

The constructor takes the member names as parameters. Only name is required.

A Computer should only be an element of one Topology. Use clone() to avoid this.

Substrate Class

The Substrate object represents a communication substrate. The elements that have interfaces connected to a substrate can communicate with one another. A Substrate may include default parameters for the latency and capacity of the communication channel.

The members are:

name
A string, the unique name of the substrate.
capacity
A Capacity object that gives the default capacity of the substrate. It may be None, incdicating that there is no default.
latency
A Latency object that gives the default latency of the substrate. It may be None, incdicating that there is no default.
attribute
A list of Attribute objects that are attached to the substrate. The list may be empty.
localname
A list of strings giving alternative names for the object. The list may be empty, meaning no such alternatives exist.
status
A string indicating the current status of the Substrate (if any). It may also be None, indicating no status is reported. Valid values are
  • "empty" - no allocation of resources has been attempted - generally not applied to Substrates
  • "active" - Substrate is functioning as part of an experiment
  • "inactive" - Substrate is allocated, but not functioning
  • "starting" - Substrate is transitioning from "inactive" to "active"
  • "terminating" - Substrate is transitioning from "active" to "inactive"
  • "failed" - an allocation of resources has failed - generally not applied to Substrates
service
A list of Service objects that describe the services this substrate uses or supplies. These are generally used by the federation system, and their definition is somewhat in flux. The list may be empty.
operation
A list of strings that describe the valid operations on the substrate. These are generally used by the federation system, and their definition is somewhat in flux.The list may be empty.
interfaces
A list of Interface objects connected to this substrate. The list may be empty, meaning that the substrate is not connected to any elements. This field is maintained by the Topology object holding this Substrate object. It is not a parameter to the constructor.

Other than interfaces, the constructor takes the member names as parameters. Only the name is required.

Because the enclosing Topology object maintains the interfaces member, it is dangerous to have the same Substrate object in more than one Topology. Use the clone() method to avoid this.

Interface Class

This class defines the attachment of an entity, e.g., a Computer, to a Substrate. As with Substrates and entities, each Interface should be attached to only one entity and be unique across a Topology. An Interface has the following members:

substrate
A list of strings, each naming a substrate to which this interface is connected. The list must have at least one entry.
capacity
A Capacity object indicating the communications capacity of the interface. It may be None to use the substrate default.
latency
A Latency object indicating the communications delay of the interface. It may be None to use the substrate default.
attribute
A list of Attribute objects that are attached to the interface. The list may be empty.
element
The entity to which this interface is attached. It may be a Computer object, Testbed object, Segment object, or Other object. This is generally managed by the Topology holding the entity and substrate with which the Interface is attached. It is wise to treat the field as read-only and valid only after a Topology has called incorporate_elements().
subs
A list of Substrates to which the interface is attached. It cannot be set through the constructor. This is generally managed by the Topology holding the entity and substrate with which the Interface is attached. It is wise to treat the field as read-only and valid only after a Topology has called incorporate_elements().

The constructor takes all the member names except subs as parameters. Most programs will not need to use element as a parameter to the constructor, either.

Attribute Class

An attribute is a user-defined annotation of one of the other classes: a key/value pair. An attribute has the following members:

attribute
A string, the key used to find the attribute.
value
A string, the value

While attributes can be manipulated directly liek any of the other topdl classes, it is most common to use the attribute manipulation functions of the annotated class to access them.

Should a program need to call the constructor, it takes the member names as named parameters.

CPU Class

This represents a CPU on a computer. It has the following members:

type
A string indicating the type of CPU. These are not yet standardized, but common formats like "i686" or "amd64" are used.
attribute
A list of Attribute objects that are attached to the CPU. The list may be empty.

The constructor takes the member names as named parameters.

OperatingSystem Class

This class captures the operating system requirements of a computer. It has the following members:

name
A string naming the operating system required. This is usually the operating system itself, e.g., Linux, FreeBSD, Windows XP. This may be None, indicating no preference or information.
version
A string identifying the version of the OS. For integrates systems like FreeBSD this is the system version. For distribution-based systems like Linux, this is the kernel version. See below for distribution versioning. This may be None, indicating no version information or preference.
distribution
A string that for operating systems that characterize OS distribution independently of kernel or base system, names the distribution in use, e.g., Ubuntu. This may be None, indicating no information or preference.
distributionversion
A string that for operating systems that characterize OS distribution independently of kernel or base system, names the version of the distribution in use, e.g., 12.04 for Ubuntu. This may be None, indicating no information or preference.
attribute
A list of Attribute objects that are attached to the operating system. The list may be empty.

The constructor takes the member names as parameters. Note that all the members may be None, but an empty !OperatingSystem object is not very useful. The flexibility is there to allow specifying an operating system version without a distribution or a distribution without a base version.

Storage Class

The storage requirements or information for a system. This class characterizes memory or permanent storage. Its members are:

amount
A floating point value indicating the number of megabytes of storage being considered. Values less than 0.000001 are treated as 0. {{{persistence}}:: A boolean value indicating whether this is persistent storage, such as a disk or SSD, or volitile such as memory.
attribute
A list of Attribute objects that are attached to the machine. The list may be empty.

Currently there is no field for media type. Use attributes.

The constructor takes the member names as named parameters.

Software Class

This indicates what software must be installed on a system. Its members are:

location
A string containing a URI from which to install the software, e.g., a file: URI pointing to a tarfile.
install
A string containing location in the local file system to base the software install. This may be None, indicating that the software distribution format includes this information, e.g., an rpm or debian software file.

The constructor takes the member names as named parameters.

Service Class

A service class encapsulates the ideas of a DETER federation service. Such services can be available from or used by a variety of topology elements. The Service class has the following members:

name
The name of the service. A few are defined.
importer
A list of entity names that are using or requesting the service. This list may be empty.
param
A list of ServiceParam objects that represent typed parameters to the service. The list may be empty.
description
A string describing the service. This may be None.
status
A string describing the current status of the service (if any). It may also be None, indicating no status is reported. Valid values are
  • "empty" - no allocation of resources has been attempted - generally not applied to Services
  • "active" - Service is functioning as part of an experiment
  • "inactive" - Service is allocated, but not functioning
  • "starting" - Service is transitioning from "inactive" to "active"
  • "terminating" - Service is transitioning from "active" to "inactive"
  • "failed" - an allocation of resources has failed - generally not applied to Services

The constructor will take the member names as named parameters.

ServiceParam Class

A description of the type of parameter to a Service will accept. It has the following members:

name
Name of the parameter, a string.
type
A string containing the type of the parameter. Valid types are:
  • string
  • int
  • float

The constructor will take the member names as named parameters.

Testbed Class

This class represents a testbed that has allocated resources or from which an allocation is requested. While it is primarily used internally by the federation system, some other applications include a Testbed as a place to attach attributes that pertain to the whole topology.

It includes the following members:

uri
A string containing the URI on which the testbed can be contacted. Usually a fedd URI. {{{type}}:: A string indicating the underlying type of the testbed. Values include "deter," "emulab," and "protoGENI".
interface
A list of Interface objects that describe connections of the Testbed to communications substrates. The list may be empty, meaning that the testbed is not connected to any substrates.
attribute
A list of Attribute objects that are attached to the Testbed. The list may be empty.
localname
A list of strings giving alternative names for the testbed. The list may be empty, meaning no such alternatives exist.
status
A string indicating the current status of the Testbed (if any). It may also be None, indicating no status is reported. Valid values are
  • "empty" - no allocation of resources has been attempted - generally not applied to Testbeds
  • "active" - Testbed is functioning as part of an experiment
  • "inactive" - Testbed is allocated, but not functioning
  • "starting" - Testbed is transitioning from "inactive" to "active"
  • "terminating" - Testbed is transitioning from "active" to "inactive"
  • "failed" - an allocation of resources has failed - generally not applied to Testbed
service
A list of Service objects that describe the services this Testbed uses or supplies. These are generally used by the federation system, and their definition is somewhat in flux. The list may be empty.
operation
A list of strings that describe the valid operations on the Testbed. These are generally used by the federation system, and their definition is somewhat in flux.The list may be empty.

The constructor takes the member names as named parameters.

Segment Class

This class represents a sub-experiment of a federated experiment about which the details are unknown except for its interfaces. It is primarily used internally by teh federation system.

It has the following members:

id
An ID object that identifies this segment.
type
A string indicating the underlying type of the testbed. Values include "deter," "emulab," and "protoGENI".
interface
A list of Interface objects that describe connections of the segment to communications substrates. The list may be empty, meaning that the segment is not connected to any substrates.
attribute
A list of Attribute objects that are attached to the segment. The list may be empty.

ID Class

This class represents a polymorphic identifier as used by the federation system. At most one of its members is set.

Those members are:

fedid
A fedid object, representing the hash of a principal key. This represents a fedid.
uuid
A string holding a UUID URI.
uri
A string holding a generic URI
localname
A string holding a locally scoped name {{{kerberosUsername}}:: A kerberos user name.

The constructor accepts the member names as named parameters.

Other Class

This class represents a topology element that is not captured above. It is a set of interfaces and a place to hang attributes, for use encoding kinds of elements unknown when topdl was specified.

The members are:

interface
A list of Interface objects that describe connections of the element to communications substrates. The list may be empty, meaning that the element is not connected to any substrates.
attribute
A list of Attribute objects that are attached to the element. The list may be empty.

The constructor accepts the member names as named parameters.

Procedures

There are several procedures in the library for inputting and outputing topologies in various formats. This section explains their use.

topology_from_xml

topology_from_xml reads a topdl-encoded topology and returns a Topology object. The encoded topology may be in a file or string and the enclosing element name may be set. It is the inverse of [topdlLibrary#topology_to_xml topology_to_xml]. The parameters are:

string
A string containing the XML to parse. If it is set file and filename must be None.
file
An open python file or file-like object that holds the XML to parse. It must be open for reading. If it is set string and filename must be None.
filename
A string containing the filename that holds the XML to parse. If it is set string and file must be None.
top
The name of the element that contains the topdl to parse. Topdl is embedded in other soap objects and XML files. This allows the library to extract the topdl from more complicated XML. If not given it defaults to "topology".

Exactly one of string, file, and filename must be given, and to avoid confusion it is best to call topology_from_xml with named parameters.

topology_to_xml

Returns a string containing the topdl representation of the given topology, the inverse of topology_from_xml. The enclosing element may be passed as a parameter. The parameters are:

t
The [Topdl#TopologyClass? Topology object] to encode.
top
A string containing the name of the element that will enclose the topdl. If not given, no element will enclose the topdl.

topology_to_ns2

Returns a string containing an ns2 representation of the topology. Computer objects are rendered as nodes and substrate objects as lans. The output can be customized with filter functions. The parameters are:

t
The topology class to encode
filters
A list of filter functions applied in order as the output is constructed. If unspecified the list is empty.
routing
A string that sets the ​experiment routing style. The Default is "Manual"; single testbed experiments will probably want "Static".

ns2 Filter Functions

topology_to_ns2 takes a list of filter functions to customize its output. Each filter must be callable and accept a single parameter and return a string. The parameter can be an element or substrate. That is, e can be a Computer object, Testbed object, Segment object, or Other object, or Substrate object. A callable class is useful if the filter needs to act on internal state.

topology_to_ns2 calls filters as it generates output. It outputs elements first by traversing its topology parameter's elements in order. For each element, the filters provided to topology_to_ns2 are called in order and their output appended to the ns2 string. After all elements are output, the substrates are traversed, and again the filters called in order. The filter output comes after the topology_to_ns2 output for each element and substrate.

Inside a filter it is often useful to translate an element or substrate name into something valid to ns2. The to_tcl_name function is in the topdl namespace for this purpose. It takes a string and returns the tcl name for it. When assigning to the name, this sequence is used: 

tname = topdl.to_tcl_name(elem.name)
out += 'set %s [$ns node]' % tname

when dereferencing the name - which is much more common: 

tname = topdl.to_tcl_name(elem.name)
out += 'tb-set-node-hardware ${%s} MicroCloud' % tname

The output of to_tcl_name should be enclosed in { } to avoid tcl/ns parsing errors.

topology_to_rspec

Converts a topology to a ​ProtoGENI ​rspec and returns it as a string. The parameters are:

t
The topology class to encode
filters
A list of filter functions applied in order as the output is constructed. If unspecified the list is empty.

The filter functions work the same way as in topology_to_ns2.

Parsing from ns2

An obvious missing element is a procedure to parse ns2 into topdl. Unfortunately, ns2 programs are full, turing-complete otcl programs, and parsing them would require copying the tcl interpreter into python (or any other implementation language - other than otcl). Rather than do that, we encourage programmers who need that translation to use the ​fedd_ns2topdl.py program. That program contacts a running fedd (like the one running on users.isi.deterlab.net) to call an otcl system to do the parsing.

Examples

This section contains some illustrative examples.

Making a Topology

This example constructs the topdl describing a simple start topology with 11 nodes, one center and 10 satellites.

Star topology

A python program to print the topdl describing that topology looks like this: 

#!/usr/bin/env python

from deter import topdl

elems = []      # elements in the topology
subs = []       # substrates in the topology

# Create the center node and put it in the element list.  (Name is set
# explicitly)
center = topdl.Computer(name='center')
elems.append(center)

# Connect 10 satellites
for i in range(0,10):
    # Create a substrate to join the new element to center
    link = topdl.Substrate('link%d' % i)
    # Create the new element (n-i) and connect it to the new substrate with an
    # Interface object
    node = topdl.Computer(name='n-%d' % i, interface=[
        topdl.Interface(substrate=['link%d' %i])])
    # Add an interface to center linking it to the new substrate
    center.interface.append(topdl.Interface(substrate=['link%d' %i]))

    # Add the new link and node to the lists
    subs.append(link)
    elems.append(node)

# Create the topology from the lists.  This names the unnamed interfaces and
# makes connections.
top = topdl.Topology(elements=elems, substrates=subs)

# Print the XML
print topdl.topology_to_xml(top, top='experiment')

The program​ builds the topology by creating a Computer object that is the center of the star, then creates other computers that are connected to the center. The connection is accomplished by creating a Substrate object that represents the connection and adding Interface objects to the center computer and the satellite that bind them to the substrate.

For comparison, here​ is the equivalent ns2 description: 

source tb_compat.tcl
set ns [new Simulator]

# Create the center node (named by its variable name)
set center [$ns node]

# Connect 10 satellites
for { set i 0} { $i < 10 } { incr i} {
    # Create node n-1 (tcl n($i) becomes n-$i in the experiment)
    set n($i) [$ns node]
    # Connect center to $n($i)
    ns duplex-link $center $n($i) 100Mb 0ms DropTail
}

# Creation boilerplate
$ns rtptoto Static
$ns run

A More Complex Topology

The next few examples will build on one another to show a more complex change to a topology. We will take a topology, find the degrees of nodes, assign operating systems to them based on that degree, and output DETER-ready ns2.

To try this we need a more complex topology. The program below creates a topology with nrouters completely connected routers, each of which has nleaves leaves, except for one central router that has none. Here is an image of the topology when nrouters=4 and nleaves=4:

Sample complex topology

We leave one router without leaves for aecthetics.

Here is the program that generates that topology: 

#!/usr/bin/env python

import sys
from deter import topdl

# Parse the user-supplied routers and leaves arguments
try:
    if len(sys.argv) > 1: arg = sys.argv[1]
    else: arg = 4
    nrouters = int(arg)
    
    if len(sys.argv) > 2: arg = sys.argv[2]
    else: arg = 3
    nleaves = int(arg)
    
except ValueError, e:
    print >>sys.stderr, 'Usage %s routers leaves' % sys.argv[0]
    sys.exit('Cannot convert %s to an int' % arg)

elems = []      # Elements in the topology
subs = []       # Substrates in the topology

# Make nrouters routers
r = []
for i in range(0,nrouters):
    r.append(topdl.Computer(name='r-%d' % i))

elems.extend(r)
# Interconnect them
n = 0
for i in range(0,nrouters):
    for j in range(i+1, nrouters):
        # Connect r[i] to r[j]
        # Pick a substrate name
        link_name = 'rlink%i' % n
        # Add interfaces to the two routers
        r[i].interface.append(topdl.Interface(substrate=[link_name]))
        r[j].interface.append(topdl.Interface(substrate=[link_name]))
        # Put a substrate into the substrate list with the name we picked
        subs.append(topdl.Substrate(name=link_name))
        n += 1

# Put nleaves leaves on each router, except the center one
for i in range(0, (nrouters-1) * nleaves):
    # Pick a link name
    link_name = 'llink%i' % i

    # add a computer with an interface on the new substrate to the element list
    elems.append(topdl.Computer(name='leaf%d' % i, interface=[
        topdl.Interface(substrate=[link_name])]))
    
    # Add an interface to a router on the new substrate
    r[i/nleaves].interface.append(topdl.Interface(substrate=[link_name]))
    
    # Put a substrate into the substrate list with the name we picked
    subs.append(topdl.Substrate(name=link_name))

# Make the topology and print it
top = topdl.Topology(elements=elems, substrates=subs)
print topdl.topology_to_xml(top, top='experiment')

The code takes a few more shortcuts in construction, e.g., most elements and substrates are appended directly to the lists rather than being assigned to variables, but is not fundamentally different from the start topology generator.

Attachments (11)

Download all attachments as: .zip