[[TOC]] = Topdl Library = This page describes the python library for manipulating topologies that follow the [TopDl topdl] model and are generally output in the topdl language. It is broken up into a reference section describing the various python classes and procedures and an examples section. = Reference = == Installing and Accessing == The topdl library is available as oart of the DETER module of the DETER federation system. Any system that has fedd installed will have topdl installed as well. In particular, {{{users.isi.deterlab.net}}} has the library installed. To install the library separately, follow the [FeddDownload#GitAccess instructions for getting fedd from git], make the distribution, and install the {{{deter-data-}}}''version''{{{.tar.gz}}} tar file using the [FeddDownload#DownloadingandInstallingthetarfile same procedure for installing the fedd tar file]. Installing all of fedd will work as well, of course. Once the library is installed, python programs can access it by: {{{ #!/usr/bin/env python from deter import topdl }}} == The Classes == The class hierarchy mirrors the entities of the [TopDl topdl model]. Where the description of the model glosses over the details of the classes that parameterize the top-level abstractions, this reference will discuss them all. Each of these classes is a python class, and can be created and manipulated independently, but in a proper topology they are grouped together. A Substrate object has members that are Capacity and Latency objects. This listing shows that nesting: * Substrate * Capacity (0 or 1) * Latency (0 or 1) * Attribute (0 or more) The other common element, a Computer object, has the following nested objects: * [TopdlLibrary#ComputerClass Computer] * [TopdlLibrary#CPUClass CPU] (0 or more) * [TopdlLibrary#AttributeClass Attribute] (0 or more) * [TopdlLibrary#OperatingSystemClass OperatingSystem] (0 or more) * [TopdlLibrary#AttributeClass Attribute] (0 or more) * [TopdlLibrary#SoftwareClass Software] (0 or more) * [TopdlLibrary#AttributeClass Attribute] (0 or more) * [TopdlLibrary#StorageClass Storage] (0 or more) * [TopdlLibrary#AttributeClass Attribute] (0 or more) * [TopdlLibrary#InterfaceClass Interface] (0 or more) * [TopdlLibrary#CapacutyClass Capacity] (0 or 1) * [TopdlLibrary#LatencyClass Latency] (0 or 1) * [TopdlLibrary#AttributeClass Attribute] (0 or more) * [TopdlLibrary#ServiceClass Service] (0 or more) * [TopdlLibrary#ServiceParamClass ServiceParam] (0 or more) Most classes can have attributes attached to them that contain extention attributes that different applications make use of. The presence or absence of an optional nested object does not mean the presence or absence of the thing it describes, but that the description is unbound. As [TopDl#Representation we have seen] a Computer without a nested CPU object does not describe a computer without a CPU, but a computer with no constraints on the CPU. Most of the nested classes can appear multiple times. This is to allow application designers to express as many interesting topologies as possible without having to redefine topdl. Interfaces can be members of more than one substrate because substrates are intended to be general enough to capture unusual communications media - e.g., interfaces with different line-of-sight connectivity. Similarly, we expect most computers to have one operating system, but want to allow some applications to express multi-boot or virtual machine capabilities. === Adaptive Constructors === The constructors of the various topdl classes will accept contained classes in several formats. They will always accept an object of the relevant class. They will also take a dict that maps constructor parameters to values for the given class. Examples make this clearer than description. The following calls to create a CPU object are equivalent: {{{ cpu = topdl.CPU(type='Intel', attribute=[topdl.Attribute(attribute='clock_rate', value='1Ghz')]) }}} and {{{ cpu = topdl.CPU(type='Intel', attribute=[{'attribute': 'clock_rate', 'value': '1Ghz'}]) }}} and {{{ cpu = topdl.CPU(**{'type': 'Intel', 'attribute' : [ {'attribute': 'clock_rate', 'value': '1Ghz'}]}) }}} That is, inner class parameters can be specified using dicts directly, and a top level dict can user the standard ** operator. All topdl class constructors take the member names as parameters, and should be called using named parameters, as above. The only exceptions to this is the [TopdlLibrary#SubstrateClass Substrate object]'s {{{interfaces}}} member and the [TopdlLibrary#InterfaceClass Interface object]'s {{{subs}}} member. An Interface's {{{element}}} member can be specified, but generally is not. === Topdl Class Features === All topdl classes implement the following methods: {{{set_attribute(self, key, value)}}}:: Attach an user-defined attribute with the given key and value to this object; value and key are strings. The method includes a key/value pair (encoded as an [TopdlLibrary#AttributeClass attribute object]) into the {{{attribute}}} member of the object. If an attribute with the given key exists, it is overwritten. This method accesses the attribute member of the given object, not any named members. Objects that do not have an {{{attribute}}} member silently discard the operation. {{{get_attribute(self, key)}}}:: Return the attribute of this object with the given key. The attribute - a string - is returned. If no such attribute exists, or the object has no {{{attribute}}} member, return None. {{{remove_attribute(self, key)}}}:: If an attribute with the given key exists, remove it. {{{clone(self)}}}:: Returns a deep copy of this object. {{{to_dict(self)}}}:: Return the object as a python dict. No copies are made, so make modifications with caution. {{{to_xml(self)}}}:: Return the XML encoding (from the topdl language) of this object (and nested objects) === Topology Class === This class holds an enitre topology. The utility functions that read and write files operate on it, and it is generally the unit of I/O for the library. It has the following members: {{{substrates}}}:: A list of [TopdlLibrary#SubstrateClass Substrate objects] that contain the substrates of the topology. The list may be empty indicating no communcations are possible. {{{elements}}}:: A list of entities make up the network. These can be of any [TopdlLibrary#ComputerClass Computer objects], [TopdlLibrary#TestbedClass Testbed objects], [TopdlLibrary#SegmentClass Segment objects], or [TopdlLibrary#OtherClass Other objects]. The list may be empty indicating no elements are in the topology. {{{attribute}}}::: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the machine. The list may be empty. {{{version}}}:: The version of topdl that this object encodes. As usual, the constructor takes the member names as parameter names. One useful specialized member is: {{{ incorporate_elements(self) }}} That member sanity checks the substrates to make sure they are uniquely named, assigns names to any interfaces without them, and connects the {{{element}}} members of the [TopdlLibrary#InterfaceClass Interface objects]. If there are inconsistencies, it raises a {{{topdl.ConsistencyError}}} with more detail in the message. {{{incorporate_elements()}}} is called by the {{{Topology}}} constructor as well and that constructor may throw the same exceptions. === Computer Class === A Computer object is a programmable computer in a topology. A webserver is well represented by a computer, as is a desktop. It has the following members: {{{name}}}:: The unique name of the computer in the topology, a string. Other names may be assigned in the {{{localname}}} attribute. {{{cpu}}}:: A list of [TopdlLibrary#CPUClass CPU objects] that describe the CPUs of the machine. The list may be empty, meaning no CPU requirements or information are present. {{{software}}}:: A list of [TopdlLibrary#SoftwareClass Software objects] that describe the software to install. The list may be empty, meaning no software is to be installed. {{{storage}}}:: A list of [TopdlLibrary#StorageClass Storage objects] that describe the storage present the machine. Storage objects can describe memory or persistent storage requirements. The list may be empty, meaning no storage requirements or information are present. {{{interface}}}:: A list of [TopdlLibrary#InterfaceClass Interface objects] that describe connections of the Computer to communications substrates. The list may be empty, meaning that the computer is not connected to any substrates. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the machine. The list may be empty. {{{localname}}}:: A list of strings giving alternative names for the object. When a machine is allocated to a testbed the DNS names assigned to the machine are given as {{{localname}}}s. The list may be empty, meaning no such alternatives exist. {{{status}}}:: A string indicating the current status of the Computer (if any). It may also be None, indicating no status is reported. Valid values are * "empty" - no allocation of resources has been attempted - generally not applied to Computers * "active" - Computer is functioning as part of an experiment * "inactive" - Computer is allocated, but not functioning * "starting" - Computer is transitioning from "inactive" to "active" * "terminating" - Computer is transitioning from "active" to "inactive" * "failed" - an allocation of resources has failed - generally not applied to Computers {{{service}}}:: A list of [TopdlLibrary#ServiceClass Service objects] that describe the services this machine uses or supplies. These are generally used by the federation system, and their definition is somewhat in flux. The list may be empty. {{{operation}}}:: A list of strings that describe the valid operations on the machine. These are generally used by the federation system, and their definition is somewhat in flux.The list may be empty. The constructor takes the member names as parameters. Only {{{name}}} is required. A {{{Computer}}} should only be an element of one {{{Topology}}}. Use {{{clone()}}} to avoid this. === Substrate Class === The {{{Substrate}}} object represents a communication substrate. The elements that have interfaces connected to a substrate can communicate with one another. A {{{Substrate}}} may include default parameters for the latency and capacity of the communication channel. The members are: {{{name}}}:: A string, the unique name of the substrate. {{{capacity}}}:: A [TopdlLibrary#CapacityClass Capacity object] that gives the default capacity of the substrate. It may be None, incdicating that there is no default. {{{latency}}}:: A [TopdlLibrary#LatencyClass Latency object] that gives the default latency of the substrate. It may be None, incdicating that there is no default. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the substrate. The list may be empty. {{{localname}}}:: A list of strings giving alternative names for the object. The list may be empty, meaning no such alternatives exist. {{{status}}}:: A string indicating the current status of the Substrate (if any). It may also be None, indicating no status is reported. Valid values are * "empty" - no allocation of resources has been attempted - generally not applied to Substrates * "active" - Substrate is functioning as part of an experiment * "inactive" - Substrate is allocated, but not functioning * "starting" - Substrate is transitioning from "inactive" to "active" * "terminating" - Substrate is transitioning from "active" to "inactive" * "failed" - an allocation of resources has failed - generally not applied to Substrates {{{service}}}:: A list of [TopdlLibrary#ServiceClass Service objects] that describe the services this substrate uses or supplies. These are generally used by the federation system, and their definition is somewhat in flux. The list may be empty. {{{operation}}}:: A list of strings that describe the valid operations on the substrate. These are generally used by the federation system, and their definition is somewhat in flux.The list may be empty. {{{interfaces}}}:: A list of [TopdlLibrary#InterfaceClass Interface objects] connected to this substrate. The list may be empty, meaning that the substrate is not connected to any elements. This field is maintained by the [TopdlLibrary#TopologyClass Topology object] holding this Substrate object. '''It is not a parameter to the constructor.''' Other than {{{interfaces}}}, the constructor takes the member names as parameters. Only the {{{name}}} is required. Because the enclosing {{{Topology}}} object maintains the {{{interfaces}}} member, it is dangerous to have the same {{{Substrate}}} object in more than one {{{Topology}}}. Use the {{{clone()}}} method to avoid this. === Interface Class === This class defines the attachment of an entity, e.g., a [TopdlLibrary#ComputerClass Computer], to a [TopdlLibrary#SubstrateClass Substrate]. As with Substrates and entities, each Interface should be attached to only one entity and be unique across a [TopdlLibrary#Topology Topology]. An Interface has the following members: {{{substrate}}}:: A list of strings, each naming a substrate to which this interface is connected. The list must have at least one entry. {{{capacity}}}:: A [TopdlLibrary#CapacityObject Capacity object] indicating the communications capacity of the interface. It may be None to use the substrate default. {{{latency}}}:: A [TopdlLibrary#LatencyObject Latency object] indicating the communications delay of the interface. It may be None to use the substrate default. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the interface. The list may be empty. {{{element}}}:: The entity to which this interface is attached. It may be a [TopdlLibrary#ComputerClass Computer object], [TopdlLibrary#TestbedClass Testbed object], [TopdlLibrary#SegmentClass Segment object], or [TopdlLibrary#OtherClass Other object]. This is generally managed by the {{{Topology}}} holding the entity and substrate with which the {{{Interface}}} is attached. It is wise to treat the field as read-only and valid only after a {{{Topology}}} has called {{{incorporate_elements()}}}. {{{subs}}}:: A list of [TopdlLibrary#SubstrateClass Substrates] to which the interface is attached. It '''cannot ''' be set through the constructor. This is generally managed by the {{{Topology}}} holding the entity and substrate with which the {{{Interface}}} is attached. It is wise to treat the field as read-only and valid only after a {{{Topology}}} has called {{{incorporate_elements()}}}. The constructor takes all the member names '''except subs''' as parameters. Most programs will not need to use {{{element}}} as a parameter to the constructor, either. === Attribute Class === An attribute is a user-defined annotation of one of the other classes: a key/value pair. An attribute has the following members: {{{attribute}}}:: A string, the key used to find the attribute. {{{value}}}:: A string, the value While attributes can be manipulated directly liek any of the other topdl classes, it is most common to use the [TopdlLibrary#TopdlClassFeatures attribute manipulation functions] of the annotated class to access them. Should a program need to call the constructor, it takes the member names as named parameters. === CPU Class === This represents a CPU on a computer. It has the following members: {{{type}}}:: A string indicating the type of CPU. These are not yet standardized, but common formats like "i686" or "amd64" are used. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the CPU. The list may be empty. The constructor takes the member names as named parameters. === !OperatingSystem Class === This class captures the operating system requirements of a computer. It has the following members: {{{name}}}:: A string naming the operating system required. This is usually the operating system itself, e.g., Linux, FreeBSD, Windows XP. This may be None, indicating no preference or information. {{{version}}}:: A string identifying the version of the OS. For integrates systems like FreeBSD this is the system version. For distribution-based systems like Linux, this is the kernel version. See below for distribution versioning. This may be None, indicating no version information or preference. {{{distribution}}}:: A string that for operating systems that characterize OS distribution independently of kernel or base system, names the distribution in use, e.g., Ubuntu. This may be None, indicating no information or preference. {{{distributionversion}}}:: A string that for operating systems that characterize OS distribution independently of kernel or base system, names the version of the distribution in use, e.g., 12.04 for Ubuntu. This may be None, indicating no information or preference. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the operating system. The list may be empty. The constructor takes the member names as parameters. Note that all the members may be None, but an empty {{{!OperatingSystem}}} object is not very useful. The flexibility is there to allow specifying an operating system version without a distribution or a distribution without a base version. === Storage Class === The storage requirements or information for a system. This class characterizes memory or permanent storage. Its members are: {{{amount}}}:: A floating point value indicating the number of megabytes of storage being considered. Values less than 0.000001 are treated as 0. {{{persistence}}:: A boolean value indicating whether this is persistent storage, such as a disk or SSD, or volitile such as memory. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the machine. The list may be empty. Currently there is no field for media type. Use attributes. The constructor takes the member names as named parameters. === Software Class === This indicates what software must be installed on a system. Its members are: {{{location}}}:: A string containing a URI from which to install the software, e.g., a file: URI pointing to a tarfile. {{{install}}}:: A string containing location in the local file system to base the software install. This may be None, indicating that the software distribution format includes this information, e.g., an rpm or debian software file. The constructor takes the member names as named parameters. === Service Class === A service class encapsulates the ideas of a [FeddAbout#ExperimentServices DETER federation service]. Such services can be available from or used by a variety of topology elements. The Service class has the following members: {{{name}}}:: The name of the service. A few [FeddAbout#ExperimentServices are defined]. {{{importer}}}:: A list of entity names that are using or requesting the service. This list may be empty. {{{param}}}:: A list of [TopdlLibrary#ServiceParamClass ServiceParam objects] that represent typed parameters to the service. The list may be empty. {{{description}}}:: A string describing the service. This may be None. {{{status}}}:: A string describing the current status of the service (if any). It may also be None, indicating no status is reported. Valid values are * "empty" - no allocation of resources has been attempted - generally not applied to Services * "active" - Service is functioning as part of an experiment * "inactive" - Service is allocated, but not functioning * "starting" - Service is transitioning from "inactive" to "active" * "terminating" - Service is transitioning from "active" to "inactive" * "failed" - an allocation of resources has failed - generally not applied to Services The constructor will take the member names as named parameters. === !ServiceParam Class === A description of the type of parameter to a [TopdlLibrary#ServiceClass Service] will accept. It has the following members: {{{name}}}:: Name of the parameter, a string. {{{type}}}:: A string containing the type of the parameter. Valid types are: * string * int * float The constructor will take the member names as named parameters. === Testbed Class === This class represents a testbed that has allocated resources or from which an allocation is requested. While it is primarily used internally by the federation system, some other applications include a {{{Testbed}}} as a place to attach attributes that pertain to the whole topology. It includes the following members: {{{uri}}}:: A string containing the URI on which the testbed can be contacted. Usually a fedd URI. {{{type}}:: A string indicating the underlying type of the testbed. Values include "deter," "emulab," and "protoGENI". {{{interface}}}:: A list of [TopdlLibrary#InterfaceClass Interface objects] that describe connections of the Testbed to communications substrates. The list may be empty, meaning that the testbed is not connected to any substrates. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the Testbed. The list may be empty. {{{localname}}}:: A list of strings giving alternative names for the testbed. The list may be empty, meaning no such alternatives exist. {{{status}}}:: A string indicating the current status of the Testbed (if any). It may also be None, indicating no status is reported. Valid values are * "empty" - no allocation of resources has been attempted - generally not applied to Testbeds * "active" - Testbed is functioning as part of an experiment * "inactive" - Testbed is allocated, but not functioning * "starting" - Testbed is transitioning from "inactive" to "active" * "terminating" - Testbed is transitioning from "active" to "inactive" * "failed" - an allocation of resources has failed - generally not applied to Testbed {{{service}}}:: A list of [TopdlLibrary#ServiceClass Service objects] that describe the services this Testbed uses or supplies. These are generally used by the federation system, and their definition is somewhat in flux. The list may be empty. {{{operation}}}:: A list of strings that describe the valid operations on the Testbed. These are generally used by the federation system, and their definition is somewhat in flux.The list may be empty. The constructor takes the member names as named parameters. === Segment Class === This class represents a sub-experiment of a federated experiment about which the details are unknown except for its interfaces. It is primarily used internally by teh federation system. It has the following members: {{{id}}}:: An [TopdlLibrary#IDClass ID object] that identifies this segment. {{{type}}}:: A string indicating the underlying type of the testbed. Values include "deter," "emulab," and "protoGENI". {{{interface}}}:: A list of [TopdlLibrary#InterfaceClass Interface objects] that describe connections of the segment to communications substrates. The list may be empty, meaning that the segment is not connected to any substrates. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the segment. The list may be empty. === ID Class === This class represents a polymorphic identifier as used by the federation system. At most one of its members is set. Those members are: {{{fedid}}}:: A fedid object, representing the hash of a principal key. This represents a [FeddAbout#GlobalIdentifiers:Fedids fedid]. {{{uuid}}}:: A string holding a UUID URI. {{{uri}}}:: A string holding a generic URI {{{localname}}}:: A string holding a locally scoped name {{{kerberosUsername}}:: A kerberos user name. The constructor accepts the member names as named parameters. === Other Class === This class represents a topology element that is not captured above. It is a set of interfaces and a place to hang attributes, for use encoding kinds of elements unknown when topdl was specified. The members are: {{{interface}}}:: A list of [TopdlLibrary#InterfaceClass Interface objects] that describe connections of the element to communications substrates. The list may be empty, meaning that the element is not connected to any substrates. {{{attribute}}}:: A list of [TopdlLibrary#AttributeClass Attribute objects] that are attached to the element. The list may be empty. The constructor accepts the member names as named parameters. == Procedures == There are several procedures in the library for inputting and outputing topologies in various formats. This section explains their use. === topology_from_xml === {{{topology_from_xml}}} reads a topdl-encoded topology and returns a [TopdlLibrary#TopologyClass Topology object]. The encoded topology may be in a file or string and the enclosing element name may be set. It is the inverse of [topdlLibrary#topology_to_xml topology_to_xml]. The parameters are: {{{string}}}:: A string containing the XML to parse. If it is set {{{file}}} and {{{filename}}} must be None. {{{file}}}:: An open python file or file-like object that holds the XML to parse. It must be open for reading. If it is set {{{string}}} and {{{filename}}} must be None. {{{filename}}}:: A string containing the filename that holds the XML to parse. If it is set {{{string}}} and {{{file}}} must be None. {{{top}}}:: The name of the element that contains the topdl to parse. Topdl is embedded in other soap objects and XML files. This allows the library to extract the topdl from more complicated XML. If not given it defaults to "topology". Exactly one of {{{string}}}, {{{file}}}, and {{{filename}}} must be given, and to avoid confusion it is best to call {{{topology_from_xml}}} with named parameters. === topology_to_xml === Returns a string containing the topdl representation of the given topology, the inverse of [TopdlLibrary#topology_from_xml topology_from_xml]. The enclosing element may be passed as a parameter. The parameters are: {{{t}}}:: The [Topdl#TopologyClass Topology object] to encode. {{{top}}}:: A string containing the name of the element that will enclose the topdl. If not given, no element will enclose the topdl. === topology_to_ns2 === Returns a string containing an {{{ns2}}} representation of the topology. [TopdlLibrary#ComputerClass Computer objects] are rendered as nodes and [TopdlLibrary#SubstrateClass substrate objects] as lans. The output can be customized with [TopdlLibrary#ns2FilterFunctions filter functions]. The parameters are: {{{t}}}:: The [TopdlLibrary#TopologyClass topology class] to encode {{{filters}}}:: A list of [TopdlLibrary#ns2FilterFunctions filter functions] applied in order as the output is constructed. If unspecified the list is empty. {{{routing}}}:: A string that sets the [https://trac.deterlab.net/wiki/Tutorial/CreatingExperiments#SettingupIProutingbetweennodes experiment routing style]. The Default is "Manual"; single testbed experiments will probably want "Static". ==== ns2 Filter Functions ==== [TopdlLibrary#topology_to_ns2 topology_to_ns2] takes a list of filter functions to customize its output. Each filter must be callable and accept a single parameter and return a string. The parameter can be an element or substrate. That is, {{{e}}} can be a [TopdlLibrary#ComputerClass Computer object], [TopdlLibrary#TestbedClass Testbed object], [TopdlLibrary#SegmentClass Segment object], or [TopdlLibrary#OtherClass Other object], or [TopdlLibrary#SubstrateClass Substrate object]. A callable class is useful if the filter needs to act on internal state. {{{topology_to_ns2}}} calls filters as it generates output. It outputs elements first by traversing its [TopdlLibrary#TopologyClass topology] parameter's elements in order. For each element, the filters provided to {{{topology_to_ns2}}} are called in order and their output appended to the ns2 string. After all elements are output, the substrates are traversed, and again the filters called in order. The filter output comes after the {{{topology_to_ns2}}} output for each element and substrate. Inside a filter it is often useful to translate an element or substrate name into something valid to ns2. The {{{to_tcl_name}}} function is in the topdl namespace for this purpose. It takes a string and returns the tcl name for it. When assigning to the name, this sequence is used: {{{ tname = topdl.to_tcl_name(elem.name) out += 'set %s [$ns node]' % tname }}} when dereferencing the name - which is much more common: {{{ tname = topdl.to_tcl_name(elem.name) out += 'tb-set-node-hardware ${%s} MicroCloud' % tname }}} The output of {{{to_tcl_name}}} should be enclosed in { } to avoid tcl/ns parsing errors. === topology_to_rspec === Converts a topology to a [https://protogeni.org ProtoGENI] [http://www.protogeni.net/trac/protogeni/wiki/RSpec rspec] and returns it as a string. The parameters are: {{{t}}}:: The [TopdlLibrary#TopologyClass topology class] to encode {{{filters}}}:: A list of [TopdlLibrary#ns2FilterFunctions filter functions] applied in order as the output is constructed. If unspecified the list is empty. The filter functions work the same way as in [TopdlLibrary#ns2FilterFunctions topology_to_ns2]. === Parsing from ns2 === An obvious missing element is a procedure to parse ns2 into topdl. Unfortunately, ns2 programs are full, turing-complete otcl programs, and parsing them would require copying the tcl interpreter into python (or any other implementation language - other than otcl). Rather than do that, we encourage programmers who need that translation to use the [http://fedd.deterlab.net/wiki/FeddCommands#fedd_ns2topdl.py fedd_ns2topdl.py] program. That program contacts a running fedd (like the one running on {{{users.isi.deterlab.net}}}) to call an otcl system to do the parsing. = Examples = This section contains some illustrative examples. == Making a Topology == This example constructs the topdl describing a simple start topology with 11 nodes, one center and 10 satellites. [[Image(ten_small.png)]] A python program to print the topdl describing that topology looks like this: {{{ #!/usr/bin/env python from deter import topdl elems = [] # elements in the topology subs = [] # substrates in the topology # Create the center node and put it in the element list. (Name is set # explicitly) center = topdl.Computer(name='center') elems.append(center) # Connect 10 satellites for i in range(0,10): # Create a substrate to join the new element to center link = topdl.Substrate('link%d' % i) # Create the new element (n-i) and connect it to the new substrate with an # Interface object node = topdl.Computer(name='n-%d' % i, interface=[ topdl.Interface(substrate=['link%d' %i])]) # Add an interface to center linking it to the new substrate center.interface.append(topdl.Interface(substrate=['link%d' %i])) # Add the new link and node to the lists subs.append(link) elems.append(node) # Create the topology from the lists. This names the unnamed interfaces and # makes connections. top = topdl.Topology(elements=elems, substrates=subs) # Print the XML print topdl.topology_to_xml(top, top='experiment') }}} The [attachment:ten.py program] builds the topology by creating a [TopdlLibrary#ComputerClass Computer object] that is the center of the star, then creates other computers that are connected to the center. The connection is accomplished by creating a [TopdlLibrary#SubstrateClass Substrate object] that represents the connection and adding [TopdlLibrary#InterfaceClass Interface objects] to the center computer and the satellite that bind them to the substrate. For comparison, [attachment:ten.tcl here] is the equivalent ns2 description: {{{ source tb_compat.tcl set ns [new Simulator] # Create the center node (named by its variable name) set center [$ns node] # Connect 10 satellites for { set i 0} { $i < 10 } { incr i} { # Create node n-1 (tcl n($i) becomes n-$i in the experiment) set n($i) [$ns node] # Connect center to $n($i) ns duplex-link $center $n($i) 100Mb 0ms DropTail } # Creation boilerplate $ns rtptoto Static $ns run }}} == A More Complex Topology == The next few examples will build on one another to show a more complex change to a topology. We will take a topology, find the degrees of nodes, assign operating systems to them based on that degree, and output DETER-ready ns2. To try this we need a more complex topology. The program below creates a topology with {{{nrouters}}} completely connected routers, each of which has {{{nleaves}}} leaves, except for one central router that has none. Here is an image of the topology ([attachment:topo.xml in topdl]) when {{{nrouters}}}=4 and {{{nleaves}}}=4: [[Image(topo.png)]] (One router is leafless because it makes the image more aesthetic). Here is the program that generates that topology: {{{ #!/usr/bin/env python import sys from deter import topdl # Parse the user-supplied routers and leaves arguments try: if len(sys.argv) > 1: arg = sys.argv[1] else: arg = 4 nrouters = int(arg) if len(sys.argv) > 2: arg = sys.argv[2] else: arg = 3 nleaves = int(arg) except ValueError, e: print >>sys.stderr, 'Usage %s routers leaves' % sys.argv[0] sys.exit('Cannot convert %s to an int' % arg) elems = [] # Elements in the topology subs = [] # Substrates in the topology # Make nrouters routers r = [] for i in range(0,nrouters): r.append(topdl.Computer(name='r-%d' % i)) elems.extend(r) # Interconnect them n = 0 for i in range(0,nrouters): for j in range(i+1, nrouters): # Connect r[i] to r[j] # Pick a substrate name link_name = 'rlink%i' % n # Add interfaces to the two routers r[i].interface.append(topdl.Interface(substrate=[link_name])) r[j].interface.append(topdl.Interface(substrate=[link_name])) # Put a substrate into the substrate list with the name we picked subs.append(topdl.Substrate(name=link_name)) n += 1 # Put nleaves leaves on each router, except the center one for i in range(0, (nrouters-1) * nleaves): # Pick a link name link_name = 'llink%i' % i # add a computer with an interface on the new substrate to the element list elems.append(topdl.Computer(name='leaf%d' % i, interface=[ topdl.Interface(substrate=[link_name])])) # Add an interface to a router on the new substrate r[i/nleaves].interface.append(topdl.Interface(substrate=[link_name])) # Put a substrate into the substrate list with the name we picked subs.append(topdl.Substrate(name=link_name)) # Make the topology and print it top = topdl.Topology(elements=elems, substrates=subs) print topdl.topology_to_xml(top, top='experiment') }}} The code takes a few more shortcuts in construction, e.g., most elements and substrates are appended directly to the lists rather than being assigned to variables, but is not fundamentally different from the start topology generator. == Annotating A Topology == The first step in our transformation is to take a topology and add attributes that will be used by other applications. In our example, the other applications will be further refinements in the same program. Here is a function that walks a topology, determines the degree of each computer, and adds an [TopdlLibrary#AttributeClass attribute] to each computer with that degree in it. It demostrates editing a topology using the [TopdlLibrary#TopdlClassFeatures set_attribute] function common to topdl classes. {{{ def annotate_degree(top): ''' Add an attribute (node_degree) to each Computer in the topology (top) that lists its degree ''' for e in top.elements: # Skip elements that are not computers if not isinstance(e, topdl.Computer): continue # degree is the number of interfaces deg = len(e.interface) # Save a string-valued attribute e.set_attribute('node_degree', '%s' % deg) return top }}} == Adding Other Classes == This function takes a topology that has been through the {{{annotate_degree}}} function above and adds an [TopdlLibrary#OperatingSystemClass OperatingSystem nested object] to each Computer in the topology, based on the {{{node_degree}}} attribute. This demonstrates adding a nested object directly, rather rthan through [TopdlLibrary#TopdlClassFeatures set_attribute]. {{{ def add_operating_system(top): ''' Add an OperatingSystem class to each computer in the topology (top). If the node is a leaf (degree 1) make it Ubuntu Linux, otherwise make it FreeBSD. annotate_degree() must have been called on the topology as this routine uses the node_degree attribute to make decisions. ''' for e in top.elements: # Skip non-Computers if not isinstance(e, topdl.Computer): continue a = e.get_attribute('node_degree') # if there is a node_degree attribute, assign an OS if a: # Covnert the string attribute into an integer try: deg = int(a) except ValueError, e: sys.exit('%s has a non-integer degree %s!?' % (e.name, a)) # Assign the os - includes a distribution for Linux if deg == 1: e.os.append(topdl.OperatingSystem(name='Linux', distribution='Ubuntu')) else: e.os.append(topdl.OperatingSystem(name='FreeBSD')) return top }}} == Changing a Topology (adding connections) == This example shows how to change a topology by adding links. This function adds a link between adjacent clusters of leaf nodes. The code walks the topology, finding routers by their [TopdlLibrary#OperatingSystem OperatingSystem] assigned above. For each router it finds, the code collects the leaves attached to that router (again, discriminating based on their OS) and collects them into a list. There will be a list of leaves for each router, all in a list. Then the code connects one computer from each per-router list to a computer in the adjacent router's list. The result is something that looks like this: [[Image(topo2.png)]] The code for this is below: {{{ def add_links(top): ''' For each router - Computer with FreeBSD assigned as an OS - select one of its Linux children and make a connection to one of the other router's children. ''' def has_os(e, os_name): ''' Return true if any of e's os classes has the given os_name ''' if e.os is None: return False else: return any([ o.name == os_name for o in e.os ]) # At the end of this loop, leaves contains a list of leaves for each router # in the topology. leaves = [ ] for e in top.elements: if not isinstance(e, topdl.Computer): continue if not has_os(e, 'FreeBSD'): continue # This is a Computer running FreeBsd my_leaves = [] # walk the interfaces for inf in e.interface: # Walk the interface's substrates for s in inf.subs: # Walk the substrate's interfaces looking for the other # elements - not this router. for sinf in s.interfaces: ee = sinf.element # all the Linux computers get attached to the leaves dict if isinstance(ee, topdl.Computer) and ee != e: if has_os(ee, 'Linux'): my_leaves.append(ee) # One router has no leaves. Ignore it. if len(my_leaves) > 0: leaves.append(my_leaves) # This loop takes the first computer in one entry in leaves and connects it # to the last computer in the next entry in leaves (wrapping around at the # top). for i in range(0, len(leaves)): link_name = 'new-link%d' % i f = leaves[i][0] t = leaves[(i+1) % len(leaves)][-1] f.interface.append(topdl.Interface(substrate=[link_name])) t.interface.append(topdl.Interface(substrate=[link_name])) top.substrates.append(topdl.Substrate(name=link_name)) # Now incorporate the new links top.incorporate_elements() return top }}} This code demonstrates the basic loop that walks through a topology, looking at neighbors of elements. To look at an element's neighbors, code must walk the element's interfaces to find the substrates it is connected to. Then for each substrate, the other elements having an interface on the substrate are examined. The {{{has_os}}} nested function is typical of this sort of loop. It encapsulates a bit of complexity - addressing the possibility that a Computer has more than one OS - in a side function. In most cases that complexity is not used, but the snippet covers the possibility. It is also important to note the {{{Topology.incorporate_elements}}} call near the end. That call completes the connections between interface objects and substrate objects (filling in the {{{Interface.subs}}} members and {{{Substrate.intrefaces}}} member) and between Interfaces and elements (by filling in the {{{Interface.element}}} member). Without that call further use of the topology is difficult. That call will also throw exceptions if the changes have introduced inconsistencies - multiple substrates with the same name, etc. == Output to NS2 == Finally, we want to output out example topology in ns2 format so that we can directly swap it in to DETER. Of course, fedd and the containers system will take (and prefer) topdl, but as an example, we show outputting ns2. This shows the use of a filter to tweak ns2 output for local requirements. The {{{os_filter}}} function defined in {{{output_ns}}} below converts a single [TopdlLibrary#OperatingSystemClass OperatingSystem object] attached to a computer into a local DETER OSID. Notice that the filter always returns a string - the empty one when no changes are made. Do not return None from a filter. Here is the output code: {{{ def output_ns(top, filename): ''' Output ns2 with the OperatingSystem specifications converted to local OSIDs when known. ''' def os_filter(e): ''' Convert OS id to local equivalents: FreeBSD -> FBSD8-STD Lunix/Ubuntu -> Ubuntu-64-STD ''' # Only work on Computers if not isinstance(e, topdl.Computer): return '' # Only handle exactly 1 specified OS if len(e.os) != 1 : return '' # String translation if e.os[0].name == 'FreeBSD': return 'tb-set-node-os ${%s} FBSD8-STD\n' % \ topdl.to_tcl_name(e.name) elif e.os[0].name == 'Linux' and e.os[0].distribution == 'Ubuntu': return 'tb-set-node-os ${%s} Ubuntu1204-64-STD\n' % \ topdl.to_tcl_name(e.name) else: return '' # Main line. Open the file and write the otcl to it. try: f = open(filename, 'w') # Include the os_filter f.write(topdl.topology_to_ns2(top, filters=[os_filter], routing='Static')) f.close() except EnvironmentError, e: sys.exit('Cannot write %s: %s' % (e.filename, e.strerror)) }}} == Putting It All Together == The complete example is [attachment:example.py attached to this page.] It consists of : * Parsing an input and output filename from the command line * Reading a topology from the input filename (you can use [attachment:topo.xml the example]). * Calling the topology manipulation subroutines developed above in sequence * Calling the ns2 output routine above using the output filename This is by no means the most efficient way to change the initial topology to the final topology; it was organized this way to demonstrate the various features. There is sample [attachment: out.tcl ns2 output] attached as well. The intent of the examples is to give you a starting point for using the topdl library. If you have questions or comments, [mailto:faber@isi.edu contact us].