CA2451042A1

CA2451042A1 - Method and apparatus for provisioning a communication path

Info

Publication number: CA2451042A1
Application number: CA002451042A
Authority: CA
Inventors: Alex Mashinsky; Daniel Wang; Erik Johnson; Keith Bemer
Original assignee: Elematics, Inc.; Alex Mashinsky; Daniel Wang; Erik Johnson; Keith Bemer
Current assignee: ELEMATICS Inc
Priority date: 2001-06-22
Filing date: 2002-06-21
Publication date: 2003-01-03
Also published as: WO2003001397A1

Abstract

The present invention enables and facilitates real-time provisioning of circuits or other communication paths across multi-carrier, multi-vendor networks. In network having network elements, each network element is associated with an intelligent provisioning node that can inventory the network element to determine the network element's topology, capabilities and connections. The intelligent provisioning node may form part of a signaling control plane that can manage network elements to provision a circuit between network elements.

Description

METHOD AND APPARATUS FOR PROVISIONING
A. COMMUNICATION PATH
CROSS-REFERENCE TO RELATED APPLICATIONS
S This patent application is related to, and claims priority to, U.S. Patent Application Serial No. 60/300,527 filed June 22, 2001, and entitled METHOD AND
SYSTEM FOR PROVISIONING OPTICAL CIRCUITS 1N A MULTI-NETWORK, MULTI-VENDOR ENVIRONMENT, the contents of which are incorporated herein by reference for all purposes.
TECHNICAL FIELD
The present invention relates generally to the field of communication systems, and more particularly, to methods and apparatus for provisioning a communication path for a communication systems.
BACKGROUND
Currently network peering - or the exchange of data between two different networks - may be divided into two groups. The first group comprises open peering in, for example, Internet exchanges. The second group comprises private, bilateral peering relationships.
Both types of peering often require the participants to over-provision capacity and negotiate each capacity transaction on a case-by-case basis. For example, a network service provider interested in entering into a peering agreement must acquire enough capacity to handle spikes in their demand (i.e. spikes in customer traffic) if they are to avoid paying extra for the extra capacity necessary to carry the added customer traffic. Further, they are in danger of a degradation in the quality of service (QoS) provided during those peek times if their demand exceeds the available capacity. Such over-provisioning often results in portions of the network sitting idle for the majority of the time (i.e. the time between the. spikes), which is disadvantageous and costly. Further, typical Internet peering agreements require lengthy and complicated agreements that are disadvantageous for a variety of reasons.
For example, it may be difficult to anticipate future needs prior to or during formation of the contract. Such agreements may also be prohibitively expensive and time consuming.
Several standards are emerging to allow systems to request and to provision circuits from nodes in the same or other networks. For example, the Internet Engineering Task Force (IETF) currently is developing the Generalized Multi-Protocol Label Switching protocol (GMPLS) for this purpose. The GMPLS protocol is an extension of the Multi-Protocol Label Switching (MPLS) protocol designed to support devices that perform switching in the packet, time, wavelength, and space domains. As another example, the Optical Internetworking Forum (OIF) is working on specifications for an optical User Network Interface (UNI) that defines the protocol for a client (user) device to request the provision for a circuit in an optical network. Similar work is taking place to define a Network-to-Network Interface (NNI) for one network to request a circuit from another. Unfortunately, these proposed solutions do not address many issues regarding the provisioning of circuits.
For example, these proposed solutions do provide unified provisioning and inventory control over managed network elements. In addition, the proposed solutions do not address legacy problems that may exist when attempting to provision a circuit using an existing group of network elements, particularly when the network elements are supplied by different vendors.
SUMMARY OF THE INVENTION
In some embodiments, the present invention enables and facilitates real-time provisioning of circuits or other communication paths across multi-Garner, multi-vendor networks. The networks may include optical and/or non-optical components, and may operate using different technologies, protocols, network elements, etc. In addition, in some embodiments, the present invention provides a signaling control plane that may be integrated with existing network infrastructure and operations support systems (OSS) to facilitate provisioning of circuits between network elements. The signaling control plane may mediate communications between a data plane having one or more network elements and a management plane having various network management applications. Furthermore, in some embodiments, the present invention allows for rapid set-up and teardown of circuits and other communication paths between end-user requested endpoints or network elements. The present invention allows unified provisioning of circuits and inventory control over managed network elements, even when the network elements are from different vendors.
In addition, the present invention allows a unified interface to a management plane for the network elements.
Additional advantages and novel features of the invention shall be set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the following or may be learned by the practice of the invention.
According to some embodiments of the present invention, a method for facilitating provisioning of a circuit between a first network element and a second network element may include receiving a request to establish a circuit between a first network element and a second network element in a network, the first network element being associated with a first provisioning node that forms part of a signaling control plane for the network, and the second network element being associated with a second provisioning node that forms part of the signaling control plane for the network; determining an available circuit between the first network element and the second network element; and providing commands for the first provisioning node and the second provisioning node regarding provisioning of the available circuit between the first network element and the second network element.
In some other embodiments, a method for facilitating provisioning of a circuit between a first network element and a second network element may include receiving a request to establish a circuit between a first network element in a first network and a second network element in a second network, the first network element being associated with a first provisioning node that forms part of a first signaling control plane for the first network, and the second network element being associated with a second provisioning node that forms part of a second signaling control plane for the second network; providing a notification indicative of the request;
determining at least a portion of an available circuit between the first network element and the second network element, the portion of the available circuit including the first network element; and providing at least one command to the first provisioning node for provisioning of the portion of the available circuit.

In some further embodiments, a method for facilitating provisioning of a circuit between a first network element and a second network element may include receiving a first notification indicative of a request to establish a circuit between a first network element in a first network and a second network element in a second network, wherein the notification is received from a first signaling control plane associated with the first network and indicates a bridge element connected to the first network; identifying the second network; identifying a second signaling control plane, the second signaling control plane being associated with the second network;
providing a second notification,indicative of the request to the second signaling control plane; receiving from the second signaling control plane data indicative of a bridge element connected to the second network; determining an available circuit between the bridge element connected to the first network and the bridge element connected to the second network; and providing commands for the first provisioning the available circuit.
In some even further embodiments, a method for facilitating provisioning of a circuit between a first network element and a second network element may include determining at least one operational characteristic of a first network element in a first network; providing information indicative of the operational characteristic to a device associated with the first network, wherein the device forms part of a signaling control plane for the first network; receiving a request to establish a requested circuit between the first network element and a second network element, wherein the second network element is associated with a provisioning node; providing data indicative of the request to the device; receiving information from the device indicative of a first command for the first network element for use in provisioning an available circuit between the first network element and the second network element; and providing data indicative of the first command to the first network element.
In some still further embodiments, a method for facilitating provisioning of a circuit between a first network element and a second network element may include receiving a request to establish a circuit between a first network element and a second network element, the first network element being associated with a first provisioning node that forms part of a signaling control plane, and the second network element being associated with a second provisioning node that forms part of the signaling control plane; determining an available circuit between the first network element and the second network element; and providing commands for the first network element and the second network element regarding provisioning of the available circuit between the first network element and the second network element.
According to some embodiments of the present invention, a system for establishing a circuit between network elements may include a first plurality of provisioning nodes associated with a'first network, wherein each of the first plurality of provisioning nodes is associated with a respective network element contained in the first network and operative to inventory the respective network element; and a first node controller in communication with each of the first plurality of provisioning nodes and operative to receive information from each of the first plurality of provisioning nodes regarding inventory of network elements included in the first network; receive a request for provisioning a requested circuit, wherein the request identifies at least one network element in the first network to be used in the requested circuit; determine an available circuit in the first network that involves the first network element and forms at least a portion of the requested circuit;
determine a one of the first plurality of provisioning nodes associated with the first network element;
provide at least one command to the one of the first plurality of provisioning nodes associated with the first network element regarding provisioning of the available circuit.
In some further embodiments, a system for establishing a circuit between network elements might include a signaling control plane associated with a network that includes a first plurality of network elements, wherein the signaling control plane includes a first plurality of provisioning nodes and a first node controller, wherein each of the first plurality of provisioning nodes is associated with a different one of the first plurality of network elements and is in communication with the node controller, and wherein the signaling control plane is operative to determine an inventory of at least one capability of each of the first plurality of network elements;
receive a request to provision a circuit that includes at least one of the first plurality of network elements; determine an available circuit that is formable using at least one of the first plurality of network elements and that satisfies at least a portion of the request; send commands to the at least one of the first plurality of network elements to facilitate establishment of the available circuit.
According to some embodiments of the present invention, an apparatus for facilitating provisioning of a circuit between a first network element and a second network element may include means for obtaining a request to establish a circuit between a first network element and a second network element in a network, the first network element being associated with a first provisioning node that forms part of a signaling control plane for the network, and the second network element being associated with a second provisioning node that forms part of the signaling control plane for the network; means for identifying an available circuit between the first network element and the second network element; and means for sending commands for the first provisioning node and the second provisioning node regarding provisioning of the available circuit between the first network element and the second network element.
In some other embodiments, an apparatus for facilitating provisioning of a circuit between a first network element and a second network element may include means for obtaining a request to establish a circuit between a first network element in a first network and a second network element in a second network, the first network element being associated with a first provisioning node that forms part of a first signaling control plane for the first network, and the second network element being associated with a second provisioning node that forms part of a second signaling control plane for the second network; means for sending a notification indicative of the request; means for identifying at least a portion of an available circuit between the first network element and the second network element, the portion of the available circuit including the first network element; and means for sending at least one command to the first provisioning node for provisioning of the portion of the available circuit.
In some further embodiments, an apparatus for facilitating provisioning of a circuit between a first network element and a second network element may include means for obtaining a first notification indicative of a request to establish a circuit between a first network element in a first network and a second network element in a second network, wherein the notification is received from a first signaling control plane associated with the first network and indicates a bridge element connected to the first network; means for determining the second network; means for determining a second signaling control plane, the second signaling control plane being associated with the second network; means for sending a second notification indicative of the request to the second signaling control plane; means for obtaining from the second signaling control plane data indicative of a bridge element connected to the second network; means for identifying an available circuit between the bridge element connected to the first network and the bridge element connected to the second network; and means for sending commands for the first provisioning the available circuit.
In some even further embodiments, an apparatus for facilitating provisioning of a circuit between a first network element and a second network element may include means for identifying at least one operational characteristic of a first network element in a first network; means for sending information indicative of the operational 1 S characteristic to a device associated with the first network, wherein the device forms part of a signaling control plane for the first network; means for obtaining a request to establish a requested circuit between the first network element and a second network element, wherein the second network element is associated with a provisioning node;
means for sending data indicative of the request to the device; means for obtaining information from the device indicative of a first command for the first network element for use in provisioning an available circuit between the first network element and the second network element; and means for sending data indicative of the first command to the first network element.
In some still further embodiments, an apparatus for facilitating provisioning of a circuit between a first network element and a second network element may include means for obtaining a request to establish a circuit between a first network element and a second network element, the first network element being associated with a first provisioning node that forms part of a signaling control plane, and the second network element being associated with a second provisioning node that forms part of the signaling control plane; means for identifying an available circuit between the first network element and the second network element; and means for sending commands for the first network element and the second network element regarding provisioning of the available circuit between the first network element and the second network element.
According to some embodiments of the present invention, a computer program in a computer readable medium for facilitating provisioning of a circuit between a first network element and a second network element may include first instructions for obtaining a request to establish a circuit between a first network element and a second network element in a network, the first network element being associated with a first provisioning node that forms part of a signaling control plane for the network, and the second network element being associated with a second provisioning node that forms part of the signaling control plane for the network; second instructions for identifying an available circuit between the first network element and the second network element; and third instructions for sending commands for the first provisioning node and the second provisioning node regarding provisioning of the available circuit between the first network element and the second network element.
1 S In some other embodiments, a computer program in a computer readable medium for facilitating provisioning of a circuit between a first network element and a second network element may include first instructions for obtaining a request to establish a circuit between a first network element in a first network and a second network element in a second network, the first network element being associated with a first provisioning node that forms part of a first signaling control plane for the first network, and the second network element being associated with a second provisioning node that forms part of a second signaling control plane for the second network;
second instructions for sending a notification indicative of the request;
third instructions for identifying at least a portion of an available circuit between the first network element and the second network element, the portion of the available circuit including the first network element; and fourth instructions for sending at least one command to the first provisioning node for provisioning of the portion of the available circuit.
In some further embodiments, a computer program in a computer readable medium for facilitating provisioning of a circuit between a first network element and a second network element may include first instructions for obtaining a first notification indicative of a request to establish a circuit between a first network element in a first network and a second network element in a second network, wherein the notification is received from a first signaling control plane associated with the first network and indicates a bridge element connected to the first network;
second instructions for determining the second network; third instructions for determining a second signaling control plane, the second signaling control plane being associated with the second network; fourth instructions for sending a second r notification indicative of the request to the second signaling control plane;
fifth instructions for obtaining from the second signaling control plane data indicative of a bridge element connected to the second network; fifth instructions for identifying an I O available circuit between the bridge element connected to the first network and the bridge element connected to the second network; and sixth instructions for sending commands for the first provisioning the available circuit.
In some even further embodiments, a computer program in a computer readable medium for facilitating provisioning of a circuit between a first network element and a second network element may include first instructions for identifying at least one operational characteristic of a first network element in a first network;
second instructions for sending information indicative of the operational characteristic to a device associated with the first network, wherein the device forms part of a signaling control plane for the first network; third instructions for obtaining a request to establish a requested circuit between the first network element and a second network element, wherein the second network element is associated with a provisioning node; fourth instructions for sending data indicative of the request to the device; fifth instructions for obtaining information from the device indicative of a first command for the first network element for use in provisioning an available circuit between the first network element and the second network element; and sixth instructions for sending data indicative of the first command to the first network element.
In some still further embodiments, a computer program in a computer readable medium for facilitating provisioning of a circuit between a first network element and a second network element may include first instructions for obtaining a request to establish a circuit between a first network element and a second network element, the first network element being associated with a first provisioning node that forms part of a signaling control plane, and the second network element being associated with a second provisioning node that forms part of the signaling control plane;
second instructions for identifying an available circuit between the first network element and the second network element; and third instructions for sending commands for the first network element and the second network element regarding provisioning of the available circuit between the first network element and the second network element.
With these and other advantages and features of the invention that will become hereina$er apparent, the nature of the invention maybe more clearly understood by reference to the following detailed description of the invention, the appended claims and to the several drawings attached herein.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are incorporated in and form a part of the specification, illustrate the preferred embodiments of the present invention, and together with the descriptions serve to explain the principles of the invention.
Figure 1 A is a block diagram of a representation of components of a system in accordance with the present invention;
Figure 1 B is a block diagram of another representation of components of a system in accordance with the present invention;
Figure 2 is a block diagram of a representative management plane, signaling control plane, and a data plane formed by or used with IPNs, INCs and nodes;
Figure 3 is a block diagram of a representative IPN of Figure 1 or Figure 2;
Figure 4 is a block diagram of a representative INC of Figure 1 or Figure 2;
Figure 5 is a bock diagram overview of a Dense Wave Division Multiplexing (DWDM) system that may be used with or in the systems of Figures 1 and 2;
Figure 6 is a tabular representation of a record used by or in the databases used in the systems of Figures 1A and Figure 1B;
Figures 7A, 7B and 7C are a flowchart of a first embodiment of a method in accordance with the present invention, for provisioning a circuit or other communication path;
Figures 8A, 8B and 8C are a flowchart of a second embodiment in accordance with the present invention for provisioning a circuit or other communication path;

Figure 9 is a block diagram of a network configuration that may be employed for testing a circuit or other communication path provisioned according to the methods of the present invention; and Figures 10A, l OB and l OC are block diagrams that depict a representative manner in which provisioning a circuit or other communication path in accordance with the present invention may affect various layers of the network.

Applicants have recognized that there is a need for systems, means and methods that allow or facilitate provisioning of circuits (also referred to herein as transmission paths or communications paths) between network nodes (also referred to herein as network elements (NE)) in one or more networks. In some embodiments, a network node may be or include one or more optical or non-optical switches, routers, cross-connects, DWDM equipment, gateways, hosts, applications, computers or computer systems, of other types of devices. A provisioned communication path may last only as long as needed for a specific data transmission or may last an indefinite period of time.
In some embodiments, the methods and apparatus of the present invention help address many economic and technical difficulties, including, but not limited to:
poor price segmentation amongst a carrier's customers that leads to poor provisioning systems/processes; over-provisioning of circuits or other communication paths as a solution for demand management; inability to make capacity available to all who need it around the clock; slow development or deployment of high-bandwidth network applications due to the inability to allocate optical capacity in real time.
Some embodiments of this invention involve creating the signaling and mediation necessary to allow a router (e.g., a CiscoTM router) to signal a request for additional peering capacity to the software management system controlling the physical layer (i.e. layer 1 of the OSI model) of the peering partner's network to allow for better, faster traffic flow, or special capacity for special types of traffic. Currently such requests often are placed by network engineers in response to alarms from the network that indicate congestion or packet loss, or based on network design applications that analyze historical network data and provide forecasts for future demand. This present invention allows actual networks, as opposed to engineers and network planers, to automatically request and provision capacity between networks.
This type of dynamic provisioning can dramatically reduce network congestion and enable a more responsive optimization and management of optical and routed S networks.
These and other features will be discussed in further detail below, by describing a system, individual devices, and processes according to embodiments of the invention.
System Now referring to Figure 1A, a communications system 100 may include various computer, data or other communication networks or network domains101, 102 and 103, various nodes or network elements 110, 111, 112, 113, 114 and 115, various intelligent provisioning nodes (IPN) 120, 121, 122, 123, 124, 125 and 128, at least one intelligent node controller (INC) 130, a database or other resource 131, and one or more customers 140 and 141. In some embodiments, the database or other resource 131 may be included in or form part of the INC 130.
As will be discussed in more detail below, the network elements 110 and 111 may form a data plane of components in or for the network 101 while the IPNs 120, 121 and the INC 130 may form part of or be included in a signaling control plane for the network 101. Typically, a realm of INC control will be referred to as an INC
domain while a collection of one or more INC domains may make up a signaling control plane. While only one INC is illustrated in Figure 1 A that includes within its domain the network elements 110, 111, 112, 113, 114 and 115, in some embodiments more than one INC may be used in or as part of signaling control plane, as will be discussed in more detail below. In addition, in some embodiments, more than one domain INC may exist in or as part of a signaling control plane, also as will be discussed in more detail below.
In some embodiments, an IPN and/or an INC may be implemented on or by a computer, server, computer system, etc. In addition, in some embodiments, more than one IPN may be implemented on or by a computer, server, computer system, etc.

For purposes of discussion, but not limitation, of the present invention, the terms intelligent provisioning node, intelligent node controller, network element, data plane, management plan, signaling control plane, and user network interface are used for convenience only and no specific limitations are implied or intended by the used of such terms.
In some embodiments, some or all of the network of IPNs 120, 121, 122, 123, 124, 125 and 128 may form a signaling control plane 150 for the networks 101, 102, 103, as will be discussed in more detail below. In some embodiments the INC

also may be included in or form part of the signaling control plane 150 and provide control plane communications for all of the IPNs within its domain. As will be discussed in more detail below, in some embodiments an INC or a signaling control plane may be connected to or in communication with a management plane. The management plane may provide additional monitoring and management functionality for networks, network elements, etc.
In some embodiments, an IPN may provide command set translation between a particular vendor's equipment and a signaling control plane command set. An IPN
may be configured with the necessary "personality package" or other software, interface, data, etc. specific to a network element under its control. Thus, the interface between an IPN and a network element may be vendor specific and in some cases may be TL 1, CORBA or GSMP. The interface may allow the IPN to discover and maintain topology, inventory, alarm, capability, and configuration information (collectively referred to herein as "inventory" information or "operational characteristic" information) regarding the network element as well as to issue provisioning commands to the network element. The IPN may provide some or all of this information to an INC. The IPN also may translate network element specific information into a protocol or other format useable or recognizable by an INC.
In addition, the IPN may develop and maintain a virtual representation or model of the network element for use in provisioning a circuit.
An IPN may receive provisioning commands from an INC, translate these commands into specific commands appropriate to the network elements) under the IPN's control, and execute the commands on the network element. Thus, the IPN
provides network element or node control. In some embodiments, an IPN may be responsible for the management and control of its associated network element or some or all of the capabilities or operational characteristics or features of the associated network element. In addition, the IPN may, collectively with its associated network element, perform security mechanisms such that access to the network element is screened. In some embodiments, only a subset of the facilities on the network element may be controlled by the IPN (or an associated signaling control plane) while the rest of the facilities or capabilities of the network element are controlled by the other systems or devices. For example, in some embodiments, network service providers may request that only a specified number of ports be made available for control by an IPN or signaling control plane. More specifically, a given service provider may only make ten percent of the ports on a network element (e.g., the node 110) available for use by IPN or signaling control plane. Thus, the network element may effectively be partitioned so that the IPN and/or signaling control plane only recognizes the identified ports as existing on the network element and is allowed to use only the identified ports in provisioning a circuit that includes the network element. Also, inventory information gathered by the IPN regarding the network element may be limited to the identified ports and not other ports available on the network element.
As will be discussed in more detail below, in some embodiments the INC 130 may maintain or include a topology database or other resource for its domain such that rapid path provisioning can be accomplished when service requests are received.
The INC 130 may retain inventory information regarding one or more network elements provided to it by one or more IPNs associated with the network elements, as well as manually entered configuration information such as link definitions and link attributes. In some embodiments, if an IPN should fail or be replaced, the new, corrected, or repaired IPN (which may be implemented in software) may discover the resources from or for its associated network elements) and forward the information to the INC 130. The INC 130 may correlate the new information with previously known information such that links or attributes will not need to be reentered. Thus, localized IPN outages may be recovered with minimal user intervention and down time.
In addition, the INC 130 may provide routing options for service requests and may issue commands to one or more IPNs in its domain once a specific path is determined or chosen for service establishment to fulfill the service request by configuring or controlling nodes or other network elements. For supporting multi-carrier provisioning, INCs may use IP-based secured INC to INC signaling to query each other for network resource availability and to coordinate provisioning of services between each other. Thus, an INC may support INC to INC communications in order to facilitate inter-domain path selection and provisioning of services between network elements in different networks.
In some embodiments, an INC may be a control element of a network domain.
The INC will communicate with all IPNs within the domain and may be responsible for maintaining all topology/inventory, usage, alarm, performance monitoring statistics, and service inventory information for the domain. An INC resource (e.g., the database 131 ) may be segmented into a resource inventory database and a service inventory database. The resource inventory database may track and retain information for or regarding the domain such as: hardware inventory for controlled network elements, listing of ports and port types for network elements, listing of optical links and link attributes, listing of cross-connects and circuits provisioned, topology information regarding the relationship between network elements , operational characteristics of the network elements, etc. The service inventory database may track and retain the following domain information: provisioned services, attributes of provisioned services, path information for provisioned services, establishment and disconnect time for provisioned services, cost metric totals for each service, performance monitoring information for each service, etc. In some embodiments, a signaling control plane may be or provide the control of a network and may include one or more INCs that maintain topology and capability (also referred to as operational characteristic) information for network elements in their respective domains (e.g., subsets of the network). Signaling between the INCs may be used to broker provisioning transactions between domains in the network.
In general, a signaling control plane is capable of signaling, or being in communication with, one or more nodes (e.g., network elements) to configure or provision a circuit or other communication path between two network elements that may be in the same or in different domains. In the system 100, the signaling control plane 150 may signal or communicate with one or more of the nodes 110, 111, 112, 113, 114 and 115 to form circuits among or across the networks 101, 102, 103 or other networks not illustrated in Figure 1 A.
A signaling control plane may allow accurate inventory of network elements controlled by the signaling control plane and set up rules/polices that govern under what terms circuits may be provisioned using one or more of the network elements.
Provisioning requests may be provided to the signaling control plane by a customer, an operator using a console as part of a network management system, another signaling control plane, an INC in another signaling control plane, etc. If the provisioning request meets the rules/policies configured by the signaling control plane, the requested circuit may be provisioned automatically. Thus, the signaling control plane may provide unified management for physical resources in a one or more networks. The networks may include network elements provided by different vendors or utilize different technologies. In some embodiments, a signaling control plane may determine routes for services through a managed network or group of network elements, account for available and used inventory, manage protection switching and restoration for managed network elements or domains, etc. In some embodiments, a signaling control plane can provide the signaling mechanisms to network elements managed by the signaling control plane. The signaling mechanisms may affect configuration changes in the managed network elements to provide services as well as necessary logic. In addition, a signaling control plane may provide a unified interface from the managed network elements to various management plane applications (e.g., network management system (NMS) applications, element management system (EMS) applications, operations support systems (OSS) applications).
In some embodiments, one or more of the networks 101, 102, 103 may be, include or represent optical networks, non-optical networks, muter networks, long-haul networks, metropolitan area networks (MANs), wide area networks (WANs), local area networks (LANs), etc. The different networks or domains 101, 102, may run on different protocols (e.g. IP, Ethernet, ATM), may have equipment from different vendors (e.g., Cisco, Corvis, CIENA, Extreme Networks) and may be owned and operated by different carriers (e.g., Sprint, Verizon, etc.).

In some embodiments, one or more of the networks or domains 101, 102, 103 might be or include some or all of the Internet, the World Wide Web, or some other public or private computer, cable, telephone, client/server, peer-to-peer, or communications network or intranet. The networks 101, 102, 103 illustrated in Figure 1A are meant only to be generally representative of cable, computer, telephone, peer-to-peer or other communication networks for purposes of elaboration and explanation of the present invention and other devices, networks, etc. may be used, configured, etc. without departing from the scope of the present invention. In some embodiments a network also can include other public and/or private wide area networks, local area networks, wireless networks, data communication networks or connections, intranets, routers, satellite links, microwave links, cellular or telephone networks, radio links, fiber optic transmission lines, optical circuits, ISDN
lines, T1 lines, DSL, etc.
Each of the networks 101, 102, 103 comprises multiple nodes (also referred to as network elements). For example, the network 101 includes the nodes 110 and 11 l, the network 102 includes the nodes 112 and 113, and the network 103 includes the nodes 114 and 11 S. A network may include more than two nodes. As previously discussed above, a node or network element may represent optical or non-optical switches, routers, cross-connects, DWDM equipment, transport equipment, etc.
As depicted in Figure 1A, each of the nodes 110, 111, 112, 113, 114, and 115 is in communication with at least one other node. The communication protocol and technology may differ between different node connections depending on the nature of the nodes involved (i.e. vendor, configuration, etc.). In some embodiments, some connections between nodes may occur within a single network, while others occur across multiple networks. Connections between nodes in different networks may occur, for example, at public or private peering points, collocation facilities, etc. It should also be noted that nodes depicted in a first network (e.g., the network 101) may be of a different model, vendor, type, etc. as compared to those nodes depicted in a second network (e.g., the network 102). In some instances, multiple connections may exist between two nodes. For example, if the nodes 111 and 112 are optical switches, they may be connected by three OC-12s links.

Each of the nodes 110, 111, 112, 113, 114 and 115 interfaces with at least one of the IPNs 120, 121, 122, 123, 124, 125 and/or 128. In some embodiments, an IPN
may be a separate network element from a node, but in some embodiments one or more IPNs may be integrated with one or more existing nodes. As will be discussed in more detail below, in some embodiments an IPN may represent a software process for controlling and managing a network element. Mutliple IPN applications may then operate on the same IPN machine or device for managing and controlling multiple network elements. For example, the IPNs 120 and 121 may be implemented by two IPN applications operating on the same device. Each network element may have one IPN application associated with it and in some embodiments an IPN application or process may be integrated with or into the network element (e.g., operating or running on the network element). For purposes of discussion of Figure 1A, each network element is illustrated as being connected to a separate IPN device that is operating an IPN application to manage the network element.
As depicted in Figure 1A, each of the nodes 110, 111, 112, 113, 114, 115 is in communication with a separate IPN. The node 110 (also referred to as network element 110) is in communication with the IPN 120, the node 111 is in communication with the IPN 121, etc. In some embodiments, more than one node may be in communication with a single IPN (e.g. if the IPN is operating more than one IPN application to support multiple nodes), and/or more than one IPN may be connected to a single node. As further depicted in Figure 1A, while not required, in some embodiments an IPN also may be in communication with another IPN. Some of these connections may take place across different networks (e.g. IPN 121 and IPN
122). In some instances, an IPN may be connected to more than one other IPN, or not connected to an IPN at all (e.g. only connected to the INC and a node).
Further discussion and description of an IPN is provided below.
Each of the IPNs 120, 121, 122, 123, 124, 125 also is in communication with the INC 130. The INC 130 may interface with IPNs deployed in its domain via IP-based connectivity.
As depicted in Figure 1A, the INC 130 does not effectively reside in any one network, but instead sits above all the networks. However, in some embodiments the INC 130 may actually reside in or be associated with a given network. Further, although Figure 1A depicts only one INC 130, more than one INC may exist. For example, as alternative to the configuration of the system 100 illustrated in Figure 1A, the IPNs 110, 111, 120 and 121 for the network elements within the networks may be in communication with one INC, while the IPNs 112, 113, 122 and 123 for the network elements in the network 102 may be in communication with a second or different INC. In such an embodiment, the multiple INCs associated with the different networks may be in communication with each other or with a "master"
INC, as will be discussed in more detail below. In some embodiments, each network may have its own associated INC that may communicate with INCs associated with other networks for provisioning circuits between nodes in the different networks, as will be discussed in more detail below.
In some embodiments, the INC 130 maintains or includes the database 131.
The database 131 may include or contain information relating to the topology and capability of each of the networks 101, 102, 103 involved in the system 100.
The database 131 may be used to calculate paths for circuits within the involved networks.
The endpoints of such paths may be within one network, or in different networks.
For example, the database 131 may be used to calculate or otherwise determine a path between the node 110 in the network 101 and the node 114 in the network 103.
In some embodiments, the database 131 may be internal to the INC 130 and/or be part of the INC 130. In addition, the database 131 may be a real time database to support quick provisioning of circuits.
The customers 140, 141 may be in communication with the networks 101 and 103, respectively. In some embodiments, the customers may be or include carriers, Internet service providers (ISPs), corporations, individuals with desktop computers, LANs, WANs, etc. As depicted in Figure 1A, the customers 140 and 141 are represented as networks of nodes. For example, the customer 140 may represent a gigabit Ethernet corporate LAN, an IP router network, etc.
In some embodiments, customers may also be in direct communication with at least one IPN, for example an IPN that is associated with a node. As depicted in Figure 1A, the customer 140 is in communication with the IPN 120. In another example, an IPN may be associated with a customer. For example, as depicted in Figure 1A, the customer 141 is in communication with IPN 128. According to some embodiments, at least one customer may be in direct communication with the INC.
Further, in some embodiments, customers may be directly connected to each other, for example at a private peering point. In some embodiments, there may be no difference between customers and nodes. For example, the node 110 may be a customer, requesting a circuit be provisioned to, for example, the node 113.
According to some embodiments of the present invention, a network of interconnected IPNs, including, in some embodiments, one or more INCs, may form a signaling control plane, such as the signaling control plane 1 SO for the interconnected networks 101, 102, 103. This control plane is capable of signaling at least one node in each of the networks for the configuration and provision of circuits that may cut across multiple networks.
In some embodiments, a signaling control plane may utilize in-band or out-of band signaling. For example, a signaling control plane may interact with a network of optical switches using a separate IP control channel (out-of band). In another 1 S example, the signaling control plane may utilize e.g. unused SONET
overhead bytes, or may make use of MPLS labels in a network of label-switched routers.
According to some embodiments, the signaling control plane may make use of a combination of in-band and out-of band signaling.
In some embodiments, each of the localized IPNs 120, 121, 122, 123, 124, 125 may perform a self discovery process according to or during which information related to local network topology and capability is discovered, including information regarding capabilities of nodes associated with the IPNs 120, 121, 122, 123, 124, 125, information regarding connections between nodes within a network as well as cross-connections to nodes in other networks, etc. Inventory of the local network topology and capability for a node is stored locally at the IPN associated with the node and, in some embodiments, the IPN may provide or upload some or all of the information to the INC 130.
During the self discovery or inventory process, an IPN may communicate directly with a network element and construct a model or virtual representation of the network element. The model provides a generic representation of the network element that can be stored in and used by the IPN. An IPN may query a network element to determine the make, model, installed software releases, capabilities, configuration, connections, etc. (referred to herein as "inventory" or "operational characteristic" information) for the network element. In addition, the network element may be capable of reporting to the IPN the facilities and capabilities the network element has as well as its configuration and/or other information.
In some embodiments, the model or virtual representation of a network element inventoried by an IPN may be created in a generic fashion or consistent fashion across a domain or a signaling control plane. That is, all models or virtual representations created by IPNs of network elements will be created in the same way or using the same format or protocol. As a result, the models or virtual representations of the network elements will be consistent and provide a generic look to the signaling control plane that includes the IPNs, even if the network elements are different (e.g., from different vendors, have different capabilities, are different types of equipment). Thus, a signaling control plane that includes the IPNs can manage and manipulate the network elements independent of the differences between the network elements. In addition, an INC that receives information from an IPN regarding a network element can receive the information in a format or protocol that is independent of the vendor that supplied the network element, the type of network element, etc. The INC for a domain can then store and use topology and capability information for network elements that is also in a format or protocol that is independent of the vendor that supplied the network element, the type of network element, etc. A network element's associated IPN will covert any provisioning commands for the network element received by the IPN from the INC into commands suitable for the specific network element, thereby allowing a generic and consistent command set or format to be used for communications between the IPN and the INC.
From the information provided by one or more IPNs, an INC can see and determine the resources available for provisioning a circuit in a vendor neutral manner and let the IPN(s) handle conversion of commands from the INC into vendor specific formats.
According to methods of the present invention, a request for the provisioning of a circuit (also referred to herein as a communication path) is received by an IPN
from a customer (which can be a node). For example, the IPN 120 may receive a request to provision a communication path from the node 110, the customer 140, etc.

A request from the node 110 to the IPN 110 may request that a communication path be provisioned between the node 110 and the node 115. In some embodiments, an IPN may receive a request directly.from the customer, from a component in a management plane (e.g., an operator using a network management system application and requesting a circuit via a console), via a node, etc. Further, in some embodiments the customer may send the request directly to the INC 130. A request may include information relating to the circuit that is to be provisioned, e.g., addresses for the origination and destination of the circuit (e.g. IP addresses), the capacity required, the quality of service required, etc.
As will be discussed in more detail below, based in part on the request received by an IPN, the IPN, operating in conjunction with other IPNs and, in some embodiments, with the INC 130 and the database 131, may determines a path over which the requested circuit may be provisioned. The determined path may cut across multiple networks, or may be entirely within one network. For example, a first network may not have enough capacity to provision a requested circuit on its own, but may be able to provision dynamically a circuit across a peering partner's network to satisfy the request. After a path is determined, signaling commands may be sent from the IPNs to the appropriate network nodes, thereby configuring the nodes to set up a circuit along the determined path.
In some embodiments, after a circuit or other communication path is established, the circuit or other communication path may be tested and, based on successful testing, may be provisioned to the customer or other requester of the circuit. Upon completion of use, the circuit is torn down, and the network segments used to make up the circuit are returned to inventory for future provisioning.
Now referring to Figure 1 B, a communications system 160 may the include the computer, data or other communication networks or network domains101, 102 and 103, the nodes or network elements 110, 111, 112, 113, 114 and 115, the intelligent provisioning nodes (IPN) 120, 121, 122, 123, 124, 125 and 128, and the one or more customers 140 and 141 as previously discussed above. The network may have an associated INC 162 in connection with the IPNs 120 and 121.
Similarly, the network 102 may have an associated INC 164 in connection with the IPNs 122 and 123 and the network 103 may have an associated INC 166 in communication with the IPNs 124 and 125. The network 101 forms a domain for the INC 164, the network 1 O1 forms a domain for the network 102, and the network 103 forms a . ~ domain for the INC 168.
The INC 164 may be in communication with the INCs 162 and 166.
Segmenting the IPNs 120 and 121, 122 and 123, and 124 and 125 across the INCs 162, 164 and 166, respectively, may reduce the management and performance burdens for any one INC and may allow enhance security measures to be used in the system 160. For example, the INC 162 may not be allowed or able to send provisioning commands to the IPNs 122, 123, 124 or 125 and may not receive or have inventory information for the network elements 112, 113, 114 and 115. In addition, changes in the network 102 and 103 and/or the network elements 112, 113, 114 and 115 do not need to be communicated to the INC 162. Each of the INCs 162, 164, may maintain topology and capability information for each of the network elements in its respective domain (e.g., the network elements 110 and 111 for the INC 162) but may not store such information for network elements outside its domain.
Signaling between INCs may be used to broker provisioning transactions between network elements in different domains. The INCs may use security mechanisms to ensure that no proprietary information passes inappropriately to another network, as well as business/policy mechanisms to determine whether a request circuit should be provisioned.
The INC 162 may be in communication with, or include, a database or other resource 168. Similarly, the INC 164 may be in communication with, or include, a database or other resource 170 and the INC 166 may be in communication with, or include, a database or other resource 172.
The network elements 110 and 111 may form a data plane of components in or for the network 1 O1 while the IPNs 120, 121 and the INC 162 may form part of or be included in a signaling control plane for the network 101. Similarly, the network elements 112 and 113 may form a data plane of components in or for the network while the IPNs 122, 123 and the INC 164 may form part of or be included in a signaling control plane for the network 102. Likewise, the network elements 114 and 115 may form a data plane of components in or for the network 103 while the IPNs 124, 125 and the INC 166 may form part of or be included in a signaling control plane for the network 103. In some embodiments, the IPNs 120, 121, 122, 123, 124 and 125, along with the INCs 162, 164 and 166, may form a signaling control plane for the network elements 110, 111, 112, 113, 114 and 115. It also is possible to consider the INC 162 and the IPNs 120, 121 as forming a first signaling control plane, while the INC 164 and the IPNs 122, 123 form a second signaling control plane, and the INC 166 and the IPNs 124, 125 form a third signaling control plane, each of which foiins a portion of the larger signaling control plane 1.80.
As will be discussed in more detail below, in some embodiments an INC or a signaling control plane may be connected to or in communication with a management plane. The management plane may provide additional monitoring and management functionality for networks, network elements, etc. managed by the INC or the signaling control plane.
As previously mentioned above, in some embodiments an IPN may be configured with the necessary "personality package" or other software, data, etc.
specific to a network element under its control. The IPN may be able to discover and maintain topology, inventory, alarm and configuration information for the network element as well as to issue provisioning commands to the network element. The IPN
also may translate network element specific information into a protocol or other format useable or recognizable by an INC. For example, the IPN 120 may determine inventory information for the network element 110 and provide such inventory information to the INC 162. The INC 162 may store the inventory information in the database 168 or in a database internal to the INC 162. Similarly, the IPN 122 may inventory the network element 112 and provide the inventory information for the network element 112 to the INC 164. As a more specific example, the IPN 120 may perform a self discovery process according to or during which information related to the network element 110 and upload or send such information to the INC 168.
During the self discovery process, the IPN 120 may communicate directly with the network element 110 and construct a model or virtual representation of the network element 110. The model can provide a generic representation of the network element 110 that is independent of the vendor that created the network element. As a result, a signaling control plane that includes multiple IPNs may form models of the IPNs' associated network elements in a consistent format and manner, regardless of differences (e.g., different types of equipment, different vendors supplying the network elements, different configurations of the network elements) in the network elements.
Also as previously mentioned above, an IPN may receive provisioning commands from an INC, translate these commands into specific commands appropriate to its associated network element and execute the command on the network'element. Thus, the IPN provides network element or node control that allows provisioning of a circuit as instructed by the INC. In some embodiments, an IPN may be responsible for the management and control of its associated network element or some or all of the capabilities or features of the associated network element. For example, the IPN 120 may receive provisioning commands from the INC 162 for the network element 110 to configure circuits in or across the network 101. Similarly, the IPN 122 may receive provisioning commands from the INC 164 for the network element 112 as part of provisioning a circuit in or across the network 102.
As previously discussed above, one or more INCs may be used to provision a circuit between network elements in different networks. For example, the IPN

may receive a request for the provisioning of a circuit may between the network element 110 and the network element 113. The IPN 120 may forward the request to the INC 162. The INC 162 may determine that such a circuit will require a communication path through or via the network 102 and may forward the request to the INC 164. The INC 162 and the INC 164 acting and communicating together may determine that the requested circuit will require a communication path using the network elements 110, 111, 112 and 113 (e.g., network elements in addition to the network elements designated in the request). The INC 162 may send provisioning commands to the IPNs 120 and 121 relative to the network elements 110 and 111, respectively. In addition, the INC 164 may send provisioning commands to the IPNs 112 and 113 relative to the network elements 112 and 113 respectively. The IPNs 120, 121, 122, 123 may send the appropriate commands to the network elements 110, 111, 112, 113, respectively, to create the desired circuit. That is, the IPN
120 sends the appropriate command to the network element 110, the IPN 121 sends the appropriate command to the network element 11 l, etc. to establish the desired circuit between the network elements 110 and 111. As previously discussed above, provisioning commands sent by an INC to an IPN regarding a network element may use a format or protocol that is vendor independent. The IPN may then convert the command into a vendor specific command for the network element and send the converted command to the network element.
As another example, the IPN 120 may receive a request for the provisioning of a circuit may between the network element 110 and the network element 115. The IPN 120 may forward the request to the INC 162. The INC 162 may not know what network the network element 115 is in and, as a result, may forward the request to the INC 164. The INC 164 may be able to determine (e.g., by querying a database or network management system) that the network element 115 is in the network 103 and may forward the request to the INC 166. Communications between the INCs 162, 164 and 166 may commence to identify the needed network elements to create the desired circuit. Various levels of security might be used for the communications across different INC domains to piotect and secure proprietary network information.
In this example, the INC 164 acts as a sort of "master" INC for the INCs 162 and 166 and the network 102 acts as a sort of backbone network for circuits provisioned between a network element in the network 101 and a network element in the network 103.
In some embodiments, a master INC for a backbone network may be used to facilitate provisioning of circuit between network elements in different regional or localized networks. Each of the regional or localized networks may have its own INC
that is in communication with the master INC. In addition, each of the regional~or localized networks may be connected to the backbone network via one or more network elements (herein referred to as "bridge elements"). For example, the network element 111 and/or the network element 112 may operate as a bridge element to connect the networks 101 and 102. Similarly, the network element 113 and/or the network element 114 may operate as a bridge element to connect the networks and 115.
When an INC receives a provisioning request for a circuit whose end point is outside the INC's domain, INC to INC signaling may be used to determine if the requested circuit can be established and/or the location of the end point (e.g., the domain containing the network element forming the end point). For example, an INC
receiving a request to provision a circuit between a first network element in the INC's domain and a second network element outside of it's domain may send all or part of a request to the master INC. The master INC may identify a network or domain that includes the second network element and provide some or all of the request to an INC
or signaling control plane associated with such network. The master INC than may coordinate provisioning of the circuit between the two network elements, which may include provisioning a portion of the circuit across a backbone network managed by the master INC. In this example, the master INC may query other INCs to determine which domain the second network element is in. Alternatively, the master INC
may keep or have access to a database or other resource that maintains information regarding the location or network affiliation of network elements or that can determine such information on behalf of the master INC.
Now referring to Figure 2, an illustration of a representative system 200 that includes a management plane, signaling plane, and data plane is provided. The system includes a data plane 202, a signaling control plane 204 and a management plane 206. In some embodiments, a data plane, signaling control plane and a management plane may form part of a network. For example, the data plane 202, a signaling control plane 204 and the management plane 206 may form part of or be included in the network 102.
As previously discussed above, a set of network elements and links between the network elements may form a data plane. For example, the nodes or network elements 112 and 113 and the link between them may form a data plane 202.
Network elements in the data plane may communicate with or be connected to other network elements (e.g., the node 112 is in communication with the node 111 ).
The IPNs 122, 123 associated with the nodes 112, 113 in the data plane 202, along with the INC 164, may comprise some or all of a signaling control plane for the data plane 202. The signaling control plane 204 (particularly the INC
164) may communicate with the management plane 206. The signaling control plane 204 may.mediate communication between the management plane 206 and the data plane 202 (e.g., between the management plane 206 and network elements in the data plane 202).

As previously discussed above, in some embodiments, the signaling control plane 204 may determine routes for services through a managed domain of network elements. The signaling control plane 204 can provide the signaling mechanisms to the network elements in the domain. The signaling mechanisms may affect configuration changes in the managed network elements to provide services. In addition, a signaling control plane may provide a unified interface from the managed network elements to various applications in the management plane 206.
The management plane 206 may include or comprise various applications and software processes and may be used to facilitate maintenance, control and operation of a network (which may include one or more domains monitored by an INC or a signaling control plane). In some embodiments, the management plane 206 may be implemented by one or more high performance servers or computer systems that may interface with other systems to provide overall network management.
The management plane 206 may comprise or include various applications and software processes to conduct, for example billing, troubleshooting, order management, etc. For example, an element management system (EMS) module 132 may be included in the management plane. The EMS module 132 may communicate with network elements directly or via the signaling control plane 204. Often, a network element in a network is packaged with an EMS module which allows or provides control of the basic operation of the network element (e.g., setting up cross connects, receiving alarms). An EMS module in a management plane may control, or facilitate control of, multiple network elements of the same type (e.g., the same type of muter). The EMS module 132 in the management plane 206 may allow or provide for the complete control of one or more network elements. The control of a network element provided by the EMS module 132 may be more complete than the control of the network element provided by an IPN or a signaling control plane. That is, there may be functions of capabilities or aspects of a network element that cannot be, or are not allowed to be, controlled by an IPN associated with the network element.
The EMS module 132 may be used to provide and manage control of such capabilities or aspects of the network element.
As another example, the management plane 206 may include a network management system (NMS) module 134 and/or an operations support systems (OSS) module 136. The NMS module 134 may be connected to or in communication with the EMS module 132 and/or the OSS module 136. Each of the modules 132, 134, and/or 136 may include or represent multiple applications, each of which handles different needs of the management plane 206 or provides different functionality to the management plane 206.
In some embodiments, the OSS module 136 may facilitate or allow communication with a network element and may communicate with the NMS module 134 and/or the EMS module 132. The OSS module 136 may include one or more applications that handle network planning/design, order management, service level agreement management, billing, etc.
The NMS module 134 may operate "above" the EMS module 132 and provide control over a network (as opposed to just the equipment of a single vendor).
For example, once a service for a network is designed, the service may be passed to an NMS module, which may then determine how to implement the service in the network and pass appropriate commands to the appropriate EMS modules to control or configure different portions of the network.
Now referring to Figure 3, a representation of a typical IPN 300 (e.g., the IPN
120) is illustrated. As previously discussed above, the IPN 300 is associated with a network element (e.g., the node 110). In some embodiments, an IPN may be implemented by or on a computer, server, computer system, etc. Such a computer, server, computer system, etc. may include a processor, one or more communication ports or hardware interfaces, one or more input devices (e.g., mouse, keyboard), one or more output devices (e.g., display, printer), and/or a data storage device or memory (e.g., hard drive, RAM, ROM). Software (e.g., IPN software processes and elements) may be resident and operating or operational on the computer, server, computer system, etc., as described in more detail below. The software may be stored on the data storage device and may include a control program, software modules, interfaces, etc. for operating the device, performing IPN functions, allowing the device to operate an IPN application, etc. The control program may control the processor and the processor preferably performs instructions of the control program. The control program may include program elements that may be necessary, such as an operating system, a database management system, and device drivers.

The IPN 300 can be implemented as software or a collection of software processes or elements. Thus, an IPN (e.g., the IPN 120) may comprise an IPN
application operating on a device. In some embodiments an IPN application may reside and operate on a network element associated with the network element.
In some embodiments, multiple IPN applications representing or implementing multiple IPNs (e.g., the IPNS 122 and 123) may operate on a single device.
The IPN 300 may include a network element driver 302 that allows the IPN to communicate or interface with a network element (e.g., the node 110) or an interface operating on the network element. Typically, the network element driver 302 may be vendor specific or include a vendor specific component to allow the IPN 300 to communicate with a specific vendor's hardware. Different IPNs communicating with network elements from different vendors may have different network element drivers.
The IPN 300 also may include one or more software interfaces 304 that allow the IPN to communicate using standard interfaces (e.g., Tl-1, SNMP, XML, CORBA). The network element driver 302 will facilitate communications between the software interfaces) 304 for the IPN 300 and the network element.
The IPN 300 also may include a virtual interface adapter 306 that may translate or parse information or communications exchanged between the network element and the IPN 300 from vendor specific formats or protocols used by the network element and the formats, protocols or information model used by the IPN
300, and vice versa. For example, the network element may send an alarm to the IPN
300 indicated the occurrence of a fiber pull on the network element. The virtual interface adapter 306 will translate the alarm information received from the network element into the information model used by the IPN 300.
The IPN 300 also may include a virtual network element model or inventory 308 that may represent an abstracted model of the network element. The model or inventory 308 may be built, maintained, and updated by the IPN 300 to reflect the current state of the network element (e.g., inventory, resources, services associated with the network element) and may be formatted in a vendor abstracted model for use by a signaling control plane. As a result, different network elements from different vendors may be controlled and used by the signaling control plane.

The IPN 300 may include one or more functional modules 310 that represent one or more functions that a signaling control plane that includes the IPN 300 may support or implement, including, but not limited to, inventory discovery and management of network elements. A signaling connectivity module may allow the IPN 300 to transmit and receive messages across a signaling control plane (e.g., to other IPNs, to an INC). A fault management module may provide functionality to monitor and manage faults generated within a data plane (e.g., by a network element) such as alarm correlation, alarm filtering, root cause analysis, etc. A
performance monitoring module may handle receiving and correlating of information relating to performance monitoring (e.g., SONET PM) from a data plane regarding provisioned services. For example, a provisioned circuit may span three "hops". The performance monitoring module may monitor all performance management data generated from the data plane regarding the circuit and correlate it as necessary. A
test and maintenance module may allow the IPN 300 or a signaling control plane to interact with test equipment that may exist in a network to facilitate testing of components or circuits. For example, a signaling control plane may configure specific network elements to establish a circuit and direct test equipment to perform tests on the circuit. Providing such functions in a modular architecture facilitates the addition or subtraction of a given function based on the needs of the network and the signaling control plane. In addition, in some embodiments, modules may allow a signaling control plane to assume some of the roles and functions traditionally handling by a management plane. For example, the signaling control plane may take over some or all of the functions of an EMS module operating in the management plane.
The IPN 300 may include a security engine 312 that represents one or more security mechanisms utilized by the IPN 300 to protect proprietary network information. The security mechanisms may include encryption, policy directives indicating which entities are capable of receiving, or permitted to receive, certain information regarding certain actions, etc.
The IPN 300 may include an interface 314 to an INC (e.g., the INC 130) that allows or facilitates communication between the IPN 300 and the INC. The IPN

may provide information regarding the network element to the INC via the interface 300. In addition, the IPN 300 may receive commands from the INC via the interface 300.
The IPN 300 may include a user network interface (UNI) 316 that provides a UNI standard compliant interface through which customer devices that support the UNI standard may communicate with or signal directly to the IPN 300.
Now referring to Figure 4, a representation of a typical INC 350 (e.g., the INC
130 or the INC 164) is illustrated. In some embodiments, an INC may be implemented by or on a computer, server, computer system, etc. Such a computer, server, computer system, etc. may include a processor, one or more communication ports or hardware interfaces, one or more input devices (e.g., mouse, keyboard), one or more output devices (e.g., display, printer), and/or a data storage device or memory (e.g., hard drive, RAM, ROM). So$ware (e.g., INC software processes and elements) may be resident and operating or operational on the computer, server, computer system, etc., as described in more detail below. The software may be stored on the data storage device and may include a control program, software modules, interfaces, etc. for operating the device, performing INC functions, allowing the device to operate an INC application, etc. The control program may control the processor and the processor preferably performs instructions of the control program. The control program may include program elements that may be necessary, such as an operating system, a database management system, and device drivers.
In some embodiments the INC 350 can be implemented as software or a collection of software processes or elements. Thus, an INC may comprise an INC
application operating on a device.
The INC 350 may include one or more IPN interfaces 352 that allow the INC
350 to communicate with IPNs. For example, the interface 352 for the INC 350 may communicate with the interface 314 on the IPN 300.
The INC 350 may include a database 354 for storing information. For example, the database may include information regarding the topology and capabilities of one or more network elements provided by IPNs. If the INC 350 is representative of the INC 130 of Figure l, the database 354 may be or include the database 131 illustrated in Figure 1.

The INC 350 may include one or more functional modules 356 that represent one or more functions that a signaling control plane that includes the INC 350 may support or implement, including, but not limited to, inventory discovery and management of network elements. Providing such functions in a modular architecture facilitates the addition or subtraction of a given function based on the needs of the network and the signaling control plane. A signaling connectivity module may allow the INC 350 to transmit and receive messages across a signaling control plane (e.g., to other IPNs and/or INCs). A fault management module may provide functionality to monitor and manage faults generated within a data plane (e.g., by a network element) such as alarm correlation, alarm filtering, root cause analysis, etc. A
performance monitoring module may handle receiving and correlating of information relating to performance monitoring (e.g., SONET PM) from a data plane regarding provisioned services. For example, a provisioned circuit may span three "hops" The performance monitoring module may monitor all performance management data generated from the data plane regarding the circuit and correlate it as necessary. A
test and maintenance module may allow the INC 350 or a signaling control plane to interact with test equipment that may exist in a network to facilitate testing of components or circuits. For example, a signaling control plane may configure specific network elements to establish a circuit and direct test equipment to perform tests on the circuit. In some embodiments, functional modules included in an INC or used by an INC may allow a signaling control plane to assume some of the roles and functions traditionally handling by a management plane. For example, the signaling control plane may take over some or all of the functions of an EMS module operating in the management plane.
The INC 350 may include a security engine 358 that represents one or more security mechanisms utilized by the INC 358 to protect proprietary network information. The security mechanisms may include encryption, policy directives indicating which entities are capable of receiving, or permitted to receive, certain information regarding certain actions, etc. The security engine 358 also may be responsible for providing security for the transfer of information across INC
domains or between INCs. This may include determining which INCs are considered "trusted", how much and which information can be provided, etc.

The INC 350 may include an interface 360 that may allow the INC 350 to communicate with other INCs. For example, the INC 350 may communicate requests for circuits to other INCs or other INC domains using the interface 360.
The INC 350 may include a command interface 362 that allows the INC 350 to translate information or communications sent to, and received from, management systems or other components (e.g., the EMS 132 of Figure 2) in a management control plane (e.g., the management control plane 204 of Figure 2). The information or communications may be sent or received by the INC via one or more interfaces 364. For example, the INC 350 may use a LJNI, NNI or GMPLS interface to communicate with a device, application, or other INC so that the device, application or other INC can sent a request to the INC 350 to provision a circuit. The INC

may use a TL-1 or CORBA interface to communicate with a management plane. The interface 362 may be used to translate communications sent by the INC 350 for one of the interfaces 364 and/or communications received via one of the interfaces 364.
Now referring to Figure 5, an overview of a typical DWDM optical switching environment is disclosed. As illustrated in the system 300 illustrated in Figure 3, a typical DWDM system may include an optical switch 400 having twelve transmission and reception ports 401, 402, 403, 404, 405, 406, 406, 408,409, 410, 411 and 412.
Similarly, the DWDM system 300 may include an optical switch 500 having twelve transmission and reception ports 501, 502, 503, 504, 505, 506, 506, 508, 509, 510, 511 and 512. Since optical connections are typically unidirectional, it is necessary to utilize two links for bi-directional communication. For this reason, each port is typically comprised of a transmitter (T) and a receiver (R). The optical switches 400 and 500 may be components connected to and inventoried by an IPN. In some embodiments, provisioning of a circuit may include inventorying and/or managing individual wavelengths used by optical switches such as the optical switches 400, 500.
The ports 401, 402, 403, 404, 405, 406, 406, 408,409, 410, 411 and 412 on the optical switch 400 are connected to DWDM equipment or devices 520, 521, 522.
Similarly, the ports 501,.502, 503, 504, 505, 506, 506, 508, 509, 510, Sl 1, 512 on the optical switch S00 are connected to DWDM equipment or devices 530, 531, 532.
The DWDM equipment or devices 520, 521, 522, 530, 531, 532 may function to "multiplex" many optical signals together for transmission on a single fiber optical link 551, 552, 553, 554, 555, 556, 557 or 558 and "de-multiplex" these signals for individual switching. Groups of the single fiber optical links 551, 552, 553, 554, 555, 556, 557, 558 are then variously bundled together in link bundles by the devices 572, 573. It is common for multiple optical links to be so bundled when they are planted in order to save on infrastructure costs. However, this bundling can be problematic when, for example, a link bundle is damaged (e.g. accidentally severed by a backhoe), thereby rendering all the optical links in the bundle inoperable. For this reason, it may be necessary to track and record which optical links are bundled together, so that bundled optical links are not included in predetermined restoration paths (Shared Risk Link Group - SRLG).
Now refernng to Figure 6, a tabular representation records that may be used by a resource associated with an INC (e.g., the resource 131, the resource 170, the database 354) is depicted. In some embodiments, the resource may contain or include current inventory and connection information related to the links between network nodes available to the system, and is typically maintained by an INC. The resource may be implemented using in-memory and/or real time database technology, similar to that offered by companies such as TimesTen, Inc. Such database technology offers the high levels of performance necessary to accomplish the provisioning of circuits in very short time periods (e.g., 50 mls). In some embodiments the resource, or portions of it, may be replicated across multiple locations within one or multiple networks for easier access. In some embodiments, a resource may form part of or be included in an IPN (e.g., as the database 354 in the INC 350).
In some embodiments, a resource may be populated using a variety of techniques, including self discovery, multicast requests, etc. According to some embodiments, each IPN performs a self discovery process in order to collect network topology and capability information related to the links and network elements that are local to the IPN, including links between networks, nodes, etc. Further, changes in network topology, such as the addition or deletion of network nodes or modification of a node (e.g., changing or addition of a new port card in a switch), may trigger a self discovery process. Each IPN forwards this information to one or more INCs (e.g., the INC 130) for storage in the resource (e.g., the resource 131).
Information stored in the resource may be analyzed in order to create a centralized map of the entire network available to the system. As another example, an IPN may be able to detect, receive or capture a report database change signal generated by a network element. The IPN may then update its database or model of the network element and forward information to an INC regarding the change to the network element.
Figure 6 illustrates examples of two records 620 and 622 that may be stored in the resource 131. Included in each record is a node ID 624 that identifies a network node. For example, the node ID may be an IP address. Associated with each node ID
is a number of port IDs 626 that identify individual ports on the node. The protocol field 628 identifies the data protocol (encapsulation method) run over the port, for example SONET, Gigabit Ethernet, etc. The bandwidth field 630 identifies the bandwidth of the connection associated with the port, for example T-3, OC-12, etc.
In some embodiments, it may be necessary to track the link bundle in which a given link between nodes resides. For example, in embodiments where the system determines two paths for a given request (one for use, one for back-up), it may be advantageous if the two paths do not include links in the same link bundles.
In this manner, should an entire link bundle become damaged (e.g. cut by a back-hoe), the second determined path should not be affected. The Shared Risk Link Group (SRLG) addressing scheme may be implemented to accomplish this. In some embodiments, it may further be necessary to develop an addressing scheme for individual wavelengths that are available over a given optical connection.
In some embodiments, the resource 131 may track a number of other parameters related to connections within and across the networks involved.
These parameters may reflect both technical concerns as well as business concerns.
Some of the parameters may be automatically discovered via a self discovery process, while others may be manually entered and associated with a given connection. Such parameters may include those a customer may request or included in a request to provision a circuit or connection, such as, for example, endpoints for the connection (e.g., IP addresses), bandwidth for the connection (e.g., OC-12), security for the connection, protocol for the connection (e.g., IP), and/or a term of the connection (e.g., one hour). In some embodiments, the customer or other requester also may designate or specify some performance related constraints. Alternatively, or in addition, there may be a performance "class-of service" (CoS) specified that encompasses one or more performance metrics. For example, performance metrics/constraints may include latency, failover flexibility, packet loss, bit error rate, number of hops for a connection, etc.
In some embodiments, additional~constraints may include constraints on resources or capabilities that may be required for a connection to be established.
These constraints may be invisible to the customer, and may include, for example, available wavelength(s), wavelength conversion, device compatibility (at client, metro, core, etc.), mux/demux speed and availability, MPLS capabilities, OSS
links, alarms, scheduling, network percentage fulfillment, etc.
In some embodiments, parameters associated with technical constraints that may be related to a connection may be stored in the resource 131 as well, such as, for example: power, polarization, type of fiber, amplification mechanism, etc.
In order to keep the information stored in the resource 131 as up-to-date as possible, periodic self discovery updates may be performed by IPNs. Updates may be received from the IPNs, through other applications, by network operators, etc.
Further, changes in a network, such as added or subtracted nodes, ports, etc.
may trigger a self discovery process by one or more IPNs so that changes in network topology may be reflected in the resource 131 in a timely manner. In some embodiments, data stored by one or more INCs, IPNs, network elements, etc. may be compared for accuracy and consistency as part of an audit process In some embodiments, the information stored in the resource 131 may be kept confidential so that, for example, competing network service providers are not given visibility into each other's networks. In such an embodiment, an entity that operates and maintains the network of IPNs and INC(s) may be a "carrier-neutral"
entity, providing the service to any/all qualifying network service providers. In other embodiments, each network may maintain a separate INC containing information related only to the given network.
In some embodiments, some or all of the information stored in or by the resource 131 may also, or instead, be stored at the IPN level. In general, various levels of hierarchical layers with regard to the nodes, IPNs and INCs may be implemented. For example, the present invention may function with no centralized control (e.g., no INC 130 or resource 131), with all decision-making happening in a distributed manner by IPNs. In other embodiments, there may be one INC
controlling all the IPNs in all the interconnected networks, or there may be a network of INCs, each controlling some portion of the interconnected networks. In embodiments with multiple INCs, the present invention may employ a single master INC to control the multiple INCs, thereby creating three levels of signaling control: (1) IPNs, (2) INCs, and (3) master INC. It should be noted that additional layers of signaling control may be implemented.
Reference is now made to Figures 7A, 7B and 7C, where a flow chart 700 is shown which represents the operation of a first embodiment of the present invention.
The particular arrangement of elements in the flow chart 700 is not meant to imply a fixed order to the steps; embodiments of the present invention can be practiced in any order that is practicable. The method 700 is particularly well suited for determining and provisioning a circuited in a centralized manner.
Processing for the method 700 begins at a step 701. During a step 702, an application, router, desktop, node, customer, network element, etc. sends a request to a local IPN requesting provisioning of a circuit or other communication path.
According to some embodiments of the present invention, a request for a circuit is sent by a customer or other device or application and received by an IPN. For example, referring to Figure 1A, the customer 140 may transmit a request for a circuit to the IPN 120. The customer 140 may represent, for example, a network of IP
routers, where the request is sent from an IP router that acts as a border for the network of routers. In other embodiments, the customer 140 may represent a desktop computer. from with a request may be initiated e.g. by launching a desktop application. For example, the request may be a web-based request, where a customer logs onto a Web page and enters a request via the Web page. As another example, the network administrator of a data center may log onto a Web page and request an additional optical circuit be provisioned between the data center and, for example, a second data center.
Typically a request may include information related to the circuit that is to be provisioned. This information may include addresses (e.g., IP addresses, BGP
addressees) of the endpoints (e.g., ingress node, egress node) of the circuit . The request may include varying levels of specificity regarding the endpoints of the requested circuit. For example, the request may specify which ports on the switches should be used, what wavelength should be used, etc. Further, a request may include the bandwidth required (e.g. 0C-12, OC-3), the term desired (e.g. thirty minutes, three months), the latency requested (e.g., 400 ms), the number of hops allowed or required for the provisioned circuit, the desired or maximum bit error rate (e.g., 10~5), the security level that should be used for the circuit, failover requirements for the circuit, directionality requirements for the circuit, an identifier associated with the request, an identifier of an IPN or other device providing the request, the date/time of the request, an identifier of a customer that made the request, etc. A request will typically also include other parameters with which the provisioned circuit must comply. These parameters may comprise technical requirements, security requirements, business requirements, etc. For example, the request may specify a required quality of service (QoS), acceptable costs, redundancy requirements, etc. An example request is illustrated in record 703 of Figure 7A for a circuit to be provisioned between an ingress node identified as "132.53.521.3" and an egress node identified as "145.234.5.3".
In some embodiments, a customer may specify acceptable parameters in the request, or the customer may have defined default parameters. For example, the customer may register a profile where the profile defines a number of default values for the various parameters. The customer's profile may be accessed when a request is received from the customer. In this way, the customer is not required to indicate all the parameters desired in every request.
According to some embodiments where requests are received from specific applications, sets of default parameters associated with each application may be maintained by an IPN, an INC, the resource 131 or other entity or device. For example, if a request is received for a circuit that is to be used for video conferencing, the INC 130 may access a set of default parameters that must typically be met for a video conferencing application to function properly.
In some embodiments, the sending of the request may be triggered in a number of ways. For example, a request may be triggered manually, for example, by a network administrator operating the border IP muter. In some embodiments an application running on or in conjunction with the border IP router may trigger a request. For example, heavy traffic levels may cause congestion along certain routes associated with the network of IP routers (and networks to which it is connected). An application may be developed that automatically launches a request for additional bandwidth along a given route when e.g. traffic or congestion levels exceed certain threshold parameters. In this manner, layer three-type activity (e.g. at the IP layer) may trigger changes in the layer one make-up of the network.
During a step 704, a local IPN receives the request sent during the step 701.
For example, the customer 140 may have a direct connection (e.g., an IP
connection) to the IPN 120 over which a request from the customer 140 can be sent to the IPN
120. In other embodiments, a customer may send the request to a network node, for example, the node 110. The request would then be forwarded from the node 110 to the IPN 120 for processing. In other embodiments, the request may instead be sent by the customer 140 directly to the INC 130, bypassing the IPN 120 all together.
During a step 706, a determination is made regarding whether a direct connection is available between the ingress node and the egress node. For example, having received the request during the step 704, the IPN 120 may first determine whether a direct connection exists between the egress node specified in the request and the ingress node to which the IPN 120 is in communication. Information related to local node reachability and capability is typically available to the IPN
120. For example, a link state database may be maintained at the node 110 (or at the IPN 120).
Such a database would contain information related to all the connections from the node 110. The IPN 120 compares the parameters specified in the request to the parameters associated with the connections currently available in order to determine whether any available connections satisfy the parameters of the request.
If the result of the step 706 is that a direct connection exists between the ingress node and the egress node, the method 700 proceeds to a step 708. If no such connection exists, the method 700 proceeds to a step 710.
During the step 708, if a direct connection is available between the egress node and the ingress node, the local IPN immediately reserves the direct connection and provisions it for the customer.

During the step 710, without a direct connection to the egress node specified in the request, the local IPN sends the request to an INC (e.g., the INC 130) for additional processing.
During a step 712, the INC that receives the request from the local IPN
queries the resource 131 (which may be internal to or part of the INC) for at least one path from the ingress to the egress node specified in the request. As previously discussed above, the resource 131 maintains information relating to the topology and capability of the networks and nodes involved in the inventive system. Based on this information, the INC is able to compute a path through at least one of the involved networks to connect the ingress and egress nodes specified in the request.
Path calculation may be based on such protocols as Open Shortest Path First (OSPF), Intermediate System to Intermediate System (ISIS), etc.
According to different embodiments of the invention, the resource 131 may contain varying amounts of additional information regarding the connections available to the system. For example, in some embodiments, the resource 131 may contain enough current information for the INC to determine a path form the ingress to the egress node that meets all of the requirements set forth in the request. In other embodiments, the resource 131 only may contain enough information for the INC
to determine at least one path between the egress and ingress node without being able to verify that the paths) fulfill the other parameters specified in the request.
In such cases, it is necessary for the INC to signal each of the nodes in the path in order to determine whether the path meets the parameters of the request.
During a step 714, if the INC determines that no paths exist between the ingress and egress nodes designated in the request, then the method 700 proceeds to a step 716. If the INC determines that at least one path does exist between the ingress and egress nodes designated in the request, then the method continues to a step 718.
During the step 716, if the INC is unable to determine a path between the ingress and egress nodes specified in the request, a failure message may be sent to the customer or other requester, indicating that the system currently cannot fulfill the 3 0 request.
During the step 718, if the INC is able to determine a path between the ingress and egress nodes specified in the request, the INC retrieves information relating to the determined paths) from the resource 131. The retrieved information may include node addresses, port IDs, wavelengths, etc. In some embodiments, the INC may retrieve at least two paths for a request. For example, the request may specify that at least two paths must be determined for protection purposes. An example response to the request is illustrated in record 719 of Figure 7A, which lists two paths for a circuit to be provisioned between an ingress node identified as "132.53.521.3" and an egress node identified as "145.234.5.3". One of the paths includes a node identified as "124.43.3.6" while the second path includes a node identified as "273.87.23.4".
During a step 720, the INC transmits resource reservation command to each node (IPN) in communication path(s)determined between the ingress and egress nodes specified in the request. A resource reservation command instructs the receiving node to reserve requested resources for the customers use. For example, the command may include the port ID, wavelength, bandwidth, etc. that must be reserved for use in the determined path. In some embodiments, reserve resource commands also may be sent to nodes that make up protection paths as well.
During steps 722A, 722B, 722C,...722N, IPNs #1, #2,...N receive resource reservation command and queries link state database for resource availability.
Each of the IPNs that make up the determined communication paths) receives the resource reservation command from the INC and determines whether the requested resources are available by querying a link state database. In some embodiments, the steps 722A, 722B; 722C, etc. may not be necessary if, for example, the resource 131 contains enough current information about the link states in each node.
During steps 724A, 724B, 724C,...724N, if the resources associated with an IPN specified in the resource reservation command are not available, then the method proceeds to a step 726. If, on the other hand, the resources specified in the resource reservation command are available, then the method 700 proceeds to steps 728A, 728b and 728C.
During the step 126, if the required resources are not available at an IPN, the IPN may signal the INC to select an alternate path for that network segment, or for the entire path. If such an alternate path exists, then processing proceeds as before. If no such path exists, than the INC signals the customer that a path cannot be provisioned at that time, and processing ends.

During the steps 728A, 728B, 728C,...728N, if the resources requested in the resource reservation command are available, the IPN signals a successful resource reservation to the INC, indicating, for example, that the resources are available, and any additional information regarding the resources necessary, such as port ID, wavelength, etc. Example resource reservations that may be sent to the INC are illustrated in records 729A, 729B and 729C of Figure 7B, each of which identifies a specific switch and a port of the switch that has been reserved.
During a step 730, having received successful resource reservation signals from each of the IPNs in the determined path, the INC issues a provisioning command, instructing the IPNs to provision the determined communication path.
During steps 732A, 732B, 732C,...732N, IPNs #1, #2,...N execute provisioning commands received from the INC and signal completion to the INC.
During a step 734, the INC receives provision completion signals from the appropriate IPNs and signals the IPNs to test the communication path. The step 1 S may be optional in some embodiments of the method 700.
During a step 736, the appropriate IPNs test the provisioned communication path. Testing of a provisioned communication path is described in further detail below. The step 736 may be optional in some embodiments of the method 700.
A determination is made during a step 738 as to whether the test of the provisioned communication path is successful. If the test is not successful, then the method 700 proceeds to a step 740. If the test is successful, then the method proceeds to a step 742.
During the step 740, the INC transmits a provisioning command to a protection circuit. That is, in some embodiments the system may determine more than one path for the entire initial path or portions of it depending on the level of protection specified in the request. In those cases where the initial path that is provisioned fails testing, the protection path may be provisioned instead. In some embodiments, additional error detection steps may be taken to determine, for example, which links in the connection are responsible for the failing, which links may be provisioned to route around the failure etc. The step 740 may be optional in some embodiments of the method 700.

During a step 742, the provisioned circuit or communication path may be handed off to the customer or requestor or is otherwise provided for use.
During a step 744, the provisioned circuit or communication path may be monitored for failure or quality of service degradation. The step 744 may be optional in some embodiments of the method 700.
During a step 746, a determination may be made as to whether a failure or quality of service degradation exists for the provisioned circuit or communication path. If there is a failure or quality of service degradation, the method 700 proceeds to the step 740 previously discussed above. If there is not a failure or quality of service degradation, the method 700 proceeds to a step 748.
During the step 748, after the term of the circuit has expired, the INC
signals each IPN to release the resources that make up each segment of the circuit. In some embodiments, the individual IPNs may track circuit terms and release resources automatically.
During a step 750, the method 700 ends.
Reference is now made to Figures 8A, 8B and 8C, where a flow chart 800 is shown which represents the operation of a second embodiment of the present invention. The particular arrangement of elements in the flow chart 800 is not meant to imply a fixed order to the steps; embodiments of the present invention can be practiced in any order that is practicable. The method 800 is particularly well suited for determining and provisioning a circuit in a distributed manner.
Processing for the method 800 begins at a step 801. During a step 802, an application, router, desktop, node, customer, network element, etc. sends a request to a local IPN requesting provisioning of a circuit or other communication path.
The step 802 is similar to the step 702 previously discussed above.
During a step 804, a local IPN receives the request. The step 804 is similar to the step 704 previously discussed above.
During a step 806, a determination is made regarding whether a direct connection is available between the ingress node and the egress node. The step is similar to the step 706 previously discussed above.

If the result of the step 806 is that a direct connection exists between the ingress node and the egress node, the method 800 proceeds to a step 808. If no such connection exists, the method 800 proceeds to a step 810.
During the step 808, if a direct connection is available between the egress node and the ingress node, the local IPN immediately reserves the direct connection and provisions it for the customer. The step 808 is similar to the step 708 previously discussed above.
During the step 810, without a direct connection to the egress node specified in the request, the local IPN sends the request to an IPN associated with the egress node.
During steps 812A and 812B, link state databases are queried for connections that fulfill the parameters specified in the request. For purposes of the method 800, the letters "A" and "B" next to step numbers refers to the same step happening in different nodes, IPNs, etc. For example, in this case "A" refers to the ingress IPN
associated with the ingress node and "B" refers to the egress IPN associated with the egress node. Each IPN queries its associated link state database to determine whether any direct connections are available that meet the parameters set forth in the request.
For example, a request may specify an amount of bandwidth, term, security, etc. The query is performed to determine whether any connections currently available fulfill those requirements.
During steps 814A and 814B, if there is not at least one such connection that fulfills the parameters specified in the request, then the method 800 proceeds to a step 816. If there is at least one such connection, then the method 800 proceeds to a step 818.
During the step 816, if no direct connection to either the ingress or egress node exists that fulfills the requirements set forth in the request, then the customer or other requester is sent a failure notice indicating that their request cannot be met. In some embodiments, additional programming may determine whether a connection that is currently set up may be torn down in favor of the current request.
During steps 818A and 818B, connection identifiers) associated with connections that fulfill request parameters are retrieved by the ingress and egress IPNs, respectively.

During steps 820A and 820B, each IPN transmits a reserve resource command to its associated node in order to reserve the connections associated with the retrieved connection identifiers.
During steps 822A and 822B, having identified the connections that fulfill the requirements set forth in the request, IDs associated with the IPNs that form the opposite ends of those connections are determined. This information may be retrieved from the link state database.
During steps 824A and 824B, the request is then sent only to those IPNs that reside on the opposite side of the connections that fulfill the requirements set forth in the request. Such IPNs receive the request during steps 826A and 826B.
During steps 828A and 828B, each IPN that receives the request first determines whether the same request was already received from an IPN in the egress/ingress request tree. If it was, that means that the IPN that received both now has a complete path from ingress to egress node that fulfills the requirements set forth in the request. If the request was not also received from the egress/ingress request tree, then the method 800 continues to steps 830A, 830B. If the request was also received from the egress/ingress request tree, then the method 800 proceeds to a step 834.
During the steps 830A and 830B, a determination is made to see if the request sent during the step 802 has timed out. Each request includes a "time to live"
value that is set when the request is created. For example, the time-to-live value may represent a time after which the request is no longer valid. In this manner, the inventive system can control the spread of requests through the interconnected networks. If the request has timed out, then the method 800 proceeds to step where the method 800 ends. If the request has not timed out, then the method proceeds to steps 812A, 812B.
During the step 834, information related to the complete path is next sent to the customer for approval. If the customer does not approve of the path, then the method 800 proceeds to a step 838. If the customer does approve of the path, then the method 800 proceeds to a step 840.

During the step 838, if the customer does not approve the determined connection or path, then the method 838 may attempt to determine another connection that the customer may approve as described above.
During the step 840, if a customer approval is received for the path, then a provision command is sent to every IPN in the path, instructing the IPNs to provision the circuit specified in the path. The step 840 is similar to the step 740 previously discussed above.
During a step 842, the provisioned communication path is tested. The step 842 is similar to the step 736 previously described above.
During a step 844, a determination is made regarding whether or not the test conducted during the step 842 is successful. The step 844 is similar to the step 738 previously discussed above. If the test is not successful, then the method 800 proceeds to a step 846. If the test is successful, then the method 800 proceeds to step a 848.
The method 800 may determine more than one path for the entire initial path or portions of it depending on the level of protection specified in the request. In those cases where the initial path that is provisioned fails testing, the protection path may be provisioned instead during a step 846. In some embodiments, additional error detection steps may be taken to determine e.g. which links in the connection are responsible for the failing, which links may be provisioned to route around the failure etc.
During a step 848, the provisioned circuit or communication path may be handed off to the customer or requestor or is otherwise provided for use. The step 848 is similar to the step 742 previously discussed above.
During a step 850, the provisioned circuit or communication path may be monitored for failure or quality of service degradation. The step 850 may be optional in some embodiments of the method 800. The step 850 is similar to the step 744 previously discussed above.
During a step 852, a determination may be made as to whether a failure or quality of service degradation exists for the provisioned circuit or communication path. If there is a failure or quality of service degradation, the method 800 proceeds to the step 846 previously discussed above. If there is not a failure or quality of service degradation, the method 800 proceeds to a step 854. The step 852 is similar to the step 746 previously discussed above.
During the step 854, a$er the term of the circuit has expired, each IPN
releases the resources that make up each segment of the circuit. In some embodiments, the S individual IPNs may.track circuit terms and release resources automatically.
During a step 856, the method 800 ends.
Now refernng to Figure 9, a method for testing a provisioned circuit is illustrated. According to some embodiments of the invention, in addition to a signaling connection, an IPN may have a data connection to its corresponding node.
In such embodiments, IPNs may test circuits set up or provisioned by the methods and apparatus of the inventive system.
An IPN 902 is connected to a node 904 via signaling a connection 910 and a data connection 912. Similarly, an IPN 906 is connected to a node 908 via a signaling connection 914 and a data connection 916. The nodes 904 and 908 are connected to each other via a data connection 918. The IPNs 902 and 906 are connected to each other via signaling a connection 920. It should be noted that data connection 918 may not be a direct connection, but may include a number of intermediate segments.
The IPN 902 may configure the node 904 to transmit data to the node 908 over the data connection 918. This configuration may take place over the signaling connection 910. To test the connection between the nodes 904 and 908, the IPN

may signal the IPN 906 over the signaling connection 920, informing the node that the node 904 is about to send a test data transmission. The IPN 902 may then transmit a data signal to the node 904 over the data connection 912 that is to be switched to the node 906. If the node 906 receives the data signal, then the signal is switched to the IPN 906 via the data connection 916. Upon receiving the data signal, the IPN 906 informs the IPN 902 via the signaling connection 920 that the data transmission was successfully received. If the IPN 902 does not receive the confirmation signal from the IPN 906 within a set time period, then the IPN

concludes that the data transmission was not received, and the test of the connection failed.

Now referring to Figures 10A, l OB and l OC, a representative depiction of how the methods and apparatus of the present invention may affects different layers of a network is illustrated.
In a communication network the nodes that accommodate routing and/or switching typically operate at up to the third layer (the network layer) of the International Standard Organization's Open System Interconnect (ISO/OSI) network model. A very common network layer protocol is the IP protocol, which forms a significant portion of all traffic in communication networks today, and accommodates the routing of individual IP datagrams in a connectionless environment.
For a very dynamic network, all nodes could support layer 3, but this is inefficient due to the high cost of operation associated per interface port.
For this reason, equipment is available that only operate at up to layer 2 (the datalink layer, e.g., SONET/SDH or Ethernet), or even only layer 1 (e.g., DWDM). Due to the lack of good management tools, the equipment can currently switch (or route) data on only one layer. Cooperation between the layers to efficiently switch/route data is currently impossible. For example, a SONET node can only switch SONET frames (it cannot simultaneously switch wavelengths or route IP datagrams), an IP router can only route IP datagrams, even if this IP muter operates with SONET as a datalink layer.
With the increase in management accomplished with the usage of an IPN, it will become possible for a switching/routing node with the correct hardware to efficiently switch and/or route data on all three layers.
Starting from the bottom of the OSI stack, there are several implementation options for the physical layer. Layer 1 switching can currently be accommodated solely in the electrical domain (for example in current LANs across coaxial cable), a combination of the photonic and electrical domain, and solely in the photonic domain (for example, with MEMS technology). Together with this there exists several layer 2 protocols that provide the framing to enable the transfer of the data across the physical medium, these protocols include SONET/SDH and Ethernet. The third layer, the network layer, is responsible for the routing of data in the network; one example is the IP protocol.
A switch/router managed by an IPN could accommodate any combination of these protocols, and it is the IPNs responsibility to ensure that a connection's agreed upon requirements will not be affected by any translations in a node. Together with this, it is also the IPNs responsibility to program the switch to enable it to effectively translate data between any layers to enable the transmission between networks that operate using different protocols.
Figure 10A depicts the interface between each layer of a switch 950 and its managing IPN 952. Also depicted is the path of data through a switch if no translation is required. Without any electrical conversion it is currently impossible to extract the data from an optical signal, this capability may be available in the future.
In this case a node is only capable of layer 1 switching. The moment a node is able to receive an electrical signal or convert an incoming photonic signal to its electronic form, it is able to extract the layer 3 data, which enables it to encapsulate and transfer it using a different datalink (layer 2) protocol.
With the availability of different network layers and the ability to program their usage, the IPN 952 can efficiently manage the switch 950. Instead of hard-provisioning the interfaces, the IPN 952 is able to initiate translations between different network layers based upon criteria that can range from cost to utilization.
The cost may, for example, be a price associated per interface where translation at the network layer is the most expensive and that at the physical layer the cheapest.
An example of the above is depicted in Figure 1 OB. In this example, the layer 1 protocol remained the same while the layer 2 protocol is changed to enable communication between the interconnected networks. The reason for this translation may be caused by high utilization on one interface, or an increase in cost associated with it. The IPN will in this case first map the connection requirements to make the connections across the different datalink layers equivalent in their quality of service, and then proceeds with the programming of the fabric to accommodate this conversion.
The two options depicted in Figure l OB shows the flexibility introduced with the usage of a managing IPN. In diagram 1000 there is no physical layer connection through the first switch 1002, but the managing IPN 1003 was able to set up a datalink layer connection between the first switch 1002 and the second switch 104, as well as the second switch 1004 and the third switch 1006, while managing the translation at the second switch 1004 to enable the data transfer between these two connections. In this example, even though there is a SONET as well as an Ethernet connection between all the switches 1002, 1004, 1006, the IPN 1003 used its intelligence to determine the cheapest connection based on all the cost parameters. In diagram 1050 the managing IPN 1051 was able to set up a physical layer connection through the first switch 1052 and the third switch 1056, while managing the translation of the data in the second switch 1054.
Another possible conversion is depicted in Figure I OC. In this conversion the data link layer needs to remain the same while the data needs to be sent across different physical layer for the switches 1102, 1104, 1106. The IPN 1108 (or one of the other IPN 1110, 1112 if the IPN 1108 is not the managing IPN for the switch 1104 and/or the switch 1106) will consider the properties of both the photonic and electrical domains to provide the requestor with its requested quality of service. The will also program the switch fabric to enable this translation to occur dynamically.
As illustrated and described above, in some embodiments an IPN may act as a link between applications requesting circuits and layers 1, 2, 3 of a network.
The methods and apparatus of the present invention may utilize a combination of in-band and out-of band signaling and can manage QoS issues for many connections, e.g., IP
transmissions by provisioning connections and by, redirecting IP traffic flows (e.g.
MPLS) to the connections in order to divert the traffic flows from congested areas of the network. The present invention allows select applications to request guaranteed QoS by provision on-demand circuits while other traffic uses shared pipes with, for example, RSVP and diffserv. Signaling between networks and IPNs (e.g.
handshakes) may take place in advance to and during transaction setup and provisioning.
In some embodiments, the methods and apparatus of the present invention may be utilized to dynamically allocate connections, e.g. legacy SONET rings, by enabling edge nodes with IPNs and/or to update router MPLS and BGP tables through IPN updates.
In some embodiments, the methods and apparatus of the present invention allow for or facilitate the dynamic concatenation of network segments from multiple service providers for failures and maintenance cutovers. In addition, the methods and apparatus of the present invention may be utilized to eliminating as many L3 and L2 transactions in order to reduce the cost of routing and switching in the network.
The methods of the present invention may be embodied as a computer program developed using an object oriented language that allows the modeling of complex systems with modular objects to create abstractions that are representative of real world, physical objects and their interrelationships. However, it would be understood by one of ordinary skill in the art that the invention as described herein could be implemented in many different ways using a wide range of programming techniques as well as general-purpose hardware systems or dedicated controllers, computers, processors, etc. In addition, many, if not all, of the steps for the methods described above are optional or can be combined or performed in one or more alternative orders or sequences without departing from the scope of the present invention and the claims should not be construed as being limited to any particular order or sequence, unless specifically indicated.
Each of the methods described above can be performed on a single computer, computer system, microprocessor, etc. In addition, two or more of the steps in each of the methods described above could be performed on two or more different computers, computer systems, microprocessors, etc., some or all of which may be locally or remotely configured. The methods can be implemented in any sort or implementation of computer software, program, sets of instructions, code, ASIC, or specially designed chips, logic gates, or other hardware structured to directly effect or implement such software, programs, sets of instructions or code. The computer software, program, sets of instructions or code can be storable, writeable, or savable on any computer usable or readable media or other program storage device or media such as a floppy or other magnetic or optical disk, magnetic or optical tape, CD-ROM, DVD, punch cards, paper tape, hard disk drive, ZipTM disk, flash or optical memory card, microprocessor, solid state memory device, RAM, EPROM, or ROM.
Although the present invention has been described with respect to various embodiments thereof, those skilled in the art will note that various substitutions may be made to those embodiments described herein without departing from the spirit and scope of the present invention.

The words "comprise," "comprises," "comprising," "include," "including," and "includes" when used in this specification and in the following claims are intended to specify the presence of stated features, elements, integers, components, or steps, but they do not preclude the presence or addition of one or more other features, elements, integers, components, steps, or groups thereof.

Claims

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method for facilitating provisioning of a circuit between a first network element and a second network element, comprising:
receiving a request to establish a circuit between a first network element and a second network element in a network, said first network element being associated with a first provisioning node that forms part of a signaling control plane for said network, and said second network element being associated with a second provisioning node that forms part of said signaling control plane for said network;
determining an available circuit between said first network element and said second network element; and providing commands for said first provisioning node and said second provisioning node regarding provisioning of said available circuit between said first network element and said second network element.

2. The method of claim 1, wherein said first network element is associated with said first provisioning node such that said first provisioning node can control at least one characteristic of said first network element.

3. The method of claim 1, wherein said wherein said available circuit includes at least one additional network element and said at least one additional network element is associated with a respective at least one additional provisioning node that forms part of said signaling control plane.

4. The method of claim 3, further comprising:
providing at least one command for said at least one additional provisioning node regarding provisioning of said available circuit between said first network element and said second network element using said at least one additional network element.

5. The method of claim 1, further comprising:
receiving inventory information regarding said first network element from said first provisioning node; and receiving inventory information regarding said second network element from said second provisioning node.

6. The method of claim 1, wherein said first provisioning node includes a first provisioning application operating on a first device and said second provisioning node includes a second provisioning application operating on said first device.

7. The method of claim 1, wherein said first provisioning node comprises a provisioning application operating on said first network element.

8. The method of claim 1, further comprising:
updating a resource to reflect provisioning of said available circuit.

9. The method of claim 1, wherein said first provisioning node is capable of sending commands to said to first network element to provision at least a portion of said available circuit.

10. The method of claim 1, further comprising:
implementing at least a part of said signaling control plane.

11. The system of claim 1, wherein said first provisioning nodes is operative to create a virtual representation of said first network element.

12. A method for facilitating provisioning of a circuit between a first network element and a second network element, comprising:
receiving a request to establish a circuit between a first network element in a first network and a second network element in a second network, said first network element being associated with a first provisioning node that forms part of a first signaling control plane for said first network, and said second network element being associated with a second provisioning node that forms part of a second signaling control plane for said second network;
providing a notification indicative of said request;
determining at least a portion of an available circuit between said first network element and said second network element, said portion of said available circuit including said first network element; and providing at least one command to said first provisioning node for provisioning of said portion of said available circuit.

13. The method of claim 12, wherein said providing a notification indicative of said request includes providing said notification to a device associated with said second signaling control plane.

14. The method of claim 12, wherein said providing a notification indicative of said request includes providing said notification to a device that can identify said second network.

15. The method of claim 14, wherein said device is not part of either said first signaling control plane or said second signaling control plane.

16. The method of claim 14, wherein said device is part of said second signaling control plane.

17. The method of claim 12, further comprising:
receiving inventory information regarding said first network element from said first provisioning node.

18. The method of claim 12, wherein said first provisioning node is capable of sending at least one command to said to first network element to provision at least a portion of said available circuit.

19. The system of claim 12, wherein said first signaling control plane is part of said second signaling control plane.

20. The method of claim 12, wherein said first signaling control plane and said second signaling control plane form at least part of a third signaling control plane.

21. A method for facilitating provisioning of a circuit between a first network element and a second network element, comprising:
receiving a first notification indicative of a request to establish a circuit between a first network element in a first network and a second network element in a second network, wherein said notification is received from a first signaling control plane associated with said first network and indicates a bridge element connected to said first network;
identifying said second network;
identifying a second signaling control plane, said second signaling control plane being associated with said second network;
providing a second notification indicative of said request to said second signaling control plane;
receiving from said second signaling control plane data indicative of a bridge element connected to said second network;
determining an available circuit between said bridge element connected to said first network and said bridge element connected to said second network; and providing commands for said first provisioning said available circuit.

22. The method of claim 21, wherein said providing commands for provisioning said available circuit includes providing commands to at least one provisioning node associated with a network element forming part of said available circuit.

23. The method of claim 21, further comprising:

providing a notification to said first signaling control plane that said first signaling control plane should establish a circuit between said first element and said bridge element connected to said first network

24. The method of claim 21, wherein said bridge element connected to said first network is part of said first network.

25. A method for facilitating provisioning of a circuit between a first network element and a second network element, comprising:
determining at least one operational characteristic of a first network element in a first network;
providing information indicative of said operational characteristic to a device associated with said first network, wherein said device forms part of a signaling control plane for said first network;
receiving a request to establish a requested circuit between said first network element and a second network element, wherein said second network element is associated with a provisioning node;
providing data indicative of said request to said device;
receiving information from said device indicative of a first command for said first network element for use in provisioning an available circuit between said first network element and said second network element; and providing data indicative of said first command to said first network element.

26. The method of claim 25, wherein said second network element is in said first network.

27. The method of claim 25, wherein said second network element is in a second network.

28. The method of claim 25, wherein said providing data indicative of said first command to said first network element includes providing data to said first network element that will cause said first network element to implement said first command.

29. A system for establishing a circuit between network elements, comprising:
a first plurality of provisioning nodes associated with a first network, wherein each of said first plurality of provisioning nodes is associated with a respective network element contained in said first network and operative to inventory said respective network element;
a first node controller in communication with each of said first plurality of provisioning nodes and operative to:
receive information from each of said first plurality of provisioning nodes regarding inventory of network elements included in said first network;
receive a request for provisioning a requested circuit, wherein said request identifies at least one network element in said first network to be used in said requested circuit;
determine an available circuit in said first network that involves said first network element and forms at least a portion of said requested circuit;
determine a one of said first plurality of provisioning nodes associated with said first network element;
provide at least one command to said one of said first plurality of provisioning nodes associated with said first network element regarding provisioning of said available circuit.

30. The system of claim 29, wherein said first plurality of provisioning nodes and said first node controller form at least part of a signaling control plane associated with said first network.

31. The system of claim 29, wherein said request identifies a second network element in said first network and said requested circuit is between said first network element and said second network element.

32. The system of claim 29, wherein said request identifies a second network element in a second network and said requested circuit is between said first network element in said first network and said second network element in said second network.

33. The system of claim 32, wherein said first node controller is operative to identify a second node controller associated with said second network.

34. The system of claim 33, wherein said first node controller is operative to provide at least a portion of said request to said second node controller.

35. The system of claim 32, wherein said first node controller is operative to identify a second node controller and wherein said second node controller is operative to identify said second network.

36. The system of claim 29, further comprising:
a second plurality of provisioning nodes associated with a second network, wherein each of said second plurality of provisioning nodes is associated with a respective network element contained in said second network and operative to inventory said respective network element;
a second node controller in communication with each of said second plurality of provisioning nodes and said first node controller and operative to:
receive information from each of said second plurality of provisioning nodes regarding inventory of network elements included in said second network;
receive information from said first node controller indicative of a request for provisioning a circuit between said first network element in said first network and a second network element in said second network;
determine an available circuit in said second network that involves said second network element and forms at least a portion of a circuit between said first network element in said first network and said second network element in said second network;
determine a one of said second plurality of provisioning nodes associated with said second network element;
provide at least one command to said one of said second plurality of provisioning nodes associated with said second network element regarding provisioning of a circuit between said first network element in said first network and said second network element in said second network.

37. The system of claim 36, wherein said second plurality of provisioning nodes and said second node controller form at least part of a signaling control plane associated with said second network.

38. A system for establishing a circuit between network elements, comprising:
a signaling control plane associated with a network that includes a first plurality of network elements, wherein said signaling control plane includes a first plurality of provisioning nodes and a first node controller, wherein each of said first plurality of provisioning nodes is associated with a different one of said first plurality of network elements and is in communication with said node controller, and wherein said signaling control plane is operative to:
determine an inventory of at least one capability of each of said first plurality of network elements;
receive a request to provision a circuit that includes at least one of said first plurality of network elements;
determine an available circuit that is formable using at least one of said first plurality of network elements and that satisfies at least a portion of said request;

send commands to said at least one of said first plurality of network elements to facilitate establishment of said available circuit.

39. The system of claim 38, wherein said signaling control plane includes a second node controller and a second plurality of provisioning nodes, each of said second plurality of provisioning nodes being in communication with said second node controller and associated with one of a second plurality of network elements in said network, and wherein said second node controller is in communication with said first node controller.

40. The system of claim 39, wherein said signaling control plane is operative to:
determine an inventory of at least one capability of each of said second plurality of network elements;
receive a request to provision a circuit that includes at least one of said first plurality of network elements and one of said second plurality of network elements;
determine an available circuit that is formable using said at least one of said first plurality of network elements and said at least one of said second plurality of network elements, wherein provisioning of said available circuit satisfies at least a portion of said request;
send commands to said at least one of said first plurality of network elements and at least one of said second plurality of network elements to facilitate establishment of said available circuit.

41. A method for facilitating provisioning of a circuit between a first network element and a second network element, comprising:
receiving a request to establish a circuit between a first network element and a second network element, said first network element being associated with a first provisioning node that forms part of a signaling control plane, and said second network element being associated with a second provisioning node that forms part of said signaling control plane;

determining an available circuit between said first network element and said second network element; and providing commands for said first network element and said second network element regarding provisioning of said available circuit between said first network element and said second network element.

42. The method of claim 41, wherein said first provisioning node and said second provisioning node are in communication with a node controller included in said signaling control plane.

43. The method of claim 41, wherein said first provisioning node is in communication with a first node controller, said second provisioning node is in communication with a second node controller, said first controller is in communication with said second node controller, and said first node controller and said second node controller are included in said signaling control plane.

44. The method of claim 43, wherein said first network element is part of a domain associated with said first node controller and second network element is part of a domain associated with said second node controller.

45. The method of claim 41, wherein said wherein said available circuit includes at least one additional network element and said at least one additional network element is associated with a respective at least one additional provisioning node that forms part of said signaling control plane, and further comprising:
providing at least one command for said at least one additional network element regarding provisioning of said available circuit between said first network element and said second network element using said at least one additional network element.