Updated: March 13,1996 |
Introduction
The Network Management Challenge
Why Instrument with SNMP ?
SNMP Review
Windows NT Extendible Agent
Extension Agents API
Summary
This session introduces developers to the Microsoft® Windows NT SNMP facilities. It assumes the developer is familiar with SNMP terminology and operation. The topics covered in this session are:
Code examples referenced in this paper can be found in the Windows NT version 3.5 SDK, in the mstools\samples\win32\snmp\testdll. Additional documentation for the API discussed in this session can be found in the Windows NT 3.5 SDK, in the doc\misc\progref.rtf.
Networks today are rarely a homogeneous collection of machines and operating systems. In fact, most common networks are literally a hodgepodge of network devices, from simple task-specific devices such as printers, routers, repeaters, and bridges, to multi-functional servers and mainframes. Since each of these devices has different problems to solve, they all evolved with different configuration and status-retrieval mechanisms. It became clear to administrators of these large complex networks that having a common method for retrieving information and configuring these network components would greatly simplify their job. Since all these devices are networking devices and interface to a network, this common management mechanism should be able to manipulate these devices via the network.
The above requirements seem simple enough; the management mechanism must present a reliable way to alter and retrieve data remotely on a variety of different platforms. The solution to this problem is not, however, all that simple. The problem implies that these devices must now, at the very least, add the intelligence to converse with a new protocol and must also share its method of instrumentation with that protocol. Put another way, the device has to support a new interface. It has to be able to support a whole new set of possible commands, and it has to be able to respond to them in a specific format. The protocol must be very powerful and flexible in order to support every different kind of data structure and request type for every network device that exists today and may exist in the future.
Many of these network devices consist of very specialized hardware that is required to do a specific task. In today's market, manufacturers of these devices are unwilling to include additional hardware to process remote requests for some universal management mechanism--especially if that mechanism requires significant processing and/or memory resources to function. If the solution to this problem is not cheap, both in cost and processing overhead, then few device manufacturers are likely to adopt it.
The solution to this problem is a management protocol that is powerful, flexible and efficient both in use of resources and processing power. SNMP is not the only proposed solution for this problem, but it is by far the most widely adopted. SNMP addresses the requirement of being powerful, yet flexible and efficient, by using a small number of simple request types. Thus the name Simple Network Management Protocol-- SNMP.
When implementing a network component, be it software services or a hardware device, sooner or later it becomes necessary to determine how you will manage the component. Inevitably you are faced with the question of whether or not it should be remotely managed and, if so, what network protocol should be used. Early network developers had little choice but to "roll their own" API sets and implement them remotely through some proprietary protocol encapsulated in their favorite network protocol. They would then write both the instrumentation of the controlled component (the agent) and the controlling console (the manager) and deliver these modules with the component itself.
As mentioned earlier, network administrators, the component developers' customers, eventually began to demand that all these proprietary management schemes work together. To have these schemes working together would enable an administrator to manage any network device, regardless of the manufacturer, from a single management console. Several management schemes have been developed to address this need, SNMP thus far being the most widely accepted.
The advantages of using SNMP as the management protocol for your network component include:
Because SNMP provides a mechanism by which management consoles can dynamically learn about new components and new component instrumentation, consoles that were written years ago are capable of managing components developed today.
In the SNMP model there is a software component residing on the network device that collects the related information about that device into a well-defined structure. This software component is referred to as the agent. The agent is responsible for responding to queries and carrying out requests to the network device. The counterpart of the agent is the software component that issues requests. Typically, the results of the requests are passed on to some sort of user interface that allows administrators to view data on the queried device. This component is referred to as the manager or management console. Keep in mind that typically the agent software and the manager software are running on different networking components and are communicating via the network and a common protocol.
The data instrument by the SNMP agent is organized into collective units that are termed management information bases or MIBs for short. MIBs are described via a precise definition language called Abstract Syntax Notation. Abstract Syntax Notation is a complete topic of its own and could not be adequately described in the context of this paper, but think of Abstract Syntax Notation as a compiled language like COBOL, FORTRAN or C. It makes possible the definition of data types and structures and arrays of structures of information on the managed device. Fundamentally, MIBs define the following for every entity in the agent:
The manager component of SNMP uses MIBs to relate the structure of the data available at the agent to the SNMP user. Simply put, a SNMP manager console can describe the data available on the SNMP agent because it uses the MIB that defines that data to describe it to the user. It is because of this relationship between agents, MIBs and managers that agents created today can work with management consoles that were written years ago.
Throughout this paper we will be discussing object names, often referring to as OIDs, or a MIB view as part of the OID namespace. An OID is a unique identifier assigned to a specific object. The identifier consists of a sequence of numbers that identify the source of the object as well as the object itself. This sequence of numbers is variable in length, so in addition to the sequence of numbers, there is a length field. OIDs are organized in a tree-like structure and the sequence of numbers identifies the various branches of the subtree that a given object comes from. The root of the tree is the ISO (International Standards Organization) trunk. Its value is 1. Each branch above the root (or below, depending on your perspective) further identifies the source of the given object. All SNMP objects are members of the subtree identified by iso.org.dod.internet or 1.3.6.l. Each additional component in this dotted notation further defines the exact location of an object. The numbers for each subtree are assigned by the IETF to ensure that all branches are unique.
The example in Figure 1.1 on the following page illustrates the OID values for the Toaster MIB. The Toaster MIB is an example MIB generated for the purposes of illustration; we will refer to it often throughout the course of this paper.
Figure 1.1 Toaster OID Diagram
To reference the ModelNumber object instance in a Toaster MIB on a network device you would specify 1.3.6.1.4.1.12.2.2.0 as the OID. The zero is appended to indicate that this a specific instance of this object.
Another data structure we will reference often in this paper is a VarBind. A VarBind consists of an OID and a value structure. The value structure of a VarBind is a two-part structure that consists of a value type and a value. The value type field identifies what type of variable the value field represents. Some examples of value types are Integer, Octet, Octet String, and Gauge. In the structure definition below the value component is referred to as an AsnObjectSyntax, which is a union of all the possible SNMP data types and a value type.
typedef struct vb { AsnObjectName name; AsnObjectSyntax value; } RFC1157VarBind;
SNMP uses a simple set of commands to set and retrieve values of objects in MIBs. There are three basic request types in SNMP: Set, Get and GetNext. The Basic SNMP protocol entity is referred to as a PDU, Protocol Data Unit. For the Set, Get and GetNext commands, a PDU consists of a VarBindList, requestType, requestId , errorStatus and errorIndex fields.
A VarBindList is an array of VarBinds. A portion of the snmp.h header file below shows the definition of a VarBindList as well as the definition for a PDU.
typedef struct { RFC1157VarBind *list; UINT len; } RFC1157VarBindList; typedef struct { RFC1157VarBindList varBinds; AsnInteger requestType; AsnInteger requestId; AsnInteger errorStatus; AsnInteger errorIndex; } RFC1157Pdu;
The requestId helps managers correlate requests with responses. The IP address indicates the IP address of the device that is being managed . The request type is one of the three: Set, Get or GetNext.
Get and Set operations are only allowed on object instances. Obviously, multiple objects may be retrieved or modified in a single PDU. SNMP specifies that, when modifying objects, if one Set fails in a PDU, none of the Set operation should be applied.
The third SNMP request type is a GetNext. A GetNext request is slightly different from Get and Set operations. A VarBindList is still passed as the argument for the GetNext operation, but unlike Get and Set commands, the OIDs present in the GetNext VarBinds do not have to identify object instances. Instead, GetNext request s can specify any OID. The SNMP protocol specifies that when a GetNext request is issued to a particular agent, it will return the first value instrument by the receiving agent following the specified OID. In Figure 1.1, if the agent for the MIB pictured receives a request for 1.3.6.1 (iso.org.dod.internet), the agent must respond with the first OID supported in its MIB(s) that follows ( is lexicographically greater than) the supplied OID--in this case iso.org.dod.internet.private.enterprises.epilogue.toaster.manufacturer.0.
Traps are SNMP messages that originate from the agent to a preconfigured management station. They are used to notify management consoles of significant events. The PDU for a trap is slightly different from that of the other SNMP request types. Trap PDUs consist of a VarBindList, enterprise OID, the IP address of the sending agent, a genericTrap identifier, a specificTrap Identifier and a timeStamp. Typical usage of traps is notification of a service starting or stopping , notification of serious error conditions, and so on. Below is the data structure defined in snmp.h for a trap PDU. The VarBindList is the same as used in Get , GetNext and Set operations. The enterprise is the OID for the enterprise that this trap belongs to. In the case of the Toaster MIB this would be the OID for Epilogue (1.3.6.4.1.12). The agent address is the IP (or IPX) address of the agent. The Generic Trap identifies what kind of trap this is. It can be one of any of the following:
In the case of the Toaster DLL we would indicate SNMP_GENERICTRAP_ ENTERSPECIFIC. The specificTrap pertains to which of Enterprise traps this specific trap is. In other words, the trap ID is the combination of the enterprise OID, genericTrap and specificTrap values. For the Toaster MIB the specificTrap value would be 0.
typedef struct { RFC1157VarBindList varBinds; AsnObjectIdentifier enterprise; AsnNetworkAddress agentAddr; AsnInteger genericTrap; AsnInteger specificTrap; AsnTimeticks timeStamp; } RFC1157TrapPdu;
The timeStamp value is the time from the instant this agent was last initialized until the time this event occurred, in increments of 100ths of a second.
The Windows NT SNMP service (snmp.exe) is an extendible SNMP agent that allows developers to add DLLs to service additional MIBs. The agent itself does not contain instrumentation for any MIBs; instead it is responsible for retrieving SNMP requests for the NT workstation or server and passing these requests on to the appropriate DLL for resolution. The response data is then returned to the agent, who is responsible for returning the request to the management station that made it. By default, four extension agent DLLs are shipped with the SNMP service. These DLLs service the MIB-II, LanMan2, DHCP and WINS MIBs. Developers who are interested in adding additional MIBs can create a DLL using the API documented in the SNMP Programmer's Reference Guide and this paper. The extendible agent is also capable of issuing traps on behalf of any of the agent DLLs.
The extendible agent interfaces to the network via the Winsock DLL (Figure 2). This DLL works over both the IP and IPX transports. This enables the extendible agent to be accessible from both IP and IPX network management consoles. The SNMP agent can be configured to accept requests only from particular IPX or IP addresses. This is done via the Network Control Panel. In addition, the valid communities that this agent will recognize can also be configured there.
The extendible agent determines what agent DLLs to load via registry values located under the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SNMP \Parameters\ExtensionAgents key. Once the registry is consulted during service initialization, the extendible agent gets the OID prefix for the MIB that the DLL supports via the SnmpExtensionInit API supplied by the DLL. The extendible agent builds a table of supported MIBs with their corresponding resolution routines and stores it sorted. Once all the DLLs are loaded and the table is initialized, the extendible agent is ready to process SNMP requests.
Although the SNMP extendible agent service is written as a service, it can be run directly as an executable. This comes in handy when attempting to debug extension agent DLLs.
Each extension agent must export at least the following three APIs:
In addition, the extension agent may support a fourth API, SnmpExtensionInitEx. This API allows a single extension agent to support multiple MIBs or multiple MIB views from a single DLL.
Below is a the SnmpExtensionInit API implementation for the Toaster MIB. The variable dwTimeZeroReference is passed into the API from the extension agent so that the extension agent has a zero reference point. If for some reason the extension agent needs to determine how long it has been running, it can determine this by subtracting this value from the value returned by GetCurrentTime(). Many MIBs need the concept of uptime, or the length of time this device has been up since last reboot or reinitialization. In addition, any MIB that supports traps will need the uptime value since this is included in the trap PDU.
The hPollForTrap event is the event that this extension agent will signal when it is ready to send a trap. Keep in mind that, because the extension agent consists of subroutines called by the extendible agent, the extension agent won't necessarily be actively processing a request when it realizes it needs to send a trap. Take, for example, a trap that is based on some outside event like an out-of-paper signal from a printer. If no management station is currently querying the printers MIB, then the printer MIB DLL needs to somehow notify that extendible agent that it needs to be called to do some processing. One obvious method that could be employed is for the extendible agent to periodically poll for traps. This method is much more efficient if the developer can somehow associate the signal from the printer with an event; then the extension agent will be notified when the signal occurs and will call into the SnmpExtensionTrap API supplied by the DLL.
Finally, the supportedView argument is simply an OID that identifies the prefix of the MIB or MIB view that this DLL supports.
BOOL WINAPI SnmpExtensionInit( IN DWORD dwTimeZeroReference, OUT HANDLE *hPollForTrapEvent, OUT AsnObjectIdentifier *supportedView) { // Record the time reference provided by the Extendible //Agent. dwTimeZero = dwTimeZeroReference; // Create an Event that will be used to communicate the // occurence of trap to the Extendible Agent. The Extension // Agent will assert this Event when a trap has occured. // This is explained further later in this file. if ((*hPollForTrapEvent = CreateEvent(NULL, FALSE, FALSE, NULL)) == NULL) { *hPollForTrapEvent = NULL; *supportedView = NULL; return FALSE; } // Indicate the MIB view supported by this Extension Agent, // an object identifier representing the sub root of the MIB // that is supported. *supportedView = MIB_OidPrefix; // NOTE! structure copy // Record the trap Event. hSimulateTrap = *hPollForTrapEvent; // Indicate that Extension Agent initialization was // sucessfull. return TRUE; } // end SnmpExtensionInit()
SnmpExtensionQuery is the API that the extendible agent will call when it determines that the OID of an SNMP request matches the supportedView OID that the extension agent DLL returned from SnmpExtensionInit.
The requestType argument indicates what type of SNMP request is being processed. The possible requestTypes are:
The routine ResolveVarBind is responsible for carrying out the actual setting or retrieving of the VarBind data. Since the DLL is capable of handling a VarBindList, multiple objects could be modified in a single request. SNMP requires that if an agent is not successful in carrying out all operation in a single PDU, then it should not carry out any. Because of this all-or-nothing requirement, the ResolveVarBind routine only modifies cached data. The Toaster SnmpExtensionQuery routine checks to make sure that the errorStatus argument is set to SNMP_ERRORSTATUS_NOERROR and, if so, it calls ActionCaches with the TOASTER_COMMIT value to take the cached data and commit it. This is an important detail to keep in mind. Many SNMP management scenarios depend on this concept to tell if an action taken was successfully accomplished. If an error was detected, calling Action Caches with the argument TOASTER_CLEAR throws away the temporarily cached data.
The second thing to note in this routine is the handling of the case of a GETNEXT request falling beyond the end of the MIB view that this DLL supports. Note that the routine checks for the error status SNMP_ERRORSTATUS_NOSUCHNAME returned from ResolveVarBind and, if it finds this to be true, will change the OID of the VarBind passed in, to a point just beyond the view of the current DLL and indicate no error. The extendible agent will check for this condition and process the request accordingly.
BOOL WINAPI SnmpExtensionQuery( IN BYTE requestType, IN OUT RFC1157VarBindList *variableBindings, OUT AsnInteger *errorStatus, OUT AsnInteger *errorIndex) { UINT I; // Iterate through the variable bindings list to resolve // individual variable bindings. for ( I=0;I < variableBindings->len;I++ ) { *errorStatus = ResolveVarBind( &variableBindings->list[I], requestType ); // Test and handle case where Get Next past end of MIB // view supported by this Extension Agent occurs. // Special processing is required to communicate this // situation to the Extendible Agent so it can take // appropriate action, possibly querying other Extension // Agents. if ( *errorStatus == SNMP_ERRORSTATUS_NOSUCHNAME && requestType == MIB_ACTION_GETNEXT ) { *errorStatus = SNMP_ERRORSTATUS_NOERROR; // Modify variable binding of such variables so the // OID points just outside the MIB view supported by // this Extension Agent.The Extendible Agent tests // for this, and takes appropriate action. SNMP_oidfree( &variableBindings->list[I].name ); SNMP_oidcpy( &variableBindings->list[I].name, &MIB_OidPrefix ); variableBindings->list[I].name.ids[MIB_PREFIX_LEN-1]++; } // If an error was indicated, communicate error status // and error index to the Extendible Agent. The // Extendible Agent will ensurethat the origional // variable bindings are returned in the response // packet. if ( *errorStatus != SNMP_ERRORSTATUS_NOERROR ) { *errorIndex = I+1; break; } } if (*errorStatus == SNMP_ERRORSTATUS_NOERROR) ActionCaches(TOASTER_COMMIT); else ActionCaches(TOASTER_CLEAR); // Indicate that Extension Agent processing was sucessfull. return TRUE; } // end SnmpExtensionQuery()
The SnmpExtensionTrap API is called by the extendible agent when it receives notification via a signaled event (event passed back in SnmpExentionInit) that a particular extension agent needs to send a trap. The API returns TRUE if has successfully initialized the arguments supplied and wishes to be called again. There are two reasons the SnmpExtensionTrap API may wish to indicate it would like to be called again. The first may be to send the same trap to multiple management stations; the second may be to give the SnmpExtensionTrap API a chance to clean up any memory allocated for VarBinds that were dynamically allocated . By keeping a static variable with state information (such as whichTime in the sample code) the API can determine whether or not it is being called to send traps or clean up after sending a trap.
In the example below the VarBind ToastUp is used to indicate that the toaster has completed toasting something or that someone has ejected the toast. Either way the trap indicates that the toaster is now ready to be reloaded or reset. Note that this VarBind allocates the memory for its name dynamically, using the SNMP_alloc API. It is important to use the SNMP_ API to alloc, realloc and free memory. Recent bugs have been found when using MSVC2.0 to build SNMP agents and managers due to conflicting run time libraries between the NT and MSVC20 run-time libraries. Using these APIs instead of the generic C run times will insure proper behavior of your app. After all the data is initialized, the routine returns true, which indicates to the extendible agent that as soon as it successfully sends the trap it should call this API again immediately. Since the state variable WhichTime is set to TRAP_CLEANUP the routine frees up the memory associated with the VarBind using the SnmpUtilVarBindFree API, resets the state variable to TRAP_GENERATION and returns FALSE, indicating to the extendible agent that no action need be taken (that is, don't send a trap, the data is not valid).
The only other notable thing about SnmpExtensionTrap is enterprise OID. This is the OID of the enterprise that is sending the trap. A single enterprise may have many traps; therefore the combination of the enterprise, genericTrap and specificTrap values constitutes a unique trap.
BOOL WINAPI SnmpExtensionTrap( OUT AsnObjectIdentifier *enterprise, OUT AsnInteger *genericTrap, OUT AsnInteger *specificTrap, OUT AsnTimeticks *timeStamp, OUT RFC1157VarBindList *variableBindings) { // Toast Up object 6, value 1, toast complete, value 2, // ejected static UINT OidList[] = { 1, 3, 6, 1, 4, 1, 12, 2, 6, 0 }; static UINT OidListLen = 10; static RFC1157VarBind ToastUpVarBind; // The following variable is used to track state info about // what phase of the trap we are processing. TRAP_GENERATION // indicates that this phase is building the Trap PDU and // TRAP_CLEANUP indicates that the trap PDU has been sent and // We are in the process of cleaning up allocated memory for // the trap static whichTime = TRAP_GENERATION; if (whichTime == TRAP_GENERATION) { whichTime = TRAP_CLEANUP; // Supports the simulation. // Communicate the trap data to the Extendible Agent. enterprise->idLength = OidListLen; enterprise->ids = (UINT *)SNMP_alloc(sizeof(UINT) * 7); memcpy(enterprise->ids, OidList, sizeof(UINT) * 7); *genericTrap = SNMP_GENERICTRAP_ENTERSPECIFIC; *specificTrap = 0; // the ToastUp trap *timeStamp = (GetCurrentTime()/10) - dwTimeZero; ToastUpVarBind->name.ids = SNMP_alloc(OidListLen); ToastUpVarBind->name.len = OidListLen; memcpy(ToastUpVarBind->name.ids, OidList, OidListLen); ToastUpVarBind->value.type = ASN_INTEGER; ToastUpVarBind->value.value = ReasonUp; variableBindings->list = ToastUpVarBind; variableBindings->len = 1; // Indicate that valid trap data exists in the // and parameters. return TRUE; } else { whichTime = TRAP_GENERATION; SnmpUtilVarBindFree(variableBindings->list); // Indicate that no more traps are available and // parameters do not refer to any valid data. return FALSE; } } // end SnmpExtensionTrap()
This API allows an extension DLL to indicate to the extendible agent that it wants to support multiple MIBs or multiple MIB views. This API returns a Boolean indicating to the extendible agent whether or not it has more views to support. The extendible agent will continue to call this API until False is returned.
BOOL SnmpExtensionInitEx(OUT AsnObjectIdentifier *supportedView)
supportedView
Points to an AsnObjectIdentifier specifying the MIB subtree supported by the extension agent.
TRUE if the extension agent has more MIB views that it wants to notify the extendible agent about.
FALSE if it has notified the extension agent of all supported views.
When the extendible agent detects that the extension agent supports this API, it will loop, calling this API as long as TRUE is returned. In this way an extension agent can indicate multiple subtree views supported by a single extension agent.
The toaster MIB DLL only supports a single view into its MIB, so included below is an example taken from the MIB-II MIB of the Windows NT extension agent. Note the use of the global variable whichView that keeps track of what views have been added. A case statement determines what view gets added on each call until all have been added, and then FALSE is returned.
int whichView; UINT MyView1[] = {1, 3, 6, 1, 2, 1, 1}; UINT MyView2[] = {1, 3, 6, 1, 2, 1, 2}; UINT MyView3[] = {1, 3, 6, 1, 2, 1, 3}; UINT MyView4[] = {1, 3, 6, 1, 2, 1, 4}; UINT MyView5[] = {1, 3, 6, 1, 2, 1, 5}; UINT MyView6[] = {1, 3, 6, 1, 2, 1, 6}; UINT MyView7[] = {1, 3, 6, 1, 2, 1, 7}; BOOL SnmpExtensionInitEx( View *supportedView) { BOOL retVal; retVal = TRUE; switch (whichView) { case 0: supportedView->idLength = 7; supportedView->ids = MyView2; break; case 1: supportedView->idLength = 7; supportedView->ids = MyView3; break; case 2: supportedView->idLength = 7; supportedView->ids = MyView4; break; case 3: supportedView->idLength = 7; supportedView->ids = MyView5; break; case 4: supportedView->idLength = 7; supportedView->ids = MyView6; break; case 5: supportedView->idLength = 7; supportedView->ids = MyView7; break; default: retVal = FALSE; break; } whichView++; return(retVal); }
Network management is an integral element in developing any kind of network service or device. This is apparent as more and more corporations are requiring network components, services and devices to work with SNMP in order to be considered for rolling out on corporation networks. Not only are network admins setting stringent requirements, but several software vendors are building up large management frameworks based on SNMP. By defining a MIB and implementing the instrumentation for that MIB using the NT extendible agent architecture, adminstrators can easily provide most major corporations with the required management functionality as well as take advantage of working with these management framework applications. Deciding to instrument via SNMP relieves developers of new network services and relieves devices of having to develop a management UI. They can instead concentrate on providing an excellent service, provide prudent instrumentation and rely on existing SNMP management consoles to provide the necessary UI.
© 1995 Microsoft Corporation.
THESE MATERIALS ARE PROVIDED "AS-IS," FOR INFORMATIONAL
PURPOSES ONLY.
NEITHER MICROSOFT NOR ITS SUPPLIERS MAKES ANY WARRANTY, EXPRESS
OR IMPLIED WITH RESPECT TO THE CONTENT OF THESE MATERIALS OR THE
ACCURACY OF ANY INFORMATION CONTAINED HEREIN, INCLUDING, WITHOUT
LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. BECAUSE SOME STATES/JURISDICTIONS DO
NOT ALLOW EXCLUSIONS OF IMPLIED WARRANTIES, THE ABOVE LIMITATION
MAY NOT APPLY TO YOU.
NEITHER MICROSOFT NOR ITS SUPPLIERS SHALL HAVE ANY LIABILITY FOR
ANY DAMAGES WHATSOEVER INCLUDING CONSEQUENTIAL INCIDENTAL, DIRECT,
INDIRECT, SPECIAL, AND LOSS PROFITS. BECAUSE SOME STATES/JURISDICTIONS
DO NOT ALLOW THE EXCLUSION OF CONSEQUENTIAL OR INCIDENTAL DAMAGES
THE ABOVE LIMITATION MAY NOT APPLY TO YOU. IN ANY EVENT, MICROSOFT'S
AND ITS SUPPLIERS' ENTIRE LIABILITY IN ANY MANNER ARISING OUT
OF THESE MATERIALS, WHETHER BY TORT, CONTRACT, OR OTHERWISE SHALL
NOT EXCEED THE SUGGESTED RETAIL PRICE OF THESE MATERIALS.
![]() |
Click Here to Search TechNet Web Contents | TechNet CD Overview | Microsoft TechNet Credit Card Order Form At this time we can only support electronic orders in the US and Canada. International ordering information. |
©1996 Microsoft Corporation |