http://www.microsoft.com/TechNet/boes/bo/winntas/technote/perfbnch.htm (PC Press Internet CD, 03/1996)
| Updated: March 13,1996 |  Go To TechNet Home Page |
The purpose of this guide is to help you understand some of the major industry standard benchmarks and how the performance of the Microsoft® Windows NT™ Server 3.5 operating system compares with other network operating systems such as Novell® NetWare™ 3.12 and LAN Server 3.0. The results in the guide should not be construed as a capacity planning guide for Windows NT Server 3.5. Rather, it only serves to compare and evaluate performance of different network operating systems under both file and application server input-output scenarios.
The guide briefly explains each benchmark, shows sample results, and explains how to interpret these results.
All tests were performed on a Compaq® Proliant™ 2000. Unless otherwise specified, the tests were performed using only the single-processor (66Mhz Pentium™ ) configuration with 4xNE3200 Ethernet cards, 32MB RAM, and 1GB disk capacity for the operating system, and 1GB capacity for the application. The clients used were 486/66 Mhz, 16MB RAM machines with NE2000 Ethernet cards. Novell's VLM 1.11 client software was used in the testing.
What is Netbench 2.1?
Netbench 2.1™ is a Ziff-Davis benchmark that is used to measure file input-output performance for different network operating systems. It is designed to test file services in order to avoid "special system calls," and it is network operating system-independent. It does not measure application server performance, as Serverbench 1.1™ does. Netbench 2.1 clients send out requests to a network server which then processes these requests. Netbench 2.1 does not use real applications, instead it is a synthetic test that models real application behavior.
There are three main parameters that are configurable: test file size, input-output record size, and ratio of reads to writes. A larger test file increases the disk area in which the test operates and also reduces the cache effects of the operating system. The record size contains the size in bytes of the block to be read/written by each client and is typically much smaller than the file size. Netbench 2.1 is usually run with 1MB data files per client and performs three random read operations followed by one random write (RRRW) request in 512Bytes and 4KB record sizes. These two record sizes represent a typical corporate network environment for both large and small data transfers.
Results
Netbench 2.1 was run with Windows NT Server 3.5 and NetWare 3.12. Chart 1 depicts the results obtained with 512Byte record sizes, 1 MB test file size and a RRRW sequence with DOS Clients. Chart 3 depicts the same test as above but with Windows™ for Workgroups 3.11 clients. Chart 2 depicts the results obtained with 4KB record sizes, 1 MB test file size, and RRRW sequence with DOS client.
Chart 1
Chart 2
Chart 3
How to interpret Netbench 2.1 results
What is Serverbench 1.1?
Serverbench 1.1 is a Ziff-Davis benchmark, and is a true client-server application which runs on NetWare 3.12, SCO UNIX®, and Windows NT Server 3.5. An application optimized for each of these operating systems runs on the server and the clients run a set of predefined tests. (By comparison, Netbench 2.1 is an operating system-independent benchmark.)
Serverbench 1.1 measures the performance of the processor, disk, and network subsystems by running different types of tests which produce different loads on the server. The processor memory sub-system test manipulates a data file containing certain fields (based on a simulated database) such as customer name, address, and phone number. The record size is 125 Bytes and there are 2800 records per client for a total of 350KB per client. This file is manipulated in memory.
The disk subsystem test measures server disk input-output performance and comprises sequential read, sequential write, random read, random write and append tests performed on the server. The atomic size and total file size are user-controllable and both determine how much data is read at one time plus the total data transferred to/from disk. The network sub-system entails sending data from the client to the server and from the server to the client. The data transferred can be tailored to a desired value. This last component resembles the file input-output test described earlier.
This test is an excellent measure of the real world simulating how real users use the network operating system as both an application server and file server. Serverbench 1.1 shows results in transactions per second (TPS) for each of the sub-systems measured - the number of transactions completed by a client in an allotted amount of time. It also computes an aggregate TPS by taking a weighted harmonic mean of the TPS of all the sub-systems tested.
Results
Serverbench 1.1 was run with both Windows NT Advanced Server 3.1 and Windows NT Server 3.5 in order to show the performance improvements of Windows NT Server 3.5. The test suite run was the same test as run by Ziff-Davis Labs and published in the May 2, 1994 issue of PC Week. The mixes used were Sequential Read, Sequential Write, Random Write and Random Read, Processor, Client to Server, and Server to Client tests and each client works with an exclusive 20MB data file. Chart 4 shows the PC Week results and Chart 5 shows the performance improvements obtained with Windows NT Server 3.5. The test was run using 4 x 1GB disk striped set with the standard equipment (as mentioned at the beginning). For the multiprocessor testing the dual-processor configuration was used.
Chart 4
Chart 5
How to interpret Serverbench results
What is the BAPCo Network Load?
The Business Applications Performance Corporation Committee on Networking (BAPCo) has developed a workload that is based on actual applications execution request patterns that mimic those measured on corporate networks. SYSMark94 for Servers can be used to measure file input-output performance for network file server operating systems. It is not an application server benchmark.
Unlike Netbench 2.1, this test uses real applications for Windows™ and DOS. The client machines execute network-based application programs that are driven by native macros in the case of DOS based-applications and by MSTEST in the case of Windows-based applications. The applications used are Lotus® 1-2-3® for DOS, Microsoft Excel for Windows, Word for Windows, CC:Mail™ for Windows, Harvard Graphics® for DOS, Paradox® for DOS, dBASE® for DOS, WordPerfect® for DOS, and Freelance® for Windows.
The clients load and execute both the Windows operating systems and the applications from the server. The MSTEST scripts, batch files, program and data files reside on the server (in each client directory) and are executed across the network. The scripts executed by the client require the client to do a certain set of local and network-based tests. Unlike Netbench 2.1, it is not possible to isolate the operations that the BAPCo suite undertakes. The BAPCo suite was designed to simulate real world application usage across the network; therefore multiple operations are performed differently by each of the applications.
Results
The BAPCo network load was run for all operating systems: LAN Server 3.0, Windows NT Server 3.5, NetWare 3.12 and NetWare 4.01. NetWare 3.12, NetWare 4.01 and Windows NT Server 3.5 were tested with the standard equipment (as explained in the Equipment Section). LAN Server 3.0 was also tested on a Compaq Proliant but using 3Com Etherlink III network cards, since LAN Server does not support the industry standard NE3200 cards. Chart 6 depicts total time for the test suite to complete for all operating systems with a 48 client load. Chart 7 is a comparison of Windows NT Server 3.5 and Windows NT Advanced Server 3.1 with a 48 client load. Chart 8 is the total time per application taken to complete the test suite with a 48 client load.
Chart 6
Chart 7
Chart 8
How to interpret the results
Again, as in Netbench 2.1, the client load does not translate into number of users. These clients are doing a great amount of work without thinktime, and better represent hundreds of typical users.
What is the NSTL Test?
This test is used to gauge file I/O performance for different network operating systems. It does not use real applications, but models the behavior of real applications: spreadsheets, word processors, Xcopy and databases. The test is not an application server benchmark.
Results
The test was run with both Windows NT Server 3.5 and NetWare 3.12. The equipment used was a DELL® machine, with a 90Mhz Pentium processor and with an additional Buslogic SCSI adapter. Chart 9 shows the total time for the test to complete for both operating systems with a single 3GB external SCSI drive. Chart 10 shows the total time for the test to complete with a 6GB external CORE RAID-5 array.
Chart 9
Chart 10
How to interpret the results
What is the LAN Test?
This test measures the performance of operating systems such as Windows NT Server 3.5 and NetWare 3.12 when functioning as file servers to Macintosh® clients. The test measures two operations - the total throughput obtained while doing Read and Write operations to a 3MB file. Macintosh clients create a load on the server, and the throughput is measured for one client as it performs the Read and Write operations under this load.
Results
The test was run against both Windows NT Server 3.5 and NetWare 3.12. The equipment used was a single processor Compaq Proliant 66MHz with 3xNE3200 Ethernet cards and 32MB RAM. Chart 11 shows the total throughput in KB/Sec while doing Write operations with a 3MB file and Chart 12 shows the total throughput in KB/Sec while doing Read operations from a 3MB file.
Chart 11
Chart 12
How to interpret the results
What is the test?
This test measures the performance of different operating systems under a heavy database transaction load. The test is modeled after the Transaction Processing Council TPC-B benchmark and uses Microsoft SQL Server™. It is an application server test.
Results
The test was run on Windows NT Advanced Server 3.1 and Windows NT Server 3.5. Chart 13 shows percentage performance increase with Windows NT Server 3.5 over Windows NT Advanced Server 3.1. The hardware used included 4 DELL 486/66 client machines that simulated a 36 client load.
Chart 13
How to interpret the results
Windows NT Server 3.5 will improve upon Windows NT Advanced Server 3.1's scaleability with the performance increase being consistent as additional processors are added.
Netbench 2.1 as Run by National Software Testing Laboratories (NSTL)
Microsoft contracted NSTL, an independent vendor, to run Netbench 2.1 with Windows NT Server 3.5 and NetWare 3.12. The equipment used was a Compaq Proliant 2000 with a single 66 Mhz Pentium processor, 4 x NE3200 Ethernet network cards, 32 MB RAM, 1GB disk capacity for the operating system and 1GB disk capacity for the data. The clients used were 486/66 Mhz with 16 MB RAM and NE2000 Ethernet network cards. The Novell client software used was VLM 1.11.
The following charts illustrate the results and also clearly support the internally generated Microsoft results (as seen earlier in this brochure). Your mileage may vary with client-side caching. The performance boost obtained will be a function of the data file being cached and the size of the client-side cache - the large boost seen herein is because the size of the client-side cache is far larger than the size of the data file.
Chart 14
Chart 15
Chart 16
Chart 17
The results obtained by NSTL, clearly show that Windows NT Server outperforms NetWare 3.12 in File I/O performance testing.
BAPCo Network Load as Run by LANQuest
Microsoft contracted LANQuest, an independent vendor, to run the BAPCo network load with Windows NT Server 3.5 and NetWare 3.12. The equipment used was a Compaq Proliant 2000 with a single 66 Mhz Pentium processor, 4 x NE3200 Ethernet network cards, 32 MB RAM, 1GB disk capacity for the operating system and 1GB disk capacity for the data. The clients used were 486/33 Mhz with 8 MB RAM and NE2000 Ethernet network cards. The Novell client software used was VLM 1.11.
The following charts illustrate the results and also clearly support the internally generated Microsoft results (as seen earlier in this brochure).
Chart 18
The results obtained by LANQuest, clearly show that Windows NT Server is as fast as NetWare 3.12 in File and Print performance testing.
Client-Side Caching
What is Client-Side Caching and how is Windows for Workgroups 3.11 designed to support this?
Operating systems make use of file caching to improve system performance. This enables the data to be available in memory, reduces disk access and can give tremendous boosts in performance, depending on how effectively the operating system manages this process. Client-side caching is simply an extension to the above argument and allows the client (in the client-server setup) to locally cache network data, whenever possible, and hence reduce the traffic over the wire. This optimization is implemented in the redirector software, is transparent to the end-user but is within the administrators control to enable very high network system performance.
How does one setup Client-Side Caching with Windows for Workgroups 3.11?
In Control Panel, click on enhanced => Virtual Memory => Change (see pg. 5 for screen-shots). At the bottom of the Virtual Memory screen, there is a setting for 32-bit File Access, which needs to be enabled. The system will then pre-decide how much memory to devote to cache size according to physical memory in the system, as shown in the following table.
Total Memory Available Default Cache Size
<= 4 MB 512 KB <= 6 MB 1024 KB <= 8 MB 2048 KB <= 12 MB 3064 KB > 12 MB 4096 KB
The above settings are also user configurable and within the administrators control.
If 32-bit File Access is not enabled the default cache size is 512 KB.
What happens to SmartDrive?
With Windows for Workgroups 3.1 and Windows 3.1, it was necessary to use SmartDrive to support disk cache functionality. Windows for Workgroups 3.11 extends the 32-bit Disk Access system architecture of Windows 3.1 and Windows for Workgroups 3.1 to provide 32-bit File Access as well. 32-bit File Access provides a 32-bit code path for Windows to access and manipulate information on disk by intercepting the MS-DOS Int 21H services in protect mode itself. The 32-bit file access functionality is implemented as two Windows virtual drivers, VFAT.386 and VCACHE.386. VFAT, working in conjunction with VCACHE.386, provides a 32-bit protect-mode replacement to SmartDrive disk cache program. When 32-bit File-Access is enabled, the 32-bit File Access driver will disable SmartDrive from caching on disk volumes that the VFAT VxD mounts. The caching routines provided as part of the 32-bit File Access differ from that offered by SmartDrive as follows:
What if I want to enable SmartDrive for CD-ROM caching?
You can continue to enable SmartDrive if you often operate in MS-DOS, or you wish to continue to cache CD-ROM drives. SmartDrive supports two different parameters - when running in MS-DOS, InitCacheSize, and when running in Windows, WinCacheSize. If SmartDrive is loaded, when Windows is started SmartDrive will change the Size of cache from the InitSizeCache value to the WinCacheSize value Windows for Workgroups 3.11 for caching CD-ROM drives. The default values used by SmartDrive are:
Total XMS Memory InitCacheSize WinCacheSize Available
1 MB 1 MB 0 2 MB 1 MB 256 KB 4 MB 1 MB 512 KB 6 MB 2 MB 1 MB >= 8MB 2 MB 2 MB
However, if you do not want to do any CD-ROM caching, you should consider reducing the size of the SmartDrive cache and this can be easily done by setting the values for InitCacheSize and WinCacheSize in Autoexec.bat. For example:
C:\WINDOWS\SMARTDRIVE.EXE 2048 128 /X
sets InitCacheSize to 2048 KB and WinCacheSize to a minimal 128 KB hence making more memory available to Windows for Workgroups 3.11.
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.
© 1994 Microsoft Corporation. All rights reserved. Printed
in the United States of America.
Microsoft and MS-DOS are registered trademarks and Windows, Win32
and Windows NT are trademarks of Microsoft Corporation.
Netbench and Serverbench are trademarks of Ziff-Davis Publishing
Company, L.P.
Macintosh is a registered trademark of Apple Computer, Inc.
dBASE and Paradox are registered trademarks of Borland International,
Inc.
Compaq is a registered trademark and Proliant is a trademark of
Compaq Computer Corporation.
Pentium is a registered trademark of Intel Corporation.
DELL is a registered trademark of DELL Computers
Lotus, 1-2-3 and Freelance are registered trademarks of Lotus
Development Corporation.
Novell and NetWare are registered trademarks of Novell, Inc.
Harvard Graphics is a registered trademark of Software Publishing
Corporation.
UNIX is a registered trademark of UNIX System Laboratories.
WordPerfect is a registered trademark of WordPerfect Corporation.
National Software Testing Laboratories Inc. (NSTL) is a division
of McGraw Hill. NSTL makes no recommendation or endorsement of
any product. The results of the NSTL test as presented in this
brochure were prepared by Microsoft using licensed testing products
from NSTL.
The BAPCo network load used in the testing described in this brochure
is a pre-release version of the software. The final release of
the BAPCo network load is currently under review by the BAPCo
committee. The BAPCo committee makes no recommendation or endorsement
of any product.
 |
Click Here to Search TechNet Web Contents | TechNet CD Overview | |
Microsoft TechNet Credit Card Order Form At this time we can only support electronic orders in the US and Canada. International ordering information. |
| Go To TechNet Home Page |
©1996 Microsoft Corporation |  Go To Microsoft Home Page |