The importance of accurate emulation of real users to evaluate the performance of server has long been recognized. However, the advent of high-capacity servers has elevated the complexity of performance analysis to a much higher level as proper analysis involves an accurate emulation of thousands, if not tens of thousands, of simultaneous and independent clients. In particular, accurate emulation requires two important elements: (1) each client is representative of real user behavior similar to our model of client's persistence, and (2) the arrival of new clients--not their requests--is independent of the progress of existing ones. 

Traditionally, client emulators are built over the popular thread abstraction, where each thread represents a single client. Threading is attractive because straight-line codes (or scripts) are easily parallelized to emulate multiple clients.  However, existing threading implementations become problematic when a very large number of clients are emulated because of their high resource and management overhead. Two mechanisms are commonly used to overcome this limitation.  One mechanism, which is used by SPECWeb, simply limits the number of threads or clients; a client waits for response from the server before issuing a new request.  This, however, is shown to significantly impact the performance numbers since it does not sustain a specific request rate. Instead of threads, an event-driven design that is based on non-blocking I/O is used to sustain the request rate (e.g., httperf) but this incurs added complexity.

The heart of the design challenge is scalability.  On one hand, client emulators should be powerful enough to model complex concepts such as persistence. On the other hand, thousands of such clients must be emulated in order to stress-test high-capacity servers. To address these issues, we designed Eve, an efficient and highly-customizable client emulation tool for creating client models. Eve consists of two powerful abstractions. The first is an optimized user-level thread library that simplifies client emulation.  This library combines the ease-of-programming of traditional threading implementations with the improved performance of event-driven counterparts.  The second abstraction is a distributed shared variable (DSV) architecture that simplifies the management of, and supports the scalability of, clients across multiple machines. This efficient communication abstraction is needed to facilitate client management, data collection, and event synchronization. Eve uses a modular design to integrate both abstractions into an extendable tool that can stress-test powerful multi-tier servers.

Design

Eve is designed with future extensibility in mind and can be viewed a programming environment that provides powerful mechanisms for specifying and emulating a wide range of client models as well as taking measurements. It is implemented using the C programming language on the Linux operating system.  With the exception of a single header file, cross-platform porting is straightforward.  Eve has the capability of running as many instances of a given client model (or a combination of models) as necessary to analyze servers under test. As the need for more resources arises, Eve can be configured to add more machines to expand its set of usable hosts.

The figure above shows a high-level design of Eve in which it is decomposed into three basic components:

1. Modules constitute the basic building blocks of Eve. They are connected through a standard interface, and offer a great deal of flexibility in extending and scaling Eve's functionality.
2. Communication Core glues all modules together and because it is distributed, Eve easily scales beyond the boundary of a single machine.
3. I/O-Threads are used to simplify the creation of a single client. This highly optimized user-level thread library abstracts away all complexity of using non-blocking I/O without losing any performance advantage.

Eve Modules

To allow a high degree of customization, Eve implements all of its components, including its core components, as modules. Eve modules are implemented as plug-in dynamic link libraries (DLLs) where each DLL is required to implement two interface points, eve_init and eve_exit, for initialization and clean-up, respectively. Modules are free to export other functions to provide added functionality. A unique identifier is associated with each module and is part of the modules configuration parameters. This identifier allows modules to reference each other's shared variables though the Communication Core (described shortly), to update or obtain measurement data. Furthermore, the identifier is used to load and unload modules using the eve_load and eve_unload primitives, respectively.

There are two types of modules: client and core. Core modules are used to initialize all client modules, facilitate communication between these modules, and at the end of an experiment, allow for the reporting of measurements. The bare-bone Eve (shaded boxes in the figure) consists of a loader module that initializes the Communication Core, sets up environment variables based on the user-defined configuration parameters, then loads the Core modules. A client model (e.g., one that mimics user shopping behavior) is encapsulated in a separate module and uses the interface points to connect to other modules.  The figure highlights a generic design of a client module where it shows the interaction between the client and the Communication Core, and other modules.

Communication Core

Eve implements its communication core using a distributed shared variable (DSV) layer. It is used by individual clients to communicate among themselves to share data and perform synchronization.  Since typical emulation data is mostly shared at the beginning--during the initialization stage--and at the end--during the statistic collection stage--with the occasional data sharing during synchronization, a simplified version of Munin is used to implement this layer. Our design follows a client/server model, where modules (the clients) use a thin stub to establish a communication channel to the DSV repository (the server).

Eve provides two standard calls to access shared and synchronization variables.  They are eve_in and eve_out, which are a restricted version of Linda's in and out method. This abstraction hides the majority of the underlying variable management from the programmer while unifying the access method to all shared variables. Release consistency is chosen based on the observation that most statistical values are updated only at the end of a simulation or at relatively coarse progress snapshots.  Enforcing stronger consistency would unnecessarily increase overhead.

To allow synchronization, Eve uses two other functions eve_lock and eve_unlock.  These provide a simple mutex context. A client that tries to lock a variable that is already locked by another client will block until that client variable is unlocked.  If more than one client are waiting for a lock, then the highest-priority one will run first. Eve supports priority inheritance to prevent priority inversion.

Collecting statistics is simplified by the DSV where measurements can be updated by all participating clients. What is important, however, is that collecting measurements does not change even in situations where clients are distributed across multiple hosts. In fact, Eve successfully hides away most of the complexity of managing distributed clients, scaling clients to stress-test powerful servers, and collecting measurements.

I/O-Threads

The thread abstraction, because of its simplicity, is an ideal solution for emulating a wide variety of client modules. However, one must be careful about the design of the thread architecture when thousands of simultaneous threads must be supported.  For example, if clients time out after 30 seconds, to sustain a rate of 200 requests/sec (which can be handled by off-the-shelf web servers), then the worst-case number of needed threads is 200 * 30 = 6000 threads, one thread for each outstanding request.

Eve implements a lightweight thread library where each new request or client can be easily handled by a separate thread. At the time of creating this multi-threading library, existing user-level libraries did not provide the necessary functionality. Recently, pth has advanced to provide similar functionality. With potentially thousands of threads that need to be supported, a natural question is why would our scheme work while others don't? We identify five key design decisions that enabled us to answer this question:

1. Integrate I/O handling into the design of the thread library to provide greater control over potentially expensive I/O calls: By transforming blocking I/O calls into non-blocking ones, Eve can schedule and execute a different task.  Whenever the I/O call is completed, the calling thread is then placed in a ready queue. These calls are then a natural place where a thread would yield to another ready thread.
2. Enforce cooperative or non-preemptive multitasking to maximize processor usage time and minimize switch overhead: Implementing cooperative multitasking was a natural consequence of using non-blocking I/O.  By avoiding the use of preemption, we eliminated the need for signals, synchronization (to lock thread control block structures) as well as the need to support reentrant functions. While implementing preemptive multitasking, as opposed to cooperative multitasking, is not difficult, as we will see later, doing so would have prevented other optimizations. One may wonder if cooperative multitasking negatively influences the accuracy of real-time events. While this is true to some extent, usual client models are I/O-heavy, implying a short period between two scheduling operations.
3. Support priority queues to give soft real-time tasks, such as timers, priority over other tasks: Supporting a large number of threads requires more than just a simple FIFO scheduler, typical of most user-level thread implementations.  Since some tasks, especially timeouts, should be handled immediately, priority scheduling was necessary to support these soft real-time requirements.
4. Use time tagging to timestamp when a thread is ready; this way the thread can compensate for the long wait caused by supporting a large number of threads:  The time delay between two scheduling -slices may be long for a large number of threads. For example, if there are 1000 ready threads and each thread takes 1 msec, then there is an average thread delay of 0.5 sec. This delay can easily degrade accuracy of measurement and is common in most client emulators. To minimize this inaccuracy, Eve uses \textit{time tagging} where each thread is tagged with the time when it became ready.  A simple call can then be used to retrieve this time and can be easily incorporated into the client code.
5. Minimize resource and memory consumption per thread to support a large number of threads. Eve provides a work stack for functions that require large stack space. Memory usage can explode easily when supporting a large number of threads. Eve uses a 4KB stack per thread. Smaller stacks can also be used. While we recognize that 4KB may not be enough in some cases, Eve introduces a large, shared, work stack that can be used during one scheduling-slice of any thread. Since threads are only switched at specific instances, they can safely take advantage of the larger stack space. This is another advantage for cooperative multitasking.

Release Notes:

We are working diligently to have an official release of Eve.  Please check again soon.

Publications on Eve

• H. Jamjoom and K. Shin. "Eve: A Scalable Network Client Emulator." University of Michigan Tech Report, CSE-TR-478-03

Copyright© 1997-2004 Hani Jamjoom. All Rights Reserved.
Hosting is provided by www.jamjoom.net