Saturday

Application Server Performance Tuning

Application Server Memory -
  • Ensure not to have too many PSAPPSRV processes booted. Extra PSAPPSRV not only take up memory space and CPU time, but also each PSAPPSRV is not sufficiently cached. Fully caching PSAPPSRV takes more time.
  • Check memory utilization on application server when there is a performamnce problem. Check for swapping using memory monitoring scripts. The total memory usage and virtual memory size should not be greater than 70% of the real memory on the server.
  • HCM uses about 100-150MB per application server process, FSCM uses about 150-300MB.
  • When page/sec is greater than 100 hard pages per second, there is a swapping problem, reduce number of PSAPPSRV processes or add more memory.

Sizing and Spawning - When a domain is started, tuxedo starts minimum number of processes, and spawns additional instances on demand upto maximum number reached. If number of services on queue exceeds high-water level for at least create_time seconds, a new server is spawned. If load drops below low-water for at least terminate_time seconds, a server is deactivated. Each server process autonomously determine whether to spawn another server processes. After processing a message from the queue, a server executes the code that does the server-spawning management.
Spawn Threshold=1, 600:1, 1

Minimum number of the servers configured should be adequate to cope with typical loads and less reliance should be placed on tuxedo spawning.


Too few Server Processes: Concurrency- If there are too few server processes then requests build up on the queues increasing contention on the queue so it is advisable to determine how many service requests executes concurrently and so a reasonable minimum number of server processes.

Too many Server Processes: Database Overload- Reduce number of application servers to a level where database can cope with the load. Also reduce number of query servers (PSQRYSRV) and other batch server processes.

Running out of CPU - Restrict number of PSAPPSRVs such that CPU on application server is not fully utilized. Thus PSAPPSRV will not contend for CPU and service time will not increase due to CPU contention.


Operating Service Priority - The priority of application server processes can be adjusted on UNIX systems only using -n option. When there is free CPU, the adjusted priority does not affect amount of CPU that the process can utilize, however, when there is no CPU, processes with a low priority get less time on CPU. The priority of a process is lowered by nice command on UNIX systems.

Service Priority - changepriority(chp) command can be used from tmadmin interface to dynamically change the priority of a service. Default priority is 50 for all the services and a possible use of this option is to give ICSript a higher priority so that navigation responds more quickly. A lower priority message does not remain forever enqueued as every 10th message is retrieved from the front of the queue.

Multiple Domains on Production System - Configure a servlet (psp) to reference a second application server domain by updating psserver value in servlet's configuration.properties file of the webserver. This makes possible the configuration changes in one domain without requiring system downtime. the users will automatically re-connected to surviving domain without any loss of service. 

Configuring Weighted Load Balancing and Failover - the psserver property in PIA's configuration.properties file enables the weighted load balancing. When we have more than one app server configured in system, we list them separated with a comma and specify the load characteristics using #n. n can be between 1 to 10. This way we direct greater load to a specific app server. e.g. pserver=app1:9000#5, app2:9050#1 Here app1 will receive 5 times more work than app2.

Failover can be configured in 2 ways - weighted backup or sequential backup. The failover domains are put within braces -{}. e.g. psserver=app1:9000#5{app3:port#5, app4:port#1}, app2:9050#1. In this case if primary app server app1 goes down, the failover servers app3 and app4 will serve requests as per the weight specified. The sequential failover means configuring multiple backup servers (separated by a semicolon-;) e.g. psserver=app1:9000#5{app3:port; app4:port}, app2:9050#1
Here if app1 goes down, app3 will handle the requests and in case app3 too goes down, app4 will start serving requests.

The DynamicConfigReload property enables PIA domain loading of psserver connect string dynamically. DynamicConfigReload=1  allows the system to automatically detect changes to psserver string without manually restarting the PIA domain.

Jolt Session Pooling - it is enabled by default and when enabled, the webserver connections are shared between user sessions reducing the usage of system resources. Session pooling is controlled by setting joltPooling=true or false in configuration.properties file.


Cycling Application Server without Shutting down AppServer - Stop and boot command of tmadmin interface can be used to stop and restart individual server process. Instead of recycling the whole domain, recycle one at a time PSAPPSRV in order to clear in-memory cache. This way there will not be any loss of service. 

Few Tuxedo Services/Functions -
tpservice()- this function is call for service routine, first parameter is the name of the service and fourth is the size of the message passed to server.
tpreturn()- Returns result from service routine.
tpalloc()- allocates memory to buffer, last parameter is number of bytes requested.
tpcall()- sends a request to a service subroutine and synchronously waits for a reply.


* You should always have at least two instances of any server process that has a non-zero recycle count - this is because when a server process reaches its recycle count, it shuts itself down and its queue (whether its a shared queue) gets deleted. Now, the services advertised by the process are also removed from the BB so if a service request arrives in app domain before the server is restarted the JSH will through an error finding after querying the BB.

The exceptions to above is -

PSSAMSRV as it is only used by windows 3-tier clients (PS/Query)
PSMSGDSP, only a single process can be configured.
PSAESRV, because in the process scheduler each PSAESRV has its own queue.



Oracle WebLogic Performance tips -
IB Performance & Tuning -

1 comment:

Followers