Application Server Performance Tuning

Application Server Memory -

• Ensure not to have too many PSAPPSRV processes booted. Extra PSAPPSRV not only take up memory space and CPU time, but also each PSAPPSRV is not sufficiently cached. Fully caching PSAPPSRV takes more time.
• Check memory utilization on application server when there is a performamnce problem. Check for swapping using memory monitoring scripts. The total memory usage and virtual memory size should not be greater than 70% of the real memory on the server.
• HCM uses about 100-150MB per application server process, FSCM uses about 150-300MB.
• When page/sec is greater than 100 hard pages per second, there is a swapping problem, reduce number of PSAPPSRV processes or add more memory.

Sizing and Spawning - When a domain is started, tuxedo starts minimum number of processes, and spawns additional instances on demand upto maximum number reached. If number of services on queue exceeds high-water level for at least create_time seconds, a new server is spawned. If load drops below low-water for at least terminate_time seconds, a server is deactivated. Each server process autonomously determine whether to spawn another server processes. After processing a message from the queue, a server executes the code that does the server-spawning management.
Spawn Threshold=1, 600:1, 1

Minimum number of the servers configured should be adequate to cope with typical loads and less reliance should be placed on tuxedo spawning.


Too few Server Processes: Concurrency- If there are too few server processes then requests build up on the queues increasing contention on the queue so it is advisable to determine how many service requests executes concurrently and so a reasonable minimum number of server processes.

Too many Server Processes: Database Overload- Reduce number of application servers to a level where database can cope with the load. Also reduce number of query servers (PSQRYSRV) and other batch server processes.

Running out of CPU - Restrict number of PSAPPSRVs such that CPU on application server is not fully utilized. Thus PSAPPSRV will not contend for CPU and service time will not increase due to CPU contention.


Operating Service Priority - The priority of application server processes can be adjusted on UNIX systems only using -n option. When there is free CPU, the adjusted priority does not affect amount of CPU that the process can utilize, however, when there is no CPU, processes with a low priority get less time on CPU. The priority of a process is lowered by nice command on UNIX systems.

Service Priority - changepriority(chp) command can be used from tmadmin interface to dynamically change the priority of a service. Default priority is 50 for all the services and a possible use of this option is to give ICSript a higher priority so that navigation responds more quickly. A lower priority message does not remain forever enqueued as every 10th message is retrieved from the front of the queue.

Multiple Domains on Production System - Configure a servlet (psp) to reference a second application server domain by updating psserver value in servlet's configuration.properties file of the webserver. This makes possible the configuration changes in one domain without requiring system downtime. the users will automatically re-connected to surviving domain without any loss of service. 

Configuring Weighted Load Balancing and Failover - the psserver property in PIA's configuration.properties file enables the weighted load balancing. When we have more than one app server configured in system, we list them separated with a comma and specify the load characteristics using #n. n can be between 1 to 10. This way we direct greater load to a specific app server. e.g. pserver=app1:9000#5, app2:9050#1 Here app1 will receive 5 times more work than app2.

Failover can be configured in 2 ways - weighted backup or sequential backup. The failover domains are put within braces -{}. e.g. psserver=app1:9000#5{app3:port#5, app4:port#1}, app2:9050#1. In this case if primary app server app1 goes down, the failover servers app3 and app4 will serve requests as per the weight specified. The sequential failover means configuring multiple backup servers (separated by a semicolon-;) e.g. psserver=app1:9000#5{app3:port; app4:port}, app2:9050#1
Here if app1 goes down, app3 will handle the requests and in case app3 too goes down, app4 will start serving requests.

The DynamicConfigReload property enables PIA domain loading of psserver connect string dynamically. DynamicConfigReload=1  allows the system to automatically detect changes to psserver string without manually restarting the PIA domain.

Jolt Session Pooling - it is enabled by default and when enabled, the webserver connections are shared between user sessions reducing the usage of system resources. Session pooling is controlled by setting joltPooling=true or false in configuration.properties file.


Cycling Application Server without Shutting down AppServer - Stop and boot command of tmadmin interface can be used to stop and restart individual server process. Instead of recycling the whole domain, recycle one at a time PSAPPSRV in order to clear in-memory cache. This way there will not be any loss of service. 

Few Tuxedo Services/Functions -
tpservice()- this function is call for service routine, first parameter is the name of the service and fourth is the size of the message passed to server.
tpreturn()- Returns result from service routine.
tpalloc()- allocates memory to buffer, last parameter is number of bytes requested.
tpcall()- sends a request to a service subroutine and synchronously waits for a reply.


* You should always have at least two instances of any server process that has a non-zero recycle count - this is because when a server process reaches its recycle count, it shuts itself down and its queue (whether its a shared queue) gets deleted. Now, the services advertised by the process are also removed from the BB so if a service request arrives in app domain before the server is restarted the JSH will through an error finding after querying the BB.

The exceptions to above is -

PSSAMSRV as it is only used by windows 3-tier clients (PS/Query)
PSMSGDSP, only a single process can be configured.
PSAESRV, because in the process scheduler each PSAESRV has its own queue.


Oracle WebLogic Performance tips -
IB Performance & Tuning -

Application Server Administration

tmadmin Command Line Interface - there are several command available via tmadmin interface. A few to list are-

1) printserver(psr) - prints information for application and administrative servers.

Different available options with this command are-

>psr -q [qaddress] 
>psr -m [machine]
>psr -g [groupname]
>psr -i [srvid]


2) printservice (psc) - prints information about the services.

Available options are -
>psc -m [machine]
>psc -g [groupname]
>psc -i [svrid]
>psc -q [qaddress]

3) serverparms (srp): prints parameters associated with servers specified with groupname and svrid.

 
>srp -g [groupname] -i [svrid]



4) bbparms (bbp) - prints summary of bulletin board parameters.
5) bbstats (bbs) - prints summary of BB statistics.
6) printclient (pclt) - prints information about client processes.
7) printgroup (pg) -m [machine name] /-g [groupname] - prints server group table for specified machine or default machine.
6) printqueue (pq) - prints information about the specified queue or all queues.
> pq -q [qaddress]
7) serviceparms (scp) - parameters associated with a service specified by groupname, svrid and service. e.g.
>scp -g APPSRV -i 100 -sICScript

8) shutdown - When a application server domain is shut down by psadmin, the tmshutdown command is issued. It has same parameters as shutdown command with tmadmin. >shutdown [-g groupname | -i svrid]

9) boot (b) - >b [-g groupname | -i svrid | -m minsrvs ]
e.g. boot -g PUBSUB,
       boot -g JSLGRP,
       boot -g JREPGRP etc.

10) boot -y  boots the domain normally, if there is an error encountered, it continues to boot unlike boot via psadmin which will shut the domain down. -w switch (boot -y -w) is for fast boot where servers are booted in parallel.
 
11) !set - command displays a list of environment variables and their current value.
12) stop -y  normal shutdown, cleans up IPC.
13) stop -y -c normal shutdown even if clients are attached.
14) stop -y -c -w0 stops domain even if clients are attached and waits for 0 (zero) seconds.
15) stop [-g groupname] - e.g. stop -g PUBSUB 
16) stop [-i  svrid] - Use this command to stop a server which is stuck trying to execute a request (e.g. ICPanel). If a request is consuming lot of memory as compared to other server processes then stop using this variant.

psadmin Command Reference -

To invoke psadmin from command line, locate PS_HOME/appserv>

psadmin -h displays help on psadmin commands.
psadmin -env displays information on current environment settings.
psadmin -c <command> -d <domain> e.g. >psadmin -c parallelboot -d hr91pd01

(psadmin -p <command> -d <domain> in case of Process Scheduler Server)

Some other psadmin commands are-
status,
qstatus,
cstatus,
cleanipc,
configure etc.


Purging Application Server Cache-Option available through psadmin under application server administration menu will purge application server cache even if application server is up and running.

Clean/Remove IPC Resources - IPC resources are OS resources e.g. message queues, shared memory, semaphores. If application server domain fails to shut down properly, stray IPC resources may remain on the system and when this happens, it is not possible to reboot the application server. There are two options to remove IPC resources. First there is an option available on psadmin menu and second the command to remove IPC resources -tmipcrm. The option -n lists all IPC resources and -y cleans IPC resources. The command reads PSTUXCFG and uses the complete path of this file as an parameter.

Application Server Performance Tuning

Application Server Configuration

These are the importnat application server configuration files -

• psappsrv.val,
• psappsrv.cfg,
• psappsrv.ubx,
• psappsrv.ubb,
• psappsrv.env,
• psappsrv.psx,
• psappsrv.lst,
• PSTUXCFG

psappsrv.val

This is the validation file for interactive configuration dialog. The responses given while configuring app server domain using psadmin utility are validated against values in this file.


psappsrv.cfg

It specifies most of the configuration of an application server domain. It is combined with psappsrv.ubx by the ubbgen utility to produce psappsrv.ubb and psappsrv.env. The variables that appear in psappsrv.ubx are used as part of tuxedo configuration, variables which are not there in psappsrv.ubx and are in psappsrv.cfg are directly read from this file by the application server processes. e.g. EnableDataMonitoring

Important sections/parameters in psappsrv.cfg
1) Startup Section -
Database Signon Settings - specifies how application server processes connect to database.
DBName
UserID - used to start the application server. [* Create a peoplesoft operator ID that has only the privilege to start the application server.]
2) Database Options - Database specific configuration options.

• UseLocalOracleDB = 0 (default); if set to 1, the server processes will attempt to make a direct shared-memory connection to the database. It works only if the application server and database server are on the same node.
• EnableDBMonitoring = 1(enabled by default); Enabling this parameter causes application server to write a string to the session information using DBMS_APPLICATION_INFO.SET_CLIENT_INFO package. Database session of peoplesoft processes can be therefore read from V$SESSION [SELECT CLIENT_INFO, PROGRAM FROM V$SESSION WHERE CLIENT_INFO IS NOT NULL;]

3) Security Section - Security settings

• Use the database for additional signon authentication.
• Validate Signon with Database - Default value is 0, If this option is set, user login will fail unless corresponding database user account is created for every application user.

4) Workstation and Jolt Listener -

• Multiplexing Factor - Tuxedo default parameter for the multiplexing factor is 10, means 10 clients per handler. Peoplesoft delivered default is 40 for both JSL and WSL.
• By reducing number of clients per handler, more WSH and JSH processes will be required, and more return queues to the handler process will be created. This will reduce contention on the return queues.
• Total number of clients that can connect to application server via the WSL is the maximum number of handlers multiplied by multiplexing factor. If the number of clients per handler is reduced, maximum number of handlers should be increased by the same factor to allow support to same number of concurrent connections.
• Timeout parameter - expressed in minutes. Peoplesoft default is 5 minutes, servlet or windows client silently authenticate itself to the application server.
• Tuxedo Jolt Compression Threshold - Set a very large compression threshold so no messages are large enough to be compressed. Servlet and application server should be located physically close to each other on same low latency network. A double compression is a waste of resource so this should be avoided if compression is enabled at hardware level.

5) Domain Settings - General setting for application server, generally the Domain name should be the name of the database. If there are multiple application servers on the same peoplesoft database, give each one a different Domain ID. DomainID appears in V$SESSION.CLIENT.INFO, and it is adentifiable which application server the session relates. Give Log Directory value here if app server logs are stored on different location.
6) Server Trace Settings - This should be enabled only in a limited test environment.  
7) Remote Call section - Synchronously initiate batch COBOL programs on application server from PIA rather than scheduling corresponding process on scheduler. When PSAPPSRV initiate sthe COBOL process, it is in fact in middle of executing routine, and PSAPPSRV cannot handle any other service until a remote call finishes and the service is complete. The risk here is if there are many remote calls and performance degrades then all the PSAPPSRV could be blocked and the application server will hang.
8) Peoplesoft Server Processes - The server definitions contain a number of parameters-

• Minimum and Maximum number of processes.
• Service Timeout - maximum amount of time that a service request can spend either on queue or active on the server process before it is timed out.
• Recycle Count - Server processes recycle after handling the number of number of services specified. Recycle count parameter helps in hiding memory leaks and relinquish allocated memory.
• Max Fetch Size - This parameter controls the amount of data that can be fetched by the SqlRequest or SqlQuery service executed by PSAPPSRV.

9) PSQCKSRV or PSQRYSRV - When a query is submitted from the PIA, the entire result set is returned to the servlet is a single tuxedo message, although a part of it may be displayed at a time.
10) Cache Settings


psappsrv.ubx:

Tuxedo template file used by ubbgen to generate psappsrv.ubb and psappsrv.env. It contains variables whose values are specified in psappsrv.cfg file. The output file contains the literal values of these variables.

Important sections/parameters in psappsrv.ubx

1) Environment Variables - are read by ubbgen with getenv() function, e.g. TUXDIR.
2) Configuration Variables - defined in Tuxedo template file. ubbgen looks up the values of these variables in psappsrv.cfg3) Special Variables - e.g. MAXSERVERS etc.4) Prompted Variables - all other variables are prompted, these specifies a number of Yes/No questions asked during configuration.5) TUXEDO Servers section - Default clause in this section specifies default parameters that apply to all servers unless overridden by the individual server definition.
CLOPT- command line options, e.g. a server can be started MAXGEN times in the GRACE period before it is marked as being in error. 

TMSYSEVT- the only server process delivered by BEA. It is not configured by default and is used only when Tuxedo Administration Console is to be used to administer the domain.
IPCKEY- this is ID of bulletin board shared memory segment. It is passed to PSWATCHSRV so that it can search BB for locked servers.

* A single Tuxedo domain can be split across more than one physical machines to permit load-balancing but peoplesoft neither uses nor support this.



Mandatory Servers in an application server domain -

1. PSWATCH SRV,
2. PSAPPSRV,
3. PSSAMSRV,
4. PSMONITOSRV.

PSWATCHSRV (Watch Server)- this does not connect to database and does not advertise or handles any service. Its job is to kill those server processes which are stuck and cant be shut down. There is only one instance of PSWATCHSRV, maximum and minimum number of instances of this server is hard-coded as one.

 Main Application Server (PSAPPSRV): The heart of application server domain which authenticates users and is responsible for generating pages, building HTML and processing business logic.

If PSQCKSRV and/or PSQRYSRV are configured in domain, then the services that they advertise are no longer advertised by PSAPPSRV. The services to be advertised are specified by -s parameter within server's command line options. There are 3 formats for this parameter-
1) -s@psappsrv.lst; this file specifies and lists all the services that are available with different servers.
2) -s<ServiceName> e.g. -sSqlQuery
3) -s<ServiceName:FunctionName> e.g. -sSqlQuery:SqlRequest


PSSAMSRV: This server process is only used by 3-tier clients and mostly when submitting requests to process scheduler. This server is mandatory and cannot be disabled. This server is never spawned.

Performance Monitor Server (PSMONITORSRV): This server is always configured and it has 3 functions-1) measuring tuxedo performance, 2) resolving Master/Slave failover for IB and 3) cancelling ad-hoc queries.



Quick Server (PSQCKSRV): Its a copy of PSAPPSRV but used for quick services. PSQCKSRV handles the very short duration services such as simple SQL queries. These queries get through the application server quickly and these have not to wait in the queue behind the larger and slower services. This server is never used by PIA.

Query Server (PSQRYSRV): Again, a copy of PSAPPSRV and handles the long running queries issued from PSQuery tool in PIA or from a 3-tier PSQuery client. It advertises two services  -sICQuery, -sSqlQuery:SqlRequest. If this server is enabled, then the service Timeout of this service is taken from the tuxedo service section for PSQRYSRV, not from PSAPPSRV.

Debug Server (PSDBGSRV): Application designer makes a persistent connection to this server and only one debug server can be configured in a app server domain. It runs on port 9500.

Real-Time Notification Server (PSRENSRV): It is a modified webserver used to send real time event notifications e.g. report related notifications to PIA users. Its primary purpose is to publish events to web browser.

Performance Collator Server (PSPPMSRV): The server collects performance metrics and inserts them into Performance Monitor tables- PSPMTRANSHIST, PSPMEVENTHIST etc.

A tools only database can be configured with multiple collators in a cluster to provide failover and scalability.

On Global Administration page (PeopleTools, Performance Monitor, Administration, Global Administration, you specify the PPMI URL stored in monitoring system database and the URL is used by PSPPMSRV to locate the PPMI servlet with which to register. The syntax is http(s)://<host:port>/ppmi/<site_name>/. You specify the PPMI userID and PPMI password which is passed by PSPPMSRV to PPMI servlet which verifies the userID/pswd and whether user has the permission to access ppmi servlet. This user must have a permission list with ppmi access selected (delivered one is PTPMCLNT)

Analytic Servers (PSANALYTICSRV): The server writes analytics - issues with running the application to log file.

Application Messaging Servers: There are up to 6 different server processes that can be configured to support application messaging(IB). Multiple instances of these server processes can be configured based on demand or in a heavy IB activity domain. These are -

PSBRKDSP, PSBRKHND
PSPUBDSP, PSPUBHND
PSSUBDSP, PSSUBHND

PSMSGDSP & PSMSGHND: Overhead of the application messaging can be significant in
a development environment so the developer template specifies these two messaging servers configuration.

7) Tuxedo SERVICES Section - This section defines different service attributes which you can set as required. These values affect load balancing, spawning of the servers and de-queuing of the requests. Peoplesoft delivers all services with same load and priority.

8) PS_ENVFILE Section -  This section contains information that peroplesoft uses to generate the server process environment file which is read by every server process as it starts and it sets the environment variables for that server process.


psappsrv.ubb: This file is generated by ubbgen and serves as domain configuration source file, all the variables from psappsrv.ubx are resolved to literals in this file. It is useful to examine this file to see what is currently set, however this file should never be modified. The changes should be made only in psappsrv.ubx and psappsrv.cfg.


psappsrv.env: Generated by ubbgen and is read by each application server process as it starts.


psappsrv.cfx (Configuration Template files): These files are template files which stores the delivered configuration templates - developer, large, medium and small. 

psappsrv.psx: This is template file for messaging section of .cfg and .ubx files.



PSSTUXCFG

The peoplesoft process ubbgen reads psappsrv.cfg and psappsrv.ubx and generates psappsrv.ubb and psappsrv.env files. Tuxedo utility - tmloadcf complies the psappsrv.ubb into binary configuration file PSTUXCFG. When the domain is started, the BBL is established and it reads the PSTUXCFG and starts the rest of the domain. ubbgen and tmloadcf processes are run back to back without a pause by psadmin utility.


pswinsrv.cfg: Configuration file that is used when app server and/or process scheduler is configured to start as a windows service. Its parameters are -

Service Start Delay - Delay in seconds. Its period of time that elapse before starting the PS windows service.
Application Server Domains- Comma separated list of domains those will be started with this service.
Process Scheduler Domains - Comma separated list of PRCS domains.


Application Server Administration

Oracle Database Basics

Physical Backups and Logical Backups: A backup is a copy of data from your database that can be used to reconstruct that data. Backups can be divided into physical backups and logical backups.

Physical backups are backups of the physical files used in storing and recovering your database, such as datafiles, control files, and archived redo logs. Ultimately, every physical backup is a copy of files storing database information to some other location.

Logical backups contain logical data (for example, tables etc.) exported from a database with an Oracle export utility and stored in a binary file, for later re-importing into a database using the corresponding Oracle import utility.

Physical backups are the foundation of any sound backup and recovery strategy. Logical backups are a useful supplement to physical backups in many circumstances but are not sufficient protection against data loss without physical backups.

Recovery from Backup

The backup strategy determines how gracefully you recover the lost data when user error such as a manual mis-step or due to an error in application logic does cause data loss. Any database file can be vulnerable to a media failure. The appropriate recovery technique following a media failure depends on the files affected and the types of backup available.

Physical Database Structures Used in Recovering Data

·         Datafiles and Data Blocks

·         Redo Logs

·         Control Files

·         Undo Segments

Database Recovery Process: Reconstructing the contents of all or part of a database from a backup typically involves two phases:

1)    Restoring datafiles from backup- To restore a datafile or control file from backup is to retrieve the file onto disk from a backup location and make it available to the database server.

2)    Reapplying changes to the file since the backup from the archived and online redo logs, to bring the database to a desired SCN since the backup.

Complete, Incomplete and Point-In-Time Recovery: Complete recovery is recovering a database to the most recent point in time, without the loss of any committed transactions. In an Incomplete recovery, also known as point-in-time recovery, the goal is to restore the database to its state at some previous target SCN or time. For example, to undo the effect of a user error, such as dropping or deleting the contents of a table, you may want to return the database to its contents before the delete occurred.

ARCHIVELOG and NOARCHIVELOG Mode

ALTER DATABASE ARCHIVELOG;

ALTER DATABASE NOARCHIVELOG;

The redo logs of your database provide a complete record of changes to the datafiles of your database (with a few exceptions, such as direct path loads).

Database can be run in one of two modes: ARCHIVELOG mode or NOARCHIVELOG mode. In ARCHIVELOG mode, Oracle copies filled online redo logs to one or more archive destinations before it can be reused. In NOARCHIVELOG mode, the online redo log groups are simply overwritten when the log is reused.

* Some performance overhead is associated with the background processes ARC0 through ARCn which copy filled online redo logs to the archiving destinations.

* In NOARCHIVELOG Mode, you cannot perform online backups of your database. You must shut your database down cleanly before you can take a backup.

Archived Redo Log: A copy of one of the filled members of an online redo log group made when the database is in ARCHIVELOG mode. After the LGWR process fills each online redo log with redo records, the archiver process (ARC0 through ARCn) copies the log to one or more redo log archiving destinations. This copy is the archived redo log.

Archived Redo Log Data Dictionary views:

-       V$ARCHIVED_LOG

-       V$PARAMETER
-       V$LOGHISTORY

Log Sequence Number: A number that uniquely identifies a set of redo records in a redo-log file, when Oracle fills one online redo log file and switches to a different one, it automatically assigns the new file a log sequence number.

Log Switch: A log switch is a point at which LGWR stops writing to the archive redo log file and switches to the next available redo log. LGWR switches when either the active log file is filled or you force a switch manually.

ALTER SYSTEM SWITCH LOGFILE;

Mirroring: Maintaining identical copies of data on one or more disks. Mirroring is performed on duplicate hard disks at the operating system level, so that if one of the disks becomes unavailable, the other disk can continue to service requests without interruptions. When mirroring files, Oracle writes once while the operating system writes to multiple disks; when multiplexing files, Oracle writes the same data to multiple files.

Multiplexing-

-       online redo logs: The automated maintenance of more than one identical copy of the online redo log.

-       control file: The automated maintenance of more than one identical copy of a database's control file.

-       archived redo logs: The Oracle archiver process is able to archive multiple copies of a redo log.

RESETLOGS: A method for opening a database that archives any current online redo logs (if using ARCHIVELOG mode), resets the log sequence number to 1, and clears the online redo logs.

An OPEN RESETLOGS operation is required after incomplete recovery or recovery with a backup control file.

System Change Number (SCN): A stamp that defines a committed version of a database at a point in time. Oracle assigns every committed transaction a unique SCN.

SYSTEM tablespace: It contains Oracle data dictionary for a database, which is the metadata that describes the complete contnents of the database. All datafiles contained in this tablespace must be online for Oracle to function.

Transportable tablespace: Transporting a tablespace into a database is like creating a tablespace with preloaded data. This is a feature that transports a set of tablespaces from one database to another, or from one database to itself.

    
Undo tablespace: A dedicated tablespace that stores only undo information when a database is run in Automatic Undo Management mode.

SYSAUX tablespace - whenever statistics in the dictionary are modified, old versions of statistics are saved automatically for future restoring. This history information is stored in SYSAUX tablespace. The MMON (Manageability Monitor) performs the automatic purge and removes all stats history older than - current time - statistics history retention (by default 31 days) or time of recent analyze in the system - 1 . MMON by default performs the purge of the optimizer stats history automatically, but it has an internal limit of 5 minutes to perform this job. MMON will do this activity once in 24 hrs. If the operation takes more than 5 minutes, then it is aborted and stats is not purged.

SYSTEM tablespace - every oracle database has a tablespace named SYSTEM, which Oracle creates automatically when the database is created. The SYSTEM tablespace is always online when the database is open. The SYSTEM tablespace always contains the data dictionary tables for the entire database.

* Once a tablespace is locally managed, it cannot be reverted back to being dictionary managed.

Default Temporary Tablespace - it is used as an intermediate storage while processing transactions e.g. sort operations etc. Wwhen the SYSTEM tablespace is locally managed, you must define a default temporary tablespace when creating a database. A locally managed SYSTEM tablespace cannot be used for default temporary storage.