This section deals with bottlenecks in the openUTM and BCAM environments which can lead to longer response times in the various phases. It indicates how these bottlenecks can be detected by means of measurements and how they can be eradicated or alleviated.
Phases 1 and 10: Network runtimes
General information on the LAN connection is provided in section "Network connection".
Phase 2: Waiting until the incoming message has been accepted by BCAM (Inbound Flow Control)
Mechanisms for controlling the data flow protect the transport system from being flooded by data from the partners. These ensure, on a connection-specific basis, that a (high-speed) sender is matched to the processing speed of a (low-speed) recipient.
In this way BCAM protects its memory pool, which is available to all applications and connections. During phases in which heavy use is made of the BCAM pool, BCAM prevents its partners from sending messages into the BS2000 system. This can result in increased response times.
Longer or persistently high utilization of the BCAM pool can be caused by applications which fetch their messages slowly. A pool size which is too small for the processing performance required can also result in a heavy workload on the BCAM pool.
openSM2 provides messages for analyzing the utilization of the BCAM pool, see the RESPONSETIME monitoring routine in the “openSM2” manual [18 (Related publications)].
BCAM itself enables you to see whether a bottleneck has occurred by entering the console command /BCMON and by analyzing the values of the console message BCA0B21.
Brief description of the procedure:
Determine the maximum values set (optional) using /SHOW-BCAM-PARAMETERS PARAMETER=*LIMITS
Activate BCAM monitoring (output of the measurement values every <n> seconds): /BCMON MODE=ON,RECORD=RES-MEM,SEC=<n>
Describe the main output values in console message BCA0B21:
USED-BCAM-MIN-I: Minimum BCAM buffer allocation for incoming messages in KB
LIMIT-ACT: Current upper limit for the buffer allocation in KB
Indicator for high utilization of the BCAM buffer by incoming messages:(USED-BCAM-MIN-I) * 4 > LIMIT-ACT
As an alternative to or to complement the above-mentioned points: /BCSET THRESHOLD-MSG=ON can be used to request a warning message.
Console message BCAB021 is then output if BCAM holds up the incoming messages for more than 5 seconds on account of a pool bottleneck (or if it cannot accept any send requests from the application, see phase 7). This warning message is disabled using /BCSET THRESHOLD-MSG=OFF.
Optimization measure in the event of a bottleneck in the BCAM pool: The maximum permissible threshold value should be significantly increased using the /BCMOD RESMEM=<n> command (<n>=3*LIMIT-ACT would be ideal).
Phases 3 and 8: Processing the incoming/outgoing message in BCAM
The time values for these phases should be in the low single-digit range (in milliseconds).
Phase 4: Waiting for openUTM
The time which elapses between a job being issued to openUTM and acceptance of the job is referred to as the INWAIT time. It is determined for each application and is included in the INPROC time.
openSM2 records the INWAIT times (and also the INPROC , REACT and OUTPROC times) in 5 time intervals which are known as buckets. Buckets can only be set up globally, not on an application-specific basis. To simplify monitoring, intervals should be classified in such a manner that all times which are not acceptable are assigned to the last interval (overrun value).
Example
The normal response time is 100 - 200 ms (typical for large /390 servers). Short-term fluctuations up to a factor of 2 should be tolerated. However, no lengthy doubling of the response time should be tolerated.
The buckets in openSM2 should consequently be defined so that they ensure that all INWAIT times which are equal to or greater than 400 ms are counted in the overrun interval, e.g. with:
/SET-BCAM-CONNECTION-PARAMETER INWAIT-BUCKETS=(50,100,200,400)
This statement defines the buckets in such a way that all wait times < 50 ms are counted in the first interval, all wait times between 50 and 100 ms in the second interval, all wait times between 100 und 200 ms in the third interval, all wait times between 200 and 400 ms in the fourth interval, and all wait times > 400 ms in the overrun interval.
The INSPECTOR or ANALYZER component of openSM2 must then be used (for every application) to monitor the measurement values of the overrun interval. The values are output as a percentage (in relation to the time values recorded in the monitoring cycle) of this application. A percentage of 10 or higher indicates that bottlenecks occurred when the jobs were accepted by the UTM tasks.
Measuring option in openUTM
The UTM command KDCINF STAT is used to output a number of useful UTM measurement values, see the openUTM manual “Administering Applications” [20 (Related publications)]. The output value %Load provides important information for the analysis of whether the number of UTM tasks could constitute a bottleneck. This value specifies the average utilization of all UTM tasks in the last monitoring cycle. At least a short-term bottleneck is indicated by a value greater than 90 (%).
Threshold value monitoring of the number of UTM tasks using openSM2
An imminent UTM task bottleneck can be recognized from the UTM application monitoring report of openSM2. For this purpose the values for the duration of a transaction in seconds (DT), the number of transactions per second (TX), and the number of UTM tasks for the application (UT) must be ascertained from this report.
This enables the average utilization of the UTM tasks of an application to be calculated: Load (in %) = 100 * DT * TX / UT.
In INSPECTOR of openSM2 this calculated value can be subjected to threshold value monitoring. If a value of 90 (%) is exceeded, an email to this effect can, for example, be generated for systems support.
Optimization measure
The UTM command KDCAPPL TASKS=<n> enables the number of UTM tasks to be increased on an application-specific and step-by-step basis until the INWAIT times are acceptable. The optimum number of tasks ascertained in this way should be entered in openUTM’s start parameter file. It will then be effective the next time the application is started. If <n> exceeds the maximum value defined in the KDCDEF run, this maximum value must be increased and a new KDCDEF run must be started.
When the number of UTM tasks changes, the number of TAC classes and the number of tasks in these TAC classes must also be taken into account. Allowance must also be made for a certain buffer for load peaks.
Note
Only if appreciably more UTM tasks than required are started can this lead to slight performance losses as a result of slightly higher utilization of the main memory and CPU.
Phases 5 and 6: Processing the transaction
The time accumulated in this phase (and in phase 7) is specified as the REACT time in the BCAM Connection Report.
Tuning I/O performance when accessing the KDCFILE
In addition to the measures relating to the hardware and in BS2000 which are described in this manual, the performance of applications with high transaction rates can also be enhanced in openUTM by means of optimized write accesses to the KDCFILE.
To do this, the page pools and/or the restart area can be exported from the KDCFILE in openUTM. These exported areas are then distributed over a number of volumes.
Example:
The page pool is to be distributed to 2 public volumes, the restart area to 4 public volumes:
/CREATE-FILE FILE-NAME=<filebase>.P01A, - SUPPORT=*PUBLIC-DISK(VOLUME=<v1>,SPACE=*RELATIVE(PRIMARY-ALLOCATION=666)) /CREATE-FILE FILE-NAME=<filebase>.P02A, - SUPPORT=*PUBLIC-DISK(VOLUME=<v2>,SPACE=*RELATIVE(PRIMARY-ALLOCATION=666)) /CREATE-FILE FILE-NAME=<filebase>.R01A, - SUPPORT=*PUBLIC-DISK(VOLUME=<v3>,SPACE=*RELATIVE(PRIMARY-ALLOCATION=300)) /CREATE-FILE FILE-NAME=<filebase>.R02A, - SUPPORT=*PUBLIC-DISK(VOLUME=<v4>,SPACE=*RELATIVE(PRIMARY-ALLOCATION=300)) /CREATE-FILE FILE-NAME=<filebase>.R03A, - SUPPORT=*PUBLIC-DISK(VOLUME=<v5>,SPACE=*RELATIVE(PRIMARY-ALLOCATION=300)) /CREATE-FILE FILE-NAME=<filebase>.R04A, - SUPPORT=*PUBLIC-DISK(VOLUME=<v6>,SPACE=*RELATIVE(PRIMARY-ALLOCATION=300))
In addition, the following parameters must be modified in the MAX statement in KDCDEF: PGPOOLSFS=2, RECBUFFS=4.
The files defined above are then used in the KDCDEF run. In this case the KDCDEF program may modify the values for PRIMARY- and SECONDARY-ALLOCATION. Without the aforementioned commands, KDCDEF would create the files itself (without volume assignment).
The new files are used after openUTM has been restarted.
Controlling the UTM jobs by means of TAC classes
Similarly to category control using PCS (see section "PCS concept"), transactions in openUTM can be assigned to so-called “TAC classes”. These TAC classes can be controlled using two methods which cannot be combined:
Priority control
Process limitation
Details of job control in openUTM are provided in the openUTM manual “Generating Applications” [21 (Related publications)].
Recommendations for use:
When the TACs are distributed to TAC classes, it must be ensured that higher-priority TAC classes are not held up by TACs from lower-priority TAC classes (use of blocking calls).
You are recommended not to define more than 3 or 4 TAC classes.
Priority control is used above all to ensure that long-running TACs from low-priority classes do not hinder short-running TACs from higher-priority TAC classes in the event of (short-term) overloads. As a result, high-priority transactions are given preference using all the started processes.
The process limitation is used to ensure that long-running TACs cannot hinder short-running TACs. The advantage of this variant is that it always guarantees that enough free UTM tasks are available for new (and important) TACs.
In addition, TACs can be controlled by specifying the RUNPRIO in the TAC statement. However, this is only recommended for highest-priority TACs and must be coordinated with the priority structure of all tasks which run in BS2000.
Further performance information on openUTM:
Monitoring the cache hit rate
The UTM command KDCINF STAT (see "Measuring option in openUTM" in phase 4 above) should be used (at least occasionally) to check whether the hit rate in the UTM cache’s page pool is high enough. When the value is below 90 (%), the UTM cache may need to be enlarged (see the openUTM manual “Generating Applications” [21 (Related publications)]).
Use of UTM-F
This UTM variant is suitable mainly for applications with “retrieval” accesses and no or only a few updates. It is used to avoid I/O operations in the case of reduced functionality (with regard to failsafe performance and restart).
Phase 7: Waiting before BCAM
This phase is included in the REACT time in the BCAM Connection Report (see phase 5).
As in phase 2, BCAM can be subject to the partner’s control mechanisms for controlling the data flow. This can be recognized from ZWR values > 0 (ZWR=Zero Window Receive) in the openSM2 report BCAM. In such situations BCAM accepts a certain volume of data to be transferred from the application, but refuses to accept further data when a threshold value is exceeded.
BCAM also delays the acceptance of data in phases in which the BCAM pool is heavily utilized. Lengthy or permanently high utilization of the BCAM pool can be caused by connections which are subject to data flow control. A pool size which is too small for the processing performance required can also result in a heavy workload on the BCAM pool.
As in phase 2, BCAM offers a monitoring option.
If the BCA0B21 message displays high values for USED-BCAM-MIN-O, the BCAM pool should be enlarged, as in phase 2.
Phases 8 and 9: Delay in transfer to the network
The time in phase 8 (100% BCAM processing) is in the single-digit millisecond range. The times for phases 8 and 9 are combined in the OUTPROC time. You are recommended to monitor the OUTPROC time like the INPROC time (see "Phase 4: Waiting for openUTM"). High values in the overrun interval indicate possible network performance problems (e.g. due to malfunctions in routers or switches).