Tab navigation
- Overview
- Objectives- selected tab,
- Test preparation
Section 1: Planning
- Given a project contract, knowledge of IBM Tivoli Composite Application Manager for Application Diagnostics V7.1 (ITCAM for AD) product and customer's computing environment information; review the contract and customer computing environment so that the project scope can be fully understood.
With emphasis on performing the following tasks:- Understand customer's requirement of the project.
- Purpose of the project
- What is required to be achieved in this project.
- Understand your responsibility as a deployment engineer.
- What IBM products is required to be deployed.
- What is the scale of the deployment.
- Is knowledge transfer to the customer part of the project.
- Understand the customer's responsibilities.
- The required environment should be ready such as IBM Tivoli Monitoring (ITM) infrastructure.
- DBA should be available if needed.
- System and network support should be available if needed.
- Working environment and system access should be provided.
- Understand what ITCAM for AD components is required to be deployed and with what scale.
- Is Managing Server (MS) needed for the project?
- Is M S installed in a single computer or should be a split installation?
- Is ITM integration is required in the project?
- Is WebSphere Application Server (WAS) monitoring defined in the project and is it a deep-dive monitoring?
- Is J2EE monitoring defined in the project and is it a deep-dive monitoring?
- Is HTTP monitoring defined in the project?
- Understand any customization is required.
- Any M S customization?
- Any Data Collector (DC) customization?
- Any ITCAM for AD Tivoli Enterprise Monitoring Agent (TEMA) customization?
- Any ITM situations and workspace customization?
- Any M Sand ITM integration such as launch in context?
- Any integration with other IBM products such as TEC and OMNIbus?
- Understand any deliverables.
- Is a formal solution design required to be completed and sign off by the customer?
- Any deliverable document at end of the project?
- Understand any milestones of the project.
- Is a solution design required to be completed before the deployment can start?
- Does the project has a development phase and production rollout phase?
- Understand the time allocated to complete the project.
- Understand the completion date of the project.
- Understand if the project is a fixed-price contract or a time-and-material contract, and the budget for the project.
- Understand customer's requirement of the project.
- Given appropriate customer monitoring environment documentation, access to the related customer's engineers and access to the customer's monitoring application; review the documentation and interview the customer engineers so that the customer's environment is defined and documented.
With emphasis on performing the following tasks:- Is ITM used in the customer's monitoring environment?
- Version of the ITM
- Architecture of the current ITM environment
- OS level of ITM servers and the monitored servers
- Vender and Type of database used
- Implement Tivoli Data Warehouse (TDW) if needed.
- If TDW is used, how historical data is collected, summarized and pruned.
- Are there separated development, QA and production ITM environment?
- Are there earlier versions of ITCAM for AD product implemented?
- Name and versions of the products
- Customer's experience on these products
- Are there firewalls in the monitoring environment?
- Ports to open between monitoring servers
- Ports to open between monitoring servers and monitored servers
- Network infrastructure
- Are there multiple datacenters around the country or around the world?
- Network speed between those datacenters
- If ITM is used, is each remote datacenter managed by a remote ITM monitoring server?
- How the customer's monitoring environment is managed?
- Are there dedicated ITM administrators?
- Are there any DBAs?
- Are there any WebSphere/J2EE administrators?
- Information about the customer applications
- What applications are currently monitored?
- What applications will be monitored in this project?
- What are the platforms of the application servers and what database are used if any?
- Integration and notification
- Is ITM integrated with any IBM and non-IBM products?
- Are there any notification send out when an ITM alert is triggered?
- Is ITM used in the customer's monitoring environment?
- Given the customer's computing environment documentation and information collected from the customer and project scope; review the customer's computing environment so that ITCAM for AD requirements are met.
With emphasis on performing the following tasks:- Verify the monitored application servers meet the ITCAM for AD product requirement.
- If the Operating Systems type and version used at the application servers is supported.
- If the required additional libraries and software, if any, can be installed.
- Any potential conflict between the software running at the application server and the ITCAM for WAS agents?
- Can the Environmental Checker be run and if so, document the results?
- Verify the network environment meet the product requirement.
- Any firewalls between the M S and the monitored application servers?
- Can required firewall ports be open?
- Verify the database used at the customer is supported.
- Verify the user authentication method used is supported.
- Verify if required system access is available.
- If an old version of ITCAM for AD components exists, verify the upgrade is supported.
- Verify the monitored application servers meet the ITCAM for AD product requirement.
- Given appropriate information of the existing ITCAM for AD environment, access to the system and customer's requirement for the new ITCAM for AD environment; analyze the existing ITCAM for AD installation and configuration so that a detailed upgrade plan is created.
With emphasis on performing the following tasks:- Collect detailed installation and configuration information for the existing ITCAM for AD components:
- Name and version of the existing ITCAM products that is required to be upgraded;
- Name of the servers that the existing monitoring agent and DCs are installed and required to be upgraded;
- Any customizations that have been performed on the MS, DCs and agents.
- Understand the scope of an upgrade installation.
- Understand the upgrade sequence.
- Understand the upgrade procedure preserves customized configuration.
- Develop an upgrade strategy.
- Collect detailed installation and configuration information for the existing ITCAM for AD components:
- Given the project contract, customer's requirement and knowledge of the customer's computing environment; review the project scope and customer's requirement so that a deployment architecture is created.
With emphasis on performing the following tasks:- Determine the ITCAM for AD components required to be implemented in order to achieve the desired monitoring function.
- Is WAS Agent needed?
- Is HTTP Agent needed?
- Is J2EE Agent needed?
- Determine if a MS is needed.
- Determine the M S architecture.
- Is the single server MS installation enough for the monitoring environment?
- If a split installation is needed, determine how the M S components will be split to multiple servers.
- Architect MS and DC customization. Based on the project contract and customer's requirement, define:
- What customization is needed on the MS.
- What customization is needed on the DCs.
- Architect the ITM integration.
- Architect the launch in context if it is in scope of the project.
- Architect alert notification and integration with other event handling tools.
- Determine the ITCAM for AD components required to be implemented in order to achieve the desired monitoring function.
- Given the number of DCs to connect to the MS, the number of requests per minutes, the archive sampling rate, the Monitoring level for DCs, and the number of days to keep in the Database (OCTIGATE); suggest the minimum required components so that the ITCAM for AD size is determined.
With emphasis on performing the following tasks:- Count the number of DCs to be deployed.
- Count the number of Application servers and their individual monitoring levels.
- Obtain the needed requests per minutes.
- Obtain the archive sampling rates.
- Obtain the number of days to keep data in the Database.
- Determine the size of the OCTIGATE Database.
- Determine the number of publishing servers required.
- Determine the number of Archive Agents required.
- Verify the MS server size: CPU, RAM, space, etc.
- Given a customer's specific requirements for reporting through situations and workspaces and the product's documentation; examine and compare the provided "out-of-the-box" situations and workspaces to determine if customization is needed.
With emphasis on performing the following tasks:- Understand and categorize customer's required Situations.
- Understand and categorize customer's required Workspaces.
- Verify that required situations exist "out of the box" or not.
- Decide whether Situations Customization is required.
- Verify that required workspaces exist "out of the box" or not.
- Decide whether Workspaces customization is required.
- Determine what custom Tivoli Enterprise Portal (TEP) user requirements are to support ITCAM for AD specifically.
- Given a target machine to deploy an agent, the location of the installation code, a predefined (or not) response file, and the level of user experience; run an assessment of options so that a deployment method is determined.
With emphasis on performing the following tasks:- Determine if an X server is available.
- Determine if the X library are available on the server to be installed on.
- If X is not available, then gather information required for silent install response file.
- Given a J2EE administrator, configuration files, and product documentations; identify and document key parameters so that the target J2EE environment configuration and security details are reviewed.
With emphasis on performing the following tasks:- Read specific J2EE product installation and configuration documentations.
- Understand the target J2EE setup, configuration and security features.
- Request Access to the deployed J2EE application from the J2EE administrator.
- Access and read specific J2EE configuration files as described in the manuals.
- Extract relevant J2EE configuration parameters.
- Document J2EE configuration and security parameters.
Section 2: ITCAM for AD V7.1 Basic Installation and Configuration
- Given all the hardware and OS pre-requisites are met, a method of installation has been chosen, and the middleware software installation media; proceed to the installation of the database management server, create the IBM Tivoli Composite Application Manager for Application Diagnostics V7.1 (ITCAM for AD V7.1) Managing Server (MS) database and tablespaces so that the server is ready for the installation of the ITCAM for AD V7.1 MS component.
With emphasis on performing the following tasks:- Verify hardware and OS platform are supported for ITCAM for AD V7.1 MS.
- Verify memory, disk space and OS Levels pre-requisites for ITCAM for AD V7.1 MS.
- Verify proper access to the server is provided.
- Verify all ports required by the pre-requisite components are not blocked, mitigate issue if they are.
- Determine if installation(s) will be performed as root or as a different user specifically created for this component.
- Create necessary user-ids and user groups.
- Pre-create installation directories.
- Install Database management server component.
- Create ITCAM for AD V7.1 database and tablespaces.
- Install WebSphere Application Server (WAS) component.
- Configure integration between WAS and LDAP for authentication (optional).
- Given that the middleware pre-requisite components have been installed and configured, install the ITCAM for AD V7.1 MS component and perform the initial configuration steps, so that the ITCAM for AD V7.1 MS is ready to control and collect information from the Data Collectors (DCs) that will be deployed to the target application servers and HTTP servers.
With emphasis on performing the following tasks:- Verify all ports required by the ITCAM for AD V7.1 MS are not blocked, mitigate issue if they are.
- Determine if installation(s) will be performed as root or as a different user specifically created for this component.
- Determine the type of installation to be done - custom or typical.
- Install ITCAM for AD V7.1 MS.
- Verify ITCAM for AD V7.1 Managing Server Visual Engine (MSVE) is up and available.
- Perform initial configuration of users and groups.
- Given a supported WAS and all other pre-requisites met, install and configure the ITCAM for AD V7.1 DC and ITCAM for AD Tivoli Enterprise Monitoring Agent (TEMA), so that it can communicate and transmit monitoring information to the IBM Tivoli Monitoring (ITM) infrastructure and optionally, to the ITCAM for AD V7.1 MS.
With emphasis on performing the following tasks:- Verify all ports required by the ITCAM for AD V7.1 DC and ITCAM for AD TEMA are not blocked, mitigate issue if they are.
- Verify OS and WAS versions are supported for ITCAM for AD V7.1 DC and ITCAM for AD TEMA.
- Determine the user id that will be used to run the ITCAM for AD V7.1 DC and ITCAM for AD TEMA and installation of the component.
- Acquire all necessary access credentials for the server, the application environment and optionally the WAS Deployment Manager if necessary.
- Verify all required directories and/or file systems are created and have the proper permissions set.
- Determine if installation and configuration are going to be performed locally or remotely.
- Install/Deploy the ITCAM for AD V7.1 DC and ITCAM for AD TEMA.
- Configure ITCAM for AD V7.1 DC and ITCAM for AD TEMA.
- Verify DC properly registered with the ITCAM for AD V7.1 MS and add the component to the proper groups.
- Verify ITCAM for AD TEMA properly registered with the ITM infrastructure.
- Given a supported J2EE Server and all other pre-requisites met, install and configure the ITCAM for AD V7.1 DC and ITCAM for AD TEMA, so that it can communicate and transmit monitoring information to the ITM Infrastructure and optionally, to the ITCAM for AD V7.1 MS.
With emphasis on performing the following tasks:- Verify all ports required by the ITCAM for AD V7.1 DC and ITCAM for AD TEMA are not blocked, mitigate issue if they are.
- Verify OS and J2EE versions are supported for ITCAM for AD V7.1 DC and ITCAM for AD TEMA.
- Determine the user id that will be used to run the ITCAM for AD V7.1 DC and ITCAM for AD TEMA and installation of the component.
- Acquire all necessary access credentials for the server, the J2EE application environment.
- Verify all required directories and/or file systems are created and have the proper permissions set.
- Determine if installation and configuration are going to be performed locally or remotely.
- Install/Deploy the ITCAM for AD V7.1 DC and ITCAM for AD TEMA.
- Configure ITCAM for AD V7.1 DC and ITCAM for AD TEMA , enable TTAPI integration if necessary.
- Verify DC properly registered with the ITCAM for AD V7.1 MS and add the component to the proper groups.
- Verify ITCAM for AD TEMA properly registered with the ITM Infrastructure.
- Given a supported HTTP server and all other pre-requisites met, install and configure the ITCAM for AD V7.1 HTTP Agent so that it can communicate and transmit monitoring information the ITM Infrastructure.
With emphasis on performing the following tasks:- Verify all ports required by the ITCAM for AD V7.1 HTTP Agent are not blocked, mitigate issue if they are.
- Verify OS and HTTP Server versions are supported for ITCAM for AD V7.1 HTTP Agent.
- Determine the user id that will be used to run the ITCAM for AD V7.1 HTTP Agent and installation of the component.
- Acquire all necessary access credentials for the server and the HTTP server environment.
- Verify all required directories and/or file systems are created and have the proper permissions set.
- Determine if installation and configuration are going to be performed locally or remotely.
- Install/Deploy the ITCAM for AD V7.1 HTTP Agent.
- Configure ITCAM for AD V7.1 HTTP Agent.
- Verify ITCAM for AD V7.1 HTTP Agent properly registered with the ITM Infrastructure.
- Given a properly functioning existing installation of ITCAM for Application Diagnostics, J2EE or WebSphere, upgrade it to Version 7.1 so that it can be running at the current level of the product.
With emphasis on performing the following tasks:- Verify all OS environments are at the supported level for ITCAM for AD V7.1, upgrade OS if necessary.
- Verify all pre-requisite middleware is at the supported level for ITCAM for AD V7.1, upgrade middleware if necessary.
- Acquire all necessary access credentials, verify all user ids and directory permissions, verify file systems have the necessary capacity for the upgrade.
- Determine the user id that will be used to run the upgrade, verify once again all directory permissions if it will not be root/Administrator user.
- Verify current version of ITCAM for AD, J2EE or WebSphere is supported for an upgrade to version 7.1, re-plan the upgrade path if not.
- Verify all newly required directories and/or file systems are created and have the proper permissions set.
- Upgrade the ITCAM for AD database following specific instructions for the current supported level of ITCAM for AD.
- Upgrade the ITCAM for AD MS and MVSE components, verify after upgrade.
- Verify OS environments in each current ITCAM for AD DC environment, upgrade if necessary.
- Acquire all necessary access credentials; verify installation directories for space and permissions.
- Verify current version of ITCAM for AD DC is supported for upgrade to ITCAM for AD V7.1.
- Upgrade DC and ITCAM for AD Agent, verify components properly register with the MS and ITM Infrastructure.
- Clean up old installation of DC and old log files.
Section 3: ITM Integration Steps
- Given the ITCAM for Application Diagnostics Installation Media, install the ITCAM for Application Diagnostics application support into the IBM Tivoli Monitoring (ITM) Infrastructure so that it will be able to properly process and display the data collected by the ITCAM for Application Diagnostics agents.
With emphasis on performing the following tasks:- Install application support on the Tivoli Enterprise Monitoring Server (TEMS).
- Install application support on the Tivoli Enterprise Portal Server (TEPS).
- Install application support on the desktop client.
- Given the install media, install the agents for ITCAM for Application Diagnostics (KYN, KYJ, KHT).
With emphasis on performing the following tasks:- Obtain the installation Media.
- Ensure system, software and OS prerequisites are met.
- Install the Tivoli Enterprise Monitoring Agent (TEMA).
- Configure the TEMA.
- Verify that the correct application support is installed onto the ITM Infrastructure.
- Start the TEMA and see it running and collecting data.
- (Optional) Setup and perform a silent installation.
- (Optional) Install language support.
- (Optional) Uninstall the TEMA.
- Given a Maintenance Delivery Vehicle (MDV) delivered from IBM Support, install it onto each and every effected ITM Infrastructure component.
With emphasis on performing the following tasks:- Obtain and stage the MDV.
- Install application support updates in the MDV to the TEMS.
- Install application support updates in the MDV to the TEPS.
- Install the MDV to the ITCAM for Application Diagnostics monitoring agent(s)
- Reconfigure Monitoring Agent after MDV installation.
- Given the Tivoli Enterprise Portal (TEP) information and a working MSVE, configure the support file for the MSVE to so that it will be able to launch the TEP in context.
With emphasis on performing the following tasks:- Locate the file "ve.properties" file on the host of the MSVE.
- Enter the properties for the TEP in the "ve.properties" file.
- Save the edited "ve.properties" file.
- Open the TEP inside the MSVE by selecting the Problem Center -> Log tab.
- Given the TEP Context Launch in the MSVE shows only ITCAM for Application Diagnostics monitoring workspaces, views and navigation items; launch the TEP inside the MSVE and view ITM Situation Events that are active for ITCAM for Application Diagnostics .
With emphasis on performing the following tasks:- Launch the MSVE.
- Select Alerts and Events.
- Select group and server name.
- Escalate the trap to Problem Center.
- Verify that the TEP Situation Event Console shows only open ITCAM for Application Diagnostics monitoring events.
- Given the proper ITCAM for Application Diagnostics documentation, know and understand each attribute collected and the purpose for each collected attribute.
With emphasis on performing the following tasks:- List the attributes collected by the ITCAM for Application Diagnostics: J2EE (KYJ) monitoring agent.
- List the attributes collected by the ITCAM for Application Diagnostics: WebSphere (KYN) monitoring agent.
- List the attributes collected by the ITCAM for Application Diagnostics: HTTP Servers (KHT) monitoring agent.
Section 4: Customization and Administration
- Given access to an ITCAM for Application deep-dive console, list settings commonly used for performance tuning and adjusting the monitoring scope for a Data Collector (DC).
With emphasis on performing the following tasks:- Diagnose the performance issue with the DC.
- Open the datacollector.properties file in a text editor: (default location DC_home/runtime/appserver_version.node_name.server_name/appserver_version.node_name.server_name.datacollector.properties)
- Adjust the internal.probe.event.packet.size property upward if the the amount of data the DC sends to the Publish Server is large enough to cause issues.
- Consider adjusting the internal.memory.limit upward if the DC goes into turbo mode within 10 minutes of starting. Alternatively, consider setting dc.turbomode.enabled to false.
- Adjust the value of am.mp.clockThreshold to match the minimum clock time value of the methods you are required to capture for method profiling (default = 30ms).
- Adjust the value of am.mp.cpuThreshold to match the minimum CPU time value of the methods you are required to capture for method profiling (default = 30ms).
- Consider adjusting the value of internal.probe.event.queue.size.limit upward if the DC is dropping events.
- Given the required ITCAM for Application Diagnostics resources, follow the steps to tune and configure the Managing Server (MS) database (OCTIGATE).
With emphasis on performing the following tasks:- Log in to the DB2 administration console with admin or equivalent privileges and perform the following operations on the OCTIGATE database.
- Change SMS tablespace(s) to DMS tablespace(s).
- Stripe tablespace(s) across as many disks as possible to parallelize I/O activities.
- Place the following on different spindles: SWAP space; TEMPORARY tablespace; Transaction log files; table data files; table index files.
- Consider prefetching index and data pages into the buffer pool, to reduce the time spent waiting for I/O to complete. (To enable prefetching, the Database Manager starts separate threads of control, known as I/O servers, to perform page reading.)
- Run the DB2 runstats utility on a regular basis for access path and space optimization.
- Run the DB2 reorgchk utility on a regular basis to update statistics used by the DB2 optimizer. (Example: Verify that the cluster ratio of the cluster index R_TIME on table requests is 80% or higher.)
- Assign enough memory to DB2 so that no swapping occurs.
- Given the required ITCAM for Application Diagnostics resources, follow the steps to set default DC monitoring levels.
With emphasis on performing the following tasks:- Log in to the Managing Server Visualization Engine (MSVE) by using credentials with sufficient authority to create custom groups.
- Using the MSVE's menu bar, navigate to Administration -> Managing Server -> System Properties.
- Under the Data Collection Settings heading, use the drop-down to select the Default Monitoring Level for all DCs.
- Under the Data Collection Settings heading, edit the other fields as desired to complete your default DC monitoring configuration.
- Click the Save button to complete the operation.
- Given the required ITCAM for Application Diagnostics resources, follow the steps to create a custom group.
With emphasis on performing the following tasks:- Log in to the MSVE by using credentials with sufficient authority to create custom groups.
- Using the MSVE's menu bar, navigate to Administration -> Server Management -> Server Groups.
- Click the Create Group link in the Menu box.
- Give the group a unique name, and an optional description in the fields provided.
- Keep the default Baseline Indicators and Baseline Definitions.
- Under the Group Members heading, select the server(s) to be included in the group, and click the Add button to move them to the Servers in Group list box.
- Under the User Access heading, use the Add and Remove buttons to place the correct users in the Granted Access list box.
- Click the Save button to complete the operation.
- Using the MSVE's menu bar, navigate to Administration -> Server Management -> Server Groups to verify that your newly created group is now in the list.
- Given the required ITCAM for Application Diagnostics resources, follow the steps to create a custom DC configuration.
With emphasis on performing the following tasks:- Log in to the MSVE by using credentials with sufficient authority to create DC configurations.
- Using the MSVE's menu bar, navigate to Administration -> Server Management -> Data Collector Configuration.
- Click the Create a Configuration link in the Menu box.
- Edit the contents of the Exclude and Exclude Override fields as desired.
- Give the configuration a unique name.
- Check the Enable MQ box, if applicable to your custom group.
- Click the Save button to add your custom configuration to the configuration library, or click the Save & Apply button if you want to apply it immediately to one or more servers.
Section 5: Troubleshooting
- Given a described problem and access to Data Collector (DC) configuration and log files, determine the need to make changes so that the logging level is adjusted.
With emphasis on performing the following tasks:- Evaluate that the Problem is not resolvable with basic troubleshooting.
- Access log files (DC).
- Read and determine a possible area cause.
- Access Configuration files (DC).
- Make required logging changes in the configuration files (DC).
- Restart DC.
- Verify that Logging level adjusted as required.
- Given a specific J2EE application, product and deployment documentation, configuration files and access to the J2EE directories; extract the relevant parameters so that the log files locations are determined.
With emphasis on performing the following tasks:- Talk to the customer.
- Read J2EE product and deployment documentations.
- Access J2EE directories and configuration files.
- Extract indications describing log files locations.
- Given abnormal DC's performance metrics, access to DC configuration and log files, and the imperative to troubleshoot deeper; determine the needed changes so that the logging level is appropriately adjusted.
With emphasis on performing the following tasks:- Evaluate that the Problem is not resolvable with basic troubleshooting.
- Access log files (DC) typical log location (IBM Tivoli Monitoring installation location)/logs.
- Read and determine a possible area cause.
- Access Configuration files (DC).
- Make required logging changes in the configuration files (DC).
- Restart DC.
- Verify that Logging level is adjusted as required.
- Given an error condition that cannot be resolved, execute gather script to collect application and environment information to have useful information to provide to support.
With emphasis on performing the following tasks:- Select component for which information has to be gathered.
- Identify OS and according location and file name of the gather script.
- Ensure that the user has sufficient rights to execute gather script and to read files and directory structures of required components.
- For Managing Server (MS) execute MS_Gather_Data.sh and pass PMR number and name of WebSphere Application Server (WAS) Server that hosts the Visualization Engine of the MS as parameters.
- For DC of type WAS execute cyn_collector.sh and pass either -a for all servers or -s [server_name] for a certain instance as parameter. Ensure that the user has sufficient rights to read from the specified server's directories.
- For DC of type J2EE execute cyn_collector_J2.sh.
- For the WebSphere agent (Tivoli Enterprise Monitoring Agent (TEMA) part) execute kyncollect.sh.
- For the J2EE Agent (TEMA part) execute kyjcollect.sh.
- Read the command line output carefully. If there are any problems report to IBM support.
- Locate the output of the gather script and send it to IBM support. The script and/or Troubleshooting guide will tell the location in the file system.
- Given a specific problem scenario on the MS, required log files and system information; determine the root cause of the problem.
With emphasis on performing the following tasks:- Check connectivity from your machine to the MS machine. Ping it or telnet the MS HTTP port (default 9080).
- Open the browser and point it to the MS. If the page does not display at all, check if the WAS component and the contained Visualization Engine (VE) application is running.
- To verify WAS run the WebSphere serverStatus -all script in your designated WAS profile bin directory or check the SystemOut.log for information.
- Log in to the MS. If the login fails it indicates a problem on the MS side. Verify that the kernel and the DB are up and running.
- To verify if the kernel processes are running run MS_HOME/bin/am-status.sh or evaluate MS Gather script.
- To verify the database connectivity run MS_HOME/bin /klctl.sh -dbtest.
- If login is successful, navigate to the Self-Diagnosis page and verify that all relevant kernel and other components' processes are running.
- If processes are not running restart failing component with the amctl.sh script.
- Locate the Tivoli Common Directory for the given Operating System and check the log files under directory CYN and the First Failure Data Capture Files under CYN\FFDC for error messages.
- Check the log files under MS_HOME\logs for logging of the individual MS processes.
- Given a specific problem scenario with the agent, required log files and system information; determine the root cause of the problem.
With emphasis on performing the following tasks:- Log on to MS and Tivoli Enterprise Portal (TEP) and check if monitored Application Server reports data is being reported.
- If there is data visible in TEP and no data visible in the MS verify the network connection between the agent and the MS. Make sure the required ports are open for proper communication.
- If there is data visible in MS and no data visible in the TEP verify the network connection between the agent and the TEP. Ensure the TEMA component is up and running.
- If there is no data visible in both TEP and MS, check the following things:
- Make sure the monitored WAS is started and running.
- Verify in the SystemOut.log of each instance, if the DC has been installed properly and a successful "join" event has been written to the log since last startup. Reconfigure and restart the instance if necessary.
- Verify the network communication between all components is working and the necessary ports are open.
- Additionally check agent logs.
- If the TEP workspaces are being populated only partially and monitoring information is missing, please verify that you are running the correct monitoring level. Modify accordingly.
- If the L3 monitoring information is not being displayed in the MS check the configuration files in the DC directory if L3 monitoring is properly configured. A WAS restart is required after changing these settings.
- Given a specific installation problem description, required log files and system information; determine the reason for the failing installation
With emphasis on performing the following tasks:- Use silent response file.
- For each component failing during installation check if a silent response file is available.
- Verify if all parameters had been set correctly.
- Redo installation if necessary and make sure a response file is being created.
- For all components failing during installation:
- Analyze the error messages displayed when the installer is failing.
- Additionally analyze the installation logs in the Tivoli Common Directory.
- Make sure that the installer does have necessary disk space and writing privileges in the desired target file system.
- If MS installation is failing:
- Verify that the required WAS and database server have been installed, are up and running for the installation and that the installer has the necessary rights to access them.
- If Agent installation or configuration is failing:
- Additionally analyze installation logs in the ITM_HOME\logs directory.
- Make sure the agent can access the MS via fully qualified host name.
- Update DNS settings or hosts file to enable server and agent for proper network communication. Ensure all components are using the same settings.
- Ensure that the installer has necessary permission to access the file system where the monitored WAS instance resides and the necessary credentials to access the server.
- Make sure the WAS Server is restarted after the configuration to complete the installation.
- Use silent response file.
- Given a situation of an expired/changed password of a runtime component, identify which one and reset it to resume normal operation.
With emphasis on performing the following tasks:- TEP part:
- If a password of a certain user on the TEP expired or changed, log on as sysadmin and reset it for the user.
- If the password for the sysadmin expired or changed, edit the settings of the Tivoli Enterprise Monitoring Server (TEMS) to not validate the password. This will give you access again to restore expired/modified passwords.
- MS part:
- If the MS is installed in the embedded WebSphere Application Server (eWAS) of the Tivoli Enterprise Portal Server (TEPS), you are required to update the TEPS/e administration password. Reset it and restart the eWAS and the MS.
- If the MS is installed in a WAS ND environment and the password gets rejected, make sure the node agent is synchronized with the Deployment Manager.
- If the database user expired or has been modified and the MS cannot connect to the database anymore, perform the following steps.
- Modify the MS_HOME\bin\setenv.sh file and update the JDBC_PASSWORD.
- Encrypt the file by using amctypto.sh.
- Verify the working connection with MS_HOME/bin/klctl.sh dbtest.
- Restart the MS.
- TEP part:
Section 6: Product Usage
- Given the Client Request, Client Request Start, Response Time (ms), Accumulated CPU (ms), Idle Time (ms), User Id, Snapshot Date, Snapshot Time, Platform CPU %, Application Server Name, Application Server IP Address, Total Thread Count, JVM CPU, # of Requests, # of Live Sessions , JVM Heap Size, and Average response Time; determine which request is taking the longest response time, longest Idle Time and highest amount of CPU time and which request is Hung and taking the longest response time, longest Idle Time and highest amount of CPU time, as well which Last Known Class, Last Known Method, Last Known Action was taken the longest time and which request is having locking issues.
With emphasis on performing the following tasks:- Log in to Managing Server Visualization Engine (MSVE).
- Using menus select Problem Determination -> Server Activity Display.
- Select Group and Server.
- Select Recent Requests Tab.
- Review Recent Requests Page.
- Select Active Request Tab.
- Review Active Request Page.
- Select Lock Contention Tab.
- Review Lock Contention Page.
- Given Throughput per hour, Throughput per Minute, Throughput per Sec, Response Time(ms), CPU Time(ms); determine which request are taking the largest Throughput per hour, Throughput per Minute, Throughput per Sec, Response Time(ms), CPU Time(ms) for a given Time Frame.
With emphasis on performing the following tasks:- Log in to MSVE.
- Select Performance Analysis -> Create Reports -> Application Report -> Request Transaction (or any other appropriate report).
- Choose if you would a recurring report.
- Select Group and Server.
- Select the metric you require (Throughput per hour, Throughput per Minute, Throughput per Sec, Response Time (ms), CPU Time (ms)).
- Select the Time Range.
- View Report.
- Given the Server Name, Client Request/Transaction, Start Date/Time, Thread/Task ID, Total Resident Time (ms), User ID, Snapshot Date, Application Server Name, Snapshot Time, Application Server IP Address, Platform CPU % Utilization, Total Task Count, Transaction, Resident Time, Priority, Last Known CPU, Task Type, Current Action, Task Status, Group/Server , Platform, Schedule Name, Current Level, Current Sampling, Schedule Change/Override, Depth, Event Type, Event Data, Elapsed Time (ms), CPU Time (ms), Elapsed Time (ms),and CPU Time (ms); determine which method calls are having the slowest response time and highest CPU time.
With emphasis on performing the following tasks:- Log in to MSVE.
- Using Menus select Administration -> Monitoring on Demand.
- Enable L3 monitoring at 100 % sampling rate for the JVM
- Using Menus select Problem Determination -> In-Flight Request Search.
- Choose the Thread/Task ID for the Hung Thread.
- Select Program/Component Trace.
- Review Method Trace.
- Given the Server Name, Client Request/Transaction, Start Date/Time, Thread/Task ID, Total Resident Time (ms), User ID, Snapshot Date, Application Server Name, Snapshot Time, Application Server IP Address, Platform CPU % Utilization, Total Task Count, Transaction, Resident Time, Priority, Last Known CPU, Task Type, Current Action, and Task Status; determine which method is causing the transaction to hang.
With emphasis on performing the following tasks:- Log in to MSVE.
- Using Menus select Problem Determination -> In-Flight Request Search.
- Choose the Thread/Task ID for the Hung Thread.
- Exam the Stack Trace.
- Given the Application Server Name, Application Server IP Address, Timestamp, Class Name, Request Name, Request Type, Allocating Class, Line Number, Estimated Bytes Used, # of Objects, # of Objects Surviving Last GC, Growth %, Growth Rate (#/sec), Growth Time Interval (sec), Oldest Object Age (sec), and Avg. Object Age (sec); determine which class is having memory leaks.
With emphasis on performing the following tasks:- Log in to MSVE.
- Select Problem Determination -> Memory Diagnosis -> Memory Leak.
- Select Change to L3.
- Enable L3.
- Return to Problem Determination -> Memory Diagnosis -> Memory Leak.
- Select Group and Server.
- Select View Memory Diagnosis.
- Review Report.
- Given the Environment Data, JDK Version, Initial Java Heap Size, Maximum Java Heap Size, Installation Directory, Class Path, Library Path, App Server, Startup Directory, Listening Port, Registered EARs, Registered WARs, Registered WARs, JDBC Connection Pools, Source Files, Source types, Unmatched and Matched, Server, Full Match, Full Pathname/Size Match, Full Pathname Match, Comparison Only and Authoritative Only use; determine what is different between the Authorative Server and the other Servers.
With emphasis on performing the following tasks:- Log in to the MSVE.
- Select Problem Determination -> Software Consistency Check -> (either Installed Binary Comparison or Runtime Environmental Comparison).
- Select the Group and Server of the Authorative Server.
- Select the Group to compare.
- Select the List of Servers to compare (up to 10).
- Review the reports.
- Given the Trap conditions: Average Platform CPU %, Average JVM CPU % , JVM Heap Size , Average JVM Heap Size after GC, Request Frequency, Number of Sessions, Application Trap Occurrence, CPU Time , Resident Time - In-Flight, Resident Time - Complete, Resident Time - Misbehaving Transaction, Uncaught - Exceptions, Lock Acquisition Time - In-Flight, Lock Acquisition Time - Completed, of occurrences of every request after which the trap will be deactivated, Component/Method Trace, Stack Trace, Thread Dump, Send SNMP Trap, and Send an email; set the appropriate trap conditions that are required.
With emphasis on performing the following tasks:- Determine which Trap Types are required to be monitored.
- Determine what action is to be taken when the Trap condition is meet such as, send an email, collect a stack trace, collect a methods trace, collect a thread dump, send and snmp trap.
- Log in to MSVE.
- Select Problem Determination -> Trap and Alert Management, Create Trap.
- Choose the Trap condition as needed.
- Choose the actions as needed.
- Select Save and Activate the trap.
- Wait until the Trap condition occurs.
- Given information from the Problem Tab, Memory Tab, Transitions Tab, Resources Tab, Traps Tab, Log Tab and Configuration Tab, determine the root cause of the problem.
With emphasis on performing the following tasks:- Log in to ITCAM Managing Sever.
- Using menu go to Problem Determination -> Problem Center.
- Select the Group and Server.
- Select the all of Report Tabs.
