How to capture Trace in IBM IIB OR ACE

 

There are two kind of traces we use to troubleshoot any issue in IIB/ACE -

User Trace:

This trace is useful for developers for debugging and it generate low level of trace capture.

Below are the steps to generate this and analyze -

1. Start trace:

mqsichangetrace ACENODE -e EGName -u -l debug -r -c 200000 

2. Re-create the runtime behavior.

3. Stop trace:

mqsichangetrace ACENODE -e EGName -u -l none

4. Verify trace status

mqsireporttrace ACENODE -e EGName

Service trace -

This is detailed trace and captures everything required to analyze the issue and required when you are working with IBM support to resolve a particular issue.

1. Start/enable trace

mqsichangetrace ACENode -e EG_Name -t -l debugTree -r -c 200000

2. Recreate the issue

3. Stop/disable the trace

mqsichangetrace ACENode -e EG_Name -t -l none

4. Verify the trace status

mqsireporttrace ACENode -e EG_Name

The generated log could be found under this dir/location - /var/mqsi/common/log

Shell Script To Retrieve IBM ACE Resource/Flow details

 

You can copy below script and save as .sh extension, replace the node name and execute it.

Note: make sure you are running the script with correct user privilege.

------------------------------- Start of script ---------------------------------------

#!/bin/sh

RESFILE="/tmp/ACE_Flow_Details.txt"

WORKDIR="/tmp/"

CURDATE=`date +"%Y-%m-%d-%H:%M:%S"`

 

echo "Starting mqsilist for all flows: " $CURDATE

mqsilist ACE-Node >> $WORKDIR/ACE_Node_list.txt

grep -i "Integration server" $WORKDIR/ACE_Node_list.txt >> $WORKDIR/ACE_EG_list.txt

while read line; do

EGname=`echo $line |cut -d"'" -f2 | cut -d"'" -f1`

mqsilist ACE-Node -e $EGname >>  $WORKDIR/ACE_AppTemp_list.txt

grep -i ": Application" $WORKDIR/ACE_AppTemp_list.txt >> $WORKDIR/ACE_App_list.txt

while read templine; do

AppName=`echo $templine | cut -d":" -f2 | cut -d"'" -f2`

echo "running mqsilist for EG: " $EGname " and App: " $AppName

mqsilist ACE-Node -e $EGname -k $AppName -d2 >>  $RESFILE

done < $WORKDIR/ACE_App_list.txt

rm $WORKDIR/ACE_App_list.txt

rm $WORKDIR/ACE_AppTemp_list.txt

done < $WORKDIR/ACE_EG_list.txt

rm $WORKDIR/ACE_EG_list.txt

rm $WORKDIR/ACE_Node_list.txt

echo "End of mqsilist, please check the file: " $RESFILE

echo "Please Remove the files: " $RESFILE " after use."

------------------------------- End of script ---------------------------------------

Download files from a remote SFTP server to local machine and upload to remote server using command prompt - putty

 

Please follow below steps to login and download files from remote server and upload to another remote server.

This guide will explain how to login to remote server using SFTP command and SSH key instead of password.

 

Local Machine - 

1. Create a dir to download the files from Source remote server and upload the same files to destination remote server 

 

[aceuser@host2ace01 ~]$ mkdir Sorce-Dest-manual-Copy

[aceuser@host2ace01 ~]$ chmod 777 Sorce-Dest-manual-Copy

[aceuser@host2ace01 Sorce-Dest-manual-Copyl]$ pwd

/home/aceuser/Sorce-Dest-manual-Copy

 

2. Login to Source server FOR first user and download the files

 

[aceuser@host2ace01 Sorce-Dest-manual-Copy]$ sftp -i /home/aceuser/.ssh/id_rsa_iib userid_01@host.com

Connected to userid_01@host.com.

sftp> ls

JobFamily_20240705.csv        JobType_20240705.csv          Org_20240705.csv              Person_20240705.csv

_staged

sftp> ls -ltr

drwxrw-rw-   1 user     group           0 Feb  1 01:57 _staged

-rw-rw-rw-   1 user     group         358 Jul  5 00:00 JobFamily_20240705.csv

-rw-rw-rw-   1 user     group      453669 Jul  5 00:00 JobType_20240705.csv

-rw-rw-rw-   1 user     group       96756 Jul  5 00:00 Org_20240705.csv

-rw-rw-rw-   1 user     group     2958835 Jul  5 01:00 Person_20240705.csv

sftp> get *20240705.csv

Fetching /JobFamily_20240705.csv to JobFamily_20240705.csv

/JobFamily_20240705.csv                                                                                           100%  358     4.0KB/s   00:00

Fetching /JobType_20240705.csv to JobType_20240705.csv

/JobType_20240705.csv                                                                                             100%  443KB   1.2MB/s   00:00

Fetching /Org_20240705.csv to TGCS_SABAOrg_20240705.csv

/Org_20240705.csv                                                                                                 100%   94KB 637.3KB/s   00:00

Fetching /Person_20240705.csv to Person_20240705.csv

/Person_20240705.csv                                                                                              100% 2889KB   1.2MB/s   00:02

sftp>exit

 

3. Login to Source server FOR user 2 and download the files

 

[aceuser@host2ace01 Sorce-Dest-manual-Copy]$ sftp -i /home/aceuser/.ssh/id_rsa_iib userid_02@host.com

Connected to userid_02@host.com.

sftp> ls -ltr

drwxrw-rw-   1 user     group           0 Dec  6  2022 _staged

-rw-rw-rw-   1 user     group         398 Jul  5 00:00 JobFamily_20240705.csv

-rw-rw-rw-   1 user     group      273250 Jul  5 00:00 JobType_20240705.csv

-rw-rw-rw-   1 user     group       22521 Jul  5 00:00 Org_20240705.csv

-rw-rw-rw-   1 user     group       39794 Jul  5 00:00 Loc_20240705.csv

-rw-rw-rw-   1 user     group     2752691 Jul  5 01:00 Person_20240705.csv

sftp> get *20240705.csv

Fetching /JobFamily_20240705.csv to JobFamily_20240705.csv

/JobFamily_20240705.csv                                                                                           100%  398     4.9KB/s   00:00

Fetching /JobType_20240705.csv to JobType_20240705.csv

/JobType_20240705.csv                                                                                             100%  267KB 560.3KB/s   00:00

Fetching /Loc_20240705.csv to Loc_20240705.csv

/Loc_20240705.csv                                                                                                 100%   39KB 192.9KB/s   00:00

Fetching /Org_20240705.csv to Org_20240705.csv

/Org_20240705.csv                                                                                                 100%   22KB 250.5KB/s   00:00

Fetching /Person_20240705.csv to Person_20240705.csv

/Person_20240705.csv                                                                                              100% 2688KB 828.5KB/s   00:03

sftp>exit

 

4. List the files downloaded from Source to local machine 

 

[aceuser@host2ace01 Sorce-Dest-manual-Copy]$ ls -ltr

total 6468

-rw-------. 1 aceuser mqbrkrs    1671 Jul  5 02:28 id_rsa_iib

-rw-r--r--. 1 aceuser mqbrkrs     358 Jul  5 03:12 JobFamily_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs  453669 Jul  5 03:12 JobType_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs   96756 Jul  5 03:12 Org_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs 2958835 Jul  5 03:12 Person_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs     398 Jul  5 03:13 JobFamily_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs  273250 Jul  5 03:13 JobType_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs   39794 Jul  5 03:13 Loc_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs   22521 Jul  5 03:13 Org_20240705.csv

-rw-r--r--. 1 aceuser mqbrkrs 2752691 Jul  5 03:13 Person_20240705.csv

[aceuser@host2ace01 Sorce-Dest-manual-Copy]$

 

 

 

5. Login to Destination remote server to upload/push the files -

[aceuser@host2ace01 Sorce-Dest-manual-Copy]$ sftp -i /home/aceuser/.ssh/id_rsa_iib -oPort=22222 destuserid@remotehost.com

Connected to destuserid@remotehost.com.

sftp> ls

sftp> cd prod/inbound/

sftp> put *20240705.csv

Uploading JobFamily_20240705.csv to /client/prod/inbound/JobFamily_20240705.csv

JobFamily_20240705.csv                                                                                            100%  398    26.2KB/s   00:00

Uploading JobType_20240705.csv to /client/prod/inbound/JobType_20240705.csv

JobType_20240705.csv                                                                                              100%  267KB   3.0MB/s   00:00

Uploading Loc_20240705.csv to /client/prod/inbound/Loc_20240705.csv

Loc_20240705.csv                                                                                                  100%   39KB   1.0MB/s   00:00

Uploading Org_20240705.csv to /client/prod/inbound/Org_20240705.csv

Org_20240705.csv                                                                                                  100%   22KB   1.0MB/s   00:00

Uploading Person_20240705.csv to /client/prod/inbound/Person_20240705.csv

Person_20240705.csv                                                                                               100% 2688KB  31.0MB/s   00:00

UploadingJobFamily_20240705.csv to /client/prod/inbound/JobFamily_20240705.csv

TGCS_SABAJobFamily_20240705.csv                                                                                            100%  358    23.1KB/s   00:00

Uploading JobType_20240705.csv to /client/prod/inbound/JobType_20240705.csv

JobType_20240705.csv                                                                                              100%  443KB   4.8MB/s   00:00

Uploading Org_20240705.csv to /client/prod/inbound/Org_20240705.csv

Org_20240705.csv                                                                                                  100%   94KB   2.9MB/s   00:00

Uploading Person_20240705.csv to /client/prod/inbound/Person_20240705.csv

Person_20240705.csv                                                                                               100% 2889KB  41.1MB/s   00:00

sftp>

 

 

6. Files are uploaded manually to Destination server under prod/inbound dir as per the ACE flow process.

[aceuser@host2ace01 Sorce-Dest-manual-Copy]$ sftp -i /home/aceuser/.ssh/id_rsa_iib -oPort=22222 destuserid@remotehost.com

Connected to destuserid@remotehost.com.

sftp> cd prod/inbound/

sftp> ls -ltr

 

-rw-r--r--    1 669      504           398 Jul  5 07:21 JobFamily_20240705.csv

-rw-r--r--    1 669      504        273250 Jul  5 07:21 JobType_20240705.csv

-rw-r--r--    1 669      504         39794 Jul  5 07:21 Loc_20240705.csv

-rw-r--r--    1 669      504         22521 Jul  5 07:21 Org_20240705.csv

-rw-r--r--    1 669      504       2752691 Jul  5 07:21 Person_20240705.csv

-rw-r--r--    1 669      504           358 Jul  5 07:21 JobFamily_20240705.csv

-rw-r--r--    1 669      504        453669 Jul  5 07:21 JobType_20240705.csv

-rw-r--r--    1 669      504         96756 Jul  5 07:21 Org_20240705.csv

-rw-r--r--    1 669      504       2958835 Jul  5 07:21 Person_20240705.csv

sftp> pwd

Remote working directory: /client/prod/inbound

sftp>

 

IBM ACE/IIB Commands

 

2. Below command is to list all the SFTP security identities configured for the broker 

mqsireportdbparms ACEORNG-TEST -n sftp::*

2. JvmMaxHeapSIze for EG in ACE

To fetch current details/values

 mqsireportproperties NODE -e EGName -o ComIbmJVMManager -r

jvmMaxHeapSize='268435456'

jvmMinHeapSize='33554432'

To change/update the valuse

mqsichangeproperties NODE -e EGNAME -o ComIbmJVMManager -n jvmMaxHeapSize -v 2147483648

BIP8491W: The changes have been accepted and persisted by the integration server 'EGNAME '. An integation server restart is required for the changes to become active.

BIP8071I: Successful command completion.

SFTP Configuration over SSH in IBM ACE/IIB

 

You can configure an App Connect Enterprise (ACE) file node (FileInput, FileOutput, FileRead) to make a connection to an SFTP server.

 

Complete the following steps:

1. Use your key generation tool (on the Linux server) to generate SSH keys (private and public) and export the public key in PEM format. 

For example:

ssh-keygen -m PEM

this will prompt for password, leave it blank if you do not want to have any password.

you can optionally supply the path and file name for the keys.

 ssh-keygen -f "filepath/filename (no extension)"-m PEM

2. once the SSH keys are generated, you can share and import the public key on the remote server.

once that is done you can verify the connection from Linux server using below command -

sftp -i "SSH-Private-key-file" -oPort=2222 userid@hostname

Note: supplying port is optional if it's different than default SFTP port.

if its successful then you can proceed with next step -

3 Run the mqsisetdbparms command:

mqsisetdbparms sftp::myIdentityName -u myUserName -i myPEMFormatKeyFile

4. Restart the Integration Node

5. Now use the security identity on the messageflow node and deploy/test.

--------------   Important Notes ------------------

If by any chance/mistake you have provided a passphrase to the SSH key while generating it.

then it will ask for the passphrase when you try to connect to the remote machine.

[aceuser@ .ssh]$ sftp -i /home/aceuser/sshkeys/id_rsa_iib User1@remote.test.com

The authenticity of host 'remote.test.com (132.42.2.123)' can't be established.

RSA key fingerprint is SHA256:uDcuqo4a2tmWXWTfRLG/pDOV7XUn7aU8/2WcapRj+CE.

Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Warning: Permanently added 'remote.test.com,132.42.2.123' (RSA) to the list of known hosts.

Enter passphrase for key '/home/aceuser/sshkeys/id_rsa_iib':

Password Authentication

Enter password for User1

Password:

Password Authentication

Enter password for User1

Password:

[aceuser@ .ssh]$

[aceuser@ .ssh] $

----------

In case if you do not know the passphrase for the private SSH key and also not sure if there is a passphrase set for the key then you can try below to confirm - for this you should have the private key.

-- below command is used to print/fetch public key from the private key, but if there is a passphrase is set for the private key then it will prompt for the passphrase.

[aceuser@ .ssh]$ ssh-keygen -y -f /home/aceuser/sshkeys/id_rsa_iib

Enter passphrase:

Load key "/home/aceuser/sshkeys/id_rsa_iib": incorrect passphrase supplied to decrypt private key

To remove or update the passphrase of the private key you can run below command -

[aceuser@ ]$ ssh-keygen -f /home/aceuser/sshkeys/id_rsa_iib -p
Enter old passphrase:

---------------------------------------------------------

In case there is any issue with known_hosts file of the OS or the EG level one, where we have old host entries and we need to remove them then, run below command -

a) for OS level known_hosts files

[aceuser@ .ssh]$ ssh-keygen -f /home/aceuser/.ssh/known_hosts -R remote.test.com

# Host transmissions.jpmorgan.com found: line 8

/home/aceuser/.ssh/known_hosts updated.

Original contents retained as /home/aceuser/.ssh/known_hosts.old

[aceuser@tuat2ace01 .ssh]$

b) for an EG level known_hosts file - 

[aceuser@ config]$ ssh-keygen -f /var/mqsi/components/Node-Name/servers/EG-Name/config/known_hosts -R [remote.test.com]:2525

# Host [remote.test.com]:2525 found: line 1

# Host [remote.test.com]:2525 found: line 2

/var/mqsi/components/Node-Name/servers/EG-Name/config/known_hosts updated.

Original contents retained as /var/mqsi/components/Node-Name/servers/EG-Name/config/known_hosts.old

[aceuser@ config]$ 

for reference on SSH-keygen tool commands - https://man7.org/linux/man-pages/man1/ssh-keygen.1.html