Check for new replies
Thread Rating:
  • 127 Vote(s) - 3.05 Average
  • 1
  • 2
  • 3
  • 4
  • 5
REFRESH CLUSTER - with scenarios and demo.
#1
What does Refresh Cluster Command Do?

REFRESH CLUSTER command discards all locally held information about a cluster and rebuilds that information from the full repositories in the cluster.

Every queue manager stores the information about the clusters and the objects in the clusters it is part of in a local queue called

SYSTEM.CLUSTER.REPOSITORY.QUEUE.

So a refresh cluster would refresh the messages stored in this queue and populate with the information received from the full repository.
This doesn't mean that for the queue managers to receive updates about cluster it is part of or for the queue manager to update full repository with the information about clustered objects it is part of would need a refresh cluster. These things happen automatically.

Basically any updates to the cluster objects are automatically propagated to the full repository, and whenever an application puts a message to a clustered queue, partial repository checks with the full repository for any updates about the clustered queue if there are any before putting the message to the destination.

Hence, everything will remain up to date and both full repositories and partial repositories will remain in sync always w.r.t the clusters they are part of.
However if a cluster is not properly configured, or in some rare scenarios where the partial repository doesn't receive any updates about clustered objects from full repository after 30 days it might require refresh cluster command to be run.

Some of the scenarios which might need a refresh cluster are:

• Same name cluster receiver channels created in different queue managers which are part of a cluster. Hence full repository will have duplicate information with different CONNAME entries for same channel name. This will result in confusion for full repositories. Once the setup is rectified it might need a refresh cluster to set things right in the cluster.

• Same name queue managers are joined to cluster. This again will lead to confusion in full repositories. This is a blunder again and should be avoided at any cost. But once the setup is rectified, it might need a refresh cluster as well.

• Cluster receiver channel’s CONNAME attribute was modified in full repositories while they could not communicate the same to rest of the queue managers in cluster due to issues like cluster sender channel in retrying mode, cluster queues corrupted or permission issues etc.
This will result in partial repositories having incorrect information since it was not propagated from full repository. This might need refresh cluster command once the setup is rectified.

• A queue manager forcefully removed from cluster by mistake (RESET CLUSTER ACTION (FORCEREMOVE)). If this happens, then to rejoin the queue manager to cluster, a refresh command will be required.

• No update received in partial repository regarding a clustered queue from full repository for more than 30 days. This will result in clustered queue info being removed from partial repository after additional 60 days grace period. So the clustered queue info will be removed from queue manager completely after a total period of 90 days which will result in 2085 error (object not found).
An automatic update will be received before 30 days from full repository, but any failure will require a refresh cluster command to refresh the cluster information.

• Removing a queue manager from cluster might sometimes lead to issues. This might need a refresh cluster command. However it should be solved by other methods.

• There are some many documented cases like queue manager start-up from a backup from a previous time which will need refresh cluster as well.

However in general refresh cluster has to be avoided if possible. Steps like changing the clustered objects attributes to push it across the cluster, suspending the queue manager with incorrect config and resuming once the configuration is corrected, even restarting the queue manager - to restart the repository process(amqrrmfa process) etc have to be taken before ultimately deciding to go with refresh cluster command.

Why Refresh cluster Shouldn’t be run genarally?

• Refresh Cluster will lead to all the locally held information to be refreshed. This might result in momentary clustered queues not being visible which will result in applications trying to put message to clustered queue getting a ‘MQRC_NO_DESTINATIONS_AVAILABLE’ error.

• Refresh cluster will lead to additional load on repository queue managers to re-propagate the cluster resources to the queue manager where the command was run. In large cluster, if this required, it should be run during non-business hours.

• Refresh cluster in full repository will lead to large volumes of messages since, it has to populate its repository with information regarding all the clusters it is part of. This will take time and also puts a lot of work load.

Command:

Refresh Cluster(Cluster_name) Repos(Yes or No)

Repos attribute is NO by default.

Repos = No --> signifies that the locally created clustered objects and clustered objects in repository queue managers won’t be refreshed. All others will be refreshed.
It Can be run in partial as well has full repository.

Repos = Yes -->signifies all the cluster information including the objects defined in full repository is refreshed.
It cannot be run in repository queue manager.


Scenario1:

A queue manager has been removed from cluster in full repository by mistake using reset cluster command. This will need a refresh cluster from partial repository for it to rejoin the cluster.

To demonstrate the use of refresh cluster, I am going to create the below setup.

Full Repository queue manager: QMA
Partial Repository Queue managers: QMB and QMC.
Cluster Name: ACLUSTER
Listener port for QMA, QMB and QMC queue managers: 1414, 1415 and 1416 respectively.

The commands to create the cluster setup have been explained in the scenarios 2 below.

Step 1: Complete the Cluster setup explained in scenario 2.

Step 2: Remove the QMB queue manager from full repository queue manager QMA forcefully.

Reset cluster(ACLUSTER) action(forceremove) qmname(QMB)

[Image: Pt4z8c0.png]

Step 3: Refresh the cluster and make the QMB queue manager part of cluster again.

Run refresh cluster from QMB queue manager and check if it is propagated to full repository by dis clusqmgr(*) command.

[Image: qQqYimk.png]

It can be seen that the queue manager QMB is successfully added to the cluster ACLUSTER again.

Scenario 2: Demonstrate refresh cluster command usage in case of duplicate Cluster receiver channel in different queue managers.

I am going to create same cluster receiver channels in two different queue managers.

Same name Cluster Receiver channel that will be created in both QMB and QMC queue managers: QMB.QMC.CLUSRCVR

Step 1: Setup clustering with duplicate cluster receiver channels in QMB and QMC queue managers

Commands to create the setup:

In QMA Queue manager:


1. Make the Queue manager as full repository:

Alter qmgr repos(ACLUSTER)

2. Create a cluster receiver channel which is shared in cluster.

Define chl(TO.QMA) chltype(clusrcvr) conname(’localhost(1414)’) cluster(ACLUSTER)

In QMB Queue manager:

1. Create the Cluster Receiver channel which is shared in cluster:

Define chl(QMB.QMC.CLUSRCVR) chltype(clusrcvr) conname(’localhost(1415)’) cluster(ACLUSTER)

2. Create the Cluster Sender channel to full repository queue manager QMA which is shared in cluster:

Define chl(TO.QMA) chltype(clussdr) conname(’localhost(1414)’) cluster(ACLUSTER)


In QMC Queue manager:

1. Create the Cluster Receiver channel which is shared in cluster:

Define chl(QMB.QMC.CLUSRCVR) chltype(clusrcvr) conname(’localhost(1416)’) cluster(ACLUSTER)

Note: the same channel is already existing in QMB Queue manager.

2. Create the Cluster Sender channel to full repository queue manager QMA which is shared in cluster:

Define chl(To.QMA) chltype(clussdr) conname(’localhost(1414)’) cluster(ACLUSTER)

[Image: cr94sOo.png]


Step 2: Validate Clustering

So with this setup, the full repository queue manager will have duplicate entries of cluster receiver channels with different CONNAME entries for QMB and QMC queue managers.

The below screenshot shows that the full repository Queue manager QMA has updated itself with information about both queue managers QMB and QMC. But the cluster receiver channel for both queue managers is same which will cause confusion in full repository.

[Image: kpTXUK6.png]

This will result in cluster receiver channel starting only in one queue manager out of QMB and QMC. In our case its QMB since it was joined to the cluster first. So QMB is successfully joined to the cluster, but same cannot be said about QMC queue manager.

[Image: 3FgvAMQ.png]

It can be observed that in QMC queue manager the cluster receiver channel won’t start. Since the connection is not established between full repository and partial repository, dis clusqmgr( *) command shows the full repository queue manager name as “SYSTEM.TEMPQMGR*” instead of QMA which indicates that there are connection problems between the two queue managers.

[Image: GAamxGt.png]

Step 3: Delete the duplicate channel and create a new cluster receiver channel and refresh the cluster.

[Image: Y6AlocP.png]

It can be seen that the issue is solved now.

Step 4: Let’s try to solve the issue without running refresh cluster command

Recreate the setup with duplicate cluster receiver channels i.e till step 2.

Without running the refresh cluster, try to remove QMC Qmanager from the cluster and then rejoin it with the correct cluster receiver channel to see if it solves the issue.

To remove queue manager QMC from cluster the following steps have to be taken.

1. Suspend QMC queue manager from cluster.
2. Remove the cluster receiver channel from cluster.
3. Stop and delete the cluster receiver channel
4. Stop and delete the cluster sender channel

The commands are as below:

Suspend qmgr cluster(ACLUSTER)
Alter chl(QMB.QMC.CLUSRCVR) chltype(clusrcvr) cluster(‘’)
Delete chl(QMB.QMC.CLUSRCVR)
Stop chl(TO.QMA)
Delete chl(TO.QMA)


[Image: vlpXSfQ.png]

Rejoin the QMC queue manager to the cluster with correct channel definitions:

Define chl(TO.QMC) chltype(clusrcvr) conname(’localhost(1416)’) cluster(ACLUSTER)
Define chl(TO.QMA) chltype(clussdr) conname(’localhost(1414)’) cluster(ACLUSTER)
Resume qmgr cluster(ACLUSTER)


[Image: IPA4AB0.png]

It can be seen the issue is solved with this method. Full repository has updated itself with the information of QMC queue manager.

[Image: XTd1O4S.png]


I have tried this with MQ 7 version. With previous versions I used to run into issues while doing it and there were a few times where i actually had to run refresh cluster. With the new versions, a simple delete the duplicate channel and creating a new one with correct parameters will also solve the issue.

Cheers,
Vinyas
Reply

Check for new replies

Forum Jump:


Users browsing this thread: 1 Guest(s)