Overview

You are attempting to perform a configuration update on a batch of devices as they are new to the ACS or the configuration is synchronized from a timer-driven policy. You notice that these batches of device configurations are taking an exponentially more extended time to finish than when you configure the devices individually. (1-2 minutes apiece, but 20-90 minutes for a batch of 20 devices)

When reviewing the logging on the server, you notice no apparent signs of any errors responsible for the increased configuration times. You are running Service Gateway on a JBoss or Wildfly environment.

 

Solution

Service Gateway uses JMS (Java Messaging System) for a lot of internal functionality, and JBoss defines a default number of 100 listeners for processing any messages posted on the JMS queues. In some situations of heavier load, the JMS messages can encounter trouble processing concurrently due to reaching the defined max listener "concurrency slots." 

Once all the message listeners are busy, any other JMS messages posted to the queue sit there until a listener is available to process them. In these cases, you can make adjustments to increase the maximum number of listeners available and resolve these bottlenecks. The steps below will guide you to modify the ejb-jar.xml deployment descriptor within your Service Gateway EAR file and your servicegateway-domain.xml.

 

Updating the ejb-jar.xml

1. Unpack the EAR and collect the ejb-jar.xml file.
   1. Locate the Service Gateway EAR file on your server.
      1. Default Path: /usr/local/sprt/<instance_name>/servicegateway/
   2. Unpack the EAR using your preferred archive management software, such as 7Zip or gZip.
   3. Extract the ejb-jar.xml file.
      1. Default Path: /servicegateway_baseplugins.jar/META-INF/
2. ​Modifying the ejb-jar.xml
   1. By default, without any existing modifications, you will see the following within this file:

      <?xml version="1.0" encoding="UTF-8"?>
      <ejb-jar version="3.2" xmlns="http://xmlns.jcp.org/xml/ns/javaee" 
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
      xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee 
      http://xmlns.jcp.org/xml/ns/javaee/ejb-jar_3_2.xsd">
        <display-name>servicegateway_baseplugins</display-name>
      </ejb-jar>

   2. The two most important listeners related to this trouble are CwmpParameterTechConfigPluginMDB and ConfigSyncTechConfigResponseListenerMDB. 

   3. Append the following <enterprise-beans> snippet within the <ejb-jar> tags below the <display-name> tag:

      <enterprise-beans>
      <message-driven>
       <ejb-name>CwmpParameterTechConfigPluginMDB</ejb-name>
       <activation-config>
        <activation-config-property>
         <activation-config-property-name>maxSession</activation-config-property-name>
         <activation-config-property-value>100</activation-config-property-value>
        </activation-config-property>
       </activation-config>
      </message-driven>
      
      <message-driven>
       <ejb-name>ConfigSyncTechConfigResponseListenerMDB</ejb-name>
       <activation-config>
        <activation-config-property>
         <activation-config-property-name>maxSession</activation-config-property-name>
         <activation-config-property-value>100</activation-config-property-value>
        </activation-config-property>
       </activation-config>
      </message-driven>
      </enterprise-beans>

      • The "activation-config-property-value" property represents the maximum number of listener sessions. The values of 100 above are the default, and you should fine-tune this value based on the needs within your environment.

      • Note: If you have existing modifications within the enterprise-beans section of your ejb-jar.xml file, you can instead add the two <message-driven> snippets within the existing section.

   4. After making the desired changes to your max listener count, save the changes to the file.

3. Repacking and redeploying the EAR file.

   1. Replace the original copy of the ejb-jar.xml file within /servicegateway_baseplugins.jar/META-INF/ with your modified copy.

   2. Then, using your chosen archive management software, repack the EAR file and redeploy the EAR to your server.

 

Updating the servicegateway-domain.xml

1. Locate the file within your managed instance.
   1. Default Path: /usr/local/sprt/<managed_instance>/servicegateway/domain/configuration
   2. Note: There are two copies of this file, but you will want to edit the copy at the path above.
2. Open the file within your preferred text editor.
3. Search for the "mdb-strict-max-pool" parameter within the "full-ha" profile.
   1. By default, you will see the following appear within the file (at approximately line 1144):
      

      <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" 
      instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

   2. Adjust this parameter to the following:
      

      <strict-max-pool name="mdb-strict-max-pool" max-pool-size="<your_desired_value>" 
      instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

   3. Note: Adjust the max-pool-size value based on your requirements and available resources. The value should be greater than or equal to the two maxSession values defined within the ejb-jar.xml above. (For example, if you defined the maxSession values to the default 100 each, this value should be at least 200.)
4. Restart your App Server (Wildfly, JBoss, etc.) to apply the changes.

 

 

Testing

After redeploying the modified EAR, modifying the XML, and restarting your Service Gateway servers, re-attempt the batch configuration of the devices and verify that they are now correctly queuing with the maximum listener count that you defined.

If needed, you can use the steps above to increase these values further and monitor the changes until you find the perfect value to accommodate your increased volume within your environment.