Day-to-day Problems & Answers: March 2017

This post explain how to setup a WSO2 ESB cluster, here I will be using ESB 4.9, MySQL as the underline DB and NginX as the load balancer. Following diagram depicts the overall distribution of the cluster components.

Following are the host names IPs of each node.

No
Node Name
Host Name
IP
Local Member Port
1
Manager
mgt.esb.cl2.wso2.com
192.168.48.243
4001
2
Worker 01
esb.cl2.wso2.com
192.168.48.244
4001
3
Worker 02
esb.cl2.wso2.com
192.168.48.249
4001

Requisites.

So for the cluster I assume you have the following prerequisites fulfilled.

1. MySQL setup
2. NginX installed and configured
3. SVN server to use as the deployment synchronizer.

Note : I will be explaining how to setup the cluster in a distributed setup, meaning one instance will host one ESB instance. So we will not see any port conflicts. If you need setup ESB cluster in a single machine or want to host multiple instnces in a single machine you need to offset each node to avoid port conflicts. To do that you can open <ESB_HOME>/repository/conf/carbon.xml and change the following property.

<Offset>0</Offset>

Also you need to specify unique local member ports for each node. This can be done in <ESB_HOME>/repository/conf/axis2/axis2.xml under the following proerty

<parameter name="localMemberPort">4100</parameter>

Configuring the Manager Node

1. Open <ESB_HOME>/repository/conf/datasources/master-datasources.xml and add following two datasources. Do not remove the existing carbon datasource.

<datasource>
<name>WSO2_REG_DB</name>
<description>The datasource used for shared config and governance registry</description>
<jndiConfig>
<name>jdbc/WSO2SharedDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://192.168.48.6:3306/ESB49_REGDB</url>
<username>esbuser</username>
<password>wso2root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>

<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UmDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://192.168.48.6:3306/ESB49_USERDB</url>
<username>esbuser</username>
<password>wso2root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>

Note : Make sure you don't remove WSO2_CARBON_DB datasource. WSO2_CARBON_DB will be used as the local registry of each node which will point to the embedded H2 Database, WSO2_UM_DB will be used for user namagement and WSO2_REG_DB will be used as the shared registry among the cluster.

2. Now open <ESB_HOME>/repository/conf/carbon.xml

a. Change the following parameters to add the necessary host names

<HostName>esb.cl2.wso2.com</HostName>
<MgtHostName>mgt.esb.cl2.wso2.com</MgtHostName>

b. Inorder to set the Deploymeny syncronizer do the changes as shown below. Note that AutoCommit option is set to true in the manager node

<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>true</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://192.168.42.273/svn/wso2qarepo/esb49repo</SvnUrl>
<SvnUser>wso2user1</SvnUser>
<SvnPassword>wso2pass123</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

3. Now open <ESB_HOME>/repository/conf/user-mgt.xml and change the following property as shown below.

<Property name="dataSource">jdbc/WSO2UmDB</Property>

Note : jdbc/WSO2UmDB is the JNDI Name of the datasource we declaired in step 01 for user management DB.

4. Open <ESB_HOME>/repository/conf/registry.xml and add the following entries. Do not delete existing entries.

<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2SharedDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9444/registry">
<id>instanceid</id>
<dbConfig>sharedregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
<cacheId>esbuser@jdbc:mysql://192.168.48.6:3306/ESB49_REGDB</cacheId>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>

5. Now lets do cluster related configs. Open <ESB_HOME>/repository/conf/axis2/axis2.xml do the following changes.

a. Enable clustering as shown below. make enable true in the following property.

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

b. Make sure that membershipScheme is set to wka (well known address based schema)

<parameter name="membershipScheme">wka</parameter>

c. Set the domain, all cluster nodes will join this domain. This is only used for the members to join the cluster.

<parameter name="domain">wso2.esb49.cl2.domain</parameter>

d. Set the local member host, add the IP of the manager node to this.

<parameter name="localMemberHost">192.168.48.243</parameter>

e. Specify the localmember port to 4100.

Note : if you are hosting all the nodes in the same machine each node should have a unique member port.

<parameter name="localMemberPort">4100</parameter>

f. Now lets add well known members.

<members>
<member>
<hostName>192.168.48.244</hostName>
<port>4100</port>
</member>
</members>

Note : Here I'm specifying worker 01 as the well-known member, ideally you can specify all the nodes as well known members. But its optional. When a node goes down the well known address will be used to reconnect to the cluster by the disrupted node.

g. Uncomment and edit the WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener and
org.apache.synapse.transport.passthru.PassThroughHttpSSLListener in the transportReceiver. Each specifies HTTP and HTTP endpoints for the WSDL url.

<parameter name="WSDLEPRPrefix" locked="false">https://esb.cl2.wso2.com:443</parameter>
<parameter name="WSDLEPRPrefix" locked="false">http://esb.cl2.wso2.com:80</parameter>

6. Now Open <ESB_HOME>repository/conf/tomcat/catalina-server.xml and add proxy ports

Under HTTP connection section add the following parameter.

proxyPort="80"

Full HTTP section will look like following,

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9763"
proxyPort="80"
redirectPort="9443"
bindOnInit="false"
maxHttpHeaderSize="8192"
acceptorThreadCount="2"
maxThreads="250"
minSpareThreads="50"
disableUploadTimeout="false"
connectionUploadTimeout="120000"
maxKeepAliveRequests="200"
acceptCount="200"
server="WSO2 Carbon Server"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
URIEncoding="UTF-8"/>

Now under HTTPs section add the following proxy port

proxyPort="443"

The Full config section will look like following

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443"
proxyPort="443"
bindOnInit="false"
sslProtocol="TLS"
maxHttpHeaderSize="8192"
acceptorThreadCount="2"
maxThreads="250"
minSpareThreads="50"
disableUploadTimeout="false"
enableLookups="false"
connectionUploadTimeout="120000"
maxKeepAliveRequests="200"
acceptCount="200"
server="WSO2 Carbon Server"
clientAuth="false"
compression="on"
scheme="https"
secure="true"
SSLEnabled="true"
keystoreFile="${carbon.home}/repository/resources/security/wso2carbon.jks"
keystorePass="wso2carbon"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
URIEncoding="UTF-8"/>

That's it, we have completed configuring the manager node. Now lets configure the worker node. Worker node is very similar to manager nodes configurations. But I will walk though worker 01's configurations so you won't miss anything :)

Configuring the Worker Node

1. Open <ESB_HOME>/repository/conf/datasources/master-datasources.xml and add following two datasources. Do not remove the existing carbon datasource.

<datasource>
<name>WSO2_REG_DB</name>
<description>The datasource used for shared config and governance registry</description>
<jndiConfig>
<name>jdbc/WSO2SharedDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://192.168.48.6:3306/ESB49_REGDB</url>
<username>esbuser</username>
<password>wso2root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>

<datasource>
<name>WSO2_UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UmDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://192.168.48.6:3306/ESB49_USERDB</url>
<username>esbuser</username>
<password>wso2root</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>

Note : Make sure you don't remove WSO2_CARBON_DB datasource.

2. Now open <ESB_HOME>/repository/conf/carbon.xml

a. Change the following parameters to add the necessary host names

<HostName>esb.cl2.wso2.com</HostName>
<MgtHostName>mgt.esb.cl2.wso2.com</MgtHostName>

b. Inorder to set the Deploymeny syncronizer do the changes as shown below. Note that AutoCommit option is set to false in the worker node

<DeploymentSynchronizer>
<Enabled>true</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>true</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://192.168.42.273/svn/wso2qarepo/esb49repo</SvnUrl>
<SvnUser>wso2user1</SvnUser>
<SvnPassword>wso2pass123</SvnPassword>
<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
</DeploymentSynchronizer>

3. Now open <ESB_HOME>/repository/conf/user-mgt.xml and change the following property as shown below.

<Property name="dataSource">jdbc/WSO2UmDB</Property>

Note : jdbc/WSO2UmDB the JNDI Name of the datasource we declaired in step 01 for user management DB.

4. Open <ESB_HOME>/repository/conf/registry.xml and add the following extries. Do not delete existing entries.

<dbConfig name="sharedregistry">
<dataSource>jdbc/WSO2SharedDB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost:9444/registry">
<id>instanceid</id>
<dbConfig>sharedregistry</dbConfig>
<readOnly>true</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
<cacheId>esbuser@jdbc:mysql://192.168.48.6:3306/ESB49_REGDB</cacheId>
</remoteInstance>
<mount path="/_system/config" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/esbnodes</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
<instanceId>instanceid</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>

Note: Note that I have specified readOnly property as false. The worker should not add any artifacts to the registry.

5. Now lets do cluster related configs. Open <ESB_HOME>/repository/conf/axis2/axis2.xml do the following changes.

a. Enable clustering as shown below. make enable true in the following property.

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

b. Make sure that membershipScheme is set to wka (well known address based schema)

<parameter name="membershipScheme">wka</parameter>

c. Set the domain, all cluster nodes will join this domain. This is only used for the members to join the cluster.

<parameter name="domain">wso2.esb49.cl2.domain</parameter>

d. Set the local member host, add the IP of the worker 01 node to this.

<parameter name="localMemberHost">192.168.48.244</parameter>

e. Specify the localmember port to 4100.

<parameter name="localMemberPort">4100</parameter>

Note : if you are hosting all the nodes in the same machine each node should have a unique member port.
f. Now lets add well known members.

<members>
<member>
<hostName>192.168.48.243</hostName>
<port>4100</port>
</member>
<member>
<hostName>192.168.48.249</hostName>
<port>4100</port>
</member>

</members>

Note : Here I'm specifying manager and worker 02 as the well-known member.

g. Uncomment and edit the WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener and
org.apache.synapse.transport.passthru.PassThroughHttpSSLListener in the transportReceiver. Each specifies HTTP and HTTP endpoints for the WSDL url.

<parameter name="WSDLEPRPrefix" locked="false">https://esb.cl2.wso2.com:443</parameter>
<parameter name="WSDLEPRPrefix" locked="false">http://esb.cl2.wso2.com:80</parameter>

6. Now Open <ESB_HOME>repository/conf/tomcat/catalina-server.xml and add proxy ports

Under HTTP connection section add the following parameter.

proxyPort="80"

Full HTTP section will look like following,

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9763"
proxyPort="80"
redirectPort="9443"
bindOnInit="false"
maxHttpHeaderSize="8192"
acceptorThreadCount="2"
maxThreads="250"
minSpareThreads="50"
disableUploadTimeout="false"
connectionUploadTimeout="120000"
maxKeepAliveRequests="200"
acceptCount="200"
server="WSO2 Carbon Server"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
URIEncoding="UTF-8"/>

Now under HTTPs section add the following proxy port

proxyPort="443"
The Full config section will look like following

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="9443"
proxyPort="443"
bindOnInit="false"
sslProtocol="TLS"
maxHttpHeaderSize="8192"
acceptorThreadCount="2"
maxThreads="250"
minSpareThreads="50"
disableUploadTimeout="false"
enableLookups="false"
connectionUploadTimeout="120000"
maxKeepAliveRequests="200"
acceptCount="200"
server="WSO2 Carbon Server"
clientAuth="false"
compression="on"
scheme="https"
secure="true"
SSLEnabled="true"
keystoreFile="${carbon.home}/repository/resources/security/wso2carbon.jks"
keystorePass="wso2carbon"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/javascript,application/x-javascript,application/javascript,application/xml,text/css,application/xslt+xml,text/xsl,image/gif,image/jpg,image/jpeg"
URIEncoding="UTF-8"/>

7. Open <ESB_HOME>/bin/wso2server.sh and serch for DworkerNode parameter and change the value to true as shown below.

-DworkerNode=false \

So we have configured one of the worker noded, worker node 02s configurations are identical to the worker 01s change the nesassary ips in relevent places when configuring worker 02.

Copying necessary Drivers and libraries

You need to copy following drivers to both Worker and Manager nodes.

JDBC Driver

Copy MYSQL JDBC driver to <ESB_HOME>/repository/components/lib directory. You can find the matching driver for your mysql version from here.

SVN related JARs

1. Download and copy SVNKit from http://product-dist.wso2.com/tools/svnkit-all-1.8.7.wso2v1.jar into the <ESB_HOME>/repository/components/dropins folder.

2. Download http://maven.wso2.org/nexus/content/groups/wso2-public/com/trilead/trilead-ssh2/1.0.0-build215/trilead-ssh2-1.0.0-build215.jar and copy it to the <ESB_HOME>/repository/components/lib folder.

Configuring NginX

So ESB has two host names, management console host and worker host entries. And ESB also exposes 4 different ports (9443/9763, 8280/8243). NginX will be listening to system default ports 80/443, and it will route all the requests to the respective ESB node. The NginX configs will look like following.

create a new NginX conf file at /etc/nginx/conf.d and add the following content to it.

Note : I'm using my IPs and host names, you should change these according to your setup.

upstream esb49wkhttps {

server 192.168.48.244:8243;
server 192.168.48.249:8243;
}

upstream esb49wkhttp {

server 192.168.48.244:8280;
server 192.168.48.249:8280;
}

upstream esb49mgt {
server 192.168.48.243:9443;
}

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

server {
listen 80;
server_name esb.cl2.wso2.com;

location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;
proxy_read_timeout 5m;
proxy_send_timeout 5m;

proxy_pass http://esb49wkhttp/;
proxy_redirect http://esb49wkhttp/ http://gw.am.wso2.com/;

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
}
}

server {

listen 443;
server_name esb.cl2.wso2.com;

ssl on;
ssl_certificate /etc/nginx/ssl/am_wso2_com.crt;
ssl_certificate_key /etc/nginx/ssl/am_wso2_com.key;

location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;
proxy_read_timeout 5m;
proxy_send_timeout 5m;

proxy_pass https://esb49wkhttps/;
proxy_redirect https://esb49wkhttps/ https://esb.cl2.wso2.com/;

#proxy_set_header Upgrade $http_upgrade;
#proxy_set_header Connection "upgrade";
#proxy_http_version 1.1;
}
}

server {

listen 443;
server_name mgt.esb.cl2.wso2.com;

ssl on;
ssl_certificate /etc/nginx/ssl/am_wso2_com.crt;
ssl_certificate_key /etc/nginx/ssl/am_wso2_com.key;

location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;
proxy_read_timeout 5m;
proxy_send_timeout 5m;

proxy_pass https://esb49mgt/;
proxy_redirect https://esb49mgt/ https://mgt.esb.cl2.wso2.com/;

#proxy_set_header Upgrade $http_upgrade;
#proxy_set_header Connection "upgrade";
#proxy_http_version 1.1;
}
}

So that's it after adding the configs to NginX you need to restart or reload NginX.

Starting the cluster and accessing the cluster

If you haven't already created the necessary tables in the DBs you can start the servers in the following way to create necessary tables automatically.

./wso2server.sh -Dsetup

After starting the manager node you can start workers one by one, if the members successfully join the cluster a log will be printed like following.

TID: [-1] [] [2015-08-21 12:07:50,815] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [6974ca1c-8403-4711-9408-9de0cfaadda2]: /192.168.48.244:4100 {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
TID: [-1] [] [2015-08-21 12:08:05,043] INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - Member joined [3ebb327b-91db-4d98-8c8a-95e2604e3a9c]: /192.168.48.249:4100 {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
TID: [-1] [] [2015-08-21 12:08:07,159] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:192.168.48.249, Remote Host:null, Port: 4100, HTTP:8280, HTTPS:8243, Domain: wso2.esb49.cl2.domain, Sub-domain:__$default, Active:true {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils}

After successfully starting the cluster you should be able to access the cluster in the following manner.
https://mgt.esb.cl2.wso2.com/carbon

Note : Make sure you add necessary host entries to your /etc/hosts file. So you can access the URLs with the host names. Your host names should resolve to the NginX node.

So that's it we have sucessfully created a ESB cluster,

Day-to-day Problems & Answers

Grep & Split log file

How to remove all .svn files recursively.?

Clustering WSO2 ESB 4.9 with NginX

How To Install Nginx on Ubuntu 16.04

See the contents of a tar.gz file without extracting