Enhanced JGroups configuration

Infinispan uses JGroups as its underlying clustering layer. In order to configure the finer details of clustering (discovery, flow control, cross-site, etc) you have to provide a separate XML file with the desired configuration and reference it from your Infinispan XML file as follows:

	<infinispan>
	<jgroups>
	<!-- Load external JGroups stacks -->
	<stack-file name="jgroups-udp" path="jgroups-udp.xml"/>
	</jgroups>
	<cache-container name="clustered">
	<!-- Use the stack -->
	<transport cluster="cluster" stack="jgroups-udp"/>
	...
	</cache-container>
	</infinispan>

view raw infinispan-jgroups-stackfile.xml hosted with ❤ by GitHub

For simple configurations this is usually fine, but configuring complex setups, such as cross-site replication, means juggling multiple files (one for the local stack, one for the cross-site stack and one for the relay configuration).

Starting with Infinispan 10 Alpha2 we have introduced a number of changes to make your life with JGroups configurations a lot easier.

Default stacks

Infinispan now comes with two pre-declared stacks: tcp and udp. Using them is as simple as just referencing their names in the <transport> element.

	<infinispan>
	<cache-container name="clustered">
	<!-- Use the default udp stack -->
	<transport cluster="cluster" stack="udp"/>
	...
	</cache-container>
	</infinispan>

view raw infinispan-jgroups-defaults.xml hosted with ❤ by GitHub

Inline stacks

Inlining a stack means you can put the JGroups configuration inside the Infinispan one as follows:

	<infinispan
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="urn:infinispan:config:10.0 http://www.infinispan.org/schemas/infinispan-config-10.0.xsd
	urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.0.xsd"
	xmlns="urn:infinispan:config:10.0"
	xmlns:ispn="urn:infinispan:config:10.0">
	<jgroups>
	<!-- Inline our own JGroups stack -->
	<stack name="mping">
	<TCP bind_port="7800" port_range="30" recv_buf_size="20000000" send_buf_size="640000"
	sock_conn_timeout="300" bundler_type="no-bundler"
	thread_pool.min_threads="0" thread_pool.max_threads="25" thread_pool.keep_alive_time="5000"
	xmlns="urn:org:jgroups"/>
	<MPING bind_addr="127.0.0.1" break_on_coord_rsp="true"
	mcast_addr="${jgroups.mping.mcast_addr:228.2.4.6}"
	mcast_port="${jgroups.mping.mcast_port:43366}"
	ip_ttl="${jgroups.udp.ip_ttl:2}"
	xmlns="urn:org:jgroups"/>
	<MERGE3 xmlns="urn:org:jgroups"/>
	<FD_SOCK xmlns="urn:org:jgroups"/>
	<FD_ALL timeout="3000"
	interval="1000"
	timeout_check_interval="1000"
	xmlns="urn:org:jgroups"
	/>
	<VERIFY_SUSPECT timeout="1000" xmlns="urn:org:jgroups"/>
	<pbcast.NAKACK2
	use_mcast_xmit="false"
	xmit_interval="100"
	xmit_table_num_rows="50"
	xmit_table_msgs_per_row="1024"
	xmit_table_max_compaction_time="30000"
	xmlns="urn:org:jgroups"/>
	<UNICAST3
	xmit_interval="100"
	xmit_table_num_rows="50"
	xmit_table_msgs_per_row="1024"
	xmit_table_max_compaction_time="30000"
	xmlns="urn:org:jgroups"
	/>
	<RSVP xmlns="urn:org:jgroups"/>
	<pbcast.STABLE stability_delay="200"
	desired_avg_gossip="2000"
	max_bytes="1M"
	xmlns="urn:org:jgroups"
	/>
	<pbcast.GMS print_local_addr="false"
	join_timeout="${jgroups.join_timeout:2000}"
	xmlns="urn:org:jgroups"/>
	<MFC max_credits="2M" min_threshold="0.40" xmlns="urn:org:jgroups"/>
	<FRAG3 xmlns="urn:org:jgroups"/>
	</stack>
	</jgroups>
	<cache-container name="clustered">
	<!-- Use the stack -->
	<transport cluster="cluster" stack="mping"/>
	...
	</cache-container>
	</infinispan>

view raw infinispan-jgroups-inline.xml hosted with ❤ by GitHub

You can use the full JGroups schema, and by using XML namespaces you get full validation.

Stack inheritance

Most of the time you want to reuse one of the pre-declared stacks but just override some of the parameters (e.g. discovery) to suit your environment. The following example creates a new tcpgossip stack which is based on the default tcp stack but replaces the discovery protocol with TCPGOSSIP:

	<infinispan>
	<jgroups>
	<!-- Use the "tcp" stack but replace the discovery -->
	<stack name="tcpgossip" extends="tcp">
	<MPING ispn:stack.combine="REMOVE" xmlns="urn:org:jgroups"/>
	<TCPGOSSIP initial_hosts="${jgroups.tunnel.gossip_router_hosts:localhost[12001]}" ispn:stack.combine="INSERT_AFTER" ispn:stack.position="TCP" xmlns="urn:org:jgroups"/>
	</stack>
	</jgroups>
	<cache-container name="clustered">
	<!-- Use the stack -->
	<transport cluster="cluster" stack="tcpgossip"/>
	...
	</cache-container>
	</infinispan>

view raw infinispan-jgroups-tcpgossip.xml hosted with ❤ by GitHub

In the above example you can see that we have enhanced the JGroups protocol declarations with two new attributes: ispn:stack.combine and ispn:stack.position which affect how and where protocol changes are applied on the parent configuration to obtain a new configuration. stack.combine can be one of COMBINE (the default, possibly overriding any specified attributes), REPLACE (which completely replaces the protocol and resets all attributes), REMOVE (removes the protocol) and INSERT_AFTER (which places this protocol in the stack immediately after the protocol specified by stack.position).

Multiple stacks and Cross-site

The inline configuration really shows its usefulness in cross-site configurations. In fact, the JGroups stack declaration has been extended with a special element which replaces the need for a separate relay XML file and can reference other stacks just by name. The following configuration uses the default udp stack for the local cluster transport and uses the default tcp stack for connecting to a remote site:

	<infinispan>
	<jgroups>
	<stack name="xsite" extends="udp">
	<relay.RELAY2 site="LON" xmlns="urn:org:jgroups"/>
	<remote-sites default-stack="tcp">
	<remote-site name="NYC"/>
	</remote-sites>
	</stack>
	</jgroups>
	<cache-container name="clustered">
	<!-- Use the stack -->
	<transport cluster="cluster" stack="xsite"/>
	...
	</cache-container>
	</infinispan>

view raw infinispan-jgroups-xsite.xml hosted with ❤ by GitHub

Having the entire configuration in a single place greatly simplifies management. Of course you can combine all of the above features to obtain the configuration you need for your environment. You can find more details and examples in the documentation.
Enjoy !
Tristan