Friday, 27 September 2013

Infinispan 6.0.0.Beta2 is released!

Dear Infinispan community,

Staying committed to the software development philosophy of "Release early. Release often. And listen to your customers" we are releasing Infinispan 6.0.0.Beta2 today. This is mainly a stabilization release after a flurry of new features released in Beta1. The Beta2 release contains a few fixes related to hotrod remote clients as well as some minor fixes in LevelDB cache store.

For a complete list of features and fixes included in this release please refer to the release notes. Visit our downloads section to find the latest release and if you have any questions please check our forums, our mailing lists or ping us directly on IRC.


Thursday, 26 September 2013

Embedded and remote queries in Infinispan 6.0.0.Beta1

If you're following Infinispan's mailing lists you've probably caught a glimpse of the new developments in the Query land: a new DSL, remote querying via Hot Rod client, a new marshaller based on Google's Protobuf. Time to unveil these properly!

The new Query DSL

Starting with version 6.0 Infinispan offers a new (experimental) way of running queries against your cached entities based on a simple filtering DSL. The aim of the new DSL is to simplify the way you write queries and to be agnostic of the underlying query mechanism(s) making it possible to provide alternative query engines in the future besides Lucene and still being able to use the same query language/API. The previous Hibernate Search & Lucene based approach is still in place and will continue to be supported and in fact the new DSL is currently implemented right on top of it. The future will surely bring index-less searching based on map-reduce and possibly other new cool search technologies.

Running DSL-based queries in embedded mode is almost identical to running the existing Lucene-based queries. All you need to do is have infinispan-query-dsl.jar and infinispan-query.jar in your classpath (besides Infinispan and its dependecies), enable indexing for your caches, annotate your POJO cache values and your're ready.

ConfigurationBuilder cfg = new ConfigurationBuilder();

DefaultCacheManager cacheManager = new DefaultCacheManager(;

Cache cache = cacheManager.getCache();
Alternatively, indexing (and everything else) can also be configured via XML configuration, as already described in the user guide, so we'll not delve into details here.

Your Hibernate Search annotated entity might look like this.


public class User {

    @Field(store = Store.YES, analyze = Analyze.NO)
    private String name;

    @Field(store = Store.YES, analyze = Analyze.NO, indexNullAs = Field.DEFAULT_NULL_TOKEN)
    private String surname;

    @IndexedEmbedded(indexNullAs = Field.DEFAULT_NULL_TOKEN)
    private List addresses;

    // .. the rest omitted for brevity
Running a DSL based query involves obtaining a QueryFactory from the (cache scoped) SearchManager and then constructing the query as follows:

import org.infinispan.query.Search;
import org.infinispan.query.dsl.QueryFactory;
import org.infinispan.query.dsl.Query;

QueryFactory qf = Search.getSearchManager(cache).getQueryFactory();

Query q = qf.from(User.class)

List list = q.list();

assertEquals(1, list.size());
assertEquals("John", list.get(0).getName());
assertEquals("Doe", list.get(0).getSurname());
That's it! I'm sure this raised your curiosity as to what the DSL is actually capable of so you might want to look at the list of supported filter operators in FilterConditionEndContext. Combining multiple conditions with boolean operators, including sub-conditions, is also possible:

Query q = qf.from(User.class)
    .and().not(qf.having("address.street").like("%Tanzania%").or().having("address.postCode").in("TZ13", "TZ22"))

The DSL is pretty nifty right now and will surely be expanded in the future based on your feedback. It also provides support for result pagination, sorting, projections, embedded objects, all demonstrated in QueryDslConditionsTest which I encourage you to look at until the proper user guide is published. Still, this is not a relational database, so keep in mind that all queries are written in the scope of the single targeted entity (and its embedded entities). There are no joins (yet), no correlated subqueries, no grouping or aggregations.

Moving further, probably the most exciting thing about the new DSL is using it remotely via the Hot Rod client. But to make this leap we first had to adopt a common format for storing our cache entries and marshalling them over the wire that would also be cross-language and robust enough to support evolving object schemas. But probably most of all, this format had to have a schema rather than just being an opaque blob otherwise indexing and searching are meaningless. Enter Protocol Buffers.

The Protobuf marshaller

Configuring the RemoteCacheManager of the Java Hot Rod client to use it is straight forward:

import org.infinispan.client.hotrod.configuration.ConfigurationBuilder;

ConfigurationBuilder clientBuilder = new ConfigurationBuilder();
    .marshaller(new ProtoStreamMarshaller());
Now you'll be able to store and get from the remote cache your User instaces encoded in protobuf format provided that:

1. a Protobuf type was declared for your entity in a .proto file which was then compiled into a .protobin binary descriptor

2. the binary descriptor was registered with your RemoteCacheManager's ProtoStreamMarshaller instance like this:

3. a per-entity marshaller was registered:

    .registerMarshaller(User.class, new UserMarshaller());
Steps 2 and 3 are closely tied to the way Protosteam library works, which is pretty straight forward but cannot be detailed here. Having a look at our UserMarshaller sample should clear this up.

Keeping your objects stored in protobuf format has the benefit of being able to consume them with compatible clients written in other languages. But if this does not sound enticing enough probably the fact they can now be easily indexed should be more appealing.

Remote querying via the Hot Rod client

Given a RemoteCacheManager configured as previously described the next steps to enable remote query over its caches are:

1. add the DSL jar to client's classpath, infinispan-remote-query-server.jar to server's classpath and infinispan-remote-query-client.jar to both
2. enable indexing in your cache configuration - same as for embedded mode
3. register your protobuf binary descriptor by invoking the 'registerProtofile' method of the server's ProtobufMetadataManager MBean (one instance per EmbeddedCacheManager)

All data placed in cache now is being indexed without the need to annotate your entities for Hibernate Search. In fact these classes are only meaningful to the Java client and do not even exist on the server.

Running the queries over the Hot Rod client is now very similar to embedded mode. The DSL is in fact the same. The only part that is slightly different is how you obtain the QueryFactory:

import org.infinispan.client.hotrod.Search;
import org.infinispan.query.dsl.QueryFactory;
import org.infinispan.query.dsl.Query;

remoteCache.put(2, new User("John", "Doe", 33));

QueryFactory qf = Search.getQueryFactory(remoteCache);

Query query = qf.from(User.class)

List list = query.list();
assertEquals(1, list.size());
assertEquals("John", list.get(0).getName());
assertEquals("Doe", list.get(0).getSurname());
Voila! The end of our journey for today! Stay tuned, keep an eye on Infinispan Query and please share your comments with us.

Friday, 20 September 2013

Heterogenous clusters with Infinispan 6.0.0.Beta1

One of the basic rules of participating in an Infinispan cluster was that everyone must pay their fair share. Sure, each node was able to run a diferent set of caches (i.e. the cluster didn't have to be symmetric). But when a node started up a distributed cache, it automatically took ownership of a share of the keys equal to all the other members of the cache.

With the implementation of ISPN-3051 in 6.0.0.Beta1, that is no longer the case. Each node can now opt to own more or less keys, using the new capacityFactor setting. The default capacity factor is 1.0, and nodes can specify both a higher and a lower value via configuration API:

Or via the XML configuration:
Note that we don't guarantee that the number of keys owned by a node will be exactly proportional to its capacity factor. In particular, custom ConsistentHashFactory implementations are free to ignore the capacityFactor setting completely. But the default ConsistentHashFactory will try to respect it as much as it can.

One interesting use case is nodes with a capacity factor of 0. This could be useful when some nodes are too short-lived to be useful as data owners, but they can't use HotRod (or other remote protocols) because they need transactions. With cross-site replication as well, the "site master" should only deal with forwarding commands between sites and shouldn't handle user requests, so it makes sense to configure it with a capacity factor of 0.

A final note: Infinispan server 6.0.0.Beta1 also supports this feature, but the setting is called capacity-factor to match the AS naming convention.

Thursday, 19 September 2013

Infinispan 6.0.0.Beta1 is out!

Dear Infinispan community,

We are proud to announce the first Beta release of Infinispan 6.0.0. This is an important milestone in the 6.0.0 lifecycle: it is feature and API complete.

Included in this release, you can find:
  • a complete implementation of the remote-query functionality, including index management through JMX .  Adrian Nistor will blog on this in the following days
  • allow configuring the number of segments per node allows one to configure an uneven load of data between the nodes int the cluster. Dan Berindei will add a blog on this shortly
Together with this release we're also launching our new website:
  •  Built with Awestruct.  Yes, it really is awesome and fun to use
  •  Hosted on GitHub Pages.  Quick and easy.
  •  Styled with Community's flavour of Twitter Bootstrap 
  •  Documentation reformatted/moved from Confluence to AsciiDoc 
Among other things, the new site reflects some changes in the way Infinispan is distributed - including several cache stores and Hot Rod clients being moved out to separate GitHub repositories and following their own release cycles. 
We think it's pretty slick and pretty sure you wouldn't guess its written by a hardcore backed developer! Kudos to Manik Surtani for such a nice job!

Last but certainly not least,  a BIG thanks to our colleagues from the Hibernate team for their support (that is enhancements, fixes and releases and unreasonable hours) in getting the support needed for Infinispan's remote query functionality in time!

For a complete list of features and fixes included in this release please refer to the release notes.
Visit our downloads section to find the latest release and if you have any questions please check our forums, our mailing lists or ping us directly on IRC.

Thanks to everyone for their involvement and contribution!


Monday, 16 September 2013

New persistence API in Infinispan 6.0.0.Alpha4

The existing CacheLoader/CacheStore API has been around since Infinispan 4.0. In this release of Infinispan we've taken a major step forward in both simplifying the integration with persistence and opening the door for some pretty significant performance improvements.

What's new

So here's what the new persistence integration brings to the table:
  • alignment with JSR-107: now we have a CacheWriter and CacheLoader interface similar to the the loader and writer in JSR 107, which should considerably help writing portable stores across JCache compliant vendors
  • simplified transaction integration: all the locking is now handled within the Infinispan layer, so implementors don't have to be concerned coordinating concurrent access to the store (old LockSupportCacheStore is dropped for that reason).
  • parallel iteration: it is now possible to iterate over entries in the store with multiple threads in parallel. Map/Reduce tasks immediately benefit from this, as the map/reduce  tasks now run in parallel over both the nodes in the cluster and within the same node (multiple threads)
  • reduced serialization (translated in less CPU usage): the new API allows exposing the stored entries in serialized format. If an entry is fetched from persistent storage for the sole purpose of being sent remotely, we no longer need to deserialize it (when reading from the store) and serialize it back (when writing to the wire). Now we can write to the wire the serialized format as read fro the storage directly


Now let's take a look at the API in more detail:

  The diagram above shows the main classes in the API:
  • ByteBuffer - abstracts the serialized form on an object
  • MarshalledEntry - abstracts the information held within a persistent store corresponding to a key-value added to the cache. Provides method for reading this information both in serialized (ByteBuffer) and deserialized (Object) format. Normally data read from the store is kept in serialized format and lazily deserialized on demand, within the MarshalledEntry implementation
  •  CacheWriter and CacheLoader  provide basic methods for reading and writing to a store
  • AdvancedCacheLoader and AdvancedCacheWriter provide operations to manipulate the underlaying storage in bulk: parallel iteration and purging of expired entries, clear and size. 
A provider might choose to only implement a subset of these interfaces:
  • Not implementing the  AdvancedCacheWriter makes the given writer not usable for purging expired entries or clear
  • Not implementing  the AdvancedCacheLoader makes the information stored in the given loader not used for preloading, nor for the map/reduce iteration
If you're looking at migrating your existing store to the new API, looking at the SingleFileStore  for inspiration can be of great help.


And finally, the way the stores are configured has changed:
  • the 5.x loaders element is now replaced with persistence
  • both the loaders and writers are configured through a unique store element  (vs loader and  store, as allowed in 5.x)
  • the preload and shared attributes are configured at each individual store, giving more flexibility when it comes to configuring multiple chained stores 

Monday, 9 September 2013

Infinispan 6.0.0.Alpha4 out with new CacheLoader/CacheWriter API!

Infinispan 6.0.0.Alpha4 is now with a few very important changes, particularly around cache stores. We've completely revamped the cache store/loader API to align it a bit better with JSR-107 (old CacheStore has become CacheWriter) and to simplify creation of new implementations. The new CacheLoader and CacheWriter should help implementors focus on the important operations and reduce the coding time. We've also created AdvancedCacheLoader and AdvancedCacheWriter in order to separate for bulk operations or purging for those implementations that wish optionally implement them. Expect a blog post from Mircea in the next few days providing many more details on this topic.

This new Infinispan version comes with other important goodies:
  • Rolling upgrades of a Infinsipan REST cluster
  • Support for Cache-Control headers for REST operations
  • Remote querying server modules and Hot Rod client update
  • REST and LevelDB stores added to Infinispan Server
  • KeyFilters can now be applied to Cache listeners
  • Allow Cache listener events to be invoked only on the primary data owner
For a complete list of features and fixes included in this release please refer to the release notes. Visit our downloads section to find the latest release and if you have any questions please check our forums, our mailing lists or ping us directly on IRC.