Another release candidate for 5.0

I've just released 5.0.0.CR7. Hopefully we won't see any more release candidates before 5.0.0.Final, so give this a good, hearty test and let us know what you think.

A number of important bugs are fixed, details are in JIRA, and you can download the release in the usual place.  Please provide feedback on the user forums.

Enjoy, and onward to 5.0.0.Final!
Manik

Infinispan @ jazoon

I've just returned from Jazoon where I spoke about in-memory data grids/Infinispan and how can they be used to complement or even replace databases. There was a good and enthusiastic crowd, and the discussions ended late in the night - of course cooled down by cold swiss beer :)

Infinispan was also present in Hardy Ferentschik presentation about Hibernate OGM: the bran new JBoss project which exposes the grid ( to be read Infinispan) through Hibernate's API.

Thank to Jazoon organizers for an excellent conference and the chance to meet other enthusiasts from all over the world!

Cheers,

Mircea

Another week, another release candidate

Infinispan 5.0.0 codenamed Pagoa has yet another release candidate out for you to play with.  CR6 was released earlier today, please switch any tests you have on the 5.0 series to this latest release candidate and provide as much feedback as possible, as we get ever closer to 5.0.0.Final.

In addition to Pete's new grouping API, we also have some changes to the way class loading works, as well as the closure of a host of bugs reported against previous release candidates.

Please provide feedback using the forums, grab the release in the usual place, and report issues on JIRA.

Enjoy
Manik

The Grouping API

Infinispan 5 CR4 (and above) includes a new Grouping API. You can read more in the documentation, but I'll introduce it quickly for you here.

In some cases you may wish to co-locate a group of entries onto a particular node. In this case, the group API will be useful for you.

How does it work?

Infinispan allocates each node a portion of the total hash space. Normally, when you store an entry, Infinispan will take a hash of the key, and store the entry on the node which owns that portion of the hash space. Infinispan always uses an algorithm to locate a key in the hash space, never allowing the node on which the entry is stored to be specified manually. This scheme allows any node to know which nodes owns a key, without having to distribute such ownership information. This reduces the overhead of Infinispan, but more importantly improves redundancy as there is no need to replicate the ownership information in case of node failure.

If you use the grouping API , then Infinispan will ignore the hash of the key when deciding which node to store the entry on, and instead use a hash of the group. Infinispan still uses the hash of the key to store the entry on a node. When the group API is in use, it is important that every node can still compute, using an algorithm, the owner of every key. For this reason, the group cannot be specified manually. The group can either be intrinsic to the entry (generated by the key class) or extrinsic (generated by an external function).

How can I use it?

If you can alter the key class, and the determination of the group is not an orthogonal concern to the key class, then you can simply annotate a method on the key class that will provide the group. For example

class User {
 
   ...
   String office;
   ...
 
   int hashCode() {
      // Defines the hash for the key, normally used to determine location
      ...
   }
 
   // Override the location by specifying a group, all keys in the same 
   // group end up with the same owner
   @Group
   String getOffice() {
      return office;
   }
 
}

Of course, you need to make sure your algorithm for computing the key is consistent, and always returns the same group for a key!

Alternatively, if you can't modify the key class, or determination of the group is an orthogonal concern, you can externalise computation of the group to an "interceptor style" class, called a "Grouper". Let's take a look an example of a Grouper:

class KXGrouper implements Grouper {
 
    // A pattern that can extract from a "kX" (e.g. k1, k2) style key
   static Pattern kPattern = Pattern.compile("(^k)(\\d)$");
 
    String computeGroup(String key, String group) {
        Matcher matcher = kPattern.matcher(key);
        if (matcher.matches()) {
            String g = Integer.parseInt(matcher.group(2)) % 2 + "";
            return g;
        } else
            return null;
    }
 
    Class getKeyType() {
        return String.class;
    }
 
}

Here, we've had to use a grouper, as we cannot modify the key class (String). Our group is still based upon the key, and established by extracting a part of the key.

Of course, you need to enable grouping support in Infinispan, and configure any groupers. The reference documentation will help you here.

So you want JPA-like access to Infinispan?

Back in the early days of Infinispan (since our first public announcement, in fact) we always had it in mind to expose a JPA-like layer to Infinispan.  Initially this was as a replacement to the fine-grained replication that JBoss Cache's POJO Cache variant offered, but it grew beyond just a technique to do fine-grained replication on complex object graphs.  The fact that it offered a familiar data storage API to Java developers was big.  Huge, in fact.

So we realised JPA-on-Infinispan was firmly on the roadmap.  The original plan was to implement the entire set of JPA APIs from scratch, but this was a daunting and Herculean task.  After much discussion with core Hibernate architects and Infinispan contributors Emmanuel Bernard and Sanne Grinovero, we came to a decision that rather than implementing all this from scratch, it served both Infinispan and the community better to fork Hibernate's core ORM engine, and replace the relational database mappings with key/value store mappings.  And we get to reuse the mature codebase of Hibernate's session and transaction management, object graph dehydration code, proxies, etc.

And Hibernate OGM (Object-Grid Mapping) was born.  After initial experiments and even a large-scale public demo at the JBoss World 2011 Keynote, Emmanuel has officially blogged about the launch of Hibernate OGM.  Very exciting times, Infinispan now has a JPA-like layer.  :-)

To reiterate a key point from Emmanuel's blog, Hibernate OGM is still in its infancy.  It needs community participation to help it grow up and mature.  This is where the Infinispan community should step in; consider Hibernate OGM as Infinispan's JPA-like layer and get involved.  For more details, please read Emmanuel's announcement.

Enjoy!
Manik

Keynote of the decade: behind the scenes, an Infinispan perspective

JBoss World 2011's much talked about keynote ended with a big bang - a live demo that many thought we were insane to even try and pull off - and has sparked off a lot of interest, many claiming JBoss has got its mojo back.  One of the things people keep asking is, what actually went on?  How did we build such a demo?  How can we do the same?

Firstly, if you did not attend the keynote or did not watch it online, I recommend that you stop reading this now, and go and watch the keynote. A recording is available online (the demo starts at about minute 35).

Ok, now that you've been primed, lets talk about the role Infinispan played in that demo.  The demo involved reading mass volumes of real-time data off a Twitter stream, and storing these tweets in an Infinispan grid. This primary grid (known as Grid-A), and ran off 3 large rack-mount servers. The Infinispan nodes were standalone, bootstrapped off a simple Main class, and formed a cluster, running in asynchronous distributed mode with 2 data owners.

Andrew Sacamano did an excellent job of building an HTML 5-based webapp to visualise what goes on in such a grid, making use of cache listeners pushing events to browsers and browsers rendering the "spinning spheres" using HTML 5's canvas tag.  So now we could visualise data and data movement within a grid of Infinispan nodes.

As Twitter data started to populate the grid, we fired up a second grid (Grid-B) consisting of 8 nodes. Again, these nodes were configured using asynchronous distribution and 2 data owners, but this time these nodes were running on very small and cheap plugtop computers.  These plugtops - GuruPlugs - are constrained devices with 512MB of RAM, a 1GHz ARM processor.

Yes, your iPhone has more grunt :-) And yes, these sub-iPhone devices were running a real data grid!

The purpose of this was to demonstrate the extremely low footprint and overhead Infinispan imposes on your hardware (we even had to run the zero assembly port of OpenJDK, an interpreted-mode JVM, since the processor only had a 16-bit bus!). We also had a server running JBossAS running Andrew's cool visualisation webapp rendering the contents of Grid-B, so people could "see" the data in both grids.

We then fired up Drools to have it mine the contents of Grid-A and send it to Grid-B applying some rules to select the interesting tweets, namely the ones having the hashtag #JBW. With this in place, we then invited the audience to participate - by tweeting with hashtag #JBW, as well as the hashtag of your favourite JBoss project - e.g., #infinispan :-)  People were allowed to vote for more than one project, and the most prolific tweeter was to win a prize.  This started a frenzy of tweeting, and was reflected in the two grid visualisations.

Not only Infinispan is very quick here: needless to say, Drools was sending the tweets from Grid-A to Grid-B using HornetQ, the fastest JMS implementation on the planet.

Jay Balunas of Richfaces built a TwitterStream app with live updates of these tweets for various devices, including iPhones, iPads, Android phones and tablets, and of course, desktop web browsers, grabbing data off Grid-B.  Christian Sadilek and Mike Brock from the Errai team also built a tag-cloud application visualising popular tags as a tag cloud, again off Grid-B, making use of Errai to push events to the browser.

After simulating Mark Proctor to try cheating the system with a script, we could recover the correct votes: clear Grid-B, update the Drools rules to have it discard the cheat tweets, and have a cleaned up stream of tweets flow to Grid-B.

All applications, including Drools and the visualizations, where using a JPA interface to store or load the tweets: it was powered by an early preview of HibernateOGM, which aims to abstract any NoSQL store as a JPA persistence store while still providing some level of consistency. As HibernateOGM is not feature complete, it was using Hibernate Search to provide query capabilities via a Lucene index, and using the Infinispan integration of Hibernate Search to distribute the index on Infinispan.

We then demonstrated failover, as we invited the winner to come up on stage to choose and brutally un-plug one of the plugtops of his choice from Grid-B - this plugtop became his prize. Important to note, the webapps running off the grid did not risk to lose any data, Drools pulling stuff off Grid-A onto Grid-B was still able to continue running, the Lucene index could continually be updated and queried by the remaining nodes.

From an Infinispan perspective, what did this demo make use of?

• Data distribution, numOwners = 2
• Async network communication via JGroups
• JTA integration with JBossTS
• Cache listeners to notify applications of changes in data and topology
• The Infinispan Lucene Directory distributing the Lucene index on the grid

So a fairly simple setup, using simple embeddable components, cheap hardware, to build a fairly complex application with excellent failover and scalability properties.

So we where depending on wi-fi connectivity, internet access, a live tweet stream, technology previews and people's cooperation!

To make things more interesting, the day before the demo one of the servers died; hardware failure: didn't survive the trip. A second server, meant to serve the UI webapps, started reporting failures on all network interfaces just before starting the demo: it could not figure out hardware addresses of cluster peers, and we had no time to replace him: its backup was already dead. Interesting enough we could tap in some advanced parameters of the JGroups configuration to workaround this issue.

Nothing was pre-recorded! Actually the backup plan was to have Mark Little dancing a tip-tap; next year we will try to stretch our demo even more so you might see that dance!

So here you can see the recording of the event: http://www.jboss.org/jbw2011keynote or listen to the behind the scenes podcast.

After the demo, we did hear of a large commercial application using Infinispan and Drools in precisely this manner - except instead of Twitter, the large data stream was flight seat pricing, changing dynamically and constantly, and eventually rendered to web pages of various travel sites - oh, and they weren't running on plugtops in case you were thinking ;)  So, the example isn't completely artificial.

How do you use Infinispan?  We'd love for you to share stories with us.

Cheers

Manik and Sanne

Faster Infinispan-based Second Level Cache coming to a JPA app near you!

Starting with Hibernate 4.0.0.Beta1, Infinispan second level cache interacts with the transaction manager as a JTA Synchronization instead of an XA resource. Based on some testing we've done in JBoss AS 7, this has resulted in a huge performance increase thanks to the optimisations the transaction manager can apply to Synchronizations, which work very well when Infinispan is used as a cache rather than as a authoritative data store.

From an Infinispan configuration perspective, nothing needs changing. The Infinispan provider still uses the same base configuration by default. Transactional configuration happens within the cache provider itself and it's here where the Infinispan is configured with Hibernate's transaction manager and where Infinispan is configured to participate as a JTA synchronization. This is the default configuration, so from an user perspective, there's nothing you have to do to take advantage of this new change.

However, you can always switch back to previous behaviour where Infinispan interacted as an XA resource via a dedicated Hibernate property called hibernate.cache.infinispan.use_synchronization

By default this property is set to true. If you set it false, Infinispan will interact with the transaction manager as an XA resource.

For more detailed information, check the "Using Infinispan As JPA Hibernate Second Level Cache Provider" wiki.

Cheers,

Galder

More release candidates: 5.0.0.CR4

Another week, another CR release for Infinispan 5.0 series. This time is the turn of CR4 including many bug fixes and the addition of JGroups JMX MBeans when Infinispan is configured with JMX statistics.

You can see the full list of issues fixed here. Thanks very much to everyone for the important feedback provided on the CR3.

As per usual, downloads are in the usual place, use the forums to provide feedback and report any issues.

Cheers,

Galder