Google Cloud Storage support in Derquinse Bocas

Starting with the just released 1.1.0 version of Derquinse Bocas, a new experimental back-end based on Google Cloud Storage is included. In this first version, the XML API 2.0 is used and only service account authentication is supported.

Besides, version 1.1.0 also includes a revision of the API, incorporating bucket support (in order to group entries) and simplifying entry creation methods. This will be described in a future post.

Announcing Conquiris

After a few months from the first release,  I'd like to announce the succesor to lucis: Conquiris. Conquiris is a set of libraries to help developing applications based on Apache Lucene. It is not a search server, as Solr fills that role really well. It is based on Java SE 6 and Lucene 3.6.1

Why the delay? Well, I was waiting to have some documentation in place to make the announcement, but it was taking so long that I have just gone for it.

So, what are the available resources to get in touch with Conquiris?:

• The project Home Page, where the documentation will be hosted.
• The Google Code Project Page, where you can get the source code and post issues.
• The Conquiris Google+ Page, for announcements.
• The API documentation, hosted on GitHub Pages.
• The Google Group for general discussion.

As with other projects, continuous integration is kindly provided by CloudBees DEV@cloud. The Jenkins instance is at https://derquinse.ci.cloudbees.com.

Lucis upgraded to Lucene 3.6.1

Just a quick note to announce that Lucis has been upgraded to Lucene 3.6.1. In the process, both Lucis and Derquinsej are now deployed to Maven Central using Sonatype OSS Repository Hosting Service, as legacy sync methods are being phased out.

Bocas now built on DEV@cloud by CloudBees

I've set up continuous integration for the Derquinse Bocas using a Jenkins instance kindly provided by CloudBees under its FOSS Program.

Everything has been very easy to set up and has worked perfectly.  You can find the provided instance at https://derquinse.ci.cloudbees.com/.

I'd like to thank CloudBees for providing this service to the Free and Open Source Software community.

"Big Memory" support and new Resources for Derquinse Bocas

Java brought automatic memory management to the mainstream. However, managing the heap is a non-trival task that can require a great amount of effort from the garbage collector. This effort grows with the heap size and may lead to long collection pauses that hurt your system's throughput and response times.

In Bocas, when using a memory-backed repository or a cache, most of the memory consumption comes from the stored values, which are opaque byte streams. If we could get those values out of the Java heap we could use that precious managed space for more value-adding objects.

Enter "big memory" support: starting with version 1.0.4, Bocas supports the use of direct buffers as the backing storage of both memory-backed repositories and caches, using off-heap memory to store the repository values. As the memory-backed repositories are used mainly for testing purposes, this feature is most useful in caches.

This support allows for better utilization of the Java heap, reducing the load for the Java garbage collector. However, take your off-heap caches into account when sizing the heap and planning the memory distribution of your machines.

I would also like to announce new resources for the project:

• A new Google+ Page, used for announcements.
• More documentation is being added to the Project Web Site.
• The General Discussion Google Group.

As a side note, version 1.0.5 has just been released and is already in Maven Central.

A small update on the Gist.GitHub Gadget

The Gadget to embed GitHub Gists in Google Properties, has just received some small updates:

• It is now hosted on a git repository.
• It has a new URL.
• Now uses HTTPS by default.

You can find more information in its home page.

New Resources for Derquinse Commons

One of the goals of open sourcing some project is getting other people to use it. However, code alone is  not enough for that, and some other resources have to be provided so that other people start looking at your code. So, I'm going to try to devote a little more time to providing documentation for the existing codebase than including new features. So for the Derquinse Commons Project, we have:

• A new Project Web Site, still work in progress, that will host the project (non-javadoc) documentation.
• Project announcements will be done through the new Google+ Page for the project.
• A new General Discussion Google group.

As before, the API documentation will continue to be hosted at GitHub Pages and the source code on Google Code.

Announcing Derquinse Bocas

I'd like announce a little project I've been working on lately. It called Bocas and it is a Java and REST API to access key-value repositories with the content-addressed storage constraint, that is, the key must be a hash of the value. In the current implementation, SHA-256 is used as the hash function.

A few releases have been already deployed in Maven Central, and documentation is being built at the project web site. As this site is still short of content, you may find the API documentation useful, above all the main API interface.

The project is hosted on Google Code and has the Apache 2 license.

Stay tuned for more news about this project.

New home for Derquinse Commons API documentation

One of the main issues in Derquinse Commons is the lack of documentation,  including the lack of a stable home for the Javadoc-generated documentation.

From now on, and thanks to the wonderful GitHub pages service, you can find the API docs at http://apis.derquinse.net/derquinse-common.

Non-API docs are still missing, and I hope to be able to write some soon.

Derquinse Commons now in Maven Central

One if the best features in Maven is dependency management. Even with its known problems it is a great step forward and many other build systems reuse Maven's repository structure.

One of the essential parts of this vision is the Maven Central Repository, your one stop shop for most of the dependencies in your projects.

Publishing to Maven Central has always had its requirement set (you can find an updated list here), one of them being that the repository must be self-contained. That is, no artifact may depend on another artifact that is not part of the repository.

Back in 2008, when I started both derquinsej and lucis, this requirement was fulfilled with no problem and synchronization with Central was provided thanks to Sourceforge's SSH access.

However, as time passed, some projects such as Hibernate starting publishing its artifacts in external repositories and many other interesting projects were hosted in repositories different of Central, such as Java.net. So, for future projects Central Publishing was not an option and I prepared an external repository for my own projects.

Fortunately, the situation has improved a lot:

• Now many interesting external repositories, such as JBoss and Java.net, are synchronized back in Central.
• Central navigation and search has been enhanced.
• Sonatype provides a repository for open source projects with the option of being published in Central. The process is very well documented and the response is fast and great (kudos to Juven Xu). 

So, from now on, you can find Derquinse Commons in Maven Central.
  

Follow up to Maven Release Plugin version problems

In a previous post, I warned about some bugs that caused the Maven Release Plugin to generate wrong tag versions. While those bugs are still opened today, the real culprit was MRELEASE-697 which is fixed in version 2.2.1, which can be used safely.

Lucis upgraded to Lucene 3.5

Well, at last the jump was made, and based on the previous 2.9.x branch, the migration to Lucene 3.5 has been done with version 3.5.2 now available in central. Apart from the migration work, the most significant changes include:

• Writer configuration suppliers, to deal with the fact that certain Lucene objects, such as merge policies, can't be shared.
• Managed readers, not based on the new searcher manager, but integrated with the lucis store architecture.

A Lucis branch for Lucene 2.9.x

As a said in the previous entry, the upgrade to Lucene 3.x may take longer than expected, so to provide an evolution path, a branch based on Lucene 2.9.x has just been released (version 2.9.5, based on Lucene 2.9.4).

It is not a mere recompile. Use of deprecated methods has been removed and the new collectors has been put in use.

A Lucis update

After a too long period of inactivity at last we have a significant update to Lucis. The migration to Lucene 3.x has been put on hold to work on some more important changes (and I don't know if it will happen before the move to Lucis' successor project). So, to avoid confusion, version numbering has changed so that the first two numbers match those of the Lucene version used.

I have just released version 2.4.31 (should be in central soon), the main changes being:

• Reindexing indexes can now hold a checkpoint.
• Reindexing indexes can know decide to skip a run.
• Cancellation policy for index services has been defined, using interruptions, providing graceful shutdowns. As always for all your java concurrency-related questions, go to the source.
• It is know possible to use external executors.
• Many more tests.

Also, as of this version, the javadoc is no longer uploaded to the repository, as it can be built from source, but the relevant parts are online for your reading convenience.

UPDATE: Already in Central.

Stick to version 2.0 of Maven Release Plugin

As reflected on bugs MRELEASE-694 (related to MRELEASE-689 and MRELEASE-691), there are some problems generating the release version numbers (like maintaining SNAPSHOT). This may bite without notice if use non-interactive mode. These bugs are still open as of version 2.2.1.

It's a nice coincidence that because of another bug, the Maven Eclipse Plugin warns you if you use a Maven Release Plugin version other than 2.0 :)

Revisiting Lucis

The lucis project is a key component of an increasing number of web sites. However, it is starting to show its age. Rushed into production in the summer of 2008 it included some "incorrect" decisions that were to be mended in its successor project, which was also going to include all the important updates. So lucis algo received small evolutions.

However, as the first release of conquiris is delayed again and again, and the number fo projects using lucis increases far more than expected, it is clear that a new impulse was needed.

So today, I've released version 0.0.31 (which should be in Maven Central in a few hours). The only (but fundamental) change is the upgrade from Lucene 2.4.1 to 3.1.0. As the new Lucene version already breaks backwards compatibility, I've gone the whole way and removed use of (now) deprecated parts of Lucene. So this version of lucis is totally incompatible with the previous stable one (0.0.29).

I hope to post about the changes real soon now.

As I said, the "only" change is the Lucene version, so this release won't be integrated in Port@l and GeNews until more improvements are added.

New Look

Three years after the First Post, I've decided to give this blog a new look, maybe as a first step in posting more often.

So, I've turned to the Blogger Template Designer, chosen a clear and simple one, removed most ads, and converted code snippets to Gist.GitHub.

I hope you like it.

Great presentation for entrepeneurs

From Zero to a Million Users - Dropbox and Xobni lessons learned

View more presentations from Adam Smith

Gist.GitHub Gadget

Some months ago I wanted to embed some code snippets stored in Gist.GitHub in some Google Sites-based documentation, just to find that you cannot use the script tag directly. So, I decided to wrap the embedding code in a Google Gadget. You can find it here.

Permgen problems and Running Eclipse on Java 1.6 update 21

See this post. Reposted here for future reference.