Getting the tools ready

Given the fact that I’m on Ubuntu 10.04 beta on my everyday working notebook, though, running “Microsoft” .NET was not an option. Fortunately, there’s Mono, an open source / software libre project aiming to provide a compatible, portable implementation of .NET for platforms not supported by Microsoft (which is any platform other than Windows, actually). In my Ubuntu installation, as well as in most other GNU/Linux distributions out there these days, one is just a few moments of package manager interaction away from having Mono installed; I personally did so in a rather rude way – by installing MonoDevelop, an open-source IDE for Mono / C#, which subsequently pulls in most of the relevant core mono packages as dependencies. A few dozens of megabytes of package downloading later, things are there, set up and ready to go. So far, so good.

Solving a real-world problem, part 1: integration client

In order to, in our environment, make a legacy (and highly proprietary) document management platform behave well and communicate bidirectionally with other applications / platforms around, we ended up hand-crafting a small “middleware” structure to get this job done. Well, “middleware” is quite an exaggeration for something which basically is HTTP-POST along with a data transfer encoding resembling a lite, stripped-down version of JSON. It’s a pretty bare-bone and rather archaic kind of approach, but then again, it works, it is easy to handle even in this very DMS platform (written in an obscure language in which a more-or-less working TCP socket implementation is the best you probably can get for such a task), and it is a way to easily link many different platforms without having to worry too much about whether or not this approach is supported in the target platform (actually, even unix-shell and netcat eventually would do here… :) ). So doing a few proof-of-concept implementations on top of various languages / frameworks is a good excuse to try the same using C# / .NET. First things first, the easier thing: Building a client posting some data structure via HTTP and dumping the response recieved to STDOUT… Firing up MonoDevelop, if one is used to a Java IDE it takes some time getting acquainted to things, but then again, world’s not all too different here. Creating an initial runnable class is simple, core syntax just slightly different to what one’s used in Java.

So now how to deal with the client side of HTTP? Fortunately, there’s documentation at hand: Along with Mono, there comes “monodoc” which allows for easily browsing pre-installed Mono documentation, mainly of course API reference. monodoc offers a rather good search facility so figuring out that System.Net.WebRequest is what I need. Adding to that, there’s a vast load of documentation available as part of the Microsoft Developers Network (MSDN) library, which, in this special case, offers an almost fool-proof step-by-step guide on how to resolve exactly the kind of problem I need. So overally, getting the C# client to talk to both the DMS and the Java server side of our integration structure took about ten minutes, including figuring out how to, in C#, convert a String object into a byte array. Not bad, and pretty intuitive given the documentation and a good deal of Java experience at hand.

Solving a real-world problem, part 2: integration server

So far, so good – running the client code from within MonoDevelop, or even from the command line, wasn’t a job too tough (even though it feels strange running a .exe file in a GNU/Linux environment… :) ). Now, however, for the server part of things. Given quite a range of servlet containers or application servers (we use Glassfish and apache tomcat for our internal purposes) at hand, a Java developer has quite a concise kind of environment in which to expose HTTP(-POST) services: Extend javax.servlet.http.HttpServlet, override the doPost method, add your code, package a .war file, deploy things to your application server – done. The good thing about Java EE, in my opinion, is that this general workflow does always apply, no matter which application server, development environment, … you are running. But how to do that in Mono/.NET?

So far, I haven’t come across the idea of something like a “generic .NET application server”, akin to what an application server in Java EE is. However, I quickly found out about ASP.NET and mono-xsp(2) server, which I installed using my package management. Fine. Reading the tutorial was rather straightforward, now for some code.

To cut things short: It turned out to be not much more complicated than building the client. Browsing the Mono documentation, and from there the MSDN articles, I quickly came across System.Web.IHttpHandler, implemented it, filled its ProcessRequest method with meaningful code (using code completion in MonoDevelop prove to be a rather good way of exploring the – at least to me – unfamiliar C#/.NET API), running stuff with xsp. Done.

Conclusions…

What to learn from this little trip?

• Well, first off, working with C#, Mono and MonoDevelop is not that much of a pain if you’re used to Java and, given at least this very use case, know what you’re about to want to do. There are tons of documentation available out there, and overally, I guess in most situations getting started with these things should be straightforward.
• As always, learning something new is a good thing, and in case of C#/.NET, I see a strong advantage in linking our existing server-sided environment with desktop-based clients – so far, Java unfortunately doesn’t do that good in terms of deep desktop integration (registering global desktop hotkeys, opening standard applications, …). Throwing in a little bit of local .NET code could ease this at the very least for Windows clients.
• I still have to see how to build a portable, self-containing application server using Mono/C#: Given the nature of .war deployment and the way application servers and JREs can be deployed, it is possible to simply make .zip file of JDK, preconfigured application server along with the applications deployed to it, unzip this file on a server machine, start the application server and have things running as desired – without the need to install additional software, configure system-wide services (apache and mod_mono?) and the like. Comparing this kind of deployment, Java EE still is way ahead of .NET/C#, Python and any other platforms I played with so far.

In the end, that’s what is left: I enjoyed my ride and liked what I saw. I still will be doing most of my work using Java, I still will enjoy working with P/Jython for most scripting things, but I figured out that, in practical use, C#/.NET/Mono is a rather smooth environment to work with, and, as pointed out, there surely are some problems I already imagine for using it… More to come.

Anyway: One of many fields in which NetBeans these days still does excel compared to Eclipse is building graphical user interfaces using the GUI builder that comes pre-installed with every NetBeans package and even from this point of view is way ahead of the Visual Editor available to Eclipse users, in case of which even installing and getting the bits to work seems an adventure of its own. And here, we’re not even talking about RAP (based upon RWT) which is the “web alternative” and, though pretty complete (and getting better with each release), always lagging a bit behind the “desktop RCP” (based upon SWT) in terms of technology, widgets, … . However, Eclipse then again is a bit ahead from our current projects point of view: The “single sourcing” approach the RAP people are trying to push forth – the idea of having one codebase and run this both “on desktop” and “in a web browser” without too much ado – is really a good approach, well, if you need to cater for both web users who still want a good feature set and to internal users to which using a browser based interface eventually is not an option. Yes, there are quite some drawbacks about the way the Eclipse runtime environment does work and scale when deployed to an application server (in example talking about one dedicated “UIThread” per user session which definitely is not the way people think of doing server-sided Java applications… :) ), but overally, it’s more than so far you can do with Netbeans unfortunately, where you’re left with either doing a desktop (Swing based) application or a web application using some framework for this purpose (JSF, Wicket, …). Works, but one of the ideas you want to address when considering single-sourcing (having programming models as well as user interfaces as close to each other as somewhat possible for web and desktop) falls short here. There have been numerous projects trying to make Swing a web application platform too, but most of them seem discontinued, and none of them integrate well with the tooling one is used to (and enthusiastic about :) ) in NetBeans.

And, talking about e4, the next major version of Eclipse IDE/platform, the current status quo might become even more apparent: e4 comes with XWT, a technology to allow for declarative UI description using XML file definitions rather than writing code, allowing for, well, “declaration” rather than coding of user interfaces, which has a bunch of more or less obvious advantages. Not that this idea is generally all new (just consider XUL used by the Mozilla applications), but along with the “usual tooling” to be found in an IDE like Eclipse, this might turn out to be a rather impressive feature. First demonstrations, like this screencast showing the visual editor for XWT, leave one end up with expectations rather high. So let’s wait and see…

… both what Eclipse people will keep on doing related to XWT until e4 is to be released, and what Oracle is likely to do to Java technologies like Swing or maybe JavaFX. Maybe the latter one might be an interesting alternative if integrated with web and desktop environments and the appropriate tooling. I am curious to see things proceed, talking about both technologies…

• First of all, NetBeans IDE has bumped up its version number to 6.8, including (not too much of a surprise, I guess…) seamless and production-ready support for Java EE 6 and Glassfish v3, along with improved or newly added toolings and plugins for maven2 (very impressive), Project Kenai, JavaFX, PHP and a whole bunch of other technologies. Adding to that, asides “just” being another multi-language/-technology IDE, NetBeans RCP also is likely to get more support as an RC platform, which surely is a good thing.
• Then, Java Enterprise Edition is available in version 6 (JSR-316), aiming at making Java EE development more slim, more easy and, thanks to the newly added “profiles”, more adaptable to given use cases than ever before. Along with this, there are a bunch of improvements in technologies such as Java Servlets (3.0), EJB (3.1), persistence (JPA 2.0), JCA and, especially notable from my point of view, support for RESTful web services (JAX-RS 1.1).
• Finally, in addition to that, along with Java EE 6 comes Glassfish v3 as its reference implementation and the “open source” platform Suns Java System Application Server is based upon. Along with (obvious) support for all the Java EE technologies, just the last couple of weeks I experienced during FishCAT that GFv3 is a pretty good platform, providing all the strengths so far provided by its predecessors, yet being updated to reflect the changes in Java EE 6 and also providing hosting for in example JRuby applications out of the box. Adding to that, Adam has a more in-depth list of goodnesses that come with GFv3.

So, overally, a lot of great technologies and tools to play (or eventually work :) ) with, but also a moment of thought: Maybe not for NetBeans, but I am pretty sure at the very least for Java EE and maybe Glassfish, this could be the last release made by Sun as an independent business entity, the last releases related to these technology before Sun gets acquired by / gets merged with Oracle, leaving their products end up in the ever-growing Oracle product and services portfolio, some driven forth maybe more enthusiastically, others maybe left out in the cold and eventually discontinued. No matter how, I guess the next year might be crucial to all of these technologies: It will show how Java EE 6 can come up against or side-by-side with latest and upcoming Spring releases, with the whole OSGi movement in general and, especially, the environment to be set up by Eclipse RT project. And it also will show what NetBeans, as an IDE, as a platform, as a community, can come up with to provide viable alternatives and approaches to a load of (unquestionably good) features provided by Eclipse ecosystem, like the whole Eclipse Modeling toolbox or the upcoming XWT and declarative UI technologies in e4. Or maybe (which eventually could be the best of all outcomes) it might show that there is a good reason for different alternatives existing to address different requirements and use cases.

And, overally, no matter how things will move on: At the moment, congratulations to all the engineers, writers, tech people behind these three releases! You did a rather great job again…

Open-ness and uncertainty

We, as a company, migrated to SAP MaxDB back in 2005 after finally dumping an old installation of MS SQL Server 97 on top of MS Windows NT 4.0 in favor of, well, an SAP MaxDB 7.5 installation on top of a Debian GNU/Linux operating system. Reasons for this decision:

• Comparing initial installation, configuration and maintaineance effort, we had to figure out that in many respects SAP MaxDB was the most simple RDBMS we could get our hands on: Straightforward installation, deployment, backup, migration, a set of rather well thought-out, well-working tools, extensive documentation and a friendly user group – what else would you expect?
• Of course, compared to Oracle 9i or a more up-to-date MSSQL, these days SAP MaxDB simply was an affordable solution, given then and now it was provided free-of-charge for use along with open-source and/or non-SAP technology.
• Wearing the label SAP, we thought that eventually there might be some way of getting “high-value” support by SAP on that given we offer some sort of MaxDB use case that justifies doing so without being into any other SAP technologies / products (as so far we simply haven’t found one in there to suit our use cases).

Getting into MaxDB, we quickly came to a rather stable and reliable environment. Subsequently, we spent a couple of money to have the manufacturer of our “old” system infrastructure to add support for MaxDB to their product for our specific environment. And we’ve been working on top of a heterogenous system built on top of SAP MaxDB for a couple of years, evaluating to buy professional support and licensing right from mySQL AB (who, by then, did offer SAP MaxDB along with mySQL as a “professional, enterprise-grade RDBMS”), until eventually: “MaxDB back under the SAP roof!” So no more licensing by mySQL AB, and a little later, there also happened to be a change in licensing, basically coming down to a construction like this, as far as I understand it:

• SAP customers to get professional, paid support for SAP MaxDB used along with any other SAP applications according to a given licensing model.
• SAP customers using SAP MaxDB along with non-SAP applications may order some sort of support contract addition to their SAP software contract specially made up for this kind of purpose.
• Non-SAP customers may use SAP MaxDB for free according to the terms of the Community License. No commercial support offerings are available to those it seems.

More uncertainty
In the end, after the licensing scheme (proprietary -> GPL/proprietary dual-licensing -> “proprietary community license”) has changed a couple of times and, now, an extensive FAQ collection outlining that the core strategy of SAP obviously is making MaxDB more suitable, more well-integrated, more “usable” along with other SAP applications (which is an obvious thing to do, no doubt about that), we are evaluating migrating our database backend to a recent Oracle release altogether. Why?

Even while we tried so, we failed managing to become an SAP customer with SAP MaxDB being the only SAP product we use. So, looking at our relation to SAP, we always will end up being the “community user” using something we get free-of-charge in an environment unknown to SAP.

Consequently, as we do not use any other SAP applications, I don’t really see where things are heading in near future. Where will SAP MaxDB development go? Could it be that, eventually, one day the “community license” will be terminated as well, making SAP MaxDB (and patches/updates) completely unavailable to anyone outside SAP again? Could it be that, in course of making the database more suitable for use with other SAP applications, one day it simply won’t be a feasible solution for our environment anymore as, this way, it is obviously not aiming at being a “general purpose RDBMS” anymore? In the past, it seems a lot of features we thought of as being pretty interesting (WebDAV/WebSQL, Synchronization tools) have either been removed altogether or aren’t maintained anymore – can we be sure this is not eventually, one day, to hit some other part of database administration tooling?

There used to be an Open MaxDB Group a while ago which, it seems, has quietly come to a halt. This makes me think that, for a “community license” user, there is next to no way of having ones interests related to MaxDB tooling, development, stragety, … represented to and eventually being recognized by SAP. Looking back at some discussions in the (kind and very helpful) MaxDB users forum, I once read a message outlining SAP not wanting to be a “general purpose DBMS seller”, which surely seems true here. But given in our situation a “general purpose DBMS” is right what we want / need, maybe SAP MaxDB is not the right strategy anymore, at least at the moment…

… and this is where, somehow, I’ll get back to the initial thoughts on open-ness and open source: In my opinion, the current state of “MaxDB open-ness” is, rudely speaking, next to useless to most customers not using SAP applications but looking for a long-term RDMBS strategy as there simply seem too many questions relating to development, long-term availability of updates, … unanswered. Having a “commercial database license” option provided by SAP would help easing these problems. If this is not possible / unwanted, having an “open-source” MaxDB at hand, along with support by external providers (so far we get MaxDB support by infolytics AG which is pretty good) would ease the pain of strategy uncertainties. Maybe the “new open-ness”(?) at SAP might change this thing for the better, somehow – I still hope for this as SAP MaxDB, after all, still is a really good database platform and really fun working with…

First impressions

Starting with the good things… By now, using Sonatype m2eclipse, Eclipse maven2 support has somewhat improved the last couple of years:

• Asides “Run as” -> “maven …”, Eclipse now seems to be able to build maven2 projects along with a straightforward workspace build / cleanup, which is really rather helpful (and actually something one would expect… :) ).
• There now is a quite powerful editor for maven2 pom.xml files which, as I have to admit, in some situations even outperforms the tooling provided by NetBeans especially when it comes to displaying dependency graphs and dependency hierarchies. The various visual editor pages also help getting closer to the full maven2 feature set if you do not want to dive all too deep into the pom.xml syntax and/or are just learning…

Likewise, there is a whole bunch of features how to “import” maven2 projects into Eclipse, also (as I think) partly due to the fact that Eclipse, unlike NetBeans, can’t simply “open” a maven2 project simply using pom.xml as the only piece of relevant information but rather requiring its custom configuration files in the project structure to be happy. So, I tried to import and get started with my maven2 projects… So far, this is an SVN trunk consisting of about 55 mvn2 jar/war/pom artifacts, which I was about to check out and import into Eclipse.

Second impressions

And this prove to be fun, for sure. First shot, most straightforward, using “Import” -> “Maven” -> “Check out Maven2 projects from SCM”. Browsed to the repository trunk, let it check out all the project, and they’re here… and they’re all red? Well. It seemed that, with checking them out of SVN, Eclipse did for each project assume to enable project specific Java Compiler settings, thus forcing them to be Java 1.4 compliant each (and ignoring what’s stated in pom.xml explicitely). Fine. Resolving this was work no less than manually changing this setting for each of the projects in question. Following this, at least some of the projects didn’t have problems anymore. The whole mess built until I eventually trashed one of the project. Well, no problem – delete it, check it out again – and see the main project failing to build?

It took a few moments to resolve what happened: My main project is a maven2 pom project containing modules like this:


<modules>
<module>../moduleA</module>
<module>../moduleB</module>
<module>../moduleC</module>
</modules>


and is referring to modules living in the same folder on the same level as this core / build pom. However, Eclipse maven2 tooling did check out all the projects initially to a folder like workspace/maven.1259135232603, thus having them all in one place. Deleting the project (not much of a surprise) removed it from this folder. Checking it out again, however, created a new folder in workspace (workspace/maven.1259151842736) and checked the project out to there. Well… I have no real understanding why this happens, and so I wouldn’t want to say it simply dumb, but at the very least in this situation it’s a behaviour highly undesirable, and I really would want to at least be capable of having this configurable. Yet, so far I fail doing so. Did, however, commit the changed projects to at least hope to have the Eclipse metadata relating to the Java Compiler settings stored along with the project.

Anyway, so let’s test another stragety: Eclipse also allows for importing existing maven2 projects, which basically just does create project related metadata in the workspace but keeps the project code wherever it is. So, manually checked out the SVN trunk to some arbitrary folder, used Eclipse to import things, saw my projects, did the same settings as above (setting Java Compiler to not be project specific… – so obviously they are kept elsewhere but in the actual project itself), built. Worked. Tried to commit the projects to SVN – to see that, this way, even though there are .svn metainformation in any of these project folders, Eclipse fails to recognize these projects as “versioned” so the only thing to be found in “Team” menu is the usual standard (“share projects” / “apply patch”). No way of reconnecting, no way of committing. Of h***. :/ So back to the initial approach, let’s just hope we don’t have to delete and checkout any project anew…

A few hours later, I am frustrated beyond belief as many features one simply takes as granted in NetBeans don’t seem to work in Eclipse:

• Even though some of the maven2 artifacts are war applications, Eclipse WTP obviously fails recognizing this out of the box, i.e. without fiddling with the project metadata, so all the code in src/main/webapp is just available via generic folder browsing and there is no real meaningful webapp tooling available to these projects.
• Classpath detection seems somewhat broken: In many situations, maven2 projects (and Java classes inside) are marked as broken because classes can’t be found, yet running a maven2 build on these projects works flawlessly and the project also has the required dependencies at hand.
• In some situations, even though I am not sure who’s to blame for that, projects checked out of SVN were listed as “unversioned” in the project explorer, yet the SVN metadata were there and trying to share them ended in a reconnect (no surprise).
• In another peculiar situation, the pom.xml editor kept me from deleting a dependency – no matter how often I tried to cut the corresponding <dependency> section – it always got back inserted into things. Resolved this by deleting and checking out the project again, see above.
• Having some projects that generate code off WSDL (JAX-WS) or XSD (JAXB) using corresponding maven2 plugins, it seems even though after adding the folder where the generated sources ended to the list of project source folders in Eclipse, the IDE seemed unable to actually find the generated classes on the classpath.
• There seems to be a general problem in redundancy of Eclipse metadata vs. pom.xml, ending up in information (Java Compiler level, “Java Project Facet” level, …) being entered more than once where, actually, pom.xml itself should suffice…

So here I am… my maven2 projects all in Eclipse, a bunch of them marked red, the whole mess building regardless of this but (obviously) code editing being somewhat difficult with not all classes correctly found on the classpath, and I know things feel somewhat non-deterministic. Maybe I have not completely understood how m2eclipse does a bunch of things. Maybe I’ll take a dive deeper to learn more about how it works, later… but now, and first off, I need to get work done, and by now I know the dissatisfactions arising from using two different IDEs are easier to address than the various troubles and issues arising from importing a complex maven2 project structure into Eclipse. Agreed, maybe most of my project structure could be set up more straightforward and “better”… but it works flawless in maven2 (which is at the core of it), and it works flawless in NetBeans maven2 tooling (which is on top of it), so it can’t be all that bad. And, in maven2, I just don’t want any more essential information (like Java Compiler version, …) placed anywhere but in the pom.xml, no matter which IDE in use… I’m off now to get all these .project files out of my SVN trunk…

VNC? Well, why not… Accepted this, so it just took a couple of seconds until the installer provided me with the IP address and port of the VNC server running on this host, and, back to my notebook, connecting there I indeed was capable of performing the server installation completely from the comfortable environment of my office rather than standing in front of the machine in the loud and cold server room. So… I have to admit that this has somehow changed my mind about “GUI based installers”, and this definitely is a ‘killer feature’ from my point of view. Hope other distributions to come up with something like this (maybe an installer via ssh?) sooner or later, as well… ever even thought about doing a Windows Server System remote installation via RDP? Wonder whether this even would be possible… ;)

Some evidence screenshots, just because: Network and disk configuration using CentOS VNC installer. Pretty neat. :)

Fortunately not, given the Mozilla Thunderbird client involved here did just move things there and mark them as ‘deleted’ but not actually expunge them. Copying things manually out using a shell or some other file manager was pretty easy, and figuring out what (cyr)reconstruct is supposed to do also wasn’t a job that though… to overally end up with most of the folders restored but most of them empty?!

Googling around and looking deeper into things, however, quickly did resolve this issue as well: Cyrus uses to store all its stuff in /var/spool/cyrus/mail/, sorted in a more or less straightforward hierarchy. And this is where my problem finally resolved:

• Mailbox Storage lives in /var/spool/cyrus/mail/s/Storage. So far, so good.
• Mailbox Storage.Contacts, however, lives in /var/spool/cyrus/mail/c/Storage/Contacts… which I missed paying attention to, in this situation.
• After having the mess moved to Trash manually, all the hierarchy was stored in /var/spool/cyrus/i/user/involvedUser/Trash/, including all subfolders. So just copying it back to its most obvious place (.../mail/s/Storage) didn’t work out as expected…

So after another examine-and-move session (copying all the folders starting with “A” from the deleted storage hierarchy to /var/spool/cyrus/mail/a/Storage/, all the “B” folders to /var/spool/cyrus/mail/b/Storage/ and so forth and finally running a cyrreconstruct -rfx Storage.*, both the hierarchy and the mail folder content seem to be back. And I am pondering a more sane mechanism of backing these things up… at the moment, without a good file manager I surely would have been doomed, so looking back at this match: Cyrus 0, gnu mc 1. Been there, done that. :)

After more or less loosely following Geertjans blog I rather early found out about the jvisualvm tool that comes with recent versions of Sun JDK 6, but so far I haven’t really seen any use cases for it in my environment. However, at the very least now I have figured out that this is rather good a solution for tracking down weird NetBeans behaviour by providing more reliable information, i.e. thread dumps. And there’s even more…

Dealing with NetBeans…
Indeed, my first obvious idea was trying to figure out what’s goin’ on inside a running NetBeans environment. Doing thread dumps is also possible in other ways, but at least the jvisualvm approach is pretty straightforward. On Unix systems, jvisualvm is found in the same place your java binary lives, so if you’re capable of running java -version from within a terminal, you’ll also be capable of running jvisualvm from there.


[kr@n428 7:56:18] ~> java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)
[kr@n428 7:56:23] ~> which jvisualvm
/home/kr/kontext/runtime/linux/java/bin/jvisualvm
[kr@n428 7:56:26] ~> which java
/home/kr/kontext/runtime/linux/java/bin/java


Once you started the application simply using the jvisualvm command, you’ll end up with a main application screen like this:

If you are a NetBeans user, the user interface pretty quickly will look familiar to you as jvisualvm basically “just” is an application based upon NetBeans RCP. Asides this, you will see an “Applications” window. In my case, the NetBeans I want to debug being already running, I can see the NetBeans process (pid 4974, in this example) below the “Local” node, so by now I am ready to get going:

Double-clicking the “NetBeans 6.7″ icon in “Local” will make jvisualvm connect to this process which might take some time but in the end provide me with a brief diagnostics overview showing basic information about the NetBeans process and its environment:

Right now, you have quite an amount of helpful tools at hand trying to figure out where your problem is, like doing CPU or memory profiling (analogous to the feature set already offered in NetBeans for developing Java applications), analyzing memory consumption and garbage collection, and, as desired in my situation, looking at threads and doing a thread dump. For that, the “Threads” window is what you want:

By now, there’s a button labeled “Thread dump” (the same can be achieved by right-clicking the application in the “Applications” window and choosing “Thread dump” from the context menu). Doing so will result in, obviously, jvisualvm generating a thread dump for the running application. As soon as this is finished, it both will appear opened in a new editor on the right side of the window and, as a child node below the application (NetBeans, in my case) in the “Applications” window:

By now, you can merrily browse through the dump and see what might be flaky in your current situation. Another good option, if required: In case you are about to file an issue against NetBeans regarding this behaviour and/or want to attach yourself to an existing issue, you might want to add this thread dump simply to provide developers with additional information to track down the problem. To do so, right-click the “[threaddump]…” node in the “Applications” window, choose “Save as…” and enter an appropriate name. This will result in generating a plain text file ending in “.tdump” containing the full thread dump as displayed in the jvisualvm window. This file, by then, might be attached to an issue filed against any application you’re likely to debug, or it might be sent to your application developer helping them getting your tool improved. The same procedure can be used i.e. for generating heap dumps or application snapshots, which all in itself is rather valuable while knee-deep into debugging work.

…and beyond.

So far, we’ve been looking at a Java application running on the same host jvisualvm runs on. While this is fine for desktop and overall development debugging, in a real-world environment your applications might run on some sort of application server, Glassfish in our case. Fortunately, jvisualvm also allows for working with remote applications i.e. making use of JMX. Masoud Kalali has written a quick guide on how to set up jvisualvm to work with Glassfish and, after goin’ through this procedure, indeed I am capable of doing more or less the same I previously did dealing with a local application now interacting with my remote production Glassfish:

So, overall conclusions and hints:

• If you ever filed an issue against NetBeans, eventually were asked to attach a thread or heap dump and have absolutely no clue how to do so, jvisualvm is a very helpful friend helping you getting this done.
• I like the chance of doing application debugging, but the capabilities in dealing with remote servers surely will make jvisualvm a tool I do not want to miss in my everyday tool box.

Lesson #1: Reconsider why you actually wanted to go for services…

Well… it might sound obvious, but personally I learnt that, actually, it isn’t. We used to get started with two different applications, both based upon Java EE web tier and Spring, implementing the notion of a “backend” to provide data access and a “frontend” to provide user interaction with these data, with just a small layer of business logic mixed into both of them (complexity of this application was pretty much limited initially). A lot of this “frontend/backend” separation, these days, arose out of the requirement to have the data access “hidden” somewhere in a well-protected, restricted environment whereas the environment available to end users needed to be publicly available. The backend system offering services to the frontend system (actually stateless beans exposed via Spring remoting) was our first take on SOA and, so, actually born out of the demand to have a distributed application (in terms of being distributed across several machines mainly for administrative reasons).

It changed. While our initial environment was running on several Apache Tomcat servers, we got away from that pretty soon, as it showed that the administrative effort caused by maintaining a couple of servlet containers, partly in cluster configuration, for front- and backend communicating with each other by far exceed the benefit gained from this structural approach. So, the next logical step was to “consolidate” this environment, which we simply did by moving all the stuff to one big (glassfishv2) Java EE application server in a clustered operation mode fronted by a fast web proxy restricting external user access. Our application, by then, still had this “frontend/backend” structure and also did still use the kind of remoting involved here (at this time mainly based upon Hessian web services) but not distributed anymore but on the same machine and in the same application server. So, the original reason to go for the system structure we chose had disappeared all of a sudden… and, adding to that, we realized that by now our requirements were somewhat different, even though we kind of “expected” our application structure to be up to this: We wanted parts of our application to be maintainable / redeployable independent upon each other as far as possible, so ideally being capable of deploying a new version of part of our system without having the whole application taken off-line or affected by this.

It prove that, overally, our architecture was not up for that requirement: Though we were capable of deploying “frontend” and “backend” separately, deploying “backend” (and, subsequently, having it offline for a couple of moments) involuntarily would cause “frontend” to fail simply because most of the functionality it required wasn’t around. Worse than that, we figured out that “backend” actually had grown to provide a lot of different service components that generally would do rather well on their own but, right now, just were to be separated with quite some effort because sharing a lot of common code. Bottom line, resolving this issue, was completely restructuring the whole “backend” module, extracting several services to modules of their own, extracting common code to shared libraries or other services, without breaking the overall structure, some glue code included. Which immediately gets me to…

Lesson #2: Careful with hierarchies of interdependent services!

I think this is a point where your mileage basically might vary, depending upon why you want to go for a service-oriented approach. In an environment like ours however, having the ability of maintaining services seperately without affecting the system as a whole all too much, services depending upon each other all too much tend to successfully break things in worst case. In our scenario in example, initially we were tempted to restructure “backend” by ripping out most of the common code and putting it to a “generic” database access service pulling data off tables (we can’t make meaningful use of an O/R mapper in our structure so this is the most comfortable way to do database access) and making all other services providing access to “special” kinds of data use this “databaseAccessService”.

Well… it didn’t take a prototype, just a second look to see this is not a good idea: Having this kind of structure immediately would make the “databaseAccessService” module the one not to be maintainable / redeployable easily without shutting most of the rest of the system down (most of what the system does is actually database access work). It wouldn’t lock all the system so generally this would have been an improvement compared to the global “backend” after all, but still it was bad enough not to use it. So, shared code again: Instead of building a custom “databaseAccessService” most of the shared code to deal with the database backend was extracted to a .jar project and used to create a bunch of specialized data access services providing access to special kinds of data, yet raising another caveat which is interesting in my opinion:

Lesson #3: Actually keep thinking on a service level throughout the project.

So, we do have these shiny new services, all featuring well-defined, (hopefully) well-documented interfaces to access them, along with (if required – EJB or Spring remoting) client stub libraries required to interact with them. And, along with this, we do have the service implementation code available in the same workspace / project group (as we’re using NetBeans for this kind of work)… so, while creating a new service module from scratch, one is always tempted to “directly” integrate, configure, use functionality and classes usually exposed in another service rather than using / developing against the services external interface. Asides contracts and conventions, I haven’t yet found a more convenient way of working against this than just creating a project structure which tries to make this approach difficult, but it’s not yet that convenient.

Lesson #4: Reduce the amount of shared code.

In some ways, this is a different facet of #3 in my opinion, looking at it from a different point of view and outlining other problems: Sure, library code shared between services might be a good thing while solving the same issues (generic frameworks come to mind), but asides the issue of not thinking on the service level anymore, in my opinion there is another pain arising from that: As soon as library or framework code written to be used in one service will be re-used to implement another one, chances are good to, while tweaking the framework in some aspects to better suit the needs of one service using it, make any other service depending upon the same code cease working in strange ways when being redeployed simply because, even if your API didn’t change, maybe a small yet important aspect of this shared code changed in a small way without all services depending upon this code being modified to know about (and honour) these changes. Simply put: Loads of shared code seem likely to make your everyday development life incredibly more difficult because, once you do changes in shared code, you’ll have to ensure all services to cleanly work with those as well sooner or later (or you utilize different SVN branches or different versions of maven2 artifacts which, however, still doesn’t eliminate but just postpone the problem of dealing with this).

Lesson #5: Try keeping extensibility at a sane level.

While working in a semi-agile process, starting with a small set of specified requirements, trying to get them worked out to a more fine-grained state following several iterations and prototypes, I have seen (or, better: followed) two different approaches:

The first, “dirty” one is a quick “front-to-back hack” of what is intended to happen: Starting with the user interface (after all, that’s the only part of the application the customer is likely to see…), introduce the functionality demanded / requested, and then hack up any services around to provide data needed to make this functionality work (which includes both external service interface, service implementation and anything “beyond” this). Do so carefully, try not to break the service interface defined so far, and try as well to keep a given semantic view on the interface throughout all the functionality added. The other one, seeming more “clean” at first sight, is trying to keep things as “generic” and extensible as possible, trying to predict any modification or requirement likely to appear in advance. This includes building “generic” service interfaces providing “generic” data, along with eventually introducing vast loads of (redundant?) transformation logic in order to get actual “non-generic” data off the service and ready to work with.

To me, both ways seem diametral and exactly opposite. The first one has proven to end up in, indeed, “hacked-up” code which is hard to refactor and hard to get straight simply because, in worst case, we did end up with a whole bunch of different views on the same service, with completely different understandings of what the service should be doing. The second one, however, ended up in the service actually doing nothing except for passing data around, which would render the service itself useless and again move most of the “essential” code to some processing layer inside, say, the user interface or frontend. Probably it’s a matter indeed of finding a balance between the two approaches…

Conclusions

Finding to an end on that: We learnt that, using a service-oriented approach, there are good chances of cleaning up a “grown”, heterogenous software infrastructure in a way pretty good in terms of modularization, independency of components, reuse of shared functions and so on. We also learnt, however, that there are indeed some more or less obvious pitfalls to take care of, and that some of the benefit provided by this kind of approach might on the long run be compensated by additional complexity arising from other unique aspects of the service-oriented way of doing things (like administering a bunch of loosely coupled services to actually make up the system they are intended to make up, in the end). Was it worth it? From a short term point of view – definitely yes, and be that just for the sake of having clarity in some of the points listed here. From a long term point of view… we still will have to learn, I guess. :) So, other experiences, thoughts, ideas on that welcome…

FRÜHER klärten die Öffentlich-Rechtlichen die Jugendlichen auf und sorgten für deren politische Bildung. HEUTE setzen sie ihnen, genauso wie die Privaten, eine Daily Soap nach der anderen vor.

FRÜHER war die Samstagabendunterhaltung geprägt von kreativen Ideen und Köpfen wie Rudi Carrell und Kulenkampff, HEUTE gibt es Schunkelshows mit Silbereisen & Co. sowie unkreative Frage-Shows, vorgelesen von Jörg Pilawa.

FRÜHER faszinierten ARD und ZDF mit tollen Tierdokus von Heinz Sielmann und anderen, HEUTE werden in Deutschlands Zoos Kameras aufgestellt und weltbewegende Bilder gezeigt, wie etwa ein Elefantenkäfig ausgemistet wird.
[...]

Deshalb:

* Schluss mit jeglicher Werbung und jeglichem Sponsoring bei ARD und ZDF.

* Schluss mit Quotenmessungen bei ARD und ZDF – öffentlich-rechtliches Fernsehen darf sich nicht nach Einschaltquoten richten

* Schluss mit Krawalljournalismus, Daily Soaps und hirnlosen Heimatschnulzen. Jedes Programm von ARD und ZDF muss einem Mindestmaß an Anspruch unterliegen. Anspruchsloses Fernsehen bieten uns die Privaten genug – und zwar kostenlos.

* Schluss mit immer mehr kostspieligen digitalen Sonderkanälen, in die Kultursendungen und Dokumentationen abgedrängelt werden. Wir zahlen dafür, dass ARD und ZDF selbst ein tiefsinniges Programm bringen.
[...]

Kann ich nur kommentarlos unterschreiben. Hoffentlich genug andere auch… Mehr hierzu auch bei telepolis.de.