Add WebSphere support

I have noticed a few requests for a cargo plugin for Websphere integration and also read the post by Malcolm Wong Ho (CARGO-212) offering to begin an implementation. I know Malcolm and have contacted him via email to see if he would like to collaborate on this initiative.

In the mean time, I have started implementing the Cargo Container API using the current Weblogic integration as a guide.

Are there any pointers that you can provide with respect to completing this implementation?

Some questions I have thus far:

1. Websphere currently provides ant tasks for interacting with the installation, with respect to starting/stopping and deploying applications to the server. (albeit with some restrictions). Should I simply wrap the ant task OR follow the model that seems to be currently employed where the server is launched through native java code i.e. calling the static void main() method in com.ibm.ws.bootstrap.WSLauncher after some environment and classpath configuration? I have done both, but the latter seems to fit better.

2. I am also having a few issues with respect to the propogation of environment variables, specifically the PATH, to the Websphere launcher, which in turn spawns another native process (using JNI) to launch the actual server class. Can anyone provide some assistance?

3. I also am forced to ensure that the java process is spawned under a JVM potentially different from the current process as Websphere requires that it is started using the IBM JDK? I use the ant setJvm() option to point to this potentially different JRE executable? Does this make sense?

Your thoughts and opinions are greatly appreciated.

Lastly what are the best steps to start placing my very early code into the subversion repository so others may offer some assistance.

My objectives would be to support both WAS 5.x, and WAS 6.x from within cargo.

Ahmed

Ahmed, that's excellent news! I'm going to answer on the cargo dev list if you don't mind (the list is a better medium for a discussion).

The preliminary code that is meant to test against a local websphere container.

The issue as described previously on the dev list is related to the Java Library Path. I have still not managed to solve this.

The startServer.log when Websphere is started from the command line is as follows:

• • • • • • • • • • • • Start Display Current Environment ************
                        WebSphere Platform 5.1 [BASE 5.1.0 b0344.02] running with process name DAMJANJ\DAMJANJ\server1 and process id 3556
                        Host Operating System is Windows XP, version 5.1
                        Java version = J2RE 1.4.1 IBM Windows 32 build cn1411-20031011 (JIT disabled), Java Compiler = , Java VM name = Classic VM
                        was.install.root = C:\java\WebSphere\AppServer
                        user.install.root = C:\java\WebSphere\AppServer
                        Java Home = C:\java\WebSphere\AppServer\java\jre
                        ws.ext.dirs = C:\java\WebSphere\AppServer\java/lib;C:\java\WebSphere\AppServer/classes;C:\java\WebSphere\AppServer/classes;C:\java\WebSphere\AppServer/lib;C:\java\WebSphere\AppServer/lib/ext;C:\java\WebSphere\AppServer/web/help;C:\java\WebSphere\AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
                        Classpath = C:\java\WebSphere\AppServer/properties;C:\java\WebSphere\AppServer/properties;C:\java\WebSphere\AppServer/lib/bootstrap.jar;C:\java\WebSphere\AppServer/lib/j2ee.jar;C:\java\WebSphere\AppServer/lib/lmproxy.jar;C:\java\WebSphere\AppServer/lib/urlprotocols.jar
                        Java Library path = C:\java\WebSphere\AppServer\java\bin;.;C:\WINNT\system32;C:\WINNT;C:\java\WebSphere\AppServer\bin;C:\java\WebSphere\AppServer\java\bin;C:\java\WebSphere\AppServer\java\jre\bin;C:\java\jdk1.5.0_06\bin;C:\java\maven-2.0.4\bin;C:\java\IBM\WebSphere MQ\Java\lib;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\java\subversion\bin;C:\java\maven-2.0.4\bin;C:\java\IBM\WebSphere MQ\bin;C:\java\IBM\WebSphere MQ\WEMPS\bin;C:\java\jdk1.5.0_06\bin;C:\Program Files\SecureCRT\;C:\PROGRA~1\Serena\vm\win32\bin;C:\PROGRA~1\Serena\vm\common\bin\win32;c:\java\apache-ant-1.6.5\bin;C:\Program Files\WinSCP3\;C:\Program Files\QuickTime\QTSystem\;C:\java\maven-1.0.2\bin;C:\Program Files\MySQL\MySQL Server 5.0\bin;C:\java\IBM\WebSphere MQ\bin;C:\java\IBM\WebSphere MQ\java\bin;C:/java/IBM/WebSphere MQ/WEMPS\bin
                        • End Display Current Environment *************

The startServer.log when Websphere is started using CARGO is as follows:

• • • • • • • • • • • • Start Display Current Environment ************
                        Host Operating System is Windows XP, version 5.1
                        Java version = J2RE 1.4.1 IBM Windows 32 build cn1411-20031011 (JIT disabled), Java Compiler = , Java VM name = Classic VM
                        was.install.root = C:\java\WebSphere\AppServer
                        user.install.root = C:\java\WebSphere\AppServer
                        Java Home = C:\java\WebSphere\AppServer\java\jre
                        ws.ext.dirs = C:\java\WebSphere\AppServer\java/lib;C:\java\WebSphere\AppServer/classes;C:\java\WebSphere\AppServer/classes;C:\java\WebSphere\AppServer/lib;C:\java\WebSphere\AppServer/lib/ext;C:\java\WebSphere\AppServer/web/help;C:\java\WebSphere\AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
                        Classpath = C:\java\WebSphere\AppServer/properties;C:\java\WebSphere\AppServer/properties;C:\java\WebSphere\AppServer/lib/bootstrap.jar;C:\java\WebSphere\AppServer/lib/j2ee.jar;C:\java\WebSphere\AppServer/lib/lmproxy.jar;C:\java\WebSphere\AppServer/lib/urlprotocols.jar
                        Java Library path = C:\java\WebSphere\AppServer\java\bin;.;C:\WINNT\system32;C:\WINNT;C:\java\jdk1.5.0_06\bin;C:\java\maven-2.0.4\bin;C:\java\IBM\WebSphere MQ\Java\lib;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\java\subversion\bin;C:\java\maven-2.0.4\bin;C:\java\IBM\WebSphere MQ\bin;C:\java\IBM\WebSphere MQ\WEMPS\bin;C:\java\jdk1.5.0_06\bin;C:\Program Files\SecureCRT\;C:\PROGRA~1\Serena\vm\win32\bin;C:\PROGRA~1\Serena\vm\common\bin\win32;c:\java\apache-ant-1.6.5\bin;C:\Program Files\WinSCP3\;C:\Program Files\QuickTime\QTSystem\;C:\java\maven-1.0.2\bin;C:\Program Files\MySQL\MySQL Server 5.0\bin;C:\Program Files\Common Files\InterSystems\Cache
                        • End Display Current Environment *************

The thing to note is the Java Library Path, everything else seems to be OK. This results in certain websphere specific DLLS not being found at runtime on the path.

Any ideas?

Try to execute the setupCmdLine.sh/bat first (To set some environment variables).

You can also try to set the java.library.path system.

Hi Ahmed,

I've started looking at your patch. Nothing much to say. I think you've pretty well understood how cargo works. Some minor comments:

• In the property set I think there are properties that shouldn't be weblogic-specific. You should try to reuse existing global properties from GeneralPropertySet, RemotePropertySet, ServletPropertySet and DataSourcePropertySet
• I think there are some checkstyle errors remaining. For example braces are put in the next line in our code style
• There are some parts missing like the configure() method but I'm sure this is because you've not started working on those yet as you're focusing on getting the container started...

What's your status? Have you been able to progress?

Vincent,

I have made much more progress with the Websphere support for cargo. I am about to post another patch which currently provides the following features:

1. Websphere 5.x support at the moment only

2. Support for a local existing installation of websphere only (at this stage)

3. The ability to start, stop, deploy, start, stop, and undeploy applications (only EARs at this stage)

Some issues that I have had and am currently still experiencing (and may need some assistance with):

1. Websphere must be executed with the IBM JDK that is provided with it, thus I override the JRE executable used on the java task. This means that the maven/cargo don't necessarily
have to also run in the IBM JVM which I think is a good thing.

2. Getting websphere to start proved to be quite a challenge. After much trial and error I now have a working configuration, however I am not entirely sure how portable this code is and more
importantly I am not sure if this covers any other edge cases or the code is superfluous.

3. Websphere needs to be installed on the machine from which the plugin is executed, for both local and remote containers, as the ability to deploy/undeploy requires many libraries, dlls, etc
from the websphere home directory. This can still target any other independent server available remotely, albeit with added complexity (such as the file may need to be copied/scp'ed/ftp'ed to the
server first as deployment can only happen from a local mount point). This also poses some design questions as the remote container is much more like a local container, specifically with regards
to the configuration it requires (the home directory, the websphere home directory, etc.) What is the best design guidance here?

4. I currently only support EAR deployments as I not entirely sure if the Websphere JACL script I am using can be used on WAR applications. This still needs to be tested.

5. The EAR deployable had to be customised as well for Websphere as the install task requires the physical EAR to be installed, but all other tasks such as start, stop, and uninstall require the symbolic
name of the EAR. Hence, I have subclassed the EAR deployable and used the display-name of the ear from the application.xml file if it is available, otherwise I fall back to the default of using the ear
filename without the .ear suffx. This requires me to subclass EAR, but the name property is private and the parseName() method is private, which I need to both be protected.

The way forward:

1. Implement Remote Container functionality (your thoughts?)
2. Attempt to add support for WAR, EJB archives?
3. Clean up the code, there seems to still be a bit of duplication
4. Write tests, tests, and more tests?
5. Attempt support for Websphere 6.x

That being said I am currently using the cargo websphere plugin to deploy onto a container within my own environment succesfully.

I don't really have much time to work on this plugin at the moment, and any help for anyone would be appreciated. I hope that make this code available will atleast create a common platform to discuss
additions, enhancements, etc.

Lastly what steps need to be taken to obtain commit access as the process continues.

The parent pom for containers, it basically ads the websphere module as a child.

The websphere container implementation.

The websphere entries are added to the generic factory.

The websphere entries are added to the generic factory.

The websphere entries are added to the generic factory.

The websphere entries are added to the generic factory.

The websphere entries are added to the extensions pom to support the maven plugin.

Hi Ahmed,

Thanks for all this. I'll check your patch but what I really need is a patch (in patch format) for all the files you have modified (all in one patch is the easiest). I don't know what svn client you're using but it you're using TortoiseSVN or Eclipse subclipse you simply need to choose the "create patch" action. Make sure that you svn add any new file so that they'll appear in the patch.

I need to check the code before I can provide any answer to the questions you asked above.

The way forward as I see it would be:

1) Check patch and ensure its design matches cargo. From your earlier patch I think there should'nt be any big issue

2) Apply your patch as is and simply run the build using m1, after ensuring that the tests will run on WebSphere. If it's doesn't pass the build we'll need to work together to fix it. The only issue for this is that 1) I will need WebSphere to be installed and 2) Our tests assume the existence of a local installed configuration existence. If it doesn't exist, the tests will not fail but they won't be run. More specifically the existing configuration tests need the standalone configuration to exist. Thus I believe that we need a 1.5) step before step 2) which is: implement standalone configuration.

3) Commit your patch once it passes the build and document the new WebSphere support on the Cargo web site

4) Start working on other missing things as you've mentioned in your "way forward" section above.

WDYT? Do you think you could create the standalone configuration?

Thanks
-Vincent

Ahmed, I forgot to answer your commit access question....

The way we do it is to have the contributor send some patches to ensure they understand cargo well enough. Then ensure that he has a lasting interest in cargo (as much as possible). Then we send a vote for making him a committer and the other committers cast their votes.

Thus I would suggest we finish applying your WebSphere patches and finish working together on it (It'll probably require a few iterations). Then we would be VERY interested to have you in as a cargo committer as we'll need someone to support and maintain this websphere implementation!

Thanks
-Vincent

Vincent,

Sorry for the delay. I am about to commit a patch of the changes necessary as discussed.

Your continued comments and advice are appreciated.

PS. Trying to support a standalone configuration is going to be extremely challenging. How configurable (in your opinion) should this standalone configuration be i.e. security, host, port, etc, etc. ?
I finally figured out how to do this in Websphere. but there is a lot of configuration to understand and then possibly filter, because of overridable settings at a cell, node and server level.

How should I proceed with the tests? Do I need integration tests or simply unit tests?

Ahmed.

Vincent,

I have not incorporated your comments, as per managing the static deployables for Websphere yet. I will do this ASAP.

Thanks
Ahmed.

> How configurable (in your opinion) should this standalone configuration be i.e. security, host, port, etc, etc. ?

It should support them all but we could start with port as it's the most important one and it's required for our existing test suite. Each container has Capability classes that tells what properties the container supports.

> How should I proceed with the tests? Do I need integration tests or simply unit tests?

Unit tests are very much appreciated They need to work in complete isolation (no running container and no expectations set on the environment, even on the file system as much as possible). For integration/functional tests we already have a generic suite of tests and thus the goal is simply to ensure that your implementation passes those tests. In order for the test suite to pick your WAS implementation you'll simply need to register the WAS classes by modifying the DefaultContainerFactory, DefaultConfigurationFactory, DefaultDeployerFactory, etc. classes. Then set the container to be WAS (using the cargo.containers property). See http://cargo.codehaus.org/Building for more details.

Thanks!
-Vincent

PS: I haven't looked at your patch yet. Will try to do asap.

Hi Ahmed,

Do you need any assistance in development of the plugin?

Thanks,
Malcolm

Can we get this into https://svn.codehaus.org/cargo-contrib/ for further work?

David, I suggest to give you access to cargo-contrib so that you can commit it there? To do so, you need to have an account on Xircles (xircles.codehaus.org) and request the access on xircles. Let me know when you've done that and I'll accept you.

I've imported the

1. cargo-websphere.patch (221 kb)

into the contrib repo @ https://svn.codehaus.org/cargo-contrib/websphere/trunk/ (only the core/containers/websphere part).

I have gotten much further with the Websphere 5x support.

In particular the Websphere 5x plugin currently has the following functionality:

• Support Websphere5x
  > This has been tested on Websphere 5.1 upto patch level 14

• Supports Existing Installed Container
  > This is based on an existing installation, with a pre-defined configuration as dictated by the CONFIGURATION_ROOT

• Supports Standalone Installed Container
  > supports the configuration of hostname, cell name, node node, server name, various ports for admin console, etc., and configuration of datasources

• The ability to start and stop the Container

• The ability to deploy, start, stop, and undeploy EAR applications

• Has support for configuration via the Generic Container support i.e. DefaultContainerFactory, DefaultConfigurationFactory

• Has support for datasource configuration (Oracle ONLY)

The Websphere5x plugin is currently missing the following functionality:

• Ability to deploy WAR applications
• Support for static deployable units (we should be able to this via specifying an alternate APP_INSTALLATION_ROOT)
• Support for locking down security (with respect to enabling global security, enforcing https access, creating users with the available roles, using a custom authentication module, etc.)
• Being able to specify an alternate location for the CONFIGURATION_ROOT in an Existing Installed Container (this could help to configure more complex properties)
• Extend support for datasource configuration beyond just an oracle database (this should be fairly easy given the structure we have now)
• Support for a Remote Container (should be able to achieve this fairly easily, since all it requires is deployment, but there are some challenges with respect to uploading the deployable unit to the server)

Some other general areas that need to still be looked into:

• The tests are nowhere near sufficient or absolutely repeatable without environment tweaks, we need more extensive unit and integration tests

• The code can still do with some general cleanup and restructuring

• There may be better ways to accomplish some things (I simply took a first stab):
  > Starting/Stopping the container is not done via the startServer/stopServer scripts or the Websphere ANT tasks (maybe this is an alternative) (we definitely have more controlling with the ant java forking process)
  > Copying the intial configuration for a Websphere Standalone Installed Container could possibly be simplified

The code has been commited cargo-contrib, and the relevant changes that were need on cargo-core have been submitted here as patch. The test may require some changes for your environment before they work, but I think it will provide some guide on how to proceed.

Anyway, I believe this is a good first pass. Hopefully, the community agrees. Please contact me via the mailing list if there are any questions, or you are prepared to offer assistance in the form of testing and/or development.

Ahmed

The patch to cargo-core to support the websphere plugin.

I have developed conjuntion with CARGRO-146 support for WAS 6.0 and also going to support 6.1 as clients request for end of this month.

• Markku Saarela

CARGO-146 contains WAS 6.1 support

At the point this module is integrated, it will support version 6.x and 7.0. IBM no longer provides a download link for Websphere 5.

There is an existing implementation in the cargo-contrib area. That said, it appears to be more of a proof of concept. I expect re-implementing this or cleaning it up to take the same effort.

No one has been working on this for a while now. Any input will be appreciated.

Note that the container probably needs to be reimplemented: many things have changed in Cargo since 2009!

The new implementation could perhaps reuse an existing container, GlassFish would be an easy example...

Since JSR88 has been deprecated since WebSphere 8, adding commands for deploy and undeploy.

Added files for creating a new profile

Also added Java commands for starting, stopping and administering WebSphere.

I think we should have all required information now

A new implementation based on CARGO 1.2.4 has been added to https://svn.codehaus.org/cargo/sandbox/websphere

It usually works, but after some point the WebSphere profile manager goes crazy and starts outputting the messages in WebSphere test log.log

The following validation errors were present with the command line arguments: 
 	profilePath: The profile path is not valid.

What is interesting is that it works with WebSphere out of the box a few times, and after some point it stops working.

I think WebSphere 8.5's profile manager is probably not stable enough for CARGO's typical use cases...

If anyone wants to take a look, feel free. To try it out, simply check out, go into the websphere directory and run mvn:

mvn clean install -Pwebsphere85x -Dcargo.websphere85x.home="C:/Program Files/IBM/WebSphere/AppServer"

There is also a much simpler sample, using Maven and existing WebSphere configuration. To run, compile the websphere container in websphere/containers/websphere and then go to websphere-test and execute:

mvn cargo:run -Dcargo.websphere85x.home="C:/Program Files/IBM/WebSphere/AppServer"

Completed: At revision: 3422