Maven 2 & 3
  1. Maven 2 & 3
  2. MNG-3989

Simple handling of external jars

    Details

    • Type: New Feature New Feature
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0.9
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Complexity:
      Intermediate
    • Number of attachments :
      2

      Description

      For whatever reason, there will always be jars that don't exist in a maven repository.

      There are numerous techniques for these - installing them in your local repo (either manually or with
      some bootstrap.sh script or special profile activation). Checking in the jars into a local maven repository that is checked into svn
      and then point to it from your settings.xml and/or top level pom (with aid of an env variable).

      But all these methods lack a very important features. You can just do: "svn co http:/myproj.com/foo; cd foo; mvn"
      If the jars change, you can't just do "svn up; mvn", you have to re-run whatever script/profile installed the repo.
      It's all rather a PITA.

      What I want, is some way to have a module of a project that contains some non-maven jars that when I
      do a "mvn install" in that project, install those jars in my local repository for use by my other modules. If the
      jars are not updated, then nothing is done.

      With something like this, projects that have external dependencies could describe them to maven and
      make them available for use, without manual steps and special scripts.

        Activity

        Hide
        Anders Kr. Andersen added a comment -

        I think it is possible to declare a maven repository in a local file structure in a maven project that is stored in your VCS
        And the declare the file structure to be a maven repository

        And then reference these as dependencies / or copy them to local repository.

        I just haven't seen a description of this.

        Show
        Anders Kr. Andersen added a comment - I think it is possible to declare a maven repository in a local file structure in a maven project that is stored in your VCS And the declare the file structure to be a maven repository And then reference these as dependencies / or copy them to local repository. I just haven't seen a description of this.
        Hide
        Greg Wilkins added a comment -

        Anders,

        you can do that (and I often do), but you need every pom that uses that repository to at least set a property to say where it is on the disk - that is an issue and prevents simply adding new modules or refactoring disk layout. At the best, it establishes a fixed directory relationship between modules (eg $

        {basedir}

        /../../project_repo

        Show
        Greg Wilkins added a comment - Anders, you can do that (and I often do), but you need every pom that uses that repository to at least set a property to say where it is on the disk - that is an issue and prevents simply adding new modules or refactoring disk layout. At the best, it establishes a fixed directory relationship between modules (eg $ {basedir} /../../project_repo
        Hide
        Brett Porter added a comment -

        hi Greg. I think I have a couple of possibilities in mind for this either through current caps or a simple plugin. Just have one question: do you have constraints on how you manage the coordinates and provision of POMs and coordinates for the dependencies?

        If it is possible to put down a repo on disk (POMs and all) and have that installed in the local repo from within the project, then there are some reasonable ways to achieve it. Is that what you are looking for?

        Show
        Brett Porter added a comment - hi Greg. I think I have a couple of possibilities in mind for this either through current caps or a simple plugin. Just have one question: do you have constraints on how you manage the coordinates and provision of POMs and coordinates for the dependencies? If it is possible to put down a repo on disk (POMs and all) and have that installed in the local repo from within the project, then there are some reasonable ways to achieve it. Is that what you are looking for?
        Hide
        Greg Wilkins added a comment -

        Hi Brett,

        yes having a repo on disk (POMs and all) is what I currently do. I define the repository in the top level pom relative to a property. Then each and every pom in that project has to set that property to locate the repository on disk.

        It would be much better if there was a way of one module that contained the local repository being able to install it in your local repository. Then other modules could just use it normally without the need for a property or a constrained disk layout.

        Show
        Greg Wilkins added a comment - Hi Brett, yes having a repo on disk (POMs and all) is what I currently do. I define the repository in the top level pom relative to a property. Then each and every pom in that project has to set that property to locate the repository on disk. It would be much better if there was a way of one module that contained the local repository being able to install it in your local repository. Then other modules could just use it normally without the need for a property or a constrained disk layout.
        Hide
        Brett Porter added a comment -

        Greg, yep that's what I meant - a module with a repo that installs to the local repo, rather than others referring to it.

        Let me see what I can come up with.

        Show
        Brett Porter added a comment - Greg, yep that's what I meant - a module with a repo that installs to the local repo, rather than others referring to it. Let me see what I can come up with.
        Hide
        Jason van Zyl added a comment -

        In what case can't you make it easier for your users by putting it in a repository? What's the case that made you ask for this?

        Show
        Jason van Zyl added a comment - In what case can't you make it easier for your users by putting it in a repository? What's the case that made you ask for this?
        Hide
        Greg Wilkins added a comment -

        In many enterprises, getting permission to run a repository is difficult (specially when you are trying to move to maven by stealth and converting projects without high level permission as a proof of principal).

        More over, I really like the ideal of maven for repeatable builds. I trust that repo.maven.org is going to exist and be well managed into the foreseeable future, but one can't expect enterprises to keep local repositories running and/or consistent, specially in todays climate where said enterprises like to lay-off technical staff using a megaphone!

        So a local repository might give me a repeatable build for a week or a month, but 1,2 or 5 years would be a harder to achieve.

        Show
        Greg Wilkins added a comment - In many enterprises, getting permission to run a repository is difficult (specially when you are trying to move to maven by stealth and converting projects without high level permission as a proof of principal). More over, I really like the ideal of maven for repeatable builds. I trust that repo.maven.org is going to exist and be well managed into the foreseeable future, but one can't expect enterprises to keep local repositories running and/or consistent, specially in todays climate where said enterprises like to lay-off technical staff using a megaphone! So a local repository might give me a repeatable build for a week or a month, but 1,2 or 5 years would be a harder to achieve.
        Hide
        Jason van Zyl added a comment -

        For a given project starting you should just check in the repository.

        If you care about your releases then generally what I'm seeing is that enterprises who understand the importance of using components put their repositories in the same class as their SCM and put it on systems that are just as reliable. They will be there 10 years from now.

        I don't really see the advantage of making a tool to take some JARs and putting it in a local repository when in the short term of a pilot project just check the JARs to a file-based repository.

        Show
        Jason van Zyl added a comment - For a given project starting you should just check in the repository. If you care about your releases then generally what I'm seeing is that enterprises who understand the importance of using components put their repositories in the same class as their SCM and put it on systems that are just as reliable. They will be there 10 years from now. I don't really see the advantage of making a tool to take some JARs and putting it in a local repository when in the short term of a pilot project just check the JARs to a file-based repository.
        Hide
        Greg Wilkins added a comment -

        Jason,

        it's not just for pilot projects. Let's say that I have a huge project with many modules with many dependencies, most of which are available in repo.maven.org

        But I have just a few jars that are not available in repo.maven.org. This may be because they just have not been published, or it may be that they are proprietary binary only jars with limited availability, or who knows the reason.

        I really do not want to have to setup a local repository just for these few jars.

        If there is the ability to have a few jars checked into SCM, that is far simpler and more reliable than creating and securing a private repository. It also means the source can be delivered across organizational boundaries without requiring a repository to be setup.

        Now I hate jars checked into SCM, but for better or for worse, it is a common practise.
        However, I hate seeing things like "../../repo" in all poms even more!

        I've seen maven dropped by a large investment bank because of issues with a local repository.
        I've frequently faced opposition to introducing maven simply because svn co; cd; mvn is not
        sufficient to build.

        I do recognize that a tool such as I'm suggesting could be misued really really badly, but then so
        can many things in every project. I do think such a tools would help the adoption of maven by
        many organizations and/or projects.

        cheers

        Show
        Greg Wilkins added a comment - Jason, it's not just for pilot projects. Let's say that I have a huge project with many modules with many dependencies, most of which are available in repo.maven.org But I have just a few jars that are not available in repo.maven.org. This may be because they just have not been published, or it may be that they are proprietary binary only jars with limited availability, or who knows the reason. I really do not want to have to setup a local repository just for these few jars. If there is the ability to have a few jars checked into SCM, that is far simpler and more reliable than creating and securing a private repository. It also means the source can be delivered across organizational boundaries without requiring a repository to be setup. Now I hate jars checked into SCM, but for better or for worse, it is a common practise. However, I hate seeing things like "../../repo" in all poms even more! I've seen maven dropped by a large investment bank because of issues with a local repository. I've frequently faced opposition to introducing maven simply because svn co; cd; mvn is not sufficient to build. I do recognize that a tool such as I'm suggesting could be misued really really badly, but then so can many things in every project. I do think such a tools would help the adoption of maven by many organizations and/or projects. cheers
        Hide
        Anders Kr. Andersen added a comment -

        Hi Greg
        It seems like it is up to you to come with a solution

        I have the same issue with weblogic jar files.

        What we need is a well defined way to load artifacts as jar,war, zip etc into a repository. mostly local repository

        Consider we made a plugin that could:
        a) read from a directory "repository" and load all files it finds there into repository. Where the maven coordinaes will be read from the file structure. (so repository should be structured as a maven 2 repository)
        b) construct poms as long as they where not there.

        I think this could be great.
        The problem I have left is to get mvn clean install executed on this pom, so developers can get started ?

        Show
        Anders Kr. Andersen added a comment - Hi Greg It seems like it is up to you to come with a solution I have the same issue with weblogic jar files. What we need is a well defined way to load artifacts as jar,war, zip etc into a repository. mostly local repository Consider we made a plugin that could: a) read from a directory "repository" and load all files it finds there into repository. Where the maven coordinaes will be read from the file structure. (so repository should be structured as a maven 2 repository) b) construct poms as long as they where not there. I think this could be great. The problem I have left is to get mvn clean install executed on this pom, so developers can get started ?
        Hide
        Anders Kr. Andersen added a comment -

        And one more thing...

        I have problems with developers that sits and uploads files into the repository from here and there, the earth, moon, or from an other planet, without documenting where the things comes from.

        So on my last project I required from all developers (sub projects) that they decared a directory "repository" with thse 3'rd party libraries.
        And then we had a shell script that uploaded the stuff to repository.

        So the point here is to give maven projects a place to document this act.

        Show
        Anders Kr. Andersen added a comment - And one more thing... I have problems with developers that sits and uploads files into the repository from here and there, the earth, moon, or from an other planet, without documenting where the things comes from. So on my last project I required from all developers (sub projects) that they decared a directory "repository" with thse 3'rd party libraries. And then we had a shell script that uploaded the stuff to repository. So the point here is to give maven projects a place to document this act.
        Hide
        Brett Porter added a comment -

        hey Greg - this structure seems closest to what you are looking for without making modifications. IS it on the right track for what you were looking for?

        You can do a bit more by using the dependency plugin or install:install-file (eg, generate POMs), but you still tend to end up with a separate module and declarations can be quite lengthy.

        Beyond that it is getting towards writing a custom plugin to copy files directly into the local repository.

        In any of these cases, it'll result in a transitive closure that won't be resolvable if the project is deployed into the repository without copying the repository module around (which is probably ok in situations like the weblogic one mentioned).

        I'd still be reluctant to build this right into Maven given the number of alternatives available, though.

        Show
        Brett Porter added a comment - hey Greg - this structure seems closest to what you are looking for without making modifications. IS it on the right track for what you were looking for? You can do a bit more by using the dependency plugin or install:install-file (eg, generate POMs), but you still tend to end up with a separate module and declarations can be quite lengthy. Beyond that it is getting towards writing a custom plugin to copy files directly into the local repository. In any of these cases, it'll result in a transitive closure that won't be resolvable if the project is deployed into the repository without copying the repository module around (which is probably ok in situations like the weblogic one mentioned). I'd still be reluctant to build this right into Maven given the number of alternatives available, though.
        Hide
        Greg Wilkins added a comment -

        Brett that's BRILLIANT!

        It works exactly how I wanted without any changes to maven!

        Thanks! I think this should be written up as THE pattern of how to
        include extra jars!

        Show
        Greg Wilkins added a comment - Brett that's BRILLIANT! It works exactly how I wanted without any changes to maven! Thanks! I think this should be written up as THE pattern of how to include extra jars!
        Hide
        Brett Porter added a comment -

        cleaned up version

        Show
        Brett Porter added a comment - cleaned up version
        Hide
        Brett Porter added a comment -

        Updating for those still referring to the ticket:

        This appears not to work in Maven 3.0 due to https://cwiki.apache.org/confluence/display/MAVEN/Maven+3.x+Compatibility+Notes#Maven3.xCompatibilityNotes-ResolutionfromLocalRepository

        You can "trick" it by giving the repository in repository/pom.xml the same repository ID as one that will be in use (though it must be the ID of your mirror if you are using a repository manager - which defeats the purpose).

        I think under Maven 3, it might be better to write an extension that supplies the additional repository as an additional source of artifacts. I took a quick hack at this here:

        There are likely to be much better ways to do this once I had time to review the code that delegates the reactor resolution, but this might work as a starting point to customise for your needs.

        Note that the same pitfalls still apply - you can't deploy the projects to a repository without the dependencies, and in this case you can't build just "my-artifact" (unless you do mvn install -pl my-artifact" from the reactor root)

        Show
        Brett Porter added a comment - Updating for those still referring to the ticket: This appears not to work in Maven 3.0 due to https://cwiki.apache.org/confluence/display/MAVEN/Maven+3.x+Compatibility+Notes#Maven3.xCompatibilityNotes-ResolutionfromLocalRepository You can "trick" it by giving the repository in repository/pom.xml the same repository ID as one that will be in use (though it must be the ID of your mirror if you are using a repository manager - which defeats the purpose). I think under Maven 3, it might be better to write an extension that supplies the additional repository as an additional source of artifacts. I took a quick hack at this here: http://svn.apache.org/repos/asf/maven/sandbox/trunk/examples/extensions/bundled-repository-extension http://svn.apache.org/repos/asf/maven/sandbox/trunk/examples/MNG-3989-maven3 There are likely to be much better ways to do this once I had time to review the code that delegates the reactor resolution, but this might work as a starting point to customise for your needs. Note that the same pitfalls still apply - you can't deploy the projects to a repository without the dependencies, and in this case you can't build just "my-artifact" (unless you do mvn install -pl my-artifact" from the reactor root)
        Hide
        Jesse Glick added a comment -

        http://jira.codehaus.org/secure/attachment/53864/MNG-1867.zip shows a related technique (using only install:install-file, no <repositories> needed); probably suffices if you have only a smallish number of external JARs to manage.

        By the way: repositoryDir.toURL().toExternalForm() should be repositoryDir.toURI().toString() lest it break when the path has funny characters in it.

        Show
        Jesse Glick added a comment - http://jira.codehaus.org/secure/attachment/53864/MNG-1867.zip shows a related technique (using only install:install-file , no <repositories> needed); probably suffices if you have only a smallish number of external JARs to manage. By the way: repositoryDir.toURL().toExternalForm() should be repositoryDir.toURI().toString() lest it break when the path has funny characters in it.
        Hide
        Basil K. added a comment - - edited

        The 'addjars-maven-plugin' simplifies handling of external jars. Just add the following declaration to your pom.xml:

         
        <plugin>
            <groupId>com.googlecode.addjars-maven-plugin</groupId>
            <artifactId>addjars-maven-plugin</artifactId>
            <version>1.0.1</version>
            <executions>
                <execution>
                    <goals>
                        <goal>add-jars</goal>
                    </goals>
                    <configuration>
                        <resources>
                            <resource>
                                <directory>lib</directory>
                            </resource>
                        </resources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        

        In the given example, all jars from the 'lib' directory (including subdirectories) will be added to project's classpath. You can also specify the dependency scope and lists of includes and excludes.

        Show
        Basil K. added a comment - - edited The 'addjars-maven-plugin' simplifies handling of external jars. Just add the following declaration to your pom.xml: <plugin> <groupId> com.googlecode.addjars-maven-plugin </groupId> <artifactId> addjars-maven-plugin </artifactId> <version> 1.0.1 </version> <executions> <execution> <goals> <goal> add-jars </goal> </goals> <configuration> <resources> <resource> <directory> lib </directory> </resource> </resources> </configuration> </execution> </executions> </plugin> In the given example, all jars from the 'lib' directory (including subdirectories) will be added to project's classpath. You can also specify the dependency scope and lists of includes and excludes.

          People

          • Assignee:
            Unassigned
            Reporter:
            Greg Wilkins
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: