Dependency resolving process "spammed" by requests

[Georg S.]

Hello,

when I tried to figure out why the process of the "resolving dependencies" is so slow, I realized that there are at least 20 HTTP requests to repo1.maven.org/maven2/org/codehaus/groovy/groovy/1.xxxxx for "xxx" in different version from 1.7.5 to never ones, even not ordered.

Both from the Maven repository maintainers' and from a developer's view, I think it's not really necessary and waisting time and net traffic resources.

Could someone explain what's the real reason behind this? What kind of way would you suggest to optimize it?
Or does it depend on the dependency structure of my project?

Thanks
 George

[Wayne Fay]

> when I tried to figure out why the process of the "resolving dependencies"
> is so slow, I realized that there are at least 20 HTTP requests to
> repo1.maven.org/maven2/org/codehaus/groovy/groovy/1.xxxxx for "xxx" in
> different version from 1.7.5 to never ones, even not ordered.

Can you share part or all of the actual build log (post to Pastebin or
Gist) that demonstrates what you are describing?

Wayne

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Georg S.]

Dear Wayne,

thanks for your reply.
First let me add that it's a Grails 1.3.7 project. The behavior could be seen when just running 'grails run-app'.

As follows a snippet from BuildConfig which might be of help:

grails.project.dependency.resolution = {
        // inherit Grails' default dependencies
        inherits("global") {
                // uncomment to disable ehcache
                // excludes 'ehcache'
        }
        log "warn" // log level of Ivy resolver, either 'error', 'warn', 'info', 'debug' or 'verbose'
        repositories {
                grailsPlugins()
                grailsHome()
                grailsCentral()

                mavenLocal()
                mavenCentral()
                mavenRepo "http://repository.codehaus.org"
                mavenRepo "http://download.java.net/maven/2/"
        }
        dependencies {

                plugins {
                        build(":lesscss:0.9.4") {
                                excludes "js"
                        }
                }

                runtime 'mysql:mysql-connector-java:5.1.5'
                runtime 'org.springframework:org.springframework.test:3.0.5.RELEASE'

                compile 'net.sf.jtidy:jtidy:r938'
                compile 'joda-time:joda-time:1.6.2'

                compile('org.apache.velocity:velocity:1.7')
                compile 'log4j:apache-log4j-extras:1.1'
                compile('xerces:xercesImpl:2.9.1') { exclude 'xml-apis'}
                compile('net.sourceforge.nekohtml:nekohtml:1.9.15') {
                        excludes "xercesImpl"
                }

                compile('org.codehaus.groovy.modules.http-builder:http-builder:0.5.1') {
                        excludes "xercesImpl"
                }
                compile('org.apache.httpcomponents:httpclient:4.1.2')
        }
}


What's the best way to create the build.log that you mentioned? Do you mean the console output? It won't be much to see. Or is there a parameter for a 'verbose' built?

Thanks
 George

[Graeme Rocher-4]

http-builder has a dependency range in it. The solution is to exclude
the transitive groovy dependency from http-builder

Cheers

On Thu, Mar 29, 2012 at 10:16 AM, Georg S. <[hidden email]> wrote:

> Dear Wayne,
>
> thanks for your reply.
> First let me add that it's a Grails 1.3.7 project. The behavior could be
> seen when just running 'grails run-app'.
>
> As follows a snippet from BuildConfig which might be of help:
>
> grails.project.dependency.resolution = {
>        // inherit Grails' default dependencies
>        inherits("global") {
>                // uncomment to disable ehcache
>                // excludes 'ehcache'
>        }
>        log "warn" // log level of Ivy resolver, either 'error', 'warn', 'info',
> 'debug' or 'verbose'
>        repositories {
>                grailsPlugins()
>                grailsHome()
>                grailsCentral()
>
>                mavenLocal()
>                mavenCentral()
>                mavenRepo "http://repository.codehaus.org"
>                mavenRepo "http://download.java.net/maven/2/"
>        }
>        dependencies {
>
>                plugins {
>                        build(":lesscss:0.9.4") {
>                                excludes "js"
>                        }
>                }
>
>                runtime 'mysql:mysql-connector-java:5.1.5'
>                runtime 'org.springframework:org.springframework.test:3.0.5.RELEASE'
>
>                compile 'net.sf.jtidy:jtidy:r938'
>                compile 'joda-time:joda-time:1.6.2'
>
>                compile('org.apache.velocity:velocity:1.7')
>                compile 'log4j:apache-log4j-extras:1.1'
>                compile('xerces:xercesImpl:2.9.1') { exclude 'xml-apis'}
>                compile('net.sourceforge.nekohtml:nekohtml:1.9.15') {
>                        excludes "xercesImpl"
>                }
>
>                compile('org.codehaus.groovy.modules.http-builder:http-builder:0.5.1') {
>                        excludes "xercesImpl"
>                }
>                compile('org.apache.httpcomponents:httpclient:4.1.2')
>        }
> }
>
>
> What's the best way to create the build.log that you mentioned? Do you mean
> the console output? It won't be much to see. Or is there a parameter for a
> 'verbose' built?
>
> Thanks
>  George
>
> --
> View this message in context: http://grails.1312388.n4.nabble.com/Dependency-resolving-process-spammed-by-requests-tp4512162p4514798.html
> Sent from the Grails - user mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

--
Graeme Rocher
Grails Project Lead
SpringSource - A Division of VMware
http://www.springsource.com

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

On 28/03/2012, at 1:39 PM, Georg S. wrote:

> Hello,
>
> when I tried to figure out why the process of the "resolving dependencies"
> is so slow, I realized that there are at least 20 HTTP requests to
> repo1.maven.org/maven2/org/codehaus/groovy/groovy/1.xxxxx for "xxx" in
> different version from 1.7.5 to never ones, even not ordered.
>
> Both from the Maven repository maintainers' and from a developer's view, I
> think it's not really necessary and waisting time and net traffic resources.
>
> Could someone explain what's the real reason behind this? What kind of way
> would you suggest to optimize it?
> Or does it depend on the dependency structure of my project?

Host likely you are using HTTPBuilder which has a version range dependency.

You should exclude this dependency.
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[alxndrsn]

On 29 March 2012 12:53, [hidden email] <[hidden email]> wrote:
>
>> when I tried to figure out why the process of the "resolving dependencies"
>> is so slow, I realized that there are at least 20 HTTP requests to
>> repo1.maven.org/maven2/org/codehaus/groovy/groovy/1.xxxxx for "xxx" in
>> different version from 1.7.5 to never ones, even not ordered.

Is it possible to temporarily disable remote dependency resolution if
you know that you have everything in local Maven or Ivy caches?

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

On 29/03/2012, at 11:13 AM, Alex Anderson wrote:

> On 29 March 2012 12:53, [hidden email] <[hidden email]> wrote:
>>
>>> when I tried to figure out why the process of the "resolving dependencies"
>>> is so slow, I realized that there are at least 20 HTTP requests to
>>> repo1.maven.org/maven2/org/codehaus/groovy/groovy/1.xxxxx for "xxx" in
>>> different version from 1.7.5 to never ones, even not ordered.
>
> Is it possible to temporarily disable remote dependency resolution if
> you know that you have everything in local Maven or Ivy caches?

See the section on “offline mode” @ http://grails.org/doc/latest/guide/conf.html#dependencyRepositories
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[pledbrook]

> http-builder has a dependency range in it. The solution is to exclude
> the transitive groovy dependency from http-builder

https://jira.codehaus.org/browse/GMOD-268

--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Georg S.]

Dear all,

thanks for your replies.  
The dependency now looks like

// use version 0.5.2 and exclude groovy: http://jira.codehaus.org/browse/GMOD-162 and GMOD-268
compile('org.codehaus.groovy.modules.http-builder:http-builder:0.5.2') { //
                        excludes "xercesImpl", "groovy"
}


I had a hard time figuring out the right maven repositories - apart from the ones mentioned on HTTPBuilder's website (http://[snapshots].repository.codehaus.org), it seems that the transitive dependencies such as httpclient 4.1.3 aren't published to the public maven servers. The only one I found was repo1.maven.org.
If you've got recommendations for other maven repos, please let me know.

Any way, my initial question is answered. Thanks to all your input.

 George

[Wayne Fay]

> dependencies such as    httpclient 4.1.3 aren't published to the public maven
> servers. The only one I found was repo1.maven.org.

The *main* public Maven server *is* repo1.maven.org. This is
frequently called Maven Central.

> If you've got recommendations for other maven repos, please let me know.

Ideally you wouldn't need many (any?) repos declared other than
Central to get your build done. I see "adding lots of Maven repos" to
a build to be an anti-pattern and not a best practice.

Wayne

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[pledbrook]

> I had a hard time figuring out the right maven repositories - apart from the
> ones mentioned on HTTPBuilder's website
> (http://[snapshots].repository.codehaus.org), it seems that the transitive
> dependencies such as    httpclient 4.1.3 aren't published to the public maven
> servers.

HttpClient 4.1.3 is in Maven Central:

    http://search.maven.org/#artifactdetails%7Corg.apache.httpcomponents%7Chttpclient%7C4.1.3%7Cjar

You may have some other issue. What is your reason for thinking the
above JAR isn't in Maven Central? That may help us diagnose.

Peter

--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Georg S.]

Peter,

> You may have some other issue. What is your reason for thinking the
> above JAR isn't in Maven Central? That may help us diagnose.

thats exactly the right question.
I've just opened a new thread to clarify this:

http://grails.1312388.n4.nabble.com/mavenCentral-and-the-dependency-resolving-mechanism-td4519287.html

Thanks
 George