Upsource Configuration Error - "Gradle executable not found"

I am trying to configure upsource for our git repo. We have multiple project repositories hence I have configured them accordingly in the administration page. We use gradle as our build tool.
Even when I have installed gradle on my linux box I get an error "Gradle executable not found" on the admin page. Below are some of the logs from the /upsource-analyzer/debug.log file

[2015-07-06 05:34:13,539] INFO  yzer:INDEX_CLUSTER-2        mysoftware onversion.gradle.GradleSupport - Generating project model for revision RowId{mysoftware, modules-936e9e8d17d3c94ce4d5da2c2f6c086fec902895}
[2015-07-06 05:34:13,539] DEBUG yzer:INDEX_CLUSTER-2        mysoftware onversion.gradle.GradleSupport - gradlew not found; trying to locate gradle installation...
[2015-07-06 05:34:13,539] DEBUG yzer:INDEX_CLUSTER-2        mysoftware onversion.gradle.GradleSupport - gradle installation not found
[2015-07-06 05:34:13,539] WARN  yzer:INDEX_CLUSTER-2        mysoftware onversion.gradle.GradleSupport - Gradle executable not found
[2015-07-06 05:34:13,540] INFO  yzer:INDEX_CLUSTER-2        mysoftware eImpl$DriverWrapperWithLogging - Interrupted: 'gradle support' OLD MAIN-{ revision: RevisionId{mysoftware, modules-936e9e8d17d3c94ce4d5da2c2f6c086fec902895} @ ~ 2015 Jul 3 09:21:51}-cluster=INDEX_CLUSTER
[2015-07-06 05:34:13,540] DEBUG yzer:INDEX_CLUSTER-2        mysoftware eImpl$DriverWrapperWithLogging - Started: 'gradle support' OLD MAIN-{ revision: RevisionId{mysoftware, modules-6caef1c8301c448c0fc14cc5430e5dfea000a3af} @ ~ 2015 Jul 3 08:55:27}-cluster=INDEX_CLUSTER
[2015-07-06 05:34:13,587] INFO  yzer:INDEX_CLUSTER-2        mysoftware onversion.gradle.GradleSupport - Patching enabled. Looking for cached snapshot from older revisions...
[2015-07-06 05:34:13,587] INFO  yzer:INDEX_CLUSTER-2        mysoftware onversion.gradle.GradleSupport - Unable to read tree info from /opt/crw/upsource/Upsource/temp/upsource-analyzer/upsource-tmp/projects/mysoftware/gradle-support/tree-info: file not found
[2015-07-06 05:34:13,587] INFO  yzer:INDEX_CLUSTER-2        mysoftware onversion.gradle.GradleSupport - Patching cannot be performed. Making full checkout.
[2015-07-06 05:34:13,718] DEBUG d; Change Observer 1        mysoftware ins.buildServer.ChangeObserver - Finished detecting changes in 0ms for File watcher for "/opt/crw/upsource/Upsource/lib/upsource/plugins/vcs-worker-impl/conf/vcs-worker.lock"


What could be missing?

Thanks,
Amit.
29 comments

Hi Amit,

Thanks for getting in touch.

Could you please clarify what Upsource version are you using? Is it android studio project?

0

I am using Upsource version - 2.0.3462.
It's not an android studio project. Its an java ee based enterprise web application. We have different git code repositories for maintaining our application code and the build code. I have configured them accordingly in the administration tab.

0

Amit,

Thanks for clarifing.

The following steps should help you to resolve the issue:

  • In the
    <upsource_home>/conf
    directory, create a file
    upsource.user.properties
  • Add the following line to that file:
    gradle.home.directory=<gradle_home>
    . Use double backslashes in the specified path (e.g.
    C:\\Users\\Jet\\Gradle
    )
  • Restart Upsource

  • Please let me know if it helps.

    0

    Thanks that works. It seems upsource can now execute gradle commands.

    I still cannot have the code navigation features like find usages etc. What could be missing?

    0

    Perhaps project is not indexed yet? If you hover on the project icon on administration page, what does it show for you?

    Does gradle.out file contain any warnings or errors? If you go to project page within Upsource UI -> "Brwose code" and navigate to /.idea/gradle/ folder in the files tree, you should find gradle.out file there.

    0

    Below is what I see when I hover over the project icon
    http://i.imgur.com/3p9yBYD.png

    Also I checked the gradle.out file but there are no warnings or errors.

    0

    I get this error message on the administration page "Project wasn't loaded: Project model not found. Code navigation and inspections will not be available."

    Looking into the logs I see below statements

    [2015-07-07 11:08:30,882] WARN  yzer:INDEX_CLUSTER-2        optymyze onversion.gradle.GradleSupport - Gradle project not found
    [2015-07-07 11:08:30,885] WARN  yzer:INDEX_CLUSTER-2        optymyze .ExternalLibrariesImportDriver - cannot find IDEA project directory
    [2015-07-07 11:08:30,887] INFO  yzer:INDEX_CLUSTER-2        optymyze eImpl$DriverWrapperWithLogging - Finished: 'Importing external libraries' OLD MAIN-{ revision: RevisionId{optymyze, optymyze-8a1dbaeef289b2060eed0fb8b610722d36c07371} @ ~ 2014 Jan 29 03:14:43}-cluster=INDEX_CLUSTER in 2ms
    [2015-07-07 11:08:30,933] INFO  yzer:INDEX_CLUSTER-2        optymyze ndexing.ProjectRevisionIndexer - Project wasn't loaded: Project model not found. Code navigation and inspections will not be available.
    [2015-07-07 11:08:30,973] INFO  yzer:INDEX_CLUSTER-2        optymyze eImpl$DriverWrapperWithLogging - Finished: 'index' OLD MAIN-{ revision: RevisionId{optymyze, optymyze-8a1dbaeef289b2060eed0fb8b610722d36c07371} @ ~ 2014 Jan 29 03:14:43}-cluster=INDEX_CLUSTER in 86ms
    [2015-07-07 11:08:30,975] INFO  yzer:INDEX_CLUSTER-2        optymyze eImpl$DriverWrapperWithLogging - Finished: 'project stats analyzer' OLD MAIN-{ revision: RevisionId{optymyze, optymyze-8a1dbaeef289b2060eed0fb8b610722d36c07371} @ ~ 2014 Jan 29 03:14:43}-cluster=INDEX_CLUSTER in 2ms

    What could be the reason why it is search for the IDEA project directory when I have configured the build system as "Gradle". My project configuration is as below. We have multiple code repositories - build-core & optymyze-modules. build-core is a standard gradle multiproject and requires a "settings.gradle" which is the entry point. The gradle files assumes optymyze code is in a folder called "modules"

    http://i.imgur.com/7nLciAV.png
    Below is a screenshot of how I see the code browser.

    http://i.imgur.com/GXqACoD.png

    What could be wrong in the config?

    0

    Hi Amit,

    Actually configuration looks good, path to project model is specified correctly and from the last screenshot .idea folder got created. All references say that code navigation should be available..

    Want to note that Upsource doesn't reindex old revisions, so code intelligence feature should be available only for those revisions which appeared after you created

    upsource.user.properties 
    file. The error in the log" Gradle project not found" and error on project administration page might refer to one of the oldest revisions.. It's one of the UI problem we are working to improve in future releases.

    Could you please navigate to the same revision where this screenshot was made (Image:http://i.imgur.com/GXqACoD.png)  open any source file there and see if code navigation is available.
    0

    Thanks for the reply.
    I still have an error on the administration page. Are you sure that the configuration is good? (Image:http://i.imgur.com/4q0NcMG.png)

    I just navigated to a side-by-side diff of a code commit (~ 10 mins back from now) and tried navigating to the code but I do not get any options on clicking the source code (Image: http://i.imgur.com/tLe50SA.png)

    btw, Is there a way to delete old data and start fresh to make sure that upsource indexes all revisions of the source code?

    0

    Yes, from the screenshots configuration looks good:

    - build.gradle is under optymyze-java folder
    - path to project model specified correctly
    - even .idea folder was generated (Note: the IDEA engine used in the Upsource doesn't understand natively project models other than IDEA. When you select maven or gradle model for you project, Upsource first converts it to the IDEA format, i.e. produces IDEA native files .idea/.and *.iml, which then are used for code intelligence. All of these idea files get loaded to Upsource DB (within a particular revision).)

    Could you please restart Upsource frontend service in order to eliminate possible cause from that part? This command should help to do that <Upsource_home_dir>/apps/upsource-frontend/bin/upsource-frontend.sh restart

    If it doesn't help, please post the logs folder from Upsource directory and gradle.out file that can be found under /.idea/gradle folder

    Thanks in advance.

    0

    P.S. There is no way to initiate reindex in the current version of the product. The only way is to delete project and recreate it.

    "Reindex feature" is already designed and will be introduced in one of the coming releases.

    0

    I tried restarting the upsource-frontend service but it didn't help. I have send the logs and the gradle.out file to upsource-support@jetbrains.com for security reasons. Below is the error details from info.log. The below text also appears on the administration page.

    [2015-07-13 06:39:14,923] INFO  zer:GLOBAL_CLUSTER-2        optymyze mportProjectFromVcsCommitsImpl - Imported new commits for project optymyze (0 revisions) in 5s 525ms
    [2015-07-13 06:39:15,199] WARN  zer:GLOBAL_CLUSTER-2        optymyze i.multi.vcs.VcsAccessorFactory - Failed to complete Import Commit for ProjectId{optymyze}. jetbrains.vcs.server.api.VcsServiceException: 2 of 2 failed. Failed to complete an action for optymyze:optymyze. Failed to collect current revision for optymyze:optymyze. List remote refs failed: java.net.SocketException: Connection reset, Failed to complete an action for optymyze:modules. Failed to collect current revision for optymyze:modules. List remote refs failed: java.net.SocketException: Connection reset
    java.lang.Error: jetbrains.vcs.server.api.VcsServiceException: 2 of 2 failed. Failed to complete an action for optymyze:optymyze. Failed to collect current revision for optymyze:optymyze. List remote refs failed: java.net.SocketException: Connection reset, Failed to complete an action for optymyze:modules. Failed to collect current revision for optymyze:modules. List remote refs failed: java.net.SocketException: Connection reset
         at com.jetbrains.upsource.backend.cli.vcs.service.VcsServiceClient.processNewRevisions(VcsServiceClient.java:67)
         at com.jetbrains.upsource.backend.cli.multi.vcs.ImportProjectFromVcsCommitsImpl.checkForNewCommits(ImportProjectFromVcsCommitsImpl.java:145)
         at com.jetbrains.upsource.backend.cli.multi.vcs.ImportProjectFromVcsCommitsImpl.checkForNewCommitsAndLog(ImportProjectFromVcsCommitsImpl.java:118)
         at com.jetbrains.upsource.backend.cli.multi.vcs.ImportProjectFromVcsCommitsImpl.executeTask(ImportProjectFromVcsCommitsImpl.java:86)
         at com.jetbrains.upsource.backend.cli.multi.vcs.VcsAccessorFactory$VcsCommitCollectorImpl$2.execute(VcsAccessorFactory.java:100)
         at com.jetbrains.upsource.backend.cli.multi.vcs.VcsAccessorFactory$VcsCommitCollectorImpl$2.execute(VcsAccessorFactory.java:95)
         at com.jetbrains.upsource.lifetimes.LifetimeImpl.runSync(LifetimeImpl.java:75)
         at com.jetbrains.upsource.backend.cli.multi.vcs.VcsAccessorFactory$VcsCommitCollectorImpl.collectChanges(VcsAccessorFactory.java:95)
         at com.jetbrains.upsource.backend.cli.multi.global.CollectChangesTask.doCollectChanges(CollectChangesTask.java:78)
         at com.jetbrains.upsource.backend.cli.multi.global.CollectChangesTask.analyze(CollectChangesTask.java:50)
         at com.jetbrains.upsource.backend.cli.stages.driver.GlobalDriverWrapper.analyze(GlobalDriverWrapper.java:43)
         at com.jetbrains.upsource.backend.cli.multi.global.GlobalPipeline$1.run(GlobalPipeline.java:60)
         at com.jetbrains.upsource.backend.cli.multi.tasks.MultiProjectTaskFactory$TaskExecutor$2.run(MultiProjectTaskFactory.java:175)
         at __.project_optymyze.__(JavaGeneratorTemplate.java:44)
         at org.jonnyzzz.stack.NamedStackFrame.frame(NamedStackFrame.java:48)
         at com.jetbrains.upsource.backend.cli.multi.tasks.MultiProjectTaskFactory$TaskExecutor.executeTask(MultiProjectTaskFactory.java:172)
         at com.jetbrains.upsource.backend.cli.multi.tasks.MultiProjectTaskFactory$TaskExecutor$1.run(MultiProjectTaskFactory.java:153)
         at com.jetbrains.upsource.backend.cli.multi.executor.ProjectSyncExecutor$ProjectTasks$1$1.run(ProjectSyncExecutor.java:224)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)
    Caused by: jetbrains.vcs.server.api.VcsServiceException: 2 of 2 failed. Failed to complete an action for optymyze:optymyze. Failed to collect current revision for optymyze:optymyze. List remote refs failed: java.net.SocketException: Connection reset, Failed to complete an action for optymyze:modules. Failed to collect current revision for optymyze:modules. List remote refs failed: java.net.SocketException: Connection reset
         at jetbrains.vcs.server.core.impl.threading.TrueParallelExecutionImpl.forEachInParallel(TrueParallelExecutionImpl.java:116)
         at jetbrains.vcs.server.core.impl.threading.ParallelExecutors$GlobalParallelExecution.forEachInParallel(ParallelExecutors.java:46)
         at jetbrains.vcs.server.core.impl.snapshot.SnapshotOfNowServiceImpl.fullSnapshotOfNow(SnapshotOfNowServiceImpl.java:43)
         at jetbrains.vcs.server.core.impl.graph.GraphServiceImpl.fetchGraph(GraphServiceImpl.java:52)
         at jetbrains.vcs.server.core.impl.graph.FetchServiceImpl.fetch(FetchServiceImpl.java:22)
         at com.jetbrains.upsource.backend.cli.vcs.service.VcsServiceClient.processNewRevisions(VcsServiceClient.java:61)
         ... 20 more
    [2015-07-13 06:39:21,610] INFO  yzer:INDEX_CLUSTER-1        optymyze onversion.gradle.GradleSupport - gradle finished successfully
    [2015-07-13 06:39:24,656] INFO  yzer:INDEX_CLUSTER-1        optymyze eImpl$DriverWrapperWithLogging - Finished: 'gradle support' OLD MAIN-{ revision: RevisionId{optymyze, modules-9637baf46f51930f86c8e10cfce5460e77650d20} @ ~ 2015 Jul 10 10:35:20}-cluster=INDEX_CLUSTER in 12s

    What could be going wrong?

    0

    Amit,

    Please take a look into 3rd line from the trace you've pasted:

    java.lang.Error: jetbrains.vcs.server.api.VcsServiceException: 2 of 2 failed. Failed to complete an action for optymyze:optymyze. Failed to collect current revision for optymyze:optymyze. List remote refs failed: java.net.SocketException: Connection reset, Failed to complete an action for optymyze:modules. Failed to collect current revision for optymyze:modules. List remote refs failed: java.net.SocketException: Connection reset
     

    It says connection reset. Could you please check if your git repository is accessible from the Upsource machine?

    Thanks.

    0

    I already verified that the connection works through "Test Connection". (Image: http://pasteboard.co/1WAkkkec.png). I have erased some details from the image for security purposes

    0

    Amit,

    Ok, thanks for clarifying. What git server are you using? There should be some logs in regards to connection reset. Could you please check records around 2015-07-13 07:48:23 Are there any?

    Thanks

    0

    Hi Artem Rokhin,

    We use git-daemon. I need to check the logs but I see that upsource is able to pull code from the configured repo periodically in the background. We are able to do code reviews, navigate to a file from the search box etc. The repository navigation features like finding usages, view hiearchy (referring to - https://www.jetbrains.com/upsource/features/java.html#navigation) are not working. Could the error and this be un-related?

    If the error meant that connecting to the git server is failing, I assume it wouldn't have been able to pull code but that's working.

    0

    Hi Amit,

    There are no other possibly related errors in the logs you sent us on Monday. Project model is converted successfully and all other things looks good.. So for now it's not really obvious on what is going on, unfortunately.

    Could you please clean logs directory, restart Upsource, give it an hour to run and send us logs folder once mote time?

    Thanks for understanding.

    0

    Hi Artem,
    I have sent the logs through mail to the support mail id after following the steps you mentioned in the previous reply. Let me know if you find the root cause.

    0

    Hi Amit,

    Thanks for sending it.

    If you open any java file, is magic wand icon enabled for it? (see screenshot)

    magic_wand.PNG

    Thanks for clarifying.

    0

    Thanks for info.

    Looks like project model doesn't recognize your files and it's not clear why..

    Could you please make a screenshot with full path to this file (Log....) in Upsource UI?

    Also in .idea folder there should be several files with iml extension, there are module based. Could you please post the iml file related to the module where this file (log...) is located?

    0

    Hi Artem,

    Here's the screenshot (Image : http://i.imgur.com/zY2NMnB.png). I have mailed the iml file to the support mail id.

    Regards,
    Amit.

    0

    Amit,

    Thanks, we got it.

    If on browse code page in Upsource could you please navigate to  */modules/users/* copy file called users.iml and send to us, so we can check that content is correct?

    Thanks.

    0

    Sent the file to support mail id.

    0

    Amit,

    Iml file that you sent us doesn’t contain path to source files, typically under <content> block in iml file there should be urls to source folders. Please take a look into example of correct iml file:

    ...

    <content url="file://$MODULE_DIR$">

          <sourceFolder url="file://$MODULE_DIR$/src"/>

         <sourceFolder url="file://$MODULE_DIR$/tests"/>

         <sourceFolder url="file://$MODULE_DIR$/resources"/>

    </content>

    ...

    While your iml doesn’t have this URLs, Upsource doesn't aware that source files exist. Your iml has only excludes URLs:

    <content url="file://$MODULE_DIR$/">

          <excludeFolder url="file://$MODULE_DIR$/build"/>

          <excludeFolder url="file://$MODULE_DIR$/.gradle"/>

        </content>

    As mentioned in one of the posts above, we don’t work directly with Gradle model, but first converts gradle model to IDEA one whichis native to Upsource. This conversion is made by gradle itself, so you might simply run the same process via cli, please refer to the corresponding doc page https://docs.gradle.org/current/userguide/idea_plugin.html, also you might find exact command which Upsource does use to run the conversion at the beginning of gradle.out file. E.g. /opt/gradle/gradle-2.4/bin/gradle, -g, /opt/crw/upsource/Upsource/temp/upsource-analyzer/upsource-tmp/gradle-home, -d, --stacktrace, --init-script, /opt/crw/upsource/Upsource/temp/upsource-analyzer/_upsource_init_script9021095754170698374.gradle, --no-color, --no-daemon, --continue, -Dcom.jetbrains.upsource.modelConversion=true, idea

    Does such launching though cli gives you the same iml files with no URLs to source files? If so, it most likely a thing for investigation on Gradle side. We are not aware of your model configuration so not really sure what exact recommendation might help in that case. Could you please see gradle files itself and check if everything is correct?

    Also one of the possible reasons is a Gradle version. At the beginning of this thread you were manually specifying path gradle wrapper. Did you install the latest gradle version and/or is this version capable with your actual gradle config?

    Thanks

    0

    Thanks for the detailed analysis Artem.

    Going through your reply I have a few questions

    1. Is this users.iml created by upsource? I am a bit confused since the "users" module is from our code base
    2. I tried executing the command you pointed out from the gradle.out file but it needs to be executed from a specific directory it is failing with "Task ',' not found in root project 'Upsource'." From which directory should this command be executed?
    3. About gradle files & configuration, I will try to elaborate a bit on the related config (Image: http://i.imgur.com/9zSPGeN.png). The first git repo is the build code and the "Path to project model" is relative to it. The optymyze-java folder contains 4 gradle files. I assume upsource tries to execute the build.gradle file first out of the rest but it seems a different gradle file named - settings.gradle is the starting point. Could upsource be configured for executing settings.gradle as the starting point?
    4. The gradle version is 2.4. Other related config details are (Image: http://i.imgur.com/R3SFZvL.png). Let me know if you find other software versions problematic.
    0

    Hi Amit,

    Is this users.iml created by upsource? I am a bit confused since the "users" module is from our code base

    Yes

    I tried executing the command you pointed out from the gradle.out file but it needs to be executed from a specific directory it is failing with "Task ',' not found in root project 'Upsource'." From which directory should this command be executed?

    Please strip those commas, i.e. the command should look like '/opt/gradle/gradle-2.4/bin/gradle -g /opt/crw/upsource/Upsource/temp/upsource-analyzer/upsource-tmp/gradle-home ...'

    About gradle files & configuration, I will try to elaborate a bit on the related config (Image: http://i.imgur.com/9zSPGeN.png). The first git repo is the build code and the "Path to project model" is relative to it. The optymyze-java folder contains 4 gradle files. I assume upsource tries to execute the build.gradle file first out of the rest but it seems a different gradle file named - settings.gradle is the starting point. Could upsource be configured for executing settings.gradle as the starting point?

    I'm afraid I don't quite get the question - you say that there are four files with *.gradle extension in the 'project model dir' (/optymyze-java), one of them is named settings.gradle and you want to configure upsource to use that file as an entry point, right?

    The gradle version is 2.4. Other related config details are (Image: http://i.imgur.com/R3SFZvL.png). Let me know if you find other software versions problematic.

    Looks ok to me

    Regards, Denis

    0
    1. I executed the gradle command by stripping the ,. It failed for a different reason now. "02:04:25.581 [ERROR] [org.gradle.BuildExceptionReporter] Could not read initialization script '/opt/crw/upsource/Upsource/temp/upsource-analyzer/_upsource_init_script9021095754170698374.gradle' as it does not exist"/ Complete logs - http://pastebin.com/WE6ab4aY
    2. I'm afraid I don't quite get the question - you say that there are four files with *.gradle extension in the 'project model dir' (/optymyze-java), one of them is named settings.gradle and you want to configure upsource to use that file as an entry point, right?


                   Yes.

    0

    I executed the gradle command by stripping the ,. It failed for a different reason now. "02:04:25.581 [ERROR] [org.gradle.BuildExceptionReporter] Could not read initialization script '/opt/crw/upsource/Upsource/temp/upsource-analyzer/_upsource_init_script9021095754170698374.gradle' as it does not exist"/ Complete logs - http://pastebin.com/WE6ab4aY

    Please create a file name 'my.gradle' with the content below and use its path as an init-script argument:

    initscript {
      allprojects {
        apply plugin: 'idea'
      
    }
    }

    I'm afraid I don't quite get the question - you say that there are four files with *.gradle extension in the 'project model dir' (/optymyze-java), one of them is named settings.gradle and you want to configure upsource to use that file as an entry point, right?


                   Yes.

    You can't do that. Moreover, it's not clear why would you want to do that as settings.gradle is an auxiliary file used primarily for multi-project configurations. It's discovered and applied by gradle automatically during working with root project's build.gradle file (check more details here)

    Regards, Denis

    0

    Please sign in to leave a comment.