gitlab 502 error on New Merge Request

[cjohnweb]

Hi there,

I'm a long time PHP programmer, 6 month newb python developer, but entirely new to git and gitlab. I've spent 2 solid work days trying to solve this issue, I've chatted with others in chat rooms, all to no avail. I'm not certain where I should post this question, but I figure it's a software issue so I'll initially place it under Software Development.

I installed XenServer (A hypervisor for virtual machines), then Ubuntu Server 14.04 LTS on that, and then the most up-to-date Omnibus Gitlab package. I've assigned the machine 4 CPU's, 4GB ram, and 50GB inital HD space. I have 1 repo called powrz that is only some 10MB in size, with 4 or 5 branches for the developers (myself included).

I'm trying to merge a dev branch to the 0.0.0.5 release branch. I do this in the gitlab web interface. I select the 2 branches to merge. When I click the "Compare Branches" button, it hangs for about a minute as if it was loading a page, and then it loads a page that gives me this 502 error. Git still works, and everything appears to be running before and after this happens.

I have looked through all the config files I could find, I've changed all the timeouts to 300. I've added the proxy timeout options as well to nginx config. I've added worker processes for nginx and unicorn...and I think sidekiq.

The 503 error is odd, when the page loads it actually changes the font in my address bar..I notice it says utf-8 true as well.

The url is: http://192.168.1.51/powrz/powrz/merge_requests/new?utf8=%E2%9C%93&merge_request%5Bsource_project_id%5D=6&merge_request%5Bsource_branch%5D=jrs_dev&merge_request%5Btarget_project_id%5D=6&merge_request%5Btarget_branch%5D=0.0.0.5-dev

The error that url gives me is: 
"502

GitLab is not responding.

Please contact your GitLab administrator if this problem persists."

Other errors I've found in the logs:

sudo cat /var/log/gitlab/nginx/gitlab_error.log:

2015/03/03 12:05:11 [error] 9243#0: *46 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.90, server: 192.168.1.51, request: "GET /powrz/powrz/merge_requests/new?utf8=%E2%9C%93&merge_request%5Bsource_project_id%5D=6&merge_request%5Bsource_branch%5D=jrs_dev&merge_request%5Btarget_project_id%5D=6&merge_request%5Btarget_branch%5D=0.0.0.5-dev HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/powrz/powrz/merge_requests/new?utf8=%E2%9C%93&merge_request%5Bsource_project_id%5D=6&merge_request%5Bsource_branch%5D=jrs_dev&merge_request%5Btarget_project_id%5D=6&merge_request%5Btarget_branch%5D=0.0.0.5-dev", host: "192.168.1.51", referrer: "http://192.168.1.51/powrz/powrz/merge_requests/new"

$sudo gitlab-ctl restart gives me
ok: run: logrotate: (pid 13291) 1s
ok: run: nginx: (pid 13294) 0s
ok: run: postgresql: (pid 13307) 1s
ok: run: redis: (pid 13315) 0s
ok: run: sidekiq: (pid 13322) 0s
ok: run: unicorn: (pid 13327) 0s

From sudo cat /var/log/gitlab/unicorn/current:

2015-03-03_19:05:45.70501 starting new unicorn master
2015-03-03_19:05:58.65386 adopted new unicorn master 9283
2015-03-03_20:07:34.35261 forwarding TERM to unicorn master 9283
2015-03-03_20:07:34.35272 forwarding CONT to unicorn master 9283
2015-03-03_20:07:35.35746 wrapper for unicorn master 9283 exiting
2015-03-03_20:07:35.37872 starting new unicorn master
2015-03-03_20:07:48.47523 adopted new unicorn master 13342

With my limited knowledge and a little intuition, I suspect that nginx is trying to talk to a rails app - http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket - and I think that app is not replying to the request - or is taking too long to perform the merge request ultimately resulting in the error. But how to solve this...I don't know anything about rails and Google has yet to help me find an appropriate resource.

Any help would be appreciated! Thank you!!

[cjohnweb]

So, with the help of some kind guys at https://gitter.im/gitlabhq/gitlabhq ...I was able to solve this issue.

First, I ran:

sudo gitlab-ctl tail unicorn/unicorn_stderr.log

And while that was sitting there, I went into the interface and tried my merge request again. The unicorn/unicorn_stderr.log log reported that it was timing out (61s > 60 ...unicorn worker thread KILLED)

To fix this, I modified the unicorn conf file (sudo pico /etc/gitlab/gitlab.rb)

I changed this line (was commented out):

# unicorn['worker_timeout'] = 60

to this:

unicorn['worker_timeout'] = 300

Finally, I ran

sudo gitlab-ctl reconfigure

I tried my merge request again, and it went through! Being new to Gitlab, this was a challenge. There is a significant learning curve and Google doesn't return very good results...there is not very clear documentation on the specifics of these issues. Having a list which log files to look at, and which conf files are important, etc...and then knowning you have to run the reconfigure command...these things are hard to find.

Good luck to anyone else facing the same or similar issue.

Some of my notes...commands that helped me figure things out:

nginx config
    /opt/gitlab/embedded/conf/nginx.conf
unicorn config
    /opt/gitlab/embedded/service/gitlab-ci/config/unicorn.rb
nginx / gitlab error log:
    sudo tail -f /var/log/gitlab/nginx/gitlab_error.log
Edit gitlaf conf (unicorn settings too):
    sudo pico /etc/gitlab/gitlab.rb
Print out unicorn log:
    sudo cat /var/log/gitlab/unicorn/current
Reconfigure (run after making any changes to logfile)
    sudo gitlab-ctl reconfigure

Restart all gitlab services:
    sudo gitlab-ctl restart

Unicorn Error Log:
    sudo gitlab-ctl tail unicorn/unicorn_stderr.log

Other Error Logs:
    sudo gitlab-ctl tail gitlab-rails/production.log
    sudo gitlab-ctl tail gitlab-rails/githost.log
    sudo gitlab-ctl tail gitlab-rails/satellites.log

[Jorge_10]

Thank you, exactly what fixed my 502 issues :)