I started a while back with setting up Gitlab Pages for replacing my personal website and setting up some doc pages. Setting everything up and getting this website to work, especially on custom domain names, was quite the adventure.
DNS
Since I’ve previously pointed the DNS for gitlab.0x43.nl and *.gitlab.0x43.nl to the gitlab server, I could simply enable pages to work on a subdomain of that.
In My case pages.gitlab.0x43.nl.
Config
The config was basically nothing more than adding/editing /etc/gitlab/gitlab.rb and adding
pages_external_url 'http://pages.gitlab.0x43.nl'
And updating Gitlab via gitlab-ctl reconfigure.
I’ve created this site project from template and pushed it to Gitlab with devops enabled. Somehow the first pipeline failed due to some ssl bitching. I updated the theme, by replacing it with a git submodule but the pipeline started bitching the following:
Cloning into '/builds/pat.vdleer/website/themes/beautifulhugo'...
fatal: unable to access 'https://github.com/halogenica/beautifulhugo.git/': SSL certificate problem: unable to get local issuer certificate
fatal: clone of 'https://github.com/halogenica/beautifulhugo.git' into submodule path '/builds/pat.vdleer/website/themes/beautifulhugo' failed
I was not in the mood so I “fixed” it quick and dirty by replacing the submodule with the actual files and committing that. One of those things for on the (never-gonna) todo-list.
The site was now up and running on http://pat.vdleer.pages.gitlab.0x43.nl/website, not a very pretty url so the next thing to do was custom DNS names
Custom domains
I dove in and figured out I needed a second IP to run the pages on since you (currently) can’t run pages and gitlab on a single IP. As far as I can tell, Pages doesn’t use nginx when you want to run it with custom domains. Since it doesn’t support using nginx as a catch all reverse proxy and pipe it to pages I figured I would go for the easier solution, a second IP.
IP config
I added (bought) a second one and added it to my VPS but didn’t know how to configure this. I figured it would add a second interface and I would have to bring it up. Checking my uptime, 1065 days, I couldn’t bring myself to rebooting and trying it out so I asked a friend to help out.
He configured it as an IP alias in /etc/network/interfaces.
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto ens3
iface ens3 inet dhcp
# This is an autoconfigured IPv6 interface
iface ens3 inet6 auto
# The secondary network interface (aliased)
auto ens3:0
iface ens3:0 inet static
address 37.97.203.76
netmask 255.255.255.0
broadcast 37.97.203.255
network 37.97.203.0
# gateway 37.97.203.1
After restarting networking via /etc/init.d/networking restart it worked!
Gitlab config
So back to /etc/gitlab/gitlab.rb and configure the pages.
external_url 'https://gitlab.0x43.nl'
registry_external_url "https://registry.gitlab.0x43.nl"
pages_external_url 'http://pages.gitlab.0x43.nl'
nginx['listen_addresses'] = ['37.97.206.228']
pages_nginx['enable'] = false
gitlab_pages['access_control'] = true
gitlab_pages['enable'] = true
gitlab_pages['external_http'] = ['37.97.203.76:80']
gitlab_pages['external_https'] = ['37.97.203.76:443']
# gitlab_pages['redirect_http'] = true
# gitlab_pages['use_http2'] = true
# gitlab_pages['inplace_chroot'] = true
It took quite some time but this is basically what I landed on.
Running gitlab-ctl reconfigure, everything was up, or so it seemed.
Pages was now redirecting to registry, which redirected to gitlab itself…
After some googling I found a bug report about the registry not listening to nginx['listen_addresses'] = ['37.97.206.228'].
Adding registry_nginx['listen_addresses'] = ['37.97.206.228'] should fix this.
But… still the same… so after a lot of screwing around I nmaped the IP of pages and found out that the 80 and 443 were closed.
At this time I started looking at the pages log Using gitlab-ctl tail gitlab-pages it showed me the following;
{
"error":"open /etc/gitlab/ssl/pages.gitlab.0x43.nl.crt: no such file or directory",
"level":"fatal",
"msg":"",
"time":"2019-07-10T22:29:40+02:00",
}
Apparently the service wasn’t starting due to a missing ssl certificate.
SSL config
So running certbot to the resque in providing me with a cert.
Running certbot I came across another issue, you can’t bind certbot to a specific IP, it binds on 0.0.0.0 and since gitlab is running it can’t do that.
$ certbot certonly --standalone --preferred-challenges http -d pages.gitlab.0x43.nl
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for registry.gitlab.0x43.nl
Cleaning up challenges
Problem binding to port 80: Could not bind to IPv4 or IPv6.
Stopping gitlab fixed this issue, but still a bit strange that there isn’t an option for this.
So once that was fixed I simply symlinked the cert and check the logs.
ln -s /etc/letsencrypt/live/pages.gitlab.0x43.nl/fullchain.pem /etc/gitlab/ssl/pages.gitlab.0x43.nl.crt
I knew there should be a key file put somewhere so I checked the logs, found the name and fixed that as well.
{
"error":"open /etc/gitlab/ssl/pages.gitlab.0x43.nl.key: no such file or directory",
"level":"fatal",
"msg":"",
"time":"2019-07-10T22:30:24+02:00",
}
ln -s /etc/letsencrypt/live/pages.gitlab.0x43.nl/privkey.pem /etc/gitlab/ssl/pages.gitlab.0x43.nl.key
But… it still kept redirecting… this was seriously driving me mad…
It took me way way too long but eventually I noticed it. Remember in the beginning when I set up the DNS to gitlab? Well… since I switched IP’s I forgot to update the pages entry to my new second IP.
Since it takes a while to update the DNS I tested it with curl.
$ curl -v --resolve 'pages.gitlab.0x43.nl:80:37.97.203.76' http://pages.gitlab.0x43.nl
* Added pages.gitlab.0x43.nl:80:37.97.203.76 to DNS cache
* Rebuilt URL to: http://pages.gitlab.0x43.nl/
* Hostname pages.gitlab.0x43.nl was found in DNS cache
* Trying 37.97.203.76...
* TCP_NODELAY set
* Connected to pages.gitlab.0x43.nl (37.97.203.76) port 80 (#0)
> GET / HTTP/1.1
> Host: pages.gitlab.0x43.nl
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Content-Type: text/html; charset=utf-8
< Location: http://projects.pages.gitlab.0x43.nl/auth?domain=http://pages.gitlab.0x43.nl&state=
FINALLY, or so I thought, this too redirect me to projects.pages which in its turn redirect me gitlab itself back again… Checking my custom domain for this project/website/pages gave me the following.
$ curl -v --resolve 'patvdleer.nl:80:37.97.203.76' http://patvdleer.nl
* Added pages.gitlab.0x43.nl:80:37.97.203.76 to DNS cache
* Trying 37.97.206.228...
* TCP_NODELAY set
* Connected to pat.vdleer.pages.gitlab.0x43.nl (37.97.206.228) port 80 (#0)
> GET /website HTTP/1.1
> Host: pat.vdleer.pages.gitlab.0x43.nl
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Wed, 10 Jul 2019 22:03:19 GMT
< Content-Type: text/html
< Content-Length: 178
< Connection: keep-alive
< Location: https://gitlab.0x43.nl:443/website
In the morning I tried again and my DNS name, patvdleer.nl just worked but as soon as I pushed a new blog post, the beginning of this one, trouble hit again. The runners were not running so the project could not go through the pipeline. Checking the config everything seemed fine so I opted for adding a new one which did fail…
ERROR: Registering runner... failed runner=CuftyRsQ status=couldn't execute POST against https://gitlab.0x43.nl/api/v4/runners: Post https://gitlab.0x43.nl/api/v4/runners: dial tcp 127.0.1.1:443: getsockopt: connection refused
PANIC: Failed to register this runner. Perhaps you are having network problems
Since nginx wasn’t running on 0.0.0.0 anymore the binding was not set on the loopback. Luckily this was quickly fixed by checking /etc/hosts which gave me:
127.0.0.1 localhost
127.0.1.1 gitlab.0x43.nl gitlab
Removing the gitlab.0x43.nl from the file fixed it, but now I wonder why I added that…
I don’t think it was done automagically, something in the back of my mind tells me it has something todo with mail but that seems to work…
For now I’m just happy it finally all works.