Why this article?
For Federation of OpenShift/Kubernetes clusters we want not only to demo some applications, but build a solution that covers all that will be needed in a real-world deployment.
Colleagues in the Solutions Engineering team have been working on demonstrating an application running on different clusters over a Federated Control Plane, allowing it to ‘roam’ between clusters with shared data access as a way to demonstrate how an application can ‘scale’ from on-premise deployment to external clouds in order to satisfy business requirements for peak demands.
This article tries to demonstrate on how to setup Quay, as a container registry for your applications, in a way that can work both for this Federated approach as well as High-Availability setup.
What is Quay?
The one liner would be:
Quay is a container registry.
The extended one, would be more in the lines of:
Quay is a container registry that can be used via the provided service at https://quay.io or can be used via Quay Enterprise as ‘on premise’ registry.
A container registry provides a point for containers to be downloaded or uploaded for later consumption.
Quay allows to be configured either to use the provided service via https://quay.io or deploy it on premise, so that you gain additional control over the setup like which namespaces to replicate, policies, etc.
clair analysis of images, allowing to detect known vulnerabilities from different sources to later check the images.
Also, allows to automate builds when a push to a repository has happened, allowing to quickly spawn an update container image when our tracked repository has a new commit that can be later updated in our OpenShift/Kubernetes image stream.
Quay can also, when having several storage backends defined, do geo-replication, that allows OpenShift to grab the needed bytes from the closest backend while handling in the background the copy of new data to the other replicas.
What is Swift?
Swift is object storage provided by OpenStack Environment, defined at https://wiki.openstack.org/wiki/Swift as:
The OpenStack Object Store project, known as Swift, offers cloud storage software so that you can store and retrieve lots of data with a simple API. It’s built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound.
Swift allows to define a
container (not in the https://www.opencontainers.org/ sense but in a ‘data storage’).
How does it come to play with Quay?
Quay allows to define storage backend based on swift that can be used together with other backends to provide geo-replication for availability.
The Setup process of Swift for Quay
At the beginning it was not easy to find out all the required configuration variables or settings to define, but what finally worked for the setup described in this article was:
Create the Swift container
First we do need to create a swift container in OpenStack that will hold the information from Quay:
Create swift container for holding the storage:
(overcloud) [[email protected] ~]$ openstack container create quay-enterprise
Configuration settings defined in Quay
|OpenStack Storage (Swift)|
|OS OPTIONS drop-down||<—– This is an important one if not it will fail|
Once done, Quay should ‘validate’ the upload and
swift list $YOURCONTAINER should give something like:
(overcloud) [[email protected] ~]$ swift list quay-enterprise quay/_verify
Showing that the verification file has been created succesfully.
Quay allows to define several backends that can be also configured as a foundation for GeoReplication.
Georeplication in quay, is used to:
- define a ‘zone name’ for a storage backend
- when booting Quay containers or apps, define which backend is ‘closest’ (via environment variable
Quay will store and replicate on the different backends defined the information uploaded. The requirement is that the backend is an object storage, but you can also define ‘mixed’ backends like for example GCE, AWS, Swift, Ceph RadosGW, etc. Quay will take care of the copy and once the backend ‘closest’ to the quay instance is ready, it will serve from that one (or fallback to the one having the actual data).
For each one of the configured backends, a different number of settings will be requested to do the proper configuration.
Google Cloud Storage
For setting Google Cloud Storage, we did use the following values:
|Storage Engine||Google Cloud Storage|
|Cloud Access Key||GOOGXXXXXXXX|
|Cloud Secret Key||pksXXXXXX|
For setting Amazon S3 storage, we did use the following values:
|Storage Engine||Amazon S3|
|AWS Access Key||AKIAXXXXXXXX|
|AWS Secret Key||/9bkfXXXXXXXXXX|
Validation of the replication
Once all the defined storage backends have been configured, there’s a button in the interface that will validate access and create the ‘marker’ files on each storage backend to verify the access permissions to them similar to what was shown on Swift configuration.
Quay SSL deployment
This procedure will guide you through the configuration of Quay with SSL certificates so the deployment is both secure and deployed in a highly available enterprise way.
Setting up certificates requires several steps involving:
- Creating a CA
- Create a Certificate Signing Request with all the possible quay hostnames
- Sign the certificates
- Upload the certificates on Quay
Execution of generate-ca.sh
2018/11/29 15:01:13 [INFO] generating a new CA key and certificate from CSR 2018/11/29 15:01:13 [INFO] generate received request 2018/11/29 15:01:13 [INFO] received CSR 2018/11/29 15:01:13 [INFO] generating key: rsa-2048 2018/11/29 15:01:13 [INFO] encoded CSR 2018/11/29 15:01:13 [INFO] signed certificate with serial number 432855461933024796113382064222588753737107796958
-rw-rw-r--. 1 iranzo iranzo 232 nov 29 15:01 ca-config.json -rw-r--r--. 1 iranzo iranzo 1001 nov 29 15:01 ca.csr -rw-rw-r--. 1 iranzo iranzo 208 nov 29 15:01 ca-csr.json -rw-------. 1 iranzo iranzo 1679 nov 29 15:01 ca-key.pem -rw-rw-r--. 1 iranzo iranzo 1359 nov 29 15:01 ca.pem
Create Quay Certs
We are going to use the same certificate for each replica, we need to know the hostname that will be used on the OCP routes beforehand in order to use it for the SAN property in the certificate.
Having each of the routes in the SAN makes it possible to use the same cert for each replica, so we can federate a single secret containing those certs rather than creating one SSL secret for each cluster.
Execution of generate-cert.sh
2018/11/29 15:01:26 [INFO] generate received request 2018/11/29 15:01:26 [INFO] received CSR 2018/11/29 15:01:26 [INFO] generating key: rsa-2048 2018/11/29 15:01:26 [INFO] encoded CSR 2018/11/29 15:01:26 [INFO] signed certificate with serial number 570248288515480165542553575432827729263232937534
Now, we have all the required certificates for Quay generated.
-rw-r--r--. 1 iranzo iranzo 1001 nov 29 15:01 quay-enterprise.csr -rw-rw-r--. 1 iranzo iranzo 208 nov 29 15:01 quay-enterprise-csr.json -rw-------. 1 iranzo iranzo 1675 nov 29 15:01 quay-enterprise-key.pem -rw-rw-r--. 1 iranzo iranzo 1809 nov 29 15:01 quay-enterprise.pem
Load Quay certificates
Now using the web interface of quay, we can start uploading the certificates required for operation.
Custom SSL Certificates
Here, load the
Server Configuration **TLS**
Certificate: load the
Key: load the
At this point Quay will be using the certificates, but by default the service created only works on port 80.
REMEMBER: We’ll need to edit:
- the app
- the route
So that both work on port 443 instead of port 80 that is used by default.
We’ll also need to load
quay-enterprise.pem into our
/etc/pki/ca-trust/source/anchors folder and then run:
update-ca-trust extract service docker restart # Now let's go login into the registry docker login https://quay-enterprise-quay-enterprise.apps.leo-fed.e2e.bos.example.com # Let's download a container: docker pull quay.io/mavazque/pacman # let's tag and upload to our registry docker images|grep pacman quay.io/mavazque/pacman latest 6c5211874850 29 hours ago 689 MB # Tag and upload # Find image docker images|grep pacman quay.io/mavazque/pacman latest 6c5211874850 29 hours ago 689 MB # Tag image docker tag 6c5211874850 quay-enterprise-quay-enterprise.apps.leo-fed.e2e.bos.example.com/admin/pacman:latest # Upload image docker push quay-enterprise-quay-enterprise.apps.leo-fed.e2e.bos.example.com/admin/pacman The push refers to a repository [quay-enterprise-quay-enterprise.apps.leo-fed.e2e.bos.example.com/admin/pacman] fb2d3be23d30: Pushed c02d067bd364: Pushed 3edd111b03f2: Pushed 2221babd26ac: Pushed 911d290ce59f: Pushed fae583f1d4e5: Pushed 9d22e51f0c6d: Pushed 58b8d417193d: Pushing [===================================> ] 225.5 MB/320.1 MB 704a8634956f: Pushing [==================================================>] 127 MB 05f0b6bcfa5c: Pushed 0972cc82b682: Pushing [=============================> ] 74.2 MB/126.8 MB
Unless we also load the SSL certificate from Quay into OpenShift, we’ll be facing issues like this:
At this point, we’ve Quay setup for using geo-replication for HA and faster speeds from each cluster, we do have an underlying postgres setup using replication across sites and the URL’s have been secured for being used by our servers downloading images from there to deploy our applications.
Hope that you’ve enjoyed it!