Setting up Harbor in Production

8 min readDec 9, 2018

Before proceeding to this setup please check previous steps:
https://medium.com/@ikod/deploy-harbor-container-registry-in-production-89352fb1a114

https://medium.com/@ikod/setting-up-the-postgres-redis-and-amazon-s3-for-harbor-852bb9ebe8f2

Harbor Setup:

From all above steps we do have the HA Postgres, Standalone Redis and S3 bucket created. I assume that you have reverse proxy say Nginx deployed with Public signed SSL certificates. Now we will do the deployment of the Harbor.

Two servers of the Harbor :

harbor101.us-west-2.compute.internal
harbor102.us-west-2.compute.internal

Execute the following steps on each server :

Step 1: SSH to the server

Step 2: Download the Harbor Offline Installer

Check the latest release of the Harbor at https://github.com/goharbor/harbor/releases

$ wget https://storage.googleapis.com/harbor-releases/release-1.6.0/harbor-offline-installer-v1.6.0.tgz
$ tar -xvf harbor-offline-installer-v1.6.0.tgz
$ cd harbor

Step 3: Edit the harbor.cfg to configure the harbor.

harbor.cfg consists the information about the authentication configuration, database configuration and etc. In the configuration, there are three main components. First, the LDAP if you are allowing to use LDAP. Then you should configure the Postgres which we configured previously. Also the configuration of Redis and Object storage.

Following is the example configuration file for the Harbor. However, I have edited the file to just hide the credentials.

On the first server: harbor101.us-west-2.compute.internal

## Configuration file of Harbor

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version = 1.6.0
#The IP address or hostname to access admin UI and registry service.
#DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
#IMPORTANT PLEASE CHANGE TO YOUR VIP.
hostname = harbor.us-west-2.compute.internal

#IMPORTANT DEPENDENT ON YOUR CONFIGURATION CHANGE IT.
#The protocol for accessing the UI and token/notification service, by default it is http.
#It can be set to https if ssl is enabled on nginx.
ui_url_protocol = http

#Maximum number of job workers in job service
max_job_workers = 10

#Determine whether or not to generate certificate for the registry’s token.
#If the value is on, the prepare script creates new root cert and private key
#for generating token to access the registry. If the value is off the default key/cert will be used.
#This flag also controls the creation of the notary signer’s cert.
customize_crt = off

#The path of cert and key files for nginx, they are applied only the protocol is set to https
ssl_cert = /root/harbor/cert/harbor101.us-west-2.compute.internal.crt
ssl_cert_key = /root/harbor/cert/harbor101.us-west-2.compute.internal.key

#The path of secretkey storage
secretkey_path = /data

#Admiral’s url, comment this attribute, or set its value to NA when Harbor is standalone
admiral_url = NA

#Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
log_rotate_count = 50
#Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
#If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
#are all valid.
log_rotate_size = 200M

#Config http proxy for Clair, e.g. http://my.proxy.com:3128
#Clair doesn’t need to connect to harbor ui container via http proxy.
#http_proxy =
#https_proxy =
http_proxy =
https_proxy =
no_proxy = 127.0.0.1,localhost,ui,registry

#NOTES: The properties between BEGIN INITIAL PROPERTIES and END INITIAL PROPERTIES
#only take effect in the first boot, the subsequent changes of these properties
#should be performed on web ui

#************************BEGIN INITIAL PROPERTIES************************

#Email account settings for sending out password resetting emails.

#Email server uses the given username and password to authenticate on TLS connections to host and act as identity.
#Identity left blank to act as username.
email_identity =
email_server = mail.us-west-2.compute.internal
email_server_port = 25
email_username =
email_password =
email_from = admin <harbor-admin@admin.com>
email_ssl = false
email_insecure = false

##The initial password of Harbor admin, only works for the first time when Harbor starts.
#It has no effect after the first launch of Harbor.
#Change the admin password from UI after launching Harbor.
harbor_admin_password = Harbor@12345

##By default the auth mode is db_auth, i.e. the credentials are stored in a local database.
#Set it to ldap_auth if you want to verify a user’s credentials against an LDAP server.
auth_mode = ldap_auth

#The url for an ldap endpoint.
ldap_url = ldap://ldap.us-west-2.compute.internal:389

#A user’s DN who has the permission to search the LDAP/AD server.
#If your LDAP/AD server does not support anonymous search, you should configure this DN and ldap_search_pwd.
ldap_searchdn = CN=example.user,OU=Special Users,DC=internal,DC=mycompany,DC=co,DC=jp

#the password of the ldap_searchdn
ldap_search_pwd = Passwordldap

#The base DN from which to look up a user in LDAP/AD
ldap_basedn = DC=internal,DC=mycompany,DC=co,DC=jp

#Search filter for LDAP/AD, make sure the syntax of the filter is correct.
#ldap_filter = (&(objectClass=user)(sAMAccountName=%s))
ldap_filter =

# The attribute used in a search to match a user, it could be uid, cn, email, sAMAccountName or other attributes depending on your LDAP/AD
ldap_uid = sAMAccountName

#the scope to search for users, 0-LDAP_SCOPE_BASE, 1-LDAP_SCOPE_ONELEVEL, 2-LDAP_SCOPE_SUBTREE
ldap_scope = 2

#Timeout (in seconds) when connecting to an LDAP Server. The default value (and most reasonable) is 5 seconds.
ldap_timeout = 5

#Verify certificate from LDAP server
ldap_verify_cert = false

#The base dn from which to lookup a group in LDAP/AD
ldap_group_basedn = DC=internal,DC=mycompany,DC=co,DC=jp

#filter to search LDAP/AD group
ldap_group_filter =

#The attribute used to name a LDAP/AD group, it could be cn, name
ldap_group_gid = cn

#The scope to search for ldap groups. 0-LDAP_SCOPE_BASE, 1-LDAP_SCOPE_ONELEVEL, 2-LDAP_SCOPE_SUBTREE
ldap_group_scope = 2

#Turn on or off the self-registration feature
self_registration = off

#The expiration time (in minute) of token created by token service, default is 30 minutes
token_expiration = 30

#The flag to control what users have permission to create projects
#The default value “everyone” allows everyone to creates a project.
#Set to “adminonly” so that only admin user can create project.
project_creation_restriction = adminonly

#************************END INITIAL PROPERTIES************************

#######Harbor DB configuration section#######
# THIS IS THE HAPROXY CONFIGURATION FOR THE POSTGRES..

#The address of the Harbor database. Only need to change when using external db.
db_host = postgressql.us-west-2.compute.internal

#The password for the root user of Harbor DB. Change this before any production use.
db_password = Harbor123

#The port of Harbor database host
db_port = 5432

#The user name of Harbor database
db_user = postgres

##### End of Harbor DB configuration#######

##########Redis server configuration.############
# THIS IS THE REDIS WE SETUP IN THE INSTALLATION STEP
redis_host = redis.us-west-2.compute.internal
redis_port = 6379

#Redis connection address
#redis_host = redis

#Redis connection port
#redis_port = 6379

#Redis connection password
redis_password =

#Redis connection db index
#db_index 1,2,3 is for registry, jobservice and chartmuseum.
#db_index 0 is for UI, it’s unchangeable
redis_db_index = 1,2,3

##########Redis server configuration.############

##########Clair DB configuration############

# USE THE SAME PASSWORD USED IN THE POSTGRES..

#Clair DB host address. Only change it when using an exteral DB.
clair_db_host = postgressql.us-west-2.compute.internal
#The password of the Clair’s postgres database. Only effective when Harbor is deployed with Clair.
#Please update it before deployment. Subsequent update will cause Clair’s API server and Harbor unable to access Clair’s database.
clair_db_password = Harbor123
#Clair DB connect port
clair_db_port = 5432
#Clair DB username
clair_db_username = postgres
#Clair default database
clair_db = postgres

#The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
clair_updaters_interval = 12

##########End of Clair DB configuration############

#The following attributes only need to be set when auth mode is uaa_auth
uaa_endpoint = uaa.mydomain.org
uaa_clientid = id
uaa_clientsecret = secret
uaa_verify_cert = true
uaa_ca_cert = /path/to/ca.pem
# STORAGE SETTING FOR THE HARBOR CHANGE ACCORDING TO YOUR USE CASE.

### Harbor Storage settings ###
#Please be aware that the following storage settings will be applied to both docker registry and helm chart repository.
#registry_storage_provider can be: filesystem, s3, gcs, azure, etc.
registry_storage_provider_name = s3
#registry_storage_provider_config is a comma separated “key: value” pairs, e.g. “key1: value, key2: value2”.
#To avoid duplicated configurations, both docker registry and chart repository follow the same storage configuration specifications of docker registry.
#Refer to https://docs.docker.com/registry/configuration/#storage for all available configuration.
registry_storage_provider_config = accesskey: your_access_key, secretkey: your_secret_key, bucket: your_bucket_name, regionendpoint: your_region_endpoint , region: your_region
#registry_custom_ca_bundle is the path to the custom root ca certificate,which will be injected into the truststore
#of registry’s and chart repository’s containers. This is usually needed when the user hosts a internal storage with self signed certificate.
registry_custom_ca_bundle =

#If reload_config=true, all settings which present in harbor.cfg take effect after prepare and restart harbor, it overwrites exsiting settings.
reload_config=true
#Regular expression to match skipped environment variables
#skip_reload_env_pattern=(^EMAIL.*)|(^LDAP.*)

Change the configuration file according to your requirement.

Deploy the Harbor on harbor101.us-west-2.compute.internal:

# Deploy the Harbor with Clair and Chartmuseum
# Note that if you want to deploy the Harbor with Notary please use certificates and mode: HTTPS in harbor.
$ rm -rf /data/registry
$ rm -rf /data/database
$ ./install.sh — with-clair — with-chartmuseum

To the rest of the harbor instances (Repeat following steps for every other instance).

Change the config of the harbor.cfg. There is no change in configuration for rest of harbor instances. Example of harbor102.us-west-2.compute.internal (You can copy the same config).
Copy the /data/secret file of harbor101.us-west-2.compute.internal to harbor102.us-west-2.compute.internal. MAKE SURE THE SECRET FILE HAS APPROPRIATE PERMISSION TO THE POSTGRES USER.
Install the harbor :

# Deploy the Harbor with Clair and Chartmuseum
# Note that if you want to deploy the Harbor with Notary please use certificates and mode: HTTPS in harbor.
$ rm -rf /data/registry
$ rm -rf /data/database
$ ./install.sh — with-clair — with-chartmuseum

Upgrade the all Harbor instances to use the reverse proxy settings.

Use the following steps to have a working reverse proxy setting. Prepare all Harbor instances with the following steps.

Solve the issue with reverse proxy:

https://github.com/goharbor/harbor/issues/3114

# In harbor.cfg:
# Make sure the hostname points to the VIP. (In my case its reverse proxy https load balancer nginx)
# ( As external source that means https nginx will do this)
customize_crt = off

#In config of nginx common you can find it in harbor main directory — /common/templates/nginx/nginx.http.conf
#Comment the following line:
proxy_set_header X-Forwarded-Proto $$scheme;
# Make sure to comment it in / , /v2/ and /service/ endpoints.
# Prepare your harbor with new settings
# Down the all docker containers : (Execute following statement in oot directory of harbor.
docker-compose -f ./docker-compose.yml -f ./docker-compose.clair.yml -f ./docker-compose.chartmuseum.yml down -v
./prepare — with-clair — with-chartmuseum
# After preparation you can find in harbor main directory this file — common/config/registry/config.yml — Change realm to https.
# PLEASE CHANGE IT ACCORDING TO YOUR SETTING. HERE THE VIP ADDRESS IS REVERSE PROXY ADDRESS FOR US.
# Previous
auth:
token:
issuer: harbor-token-issuer
realm: http://vip-address/service/token
rootcertbundle: /etc/registry/root.crt
service: harbor-registry

# Change http to https (After change):
auth:
token:
issuer: harbor-token-issuer
realm: https://vip-address/service/token
rootcertbundle: /etc/registry/root.crt
service: harbor-registry

# Start the harbor.
docker-compose -f ./docker-compose.yml -f ./docker-compose.clair.yml -f ./docker-compose.chartmuseum.yml up -d

The Harbor will run successfully with the deployments. This completes the HA setup for the Harbor with HA Postgresql, Standalone Redis, and Amazon S3 service.

You can access the harbor with https://harbor.us-west-2.compute.internal url.

If you are facing issues you can email me at jakirpatel@outlook.com (Please ensure the title of email : Related to Harbor)