Git pull through a proxy

I found myself this afternoon trying to do a git pull from an internal host with a self-signed (or at least, signed by an authority I don't have locally) certificate. I also needed to do this through a proxy. This required a git incantation I hadn't come across before, and also the trusty https_proxy environment variable:

$ GIT_SSL_NO_VERIFY=true \
  https_proxy=http://proxy:3128/ \
  git pull https://self-signed.example.com/gerrit/nova refs/changes/X/Y/Z

Diffing diffs with bash process substitution

I found myself wanting to examine the differences between different git commits. Specifically these commits represent the same change applied to 2 different branches, so I'm interested in what changes the backport author had to make. I was initially using temporary files for this, but stumbled across this bash gem:

$ diff -u <(git show [original]) <(git show [backport])

Note that this isn't your regular redirection, because I'm passing 2 filenames to diff. This is bash's process substitution syntax. It essentially runs a command and substitutes the path of a temporary named pipe connected to the command.

Stick it in a bash function, and you get:

function diffdiff() {
    diff -u <(git show "$1") <(git show "$2") | less
}

Now I can run:

$ diffdiff 01234567 89abcdef

and see my gloriously diffed diffs.

Configure a simple Galera cluster on Fedora

This post documents the most basic possible configuration of a 2-node Galera cluster on Fedora 22. The resulting cluster is good enough for playing with Galera, but should not be considered ready for production.

The required tasks are:

• Install Fedora and Galera packages
• Configure firewall
• Configure SELinux
• Configure Galera
• Start Galera cluster

Install Fedora and Galera packages

For the purposes of testing I am using 2 virtual machines, each with 1 CPU, 1 NIC, 2GB RAM and a single 5GB disk. I have installed both of these with Fedora 22 Server using the minimal install + standard packages as defined in the installer. Configure networking such that both nodes can reach each other and DNS is working correctly.

Install Galera and its dependencies with dnf:

# dnf install mariadb-galera-server

Configure firewall

Galera uses the following TCP ports:

• 3306 for MySQL network connections
• 4567 for cluster traffic
• 4444 for state snapshot transfer using rsync

Note that it does not keep the rsync port open, as it is only used when joining the cluster. Other backends may use different ports.

Open the above ports in the firewall with:

# firewall-cmd --add-port 4567/tcp --add-port 4444/tcp
# firewall-cmd --permanent --add-port 4567/tcp --add-port 4444/tcp
# firewall-cmd --add-service mysql
# firewall-cmd --permanent --add-service mysql

Configure SELinux

Note that the default targeted SELinux policy shipped with Fedora 22 will currently prevent Galera from starting on all but the first node. I have reported this in Fedora bug 1229794. There is a reasonable chance that by the time you read this the bug will have been fixed, or the steps below replaced with an SELinux boolean. Please check first.

Copy the following into a file called galeralocal.te:

module galeralocal 1.0;

require {
    type rsync_exec_t;
    type mysqld_safe_exec_t;
    type kerberos_port_t;
    type mysqld_t;
    class tcp_socket name_bind;
    class file { getattr read open execute execute_no_trans };
}

#============= mysqld_t ==============
allow mysqld_t kerberos_port_t:tcp_socket name_bind;
allow mysqld_t mysqld_safe_exec_t:file getattr;
allow mysqld_t rsync_exec_t:file { read getattr open execute execute_no_trans };

Install the required tools to compile and install local SELinux policy:

# dnf install checkpolicy policycoreutils-python

Compile and load the custom SELinux policy:

# checkmodule -M -m -o galeralocal.mod galeralocal.te
# semodule_package -o galeralocal.pp -m galeralocal.mod
# semodule -i galeralocal.pp

The module must be installed on all nodes. However, after compiling the module on 1 node it is sufficient to copy galeralocal.pp to the other and just install it with semodule, meaning it isn't necessary to install the additional packages.

Configure Galera

Fedora puts the default galera configuration /etc/my.cnf.d/galera.cnf. For the purposes of this setup we leave almost everything untouched. We need to set the wsrep_provider to Galera:

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

Comment out wsrep_sst_auth, which is not used by the rsync sst method:

#wsrep_sst_auth=root:

Finally, set wsrep_cluster_address to a list of both nodes:

wsrep_cluster_address="gcomm://galera-1.example.com,galera-2.example.com"

Make these changes on both nodes. 

Starting the cluster

With the above configuration we are almost ready to start the cluster. However, if you try to start the database you will notice that it fails. At startup, the database will attempt to connect to any of the hosts listed in wsrep_cluster_address other than itself, to get the current database state. If it can't do this, it can't safely join the database. However, if none of the database nodes are running we have a bootstrapping problem.

To get round this, after ensuring that the database is most definitely not running anywhere, we edit the configuration on the node with the most up to date state to tell it to start without contacting any other database node. When initialising the cluster this can be any node.

Edit /etc/my.cnf.d/galera.cnf again to set:

wsrep_cluster_address="gcomm://"

With this in place, start the database with:

# systemctl start mariadb

Once the database has come up, you should put wsrep_cluster_address back to its original setting immediately. The reason for this is that it allows the database to come up without synchronising. If the database is, in fact, running somewhere, this will result in diverged state.

With 1 node running, you can now start all the other nodes with:

# systemctl start mariadb

Using the database

You now have a naively configured multi-master Galera cluster. Connect to mysql on any node and use it as normal:

# mysql
MariaDB [(none)]> create database foo;
Query OK, 1 row affected (0.00 sec) 

MariaDB [(none)]> connect foo 

MariaDB [foo]> create table bar (id int auto_increment primary key);
Query OK, 0 rows affected (0.02 sec)

New databases and data will be propagated immediately to all other nodes, and updates can be made on any node.