Cloudpipe – Per Project Vpns

Cloudpipe is a method for connecting end users to their project instances in vlan mode.

Overview

The support code for cloudpipe implements admin commands (via nova-manage) to automatically create a vm for a project that allows users to vpn into the private network of their project. Access to this vpn is provided through a public port on the network host for the project. This allows users to have free access to the virtual machines in their project without exposing those machines to the public internet.

Cloudpipe Image

The cloudpipe image is basically just a linux instance with openvpn installed. It needs a simple script to grab user data from the metadata server, b64 decode it into a zip file, and run the autorun.sh script from inside the zip. The autorun script will configure and run openvpn to run using the data from nova.

It is also useful to have a cron script that will periodically redownload the metadata and copy the new crl. This will keep revoked users from connecting and will disconnect any users that are connected with revoked certificates when their connection is renegotiated (every hour).

Creating a Cloudpipe Image

Making a cloudpipe image is relatively easy.

# install openvpn on a base ubuntu image. # set up a server.conf.template in /etc/openvpn/

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
port 1194
proto udp
dev tap0
up "/etc/openvpn/up.sh br0"
down "/etc/openvpn/down.sh br0"

persist-key
persist-tun

ca ca.crt
cert server.crt
key server.key  # This file should be kept secret

dh dh1024.pem
ifconfig-pool-persist ipp.txt

server-bridge VPN_IP DHCP_SUBNET DHCP_LOWER DHCP_UPPER

client-to-client
keepalive 10 120
comp-lzo

max-clients 1

user nobody
group nogroup

persist-key
persist-tun

status openvpn-status.log

verb 3
mute 20

# set up.sh in /etc/openvpn/

1
2
3
4
5
6
7
#!/bin/sh

BR=$1
DEV=$2
MTU=$3
/sbin/ifconfig $DEV mtu $MTU promisc up
/usr/sbin/brctl addif $BR $DEV

# set down.sh in /etc/openvpn/

1
2
3
4
5
6
7
#!/bin/sh

BR=$1
DEV=$2

/usr/sbin/brctl delif $BR $DEV
/sbin/ifconfig $DEV down

# download and run the payload on boot from /etc/rc.local

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
####### These lines go at the end of /etc/rc.local #######
. /lib/lsb/init-functions

echo Downloading payload from userdata
wget http://169.254.169.254/latest/user-data -O /tmp/payload.b64
echo Decrypting base64 payload
openssl enc -d -base64 -in /tmp/payload.b64 -out /tmp/payload.zip

mkdir -p /tmp/payload
echo Unzipping payload file
unzip -o /tmp/payload.zip -d /tmp/payload/

# if the autorun.sh script exists, run it
if [ -e /tmp/payload/autorun.sh ]; then
  echo Running autorun.sh
  cd /tmp/payload
  sh /tmp/payload/autorun.sh

else
  echo rc.local : No autorun script to run
fi


exit 0

# setup /etc/network/interfaces

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual
  up ifconfig $IFACE 0.0.0.0 up
  down ifconfig $IFACE down

auto br0
iface br0 inet dhcp
  bridge_ports eth0

# register the image and set the image id in your flagfile:

--vpn_image_id=ami-xxxxxxxx

# you should set a few other flags to make vpns work properly:

--use_project_ca
--cnt_vpn_clients=5

Cloudpipe Launch

When you use nova-manage to launch a cloudpipe for a user, it goes through the following process:

  1. creates a keypair called <project_id>-vpn and saves it in the keys directory
  2. creates a security group <project_id>-vpn and opens up 1194 and icmp
  3. creates a cert and private key for the vpn instance and saves it in the CA/projects/<project_id>/ directory
  4. zips up the info and puts it b64 encoded as user data
  5. launches an m1.tiny instance with the above settings using the flag-specified vpn image

Vpn Access

In vlan networking mode, the second ip in each private network is reserved for the cloudpipe instance. This gives a consistent ip to the instance so that nova-network can create forwarding rules for access from the outside world. The network for each project is given a specific high-numbered port on the public ip of the network host. This port is automatically forwarded to 1194 on the vpn instance.

If specific high numbered ports do not work for your users, you can always allocate and associate a public ip to the instance, and then change the vpn_public_ip and vpn_public_port in the database. This will be turned into a nova-manage command or a flag soon.

Certificates and Revocation

If the use_project_ca flag is set (required to for cloudpipes to work securely), then each project has its own ca. This ca is used to sign the certificate for the vpn, and is also passed to the user for bundling images. When a certificate is revoked using nova-manage, a new Certificate Revocation List (crl) is generated. As long as cloudpipe has an updated crl, it will block revoked users from connecting to the vpn.

The userdata for cloudpipe isn’t currently updated when certs are revoked, so it is necessary to restart the cloudpipe instance if a user’s credentials are revoked.

Restarting Cloudpipe VPN

You can reboot a cloudpipe vpn through the api if something goes wrong (using euca-reboot-instances for example), but if you generate a new crl, you will have to terminate it and start it again using nova-manage vpn run. The cloudpipe instance always gets the first ip in the subnet and it can take up to 10 minutes for the ip to be recovered. If you try to start the new vpn instance too soon, the instance will fail to start because of a NoMoreAddresses error. If you can’t wait 10 minutes, you can manually update the ip with something like the following (use the right ip for the project):

euca-terminate-instances <instance_id>
mysql nova -e "update fixed_ips set allocated=0, leased=0, instance_id=NULL where fixed_ip='10.0.0.2'"

You also will need to terminate the dnsmasq running for the user (make sure you use the right pid file):

sudo kill `cat /var/lib/nova/br100.pid`

Now you should be able to re-run the vpn:

nova-manage vpn run <project_id>

Logging into Cloudpipe VPN

The keypair that was used to launch the cloudpipe instance should be in the keys/<project_id> folder. You can use this key to log into the cloudpipe instance for debugging purposes.

The nova.cloudpipe.pipelib Module

The nova.api.cloudpipe Module

The nova.crypto Module

Wrappers around standard crypto data elements.

Includes root and intermediate CAs, SSH key_pairs and x509 certificates.

ca_folder(project_id=None)
ca_path(project_id=None)
compute_md5(fp)

Compute an md5 hash.

Parameters:fp (file) – File pointer to the file to MD5 hash. The file pointer will be reset to the beginning of the file before the method returns.
Return type:tuple
Returns:the hex digest version of the MD5 hash
crl_path(project_id=None)
decrypt_text(project_id, text)
ensure_ca_filesystem()

Ensure the CA filesystem exists.

fetch_ca(project_id=None)
fetch_crl(project_id)

Get crl file for project.

generate_fingerprint(public_key)
generate_key_pair(bits=None)
generate_vpn_files(project_id)
generate_x509_cert(user_id, project_id, bits=1024)

Generate and sign a cert for user in project.

key_path(project_id=None)
revoke_cert(project_id, file_name)

Revoke a cert by file name.

revoke_certs_by_project(project_id)

Revoke all project certs.

revoke_certs_by_user(user_id)

Revoke all user certs.

revoke_certs_by_user_and_project(user_id, project_id)

Revoke certs for user in project.

sign_csr(csr_text, project_id=None)