Protogeniinelabinelab

Notes about swapping the inner Emulab in and out

The ProtoGENI software is currently under heavy development, but is already available for testing. A convenient environment for experimenting with ProtoGENI is to use an existing Emulab facility to create an inner Emulab of your own, and configure that Emulab to run ProtoGENI facilities.

When configuring an inner elab, that probably means you want to do some development that can break things. We have set up some tools to help you get back to an consistent state and cleanup slices, tickets, and slivers. See RepairingEmulabProtoGENI for instructions.

The following steps allow you to join the ProtoGENI federation.

Create a user account in the outer Emulab, if you do not have one already. Those instructions refer to the Emulab at the University of Utah, although in principle any sufficiently recent Emulab will do.

Create an experiment to hold the inner (ProtoGENI-capable) Emulab. The following .ns file is adequate:

source tb_compat.tcl
set ns [new Simulator]

tb-elab-in-elab 1
tb-elabinelab-singlenet
tb-set-elabinelab-cvstag HEAD

namespace eval TBCOMPAT {
    set elabinelab_maxpcs 3
}

$ns run

Currently, that file will set up inner boss and ops nodes running FreeBSD 6.3.

Swap in the experiment you created in step 2.
On the inner boss node, make the FreeBSD 6.3 ports tree available under /usr/ports. On the Utah Emulab, this is possible with the command:
```
sudo mount ops.emulab.net:/share/freebsd/6.3/ports /usr/ports
```
On other systems, you can access the tree by anonymous CVS.

Install the libdnet port in the inner boss, as follows:

cd /usr/ports/net/libdnet
sudo make
sudo make install

Configure and build the Emulab/ProtoGENI software on the inner boss:
1. The source is available under /usr/testbed/src/testbed; you will need to edit the defs-elabinelab file in that directory to add ProtoGENI support. That is done by appending the following two lines:
```
PROTOGENI_SUPPORT=1
PROTOGENI_DOMAIN="mygeni"
```
  where mygeni should be changed to a short descriptive name for your installation.
2. Once you have updated the configuration, compilation is performed as follows:
```
cd /usr/testbed/src
mkdir obj && cd obj
../testbed/configure --with-TBDEFS=../testbed/defs-elabinelab
gmake
sudo gmake boss-install
```
  The gmake command in particular will take some time.
  Important Note: If you are planning to develop from a source tree on the inner elab, first clone a repository. Build and install all main Emulab updates *before* enabling PROTOGENI in the defs file.Then make sure to keep using the defs file in /usr/testbed/src/defs-elabinelab which has the proper settings for your inner elab.
3. Reboot the inner boss to start the newly installed system:
```
sudo reboot
```
Remount the /usr/ports tree (i.e. repeat step 4).
The remainder of the procedure is mostly automated: you should repeatedly invoke the following command:
```
sudo /usr/testbed/sbin/protogeni/initsite
```
and follow the instructions it provides as necessary. (Typically, you will need to join the ProtoGENI federation by sending your site key certificate to the Utah Emulab administrators; install a few more ports; and edit system configuration files such as /etc/rc.conf and /etc/crontab.)
Once initsite runs to completion, the site itself is ready, and all that is needed is the maintenance of user accounts. Firstly, you should ensure that all existing accounts are assigned unique IDs, by executing the following on the inner boss:
```
mysql
mysql> use tbdb;
mysql> update users set uid_uuid=UUID();
```
Install and start the protogeni expiration daemon:
```
cd /usr/testbed/src/obj/rc.d
gmake
sudo gmake install
sudo /usr/testbed/sbin/protogeni/expire_daemon
```
the daemon will background itself and return immediately, and will be automatically started when boss reboots.
If you want local resources to be registered with the clearinghouse (which is required to create slices which include resources handled by multiple component managers, for instance), then you need to run:
```
/usr/testbed/sbin/protogeni/register_resources
```
on the inner boss.
If you are building out of a git repository, make sure it is up to date and rebuild. You may be required to rerun the update script to accomodate changes to the geni database schema.
Any user who wishes to access ProtoGENI resources should create a personal key. This is done through the WWW interface on the inner boss, accessed with a URL such as
```
https://myboss.myexperiment.myproject.emulab.net/
```
by logging in, then following the My Emulab link in the upper left and then Generate SSL Cert on the far left. A secure passphrase must be supplied by the user.
Any user who has completed step 12 can now access ProtoGENI resources! This should be performed on the inner ops host. Some example facilities are available under the /usr/testbed/src/testbed/protogeni/test directory as Python scripts. These scripts require the the passphrase that the user chose in step 12 to be saved in $HOME/.ssl/password.

Notes about swapping the inner Emulab in and out

Unfortunately, the ProtoGENI installation maintains a large amount of state, so it is a relatively involved procedure to swap it in and out. The simplest approach is to leave it swapped in whenever possible.

To keep the resources registered at the clearinghouse up to date, it is best to run the remauthority script on the clearinghouse when swapping out the inner Emulab, if you have permission:

/usr/testbed/sbin/protogeni/remauthority -c <uuid of inner CM>
/usr/testbed/sbin/protogeni/remauthority -c <uuid of inner SA>
/usr/testbed/sbin/protogeni/remauthority -c <uuid of inner SES>

The current prototype is undergoing heavy development, so the above instructions are subject to change. Please send any comments to the developers.