= DC3 Development with Virtual Machines =

[[PageOutline]]

Virtual machines, using [http://vmware.com/ VMware] tools and running [http://centos.org/ CentOS] Linux, are a supported platform in DC3.  To get started, follow these steps:

== Getting started with VMware ==

1. Install [http://vmware.com VMware] software
   * Mac
      * [http://www.vmware.com/mac/ VMware Fusion] - $80, with a free 30-day trial
      * Alternative: [http://www.parallels.com/ Parallels Desktop for Mac] - $80, but you will have to convert the image to a Parallels image, using [http://www.virtualizationdaily.com/archives/73_how-to-convert-a-vmware-virtual-appliance-to-work-with-parallels.html elbow grease and open-source software] or [http://www.parallels.com/en/products/desktop/features/transporter/ Parallels Transporter] ($80)
   * Windows & Linux
      * [http://vmware.com/download/player/ VMware Player] - free; runs a single virtual machine nicely on a desktop
      * [http://vmware.com/download/server/ VMware Server] - free; runs multiple virtual machines; allows multiple administrators and remote access
      * [http://vmware.com/products/ws/ VMware Workstation] - $190 (cheaper academic license available); can run multiple virtual machines simultaneously; single-user; has more powerful checkpointing abilities than vmware-server
      * [http://vmware.com/products/server_virtualization.html VMware's higher-end options] - dynamic failover, load balancing, sophisticated management tools
   * Alternative: [http://www.virtualbox.org/ VirtualBox], from Sun
      * Free for [http://www.virtualbox.org/wiki/VirtualBox_PUEL personal use and evaluation]
      * Pure open-source (GPL) edition available
      * Runs on Windows, Mac, Linux, and Solaris
      * [https://wiki.ubuntu.com/UbuntuMagazine/HowTo/Switching_From_VMWare_To_VirtualBox:_.vmdk_To_.vdi_Using_Qemu_+_VdiTool Converting a VMware image to run in VirtualBox]

2. Download a virtual machine ''image'' -- that is, a snapshot of a virtual machine:

   * 64-bit (requires a 64-bit host machine): http://dev.lsstcorp.org/images/lsst-dc3-64.tar.gz [1.7 GB] (out of date; update coming soon)
   * 32-bit (works on either 32- or 64-bit host): http://dev.lsstcorp.org/images/lsst-dc3-32.tar.gz [1.5 GB] (out of date; update coming soon)
   * About these VMs
     * Installed OS: [http://centos.org/ Centos version 5.3] (CentOS tracks and is compatible with [http://redhat.com/ Red Hat Enterprise Linux]), with `gcc 4.3.3`.
     * LSST tools installed under `/lsst`
     * Snapshot is out-of-date. Once you have the VM setup, you should get rid of the existing /lsst and rebuild as directed in [http://dev.lsstcorp.org/GettingStarted.html Getting Started].

3. Unpack the image locally and open it in your virtualization software.

   * VMware may alert you that ''"The location of this virtual machine's configuration file has changed since it was last powered on."'' and ask whether you want to change or keep its UUID.

     [[Image(vmware-uuid.png, 33%)]][[Image(vmware-uuid-mac.png, 33%)]]

   * The UUID will determine the new virtual machine's MAC address.  Click '''Moved''' or '''Keep''' to keep the MAC address unchanged -- the network drivers in CentOS 4 can get confused if it changes.  Note that if you intend to run the VM in ''Bridged'' mode, however, you may need to change it so that it can coexist on the network with other copies of itself.

4. On first boot, you may encounter '''Kudzu''', the Red Hat hardware management tool.  In particular, Kudzu may detect a change in the VM's ethernet card.  Don't worry; it's just adjusting to its new physical host.  Go along with Kudzu's default actions, which will usually be:

   a. remove the old card
   a. add the new card (which may be indistinguishable from the old card)
   a. if it's an ethernet card, configure TCP/IP; to use DHCP, be sure to press `space` to select "Use dynamic IP configuration (BOOTP/DHCP)"

     [[Image(kudzu-welcome.png, 25%)]][[Image(kudzu-removed.png, 25%)]][[Image(kudzu-added.png, 25%)]][[Image(kudzu-tcpip.png, 25%)]]

5. On first boot, the virtual machine will ask you to:

   a. establish a root password
   a. create a user account, which will be added to the `lsst` group in order to access the LSST tools installed in `/lsst`

6. Log in and adjust your environment as you wish.  

7. Once logged in you should get any updates to the LSST software stack.  To do this:
   1. Open a terminal window (Applications->Accessories->Terminal)
   1. Install updates by typing "`lsstpkg install LSSTPipe`".

   If you get `lsstpkg: command not found`, load the LSST environment by typing "`source /lsst/stacks/default/loadLSST.sh`" (tcsh/csh users source `loadLSST.csh`).  It should be loaded already by default for bash users due to a setting in `/etc/profile.d/lsst.sh`.

=== Networking Options ===

   * The LSST DC3 virtual machine is configured to use VMware's internal NAT, which means it will only be accessible from your VMware host machine.
   * To assign it an externally addressable IP address:
     i. change the virtual machine's settings in your VMware client to "Bridged Networking"
     i. reboot it (or execute `service network restart`); you may have to manually assign it an address
     i. assign it a DNS entry as if it was just another machine on your network

   See FAQ Q.9 below for a note on security implications.

== Software Development for DC3 ==

=== LSST Tools ===

The virtual machine described above includes all tools, [SwTools as described here], to do LSST development work:

   * The build system has been in stalled in `/lsst/stack`
   * `/lsst/stack/loadlsst.(sh|csh)` has ''not'' been executed.

These LSST software development guides describe the next steps to take:

   * [http://dev.lsstcorp.org/GettingStarted.html Getting Started]
   * [SwTools LSST Software Tools]

=== Sharing Files ===

You can share files between your host computer and the virtual machine in at least two ways: through a network file system, and through VMware's own tools.

 1. Sharing files over a network.  The easiest way to access your VM's files from a host machine is probably via SSH, using [http://fuse.sourceforge.net/ Fuse] or [http://code.google.com/p/macfuse/ Macfuse].  Macfuse, in particular, is easy to install and use.

 2. Sharing files using VMware's tools.  Only the paid VMware products support file sharing, though -- notably, ''Fusion'' and ''Workstation''.
    a. Make sure that VMware tools are installed (they are pre-installed in the LSST images for Fusion)
    a. In the VM's settings, enable shared folders
    a. Add a shared folder (in ''Fusion'', click the '''+''' icon and choose a folder on your host)
    a. Be sure to click the new shared folder's ''Enabled'' checkbox
    a. The folder will show up in the VM's file system under {{{/mnt/hgfs}}}.

== Frequently Asked Questions ==

1. How do I '''open a terminal''' in the virtual machine?
   Answer:: log in, right-click on the desktop, and select "Open Terminal"

2. How can I '''ssh''' to the virtual machine?
   Answer:: The virtual machine runs under NAT by default, and is only reachable from the host.  To determine its IP address, open a terminal in the virtual machine and run `ifconfig`.

3. How can I give the virtual machine an '''externally reachable IP address'''?
   Answer:: Change from NAT to Bridged networking in VMware:
      a. In VMware software, change the settings on the virtual machine: change Network to be ''Bridged'' instead of ''NAT''.
      a. Either restart the virtual machine or run `service network restart` in a terminal.
      a. Assign DNS normally.
      a. If you want to assign a static address instead of DHCP, in your virtual machine run `system-config-network` and change `eth0` to use a static address instead of DHCP.

5. Can I still use my desktop '''development tools'''?
   Answer:: This can be an excellent way to work.  You will need to find a way to share files with the virtual machine ''(see [#SharingFiles], above)''.  Also consider running the virtual machine in text mode (see below), to free up CPU cycles on your host.

6. How can I change the '''screen size''' of my virtual machine?  800x600 is too small.  Can I make it dynamically adapt to the size of the VMware window?
   Answer:: Run `vmware-config-tools.pl` in your virtual machine and choose to change your X resolution.  Unfortunately, it can't adapt dynamically, due to limitations of X Windows on the virtual machine.

7. Can I run the virtual machine in '''text mode'''?
   Answer:: Yes, and you can save significant CPU cycles on your host machine, if you don't need local graphics in the VM.  Edit `/etc/inittab`, changing the line that says `id:5:initdefault:` to `id:3:initdefault:`.

8. A little window is telling me that there are new packages available.  Should I do automated updates to the operating system?  

   Answer::  We recommend that you ''do'' keep your system up to date by installing updates as advised by the Update Manager icon. This should not affect your LSST software stack; however, if we become aware of issues related to OS updates, we will let you know via this page.  

9. Are there any security issues I should be worried about while using this VM?

 Answer::  The VM comes using so-called "NAT" networking.  This means that you can from inside the VM get out to the outside world (e.g. via a browser, the `lsstpkg` tool, etc.), but you and others cannot get into the machine from the outside.  This is because your machine does not have its own IP address.  This makes the VM more secure than a real workstation or laptop.  There are still the usual concerns about downloading viruses (to which Linux is historically less prone) and getting caught by phishing scams.  

== Troubleshooting ==

My virtual machine doesn't have '''network access'''.  (You may have seen a message about a mismatched MAC address.)

   a. In your virtual machine, run {{{system-config-network}}} as {{{root}}}.
   a. ''Delete'' your existing ethernet device (you may have to ''Deactivate'' it first).
   a. Create a ''New'' network device (''New'' -> ''Ethernet Connection'' -> ''Forward'' x3 -> ''Apply''
   a. ''Activate'' the new Ethernet device

I switched to '''Bridged Networking''', but something isn't working right.

    You may be having a problem with your or switch; some are configured to only allow one computer per port, but with Bridged Networking, your host computer is pretending to be two different devices, each with its own MAC address.  You will have to configure your networking equipment to allow multiple MAC addresses per port.

Some VMware services running ''inside the VM'' are failing to start up during boot.  Is this a problem?  How can I fix it?

    After a kernel upgrade, usually thanks to `yum`, the VMware tools installed inside the virtual machine, which allow for file sharing and better performance, need to be updated as well.  To update them, as `root`, run `vmware-config-tools.pl`.

== How these VMs were built ==

=== Installed Software ===

 * [NotesOnBuildingGcc gcc 4.3.3] as default compiler (OS-provided gcc removed before LSST stack installed)
 * Current LSST software stack as of release date
    * built with gcc 4.3.3
    * installed under `/lsst/stacks/default`
    * set to automatically initialize via `/etc/profile.d/lsst.sh`
    * Packages
       * umbrella package `LSSTPipe` (includes `ctrl_dc3pipe, ap, sdqa, afw,` etc.)
       * [http://java.sun.com/ Sun Java] installed as an [InstallingOptionalPackages optional package]
       * `activemq` & `jython` installed as external packages
       * in future plan to install `ctrl_evmon` when it is available

=== Settings ===

 * VMware virtual machine version 4 (created with [http://www.vmware.com/products/server/ VMware Server])
 * 512MB RAM (32-bit), 768MB RAM (64-bit)
 * 40GB hard disk
 * 1 cpu
 * Before release
    * run VMware Shrink Disk to reclaim unused space
    * switch networking to NAT
    * trigger firstboot (via /root/bin/reset-firstboot.sh)

And finally, thanks to Mike Freemon for his [http://mywiki.ncsa.uiuc.edu/wiki/VM_Distribution_Procedure VM distribution procedure].