======
Clouds
======

The DoE-Suite supports multiple _clouds_.
A cloud refers to an environment in which servers are set up and configured.
So far, we support AWS (aws), Euler (euler, ETH compute cluster), and clouds described by an Ansible inventory file.

Set the environment variable ``DOES_CLOUD`` to control the default cloud.
On every invocation of the DoE-Suite, you can set the cloud environment,
e.g., ``make run suite=XYZ id=new cloud=euler``.
For clouds described by an Ansible inventory file, the name of the inventory is the cloud name.
Note, that if you started a suite on a cloud, than this suite run cannot switch to another cloud.


Supported clouds
----------------

+-------+-------------------------------+
| Cloud | Description                   |
+-------+-------------------------------+
| aws   | Amazon Web Services (EC2)     |
+-------+-------------------------------+
| euler | Euler (ETHZ compute cluster)  |
+-------+-------------------------------+
| <INV> | Ansible Inventory named <INV> |
+-------+-------------------------------+


Cloud multiplexing
------------------

Doe-suite can load cloud-specific functionality depending on the active cloud.

For some roles, the DoE-Suite will look for the cloud specific  ``{cloud}.yml`` file.
If no such file exists, ``main.yml`` will be used as a fallback.

This cloud multiplexing is enabled for the following roles:
- ``setup-suite``
- Any ``suite-cloud-*`` role
- Any host-specific role, those in ``doe-suite-config/roles/setup-*``.

Schedulers
----------

We now have the concept of schedulers that handles the scheduling of jobs.
The default scheduler is ``tsp`` which will be set-up on the OS by the DoE-Suite.
In some cases, the cloud has its own scheduler, for example ``bsub`` or ``slurm`` in Euler.
The variable ``_job_scheduler`` in ``doe-suite/src/group_vars/all`` is used to control which scheduler is used.

For ansible inventories, you can set the scheduler in the inventory file as a host variable: `job_scheduler: slurm`

We support the same type of multiplexing as for clouds.
The Doe-Suite looks for the specific scheduler implementation in the following roles:
- ``suite-scheduler-enqueue`` Enqueue the pending jobs
- ``suite-scheduler-remove`` Remove a finished job (unused in ``slurm``, ``bjobs``, but used for ``tsp``)
- ``suite-scheduler-status`` Get status of a specific job



Adding new clouds
=================

``main`` represents all supported clouds except the ones that have a specific file.

In ``doe-suite-config/roles/**``, rename ``main.yml`` to ``aws.yml``

Add setup tasks for all cloud-related tasks: ``suite-cloud-*`` in the suite.