The Cloud42 Web Service

Introduction

The Web service interface is the actual core of Cloud42, since it allows to invoke Cloud42's enhanced functionalities from within other software or BPEL processes very easily. This way, Cloud42 can be used to compose large software systems, exploiting all the benefits of Cloud Computing.

Design

The main challenge when designing the Cloud42 Web service layer was state handling. In order to interact with EC2, several user information like the AWS credentials (used to identify an user at the EC2 API) or RSA private keys have to be provided.
This raises the question whether such information should be stored within Cloud42 or whether a calling process should provide it within each request.
Stateful Web service interfaces are more difficult to handle in general, because the caller must be identified and some kind of session information must be transferred.

As a conclusion, the Cloud42 Web service interface was supposed to be fully stateless. Finally, this means the caller has to handle access information for itself. AWS credentials and other data are passed to the Cloud42 API within each actual request.

This solution guarantees full flexibility and scalability by delegating state management and responsibility to the calling process, allowing Cloud42 to focus on its main capabilities.

Available Services

The Cloud42 Web service interface is logically separated into four particular services, each of them providing its own WSDL file for full modularity.

  • Cloud42BaseService (http://localhost:8080/Cloud42WS/Cloud42BaseService?wsdl) contains the base methods to interact with EC2. Querying instance information, launching new instances or creating security groups are some examples.
  • Cloud42FileService (http://localhost:8080/Cloud42WS/Cloud42FileService?wsdl) provides methods for up- and downloading files using MTOM.
  • Cloud42NotificationService (http://localhost:8080/Cloud42WS/Cloud42NotificationService?wsdl) is the implementation of the Cloud42 notification mechanism (see the corresponding documentation). It consists of the subscribe and unsubscribe methods. Furthermore, it allows configuring the address of the endpoint published by Cloud42 in order to retrieve event messages from running AMI instances.
  • Cloud42RemotingService (http://localhost:8080/Cloud42WS/Cloud42RemotingService?wsdl) finally can be used to execute commands or batch files remotely on an EC2 server instance. Bundling new AMIs is also supported by this service.

Sample SOAP Messages

The Web service interface uses SOAP over HTTP to exchange information. There are bindings for SOAP 1.1 as well as for SOAP 1.2.

It's recommended that you browse the JavaDocs for the webservice package to see which methods are available and which parameters are expected.

Since the Web service interface is stateless, most methods require AWS credentials. These credentials are transferred in the SOAP body.

General Example

The example below illustrates a typical flow of SOAP messages for the Cloud42BaseService with the SOAP 1.1 binding.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservice.cloud42.jw.de" xmlns:xsd="http://domain.core.cloud42.jw.de/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <web:listImages>
         <web:credentials>
            <xsd:id>1</xsd:id>
            <xsd:awsAccessKeyId>your accessKey</xsd:awsAccessKeyId>
            <xsd:secretAccessKey>your secretAccessKey</xsd:secretAccessKey>
            <xsd:userID>your userId</xsd:userID>
         </web:credentials>
         <!--the AWS Region to use -->
         <web:regionUrl>ec2.eu-west-1.amazonaws.com</web:regionUrl>
    <!--only list own AMIs -->
         <web:all>false</web:all>
      </web:listImages>
   </soapenv:Body>
</soapenv:Envelope>

This request lists all registered AMIs belonging to your user account.

Since Amazon separates between the different regions very strictly, it is important that you specify the correct region with the regionUrl-parameter in each request. You can list the available regions with the function listRegions. Here are the URLs:

  • ec2.eu-west-1.amazonaws.com for eu-west-1
  • ec2.us-east-1.amazonaws.com for us-east-1
  • ec2.us-west-1.amazonaws.com for us-west-1

The response could look like:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:listImagesResponse xmlns:ns="http://webservice.cloud42.jw.de" xmlns:ax21="http://domain.core.cloud42.jw.de/xsd" xmlns:ax25="http://wrapper.webservice.cloud42.jw.de/xsd" xmlns:ax23="http://ec2.amazonws.xerox.com/xsd">
         <ns:return type="com.xerox.amazonws.ec2.ImageDescription">
            <ax23:imageId>ami-c445a1ad</ax23:imageId>
            <ax23:imageLocation>my-ami/my-ami.manifest.xml</ax23:imageLocation>
            <ax23:imageOwnerId>932553346941</ax23:imageOwnerId>
            <ax23:imageState>available</ax23:imageState>
            <ax23:productCodes type="java.util.ArrayList">
               <empty xmlns="http://www.w3.org/2001/XMLSchema">true</empty>
            </ax23:productCodes>
            <ax23:public>false</ax23:public>
         </ns:return>
      </ns:listImagesResponse>
   </soapenv:Body>
</soapenv:Envelope>

Now you could use this information trigger a launch of an instance of your AMI:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservice.cloud42.jw.de" xmlns:xsd="http://domain.core.cloud42.jw.de/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <web:runInstance>
         <web:credentials>
            <xsd:id>1</xsd:id>
            <xsd:awsAccessKeyId>your accessKey</xsd:awsAccessKeyId>
            <xsd:secretAccessKey>your secretAccessKey</xsd:secretAccessKey>
            <xsd:userID>your userId</xsd:userID>
         </web:credentials>
         <!--the AWS Region to use -->
         <web:regionUrl>ec2.eu-west-1.amazonaws.com</web:regionUrl>
         <web:imageId>ami-c445a1ad</web:imageId>
         <web:groups>default</web:groups>
         <web:keyName>kp1</web:keyName>
         <web:userData>hello world!</web:userData>
         <web:instanceType>m1.small</web:instanceType>
         <web:count>1</web:count>
         <!--Optional:-->
         <web:availabilityZone></web:availabilityZone>
         <web:kernelId></web:kernelId>
         <web:ramdiskId></web:ramdiskId>
      </web:runInstance>
   </soapenv:Body>
</soapenv:Envelope>

Note that you can omit the values for availability-zone, kernelId and ramdiskId if you don't want to provide special values. Amazon EC2 will pick default values then.
As response, you will retreive a description of the instance you launched:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:runInstanceResponse xmlns:ns="http://webservice.cloud42.jw.de" xmlns:ax21="http://domain.core.cloud42.jw.de/xsd" xmlns:ax25="http://wrapper.webservice.cloud42.jw.de/xsd" xmlns:ax23="http://ec2.amazonws.xerox.com/xsd">
         <ns:return type="de.jw.cloud42.core.domain.Instance">
            <ax21:availabilityZone>us-east-1c</ax21:availabilityZone>
            <ax21:dnsName/>
            <ax21:groups>default</ax21:groups>
            <ax21:imageId>ami-c445a1ad</ax21:imageId>
            <ax21:instanceId>i-df43c5b6</ax21:instanceId>
            <ax21:instanceType>m1.small</ax21:instanceType>
            <ax21:kernelId xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
            <ax21:keyName>kp1</ax21:keyName>
            <ax21:launchTime>2009-01-13T11:02:26.000Z</ax21:launchTime>
            <ax21:owner>932553346941</ax21:owner>
            <ax21:pending>true</ax21:pending>
            <ax21:privateDnsName/>
            <ax21:ramdiskId xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
            <ax21:reason/>
            <ax21:reservationId>r-0eef4567</ax21:reservationId>
            <ax21:running>false</ax21:running>
            <ax21:shuttingDown>false</ax21:shuttingDown>
            <ax21:state>pending</ax21:state>
            <ax21:stateCode>0</ax21:stateCode>
            <ax21:terminated>false</ax21:terminated>
         </ns:return>
      </ns:runInstanceResponse>
   </soapenv:Body>
</soapenv:Envelope>

The instance is not started yet, but let's do another query to check its state:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservice.cloud42.jw.de" xmlns:xsd="http://domain.core.cloud42.jw.de/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <web:describeInstance>
           <web:credentials>
            <xsd:id>1</xsd:id>
            <xsd:awsAccessKeyId>your accessKey</xsd:awsAccessKeyId>
            <xsd:secretAccessKey>your secretAccessKey</xsd:secretAccessKey>
            <xsd:userID>your userId</xsd:userID>
         </web:credentials>
         <!--the AWS Region to use -->
         <web:regionUrl>ec2.eu-west-1.amazonaws.com</web:regionUrl>
         <web:instanceId>i-df43c5b6</web:instanceId>
      </web:describeInstance>
   </soapenv:Body>
</soapenv:Envelope>

Here we use the previously retrieved instance-id to get instance information. The response

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:describeInstanceResponse xmlns:ns="http://webservice.cloud42.jw.de">
         <ns:return type="de.jw.cloud42.core.domain.Instance" xmlns:ax21="http://domain.core.cloud42.jw.de/xsd" xmlns:ax25="http://wrapper.webservice.cloud42.jw.de/xsd" xmlns:ax23="http://ec2.amazonws.xerox.com/xsd">
            <ax21:availabilityZone>us-east-1c</ax21:availabilityZone>
            <ax21:dnsName>ec2-174-129-142-242.compute-1.amazonaws.com</ax21:dnsName>
            <ax21:groups>default</ax21:groups>
            <ax21:imageId>ami-c445a1ad</ax21:imageId>
            <ax21:instanceId>i-df43c5b6</ax21:instanceId>
            <ax21:instanceType>m1.small</ax21:instanceType>
            <ax21:kernelId xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
            <ax21:keyName>kp1</ax21:keyName>
            <ax21:launchTime>2009-01-13T11:02:26.000Z</ax21:launchTime>
            <ax21:owner>932553346941</ax21:owner>
            <ax21:pending>false</ax21:pending>
            <ax21:privateDnsName>domU-12-31-39-02-F5-D5.compute-1.internal</ax21:privateDnsName>
            <ax21:ramdiskId xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
            <ax21:reason/>
            <ax21:reservationId>r-0eef4567</ax21:reservationId>
            <ax21:running>true</ax21:running>
            <ax21:shuttingDown>false</ax21:shuttingDown>
            <ax21:state>running</ax21:state>
            <ax21:stateCode>16</ax21:stateCode>
            <ax21:terminated>false</ax21:terminated>
         </ns:return>
      </ns:describeInstanceResponse>
   </soapenv:Body>
</soapenv:Envelope>

tells us that the AMI instance is already in state "running" and provides further data such as DNS name.

If you want to try the unique remoting capabilities of Cloud42 now, jump to this section.

Finally, let's terminate our instance again.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservice.cloud42.jw.de" xmlns:xsd="http://domain.core.cloud42.jw.de/xsd">
   <soapenv:Header/>
   <soapenv:Body>
      <web:stopInstance>
         <web:credentials>
            <xsd:id>1</xsd:id>
            <xsd:awsAccessKeyId>your accessKey</xsd:awsAccessKeyId>
            <xsd:secretAccessKey>your secretAccessKey</xsd:secretAccessKey>
            <xsd:userID>your userId</xsd:userID>
         </web:credentials>
         <!--the AWS Region to use -->
         <web:regionUrl>ec2.eu-west-1.amazonaws.com</web:regionUrl>
         <web:instanceId>i-df43c5b6</web:instanceId>
      </web:stopInstance>
   </soapenv:Body>
</soapenv:Envelope>

The Cloud42 Web service responds with a description of the now terminating instance.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:stopInstanceResponse xmlns:ns="http://webservice.cloud42.jw.de">
         <ns:return type="com.xerox.amazonws.ec2.TerminatingInstanceDescription" xmlns:ax21="http://domain.core.cloud42.jw.de/xsd" xmlns:ax25="http://wrapper.webservice.cloud42.jw.de/xsd" xmlns:ax23="http://ec2.amazonws.xerox.com/xsd">
            <ax23:instanceId>i-df43c5b6</ax23:instanceId>
            <ax23:previousState>running</ax23:previousState>
            <ax23:previousStateCode>16</ax23:previousStateCode>
            <ax23:shutdownState>shutting-down</ax23:shutdownState>
            <ax23:shutdownStateCode>32</ax23:shutdownStateCode>
         </ns:return>
      </ns:stopInstanceResponse>
   </soapenv:Body>
</soapenv:Envelope>

Note: the SOAP messages were created with soapUI. If you want to test some other functions of the Web service interface, this tool is an ideal choice.

Controlling instances remotely

One of the most interesting features of Cloud42 is the remoting mechanism offered by the Cloud42RemotingService. This way, you can execute arbitrary commands on a running AMI instance by using the Web service interface of Cloud42.

The Web service returns all outputs of the command to the caller, including the exit code. Let's see an example:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservice.cloud42.jw.de">
   <soapenv:Header/>
   <soapenv:Body>
      <web:executeCommand>
         <web:dnsName>ec2-67-202-53-93.compute-1.amazonaws.com</web:dnsName>
         <web:rsaKey>-----BEGIN RSA PRIVATE KEY----- ... </web:rsaKey>
         <web:command>ls</web:command>
      </web:executeCommand>
   </soapenv:Body>
</soapenv:Envelope>

This requests lists the contents of the home directory using ls. For the rsaKey-Parameter you have to enter the private key for the keypair used to launch the corresponding instance. The result comes as follows:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Body>
      <ns:executeCommandResponse xmlns:ns="http://webservice.cloud42.jw.de">
         <ns:return type="de.jw.cloud42.core.domain.RemoteResult" xmlns:ax210="http://domain.core.cloud42.jw.de/xsd">
            <ax210:exceptionMessage xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
            <ax210:stdErr/>
            <ax210:stdOut>testfile.zip</ax210:stdOut>
            <ax210:exitCode>0</ax210:exitCode>
         </ns:return>
      </ns:executeCommandResponse>
   </soapenv:Body>
</soapenv:Envelope>

As you can see, the home folder contains only one file, testfile.zip.

Of course you can use this remoting mechanism not only to run trivial commands, but also to install and control any application you want. Furthermore, for executing batchs, the function executeBatch is suited. Instead of a single command, it is possible to pass a batch file or a shell script using MTOM.

File Transfer

Some functions of the Cloud42 Web service inteface require transferring files. Besides uploading batch files for the remoting mechanism, for instance the function uploadFile in the Cloud42FileService expects a file to be uploaded to the desired AMI instance.

Cloud42 uses MTOM to handle files. This means you have to enable MTOM support in the tool you use to access the Web service (for instance in soapUI). Then, the file can be passed as an attachment to the SOAP message. In the message itself, the attachment is referenced by an ID.

BPEL Example

The following section illustrates an example BPEL process using the Cloud42 Web service API, since invoking EC2's functionalities from within BPEL was one of the main motivations when developing Cloud42.

A small and simple scenario was chosen: suppose a particular AMI is needed in order to do some calculations. However, it is not sure whether an instance of this AMI is already running or whether it is required to start a new one. The BPEL process presented below can be used to automatically do this check.

BPEL workflow with EC2

The process requires AWS credentials (user id, key, secret key), a keypair to use and an image location (on Amazon S3) as input.
At first, it retrieves the EC2 imageId for the given AMI location by querying listImages.
Then, it checks whether an instance of this AMI is already running.
If so, it returns the EC2 instanceId of the first instance of this AMI.
Otherwise an instance of the given AMI is started by invoking the runInstance function. In this case, the instanceId of the newly created instance is returned to the client.

Although the sample process does only invoke functions from the Cloud42BaseService part of the Cloud42 Web service interface, it shows the power of this approach. More complex BPEL processes can be developed using the enhanced features of Cloud42 like uploading files or even executing arbitrary commands on AMI instances. This way, orchestrating EC2 becomes easy and convenient.

You can download the illustrated process as a JDeveloper project from here. The package contains a readme-file with further information.

Note: The BPEL example was designed to work with Cloud42 v1.1.0 and was not tested with newer versions.