MATLAB® Distributed Computing Server™ Cloud Center User’s Guide
How to Contact MathWorks Latest news: www.mathworks.com Sales and services: www.mathworks.com/sales_and_services User community: www.mathworks.com/matlabcentral Technical support: www.mathworks.com/support/contact_us Phone: 508-647-7000 The MathWorks, Inc. 3 Apple Hill Drive Natick, MA 01760-2098 MATLAB® Distributed Computing Server™ Cloud Center User’s Guide © COPYRIGHT 2012–2015 by The MathWorks, Inc. The software described in this document is furnished under a license agreement.
Revision History March 2012 September 2012 November 2012 March 2013 December 2013 February 2014 March 2014 October 2014 November 2014 February 2015 May 2015 Online only Online only Online only Online only Online only Online only Online only Online only Online only Online only Online only
Contents 1 Cloud Computing Create and Manage Cloud Clusters with MathWorks Cloud Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Access Your Cloud Center Account . . . . . . . . . . . . . . . . . . . . Cloud Center Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Your AWS Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . Create a Cloud Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . View Your Clusters . . .
1 Cloud Computing • “Create and Manage Cloud Clusters with MathWorks Cloud Center” on page 1-2 • “Access Cloud Cluster from MATLAB” on page 1-18 • “Transfer Data to or from a Cloud Cluster” on page 1-21
1 Cloud Computing Create and Manage Cloud Clusters with MathWorks Cloud Center In this section...
Create and Manage Cloud Clusters with MathWorks Cloud Center 2 Enter your MathWorks Account email address and password. 3 Click Log In. 4 The first time you log in, you must read and accept the Cloud Center Software License Agreement and the Terms of Use. To proceed, select I Accept for each, and click Continue.
1 Cloud Computing 2 Enter data for the following fields according to your AWS account setup. For your particular AWS information, see your administrator who set up your Amazon Web Services account. It is possible that several people will be using the same AWS account and credentials, even if using separate MathWorks accounts to access Cloud Center. • AWS Access Key ID. • AWS Secret Access Key. • Description — You can enter any text here for a description of your account or credentials.
Create and Manage Cloud Clusters with MathWorks Cloud Center • Machine type: types vary by hardware specification including number of cores, memory, and GPU support. • MATLAB version. • An optional timeout for the cluster so that it shuts down automatically: • When cluster is idle: When the cluster no longer has any jobs to process, it will eventually shut itself down after a few minutes if no more work is submitted. This option is available only on clusters of MATLAB version R2014a and later.
1 Cloud Computing about downloading the user access key file, see “Download SSH Key Identity File” on page 1-22. The settings for a standard 32-worker cluster with a 2-hour time limit might look like this: 3 Click Advanced Options to specify further optional details: • Number of workers per node. • Persisted storage space. For more information on persisted storage, see “Cluster Shared File System” on page 1-10. • Operating system image.
Create and Manage Cloud Clusters with MathWorks Cloud Center The full set of settings for a cluster might look like the following. With 16 workers per node, this 32-worker cluster requires only two nodes. 4 Click Create Cluster to create and start your cluster nodes. The cluster starts a number of nodes (instances) determined by your choices of number of workers and workers per node.
1 Cloud Computing It can take up to several minutes for a cluster to completely start up, with the status indicating the particular stages of the process. 5 You can click More Details to see further information about your cluster, including any status messages. To get further status information on any individual cluster node, click the appropriate Headnode or Worker expanders. When the cluster is started and ready for use, the Cloud Center indicates the cluster status as Online.
Create and Manage Cloud Clusters with MathWorks Cloud Center View Your Clusters You can have more than one cluster, some running (online) and some shut down (offline). Click My Clusters to see a list of your clusters. The following listing shows a pair of clusters, one currently online and ready, the other offline: For detailed information about a particular cluster, click its name in the list.
1 Cloud Computing Cluster Shared File System When you create and start a cluster, shared folders are created for the storage of your data. These folders are shared by all the nodes of your cluster, but have different longevity, according to the following table. The path is the same for all nodes of the cluster. Location Size Longevity /shared/persisted Specified at cluster creation Deleted when cluster is deleted /shared/tmp 1.
Create and Manage Cloud Clusters with MathWorks Cloud Center Cluster Access and Security Groups Set Cluster Access Cluster Access settings control which computers can access your Amazon cluster from the Internet. To access the Cluster Access setting in the Cloud Center, click Cluster Access beneath Preferences in the navigator. Cluster access comprises a list of IP ranges for the computers that can access your cloud cluster.
1 Cloud Computing 123.123.0.0/16 Caution Make your address formats as strict as possible, using as many fields and bits as you can. Address formats that are too open can increase the risk of unauthorized access to your cluster. A format that uses no bits for matching (e.g., 0.0.0.0/0, or 123.123.234.56/0) allows all machines on the Internet to access your cluster. The computer you are currently accessing the Cloud Center from is automatically added to the access list.
Create and Manage Cloud Clusters with MathWorks Cloud Center Server Sockets Accessed by Client MATLAB Distributed Computing Server™ and the Cloud Center require the client to contact servers listening on certain sockets within the cloud.
1 Cloud Computing AWS Identity and Access Management (IAM) If you are the administrator setting up Amazon Web Services (AWS) groups and user accounts, each user needs particular policies for using MATLAB Distributed Computing Server EC2 clusters. You can define a group of users, and attach the policies to that group.
Create and Manage Cloud Clusters with MathWorks Cloud Center 8 Click Create Policy at the bottom of the page. You now have a policy to apply to your groups. 9 In the left pane, click Groups. 10 Click Create New Group. 11 Specify a group name. For example, CloudCenterUsers. 12 Click Next Step at the bottom of the page. 13 Now on step 2 (Attach Policy), use the search field to find the policy you just created (CloudCenterUserPolicy or whatever name you gave it).
1 Cloud Computing Create an Amazon Machine Image (AMI) You can use an Amazon Machine Image (AMI) when creating a cloud cluster. To create your custom AMI for this purpose, you modify an AMI provided in Cloud Center. Then you can install drivers, libraries, or other utilities, so that they are available for subsequent clusters without having to reinstall them each time. Use the following procedure to create and customize a cluster AMI in the Amazon EC2® Dashboard of the AWS Management Console.
Create and Manage Cloud Clusters with MathWorks Cloud Center 11 Modify or add a security rule with the following settings: Type SSH Protocol TCP Port Range 22 Source My IP 12 Click Review and Launch. If you see a dialog box asking about booting from General Purpose (SSD), select your preferred option and click Next. 13 If everything looks correct in the review dialog box, click Launch. 14 You will be asked to select a key pair. You can use an existing key pair that you have access to.
1 Cloud Computing Access Cloud Cluster from MATLAB In this section... “Import Cluster Profiles” on page 1-18 “Create Job on Cloud Cluster Using Command-Line Access” on page 1-19 When your cloud cluster is running, you access it and workers in much the same way you access a cluster in your own on-site network. Through cluster profiles or parallel computing functions, you identify available clusters for creating and submitting jobs.
Access Cloud Cluster from MATLAB With your cloud cluster profile set as your default, you can now run parallel computing applications on the cloud with functions such as parpool (and therefore parfor and spmd), batch, createJob, etc.
1 Cloud Computing Number Finished: 0 - MJSComputeCloud Specific Properties Name: State: NumBusyWorkers: NumIdleWorkers: MyCluster online 0 32 Now you can use the cluster object to create jobs and tasks in the usual manner. For example: parpool(c,8) or createJob(c) Use Downloaded Profile for Cloud Cluster As an alternative to fetchCloudClusters, you can use the a profile downloaded from the Cloud Center to identify the cluster you want to access.
Transfer Data to or from a Cloud Cluster Transfer Data to or from a Cloud Cluster In this section...
1 Cloud Computing Data that is stored in job and task properties is available to the client, so your task or batch function results are accessible from the finished job’s fetchOutputs function or the tasks’ OutputArguments property. For batch jobs that run on the cloud, you can access the job’s workspace variables with the load function in your client session. Download SSH Key Identity File Among the cluster information in Cloud Center is the SSH key for that cluster, for nonroot user access.
Transfer Data to or from a Cloud Cluster SFTP The sftp utility is a command-line interactive interface, similar to ftp, that lets you connect to a remote host, navigate its file system, and transfer files. The following example shows how to use sftp at a UNIX command prompt: cd /home/cloudtmp sftp -i /home/.ssh/your-key.pem \ clouduser@ec2-67-202-5-207.compute-1.amazonaws.com:/shared/persisted sftp> put emem.mat sftp> ls emem.
1 Cloud Computing 3 Do not provide a password, but instead provide your SSH key identity file under Edit > Settings. In the Select pane of the Settings dialog box, choose SFTP. In the Public Key Authentication pane, click Add keyfile. Navigate to the key file that you downloaded from the Cloud Center for this cluster. (Note: On Windows, the .
Transfer Data to or from a Cloud Cluster The -remotehost name is available in Cloud Center under the details for the head node of a running cluster. 2 With the data files in place on the cloud cluster nodes, you can specify their location in the job’s AdditionalPaths property to provide access to them for the MATLAB workers. Retrieve Data from the Cloud This example shows how to copy the file /shared/persisted/emem.mat from a cloud cluster node to a local UNIX machine as /home/cloudtmp/return_emem.mat.
1 Cloud Computing 4 Search for your snapshot: • In the Filter list, select Owned By Me. • In the Search Snapshots field, enter your cluster name from the Cloud Center. • Sort the Started column in descending order. 5 In the lower half of the page, review the Tags for the top result in the list, and verify that the ClusterInfo value has the correct cluster name. For example, the result when your cluster name is MyR12b might look like this: MyR12b / first.last__AT__company.
Transfer Data to or from a Cloud Cluster • In the Snapshot field, enter the snapshot ID you copied earlier; for example, snap-20cd6642. • In the Size field, enter a value equal to the size of the snapshot; for example, 100 GiB. 5 (optional) On the Tag Instance tab, give the instance a Name value so you can more easily find the instance in the Amazon Management Console. 6 On the Configure Security Group tab, use a security group to define firewall rules for your instance.
1 Cloud Computing 3 Create a volume from your snapshot: a Click Actions > Create Volume. b Set the Availability Zone to match that of your instance. You can accept the defaults for the other settings. c A confirmation indicates that the volume was successfully created. Note the volume ID, for example, vol-8a9d6642). Wait until the state of your volume is Available. 4 In the left-side navigation pane of the EC2 Dashboard, select Elastic Block Store > Volumes.
Transfer Data to or from a Cloud Cluster 4 Use the following command to mount the volume at the location you just created. $ sudo mount device_name mount_point For example, $ sudo mount /dev/xvdf /data For more information on using EBS volumes, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html.
