Run Apache Spark SQL workloads on Azure Cobalt 100 Arm64 using Gluten and Velox for accelerated analytics
Introduction
Understand Azure Cobalt 100 and Apache Spark with Gluten and Velox
Create an Azure Cobalt 100 Arm64 virtual machine
Deploy Apache Spark SQL with Gluten and Velox on Arm64
Run TPC-DS Benchmark on Spark with Gluten and Velox on Arm64
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

Prerequisites and setup

You can create an Arm-based Cobalt 100 virtual machine using any of the following methods that best fits your workflow or requirements:

  • The Azure Portal
  • The Azure CLI
  • An infrastructure as code (IaC) tool

In this section, you’ll launch the Azure Portal to create a virtual machine (VM) with the Arm-based Azure Cobalt 100 processor.

This Learning Path focuses on general-purpose virtual machines in the Dpsv6 series. For more information, see the Microsoft Azure guide for the Dpsv6 size series .

While the steps to create this instance are included here for convenience, for detailed steps, see the Deploy a Cobalt 100 virtual machine on Azure Learning Path . To learn more about Arm-based virtual machines in Azure, see “Getting Started with Microsoft Azure” in Get started with Arm-based cloud instances .

Create an Arm-based Azure virtual machine on the Azure portal

To create an Azure virtual machine:

  1. Launch the Azure portal and navigate to Virtual Machines.
  2. Select Create, and select Virtual Machine from the drop-down list.
  3. Inside the Basic tab, enter the instance details such as Virtual machine name and Region.
  4. Select the image for your virtual machine (for example, Ubuntu 22.04 LTS Server) and select Arm64 as the VM architecture.
  5. In the Size field, select See all sizes and select the D-Series v6 family of virtual machines.
  6. Select D8ps_v6 from the list as shown in the following diagram:

Image Alt Text:Azure Portal showing D-Series v6 VM size selection with D8ps_v6 highlightedSelect D8ps_v6 from the D-Series v6 family

  1. For Authentication type, select SSH public key.
Note

Azure generates an SSH key pair for you and lets you save it for future use. This method is fast, secure, and easy for connecting to your virtual machine.

  1. Fill in the Administrator username for your VM.
  2. Select Generate new key pair, and select RSA SSH Format as the SSH Key Type.
Note

RSA offers better security with keys longer than 3072 bits.

  1. Give your SSH key a key pair name.
  2. In the Inbound port rules, select HTTP (80) and SSH (22) as the inbound ports, as shown in the following screenshot:

Image Alt Text:Azure Portal showing inbound port rules with HTTP (80) and SSH (22) selectedConfigure inbound port rules for HTTP and SSH access

  1. Select the Review + Create tab and review the configuration for your virtual machine. It should look like the following:

Image Alt Text:Azure Portal Review + Create tab showing VM configuration summary ready for deploymentReview VM configuration before creation

  1. After reviewing the configuration, select the Create button and then the Download Private key and Create Resource button.

Image Alt Text:Azure Portal showing Create button and SSH key download dialogDownload SSH key and create the virtual machine

Your virtual machine should be ready and running in a few minutes. You can SSH into the virtual machine using the private key, along with the public IP details.

Image Alt Text:Azure Portal showing successful VM deployment with confirmation detailsSuccessful VM deployment confirmation

What you’ve learned and what’s next

You’ve now successfully created an Azure Cobalt 100 Arm64 virtual machine running Ubuntu 22.04 LTS Server with SSH authentication configured. The VM is now fully prepared for running distributed data processing workloads.

Next, you’ll build a high-performance Spark SQL analytics platform using modern acceleration technologies.

Back
Next