Flintrock - command-line tool for launching Apache Spark clusters

Ruslan Osipov
Author: Ruslan Osipov

What is Flintrock

flintrock command-line tool

Flintrock is a command-line tool for launching Apache Spark clusters

Usage

Flintrock works best with Amazon Linux.

One way to establish a cluster on EC2 is by saying: flintrock launch test-cluster

If you persist options to a file, you can do the same thing:

Once you’re done using the cluster, destroy it with "flintrowDestroyTestClustered.cfg"

Other things to do:

  • Flintrock login

  • describe

  • add slaves

  • remove slaves

  • run command

  • copy-file

  • etc

Accessing data on S3

Setup an IAM Role that grants access to S3 as desired

Reference this role when you launch your cluster using the "--ec2-instance-profile-name" option and reference S3 paths in your Spark code using the "s3a:// prefix"

Call Spark with the Hadoop-aws package to enable s3 a://

Installation

Flintrock requires Python 3.7 or newer unless you use one of our standalone packages.

Standalone version

If you don’t have a recent enough version of Python, or if you don't have Python installed at all, you can still use Flintrock.

Publish standalone packages of Flintrock on GitHub

Unzip the standalone package, unzip it to a location of your choice, and run the flintrock executable inside

Community-supported distributions

Flintrock is also available via the following package managers: Homebrew:

"brew install flintrock"

Automated Pipelines

Flintrock is designed to be used as part of an automated pipeline.

Managing permanent infrastructure

Flintrock is not for managing long-lived clusters or any infrastructure that is a permanent part of some environment.

If looking for ways to manage permanent infrastructure, look at tools like Terraform, Ansible, SaltStack, or Ubuntu Juju.

Launching non-Spark-related services

Flintrock is meant for launching Spark clusters that include closely related services like HDFS, Mesos, and YARN.

Configurable CLI Defaults

Flintrock lets you persist your desired configuration to a YAML file, so you don't have to keep typing options in the command line.

Flintrock'se typical launch time will be a minute or two longer.

Flintrock is a single-purpose tool with minimal focus.

Repository: https://github.com/nchammas/flintrock

Video:

FAQs

What is Flintrock?

Flintrock is a command-line tool for launching and managing Apache Spark clusters.

Who can use Flintrock?

Flintrock is designed for developers and data scientists who work with Apache Spark and need to streamline their cluster management.

What are the benefits of using Flintrock?

Using Flintrock can increase efficiency, flexibility, and reduce manual tasks in Apache Spark cluster management.

How do I install Flintrock?

You can install Flintrock on a Linux or macOS machine using pip, the Python package manager.

What are the prerequisites for installing Flintrock?

You will need Python 3.6 or higher and an AWS account with the necessary permissions.

How do I launch an Apache Spark cluster with Flintrock?

You can use the 'flintrock launch' command to launch a cluster with your desired configuration.

Can I configure my Flintrock cluster?

Yes, you can use the 'flintrock configure' command to customize your cluster's settings, such as the number of nodes and instance types.

How do I connect to my Flintrock cluster?

You can use the 'flintrock ssh' command to connect to your cluster via SSH.

Can I terminate my Flintrock cluster when I'm done?

Yes, you can use the 'flintrock destroy' command to terminate your cluster and avoid additional charges.

How can I get help with using Flintrock?

You can consult the official Flintrock documentation or join the Flintrock community on GitHub to ask questions and get support.

Related articles