SELF-SERVICE, TEMPLATE-BASED CLUSTER PROVISIONING SYSTEM
At Cask, we needed a fast, self-service way to provision clusters, because testing against real clusters is such a critical and frequent component of our development cycle. We started off with simple scripts. But scripts had multiple issues—they required a lot of setup, there was no central management keeping track of available clusters, and scripts had to be changed whenever we moved from one cloud to another, or one software deployment to another. Next, we started looking at different SCM (software configuration management) technologies and picked Chef. Combining scripts and Chef recipes, we were able to create a cluster by running a single script, but it became extremely complex for two reasons: there were an increasing number of customizations needed for each new type of cluster, and a significant amount of setup was needed on every developer’s box.
When we launched our CDAP product, we wanted to enable developers to deploy our standalone Sandbox CDAP to a public cloud directly from the website. This forced us to think harder about designing a complete production system while still providing an easy and flexible solution for our developers. We wrote a combination of scripts and Chef recipes, designed a system that can templatize the Hadoop cluster, added capabilities to make it extendable to other services and providers, and made it very developer and ops friendly—and that’s when Coopr was born.
Today, Coopr is used internally as a self-service cluster provisioner for our developers, as a CI tool for integration testing, and as a DevOps tool to test new and incremental features. We’ve been developing and using Coopr for over a year, and we’re noticing the benefits: our developer and DevOps productivity have doubled, and our development and test cycles have become faster.
We built Coopr for ourselves, but as we believe that the broader developer community can benefit from it, we’ve made it available for everyone to use.
Why Coopr ?
Coopr simplifies the installation and configuration of any software stack, including Hadoop, and ensures that all installations are verified before a cluster is made available. Developers can create custom cluster types ranging from single VM application servers to large-scale, distributed, heterogeneous clusters with the click of a mouse.
Coopr enables the creation of custom cluster templates ranging from Hadoop to LAMP to Solr search clusters. Templates define rules describing how hardware, images, and services should be laid out in a cluster, leaving Coopr to solve and coordinate which tasks need to be done—and in which order—to create a cluster. Services are automatically placed on the correct nodes and are configured, initialized, and started in the correct order. In this way, the same template used to create a three-node Hadoop cluster with only HDFS and YARN can also be used to create a 300-node, kerberos-enabled cluster with HDFS, YARN, and HBase.
Coopr integrates with various IaaS providers (currently AWS, GCE, Openstack, Rackspace, and Joyent) and SCM tools (currently just Chef) out of the box. Coopr has a pluggable architecture, so adding a provider or SCM tool can be done by writing a plugin. If your provider or SCM tool is not supported out of the box, simply implement a plugin for it (and contribute back!) Switching between supported providers is even simpler: you simply select a different provider from a drop-down menu.
Coopr is multi-tenant, allowing you to create multiple tenants with their own administrators, users, templates, and clusters. Limits on clusters and nodes can be set for each tenant. Lease times can be set on clusters, which will auto-delete them after some time, preventing leaking resources. Admins can manage different versions of scripts, cookbooks, keys, and other resources, giving them control over how new code is pushed out or rolled back. Admins can also see all clusters, who owns each cluster, when they were created, and other useful information.
Run it in a VM
We have created a sandbox VM with Coopr pre-installed and initialized with several cluster templates that you can download. Follow the Quickstart Guide, which steps you through an example of adding workers to your tenant, configuring a provider, and creating a Hadoop cluster.
Run it on your Mac
After downloading the standalone version of Coopr, unzip it and follow the instructions in the Standalone’s README.md to start up Coopr on your own machine. It comes pre-packaged with templates for Hadoop, LAMP, and MongoDB. In order to create an actual cluster, you will need an account with a supported provider (AWS, GCE, Openstack, Rackspace, and Joyent), and perform a couple steps to setup Coopr to use the correct credentials to integrate with your provider. Follow the Quickstart Guide, which steps you through an example of adding workers to your tenant, configuring a provider, and creating a Hadoop cluster.
Build it from Source
$ git clone http://github.com/caskdata/coopr.git $ cd coopr/coopr-standalone $ mvn clean package assembly:single
The build places a zip of the standalone version of Coopr into the target directory.
- User Guide – Describe how to install and setup Coopr, as well as how administrators and users can use Coopr
- Installation Guide – Instructions for installing Coopr
- Tech Guide – How Coopr works
- REST Endpoints – Documents REST APIs for managing your Coopr instance
- Javadocs – Server Java documentation
There have been three releases of Coopr. You can see full details on the repository’s releases page. Release highlights are summarized below:
- Docker support, ability to deploy Docker services and manage Docker containers
- Provider plugin for Digital Ocean added
- New UI, rewritten to be more modular and extensible
- CLI tool, allows interaction via simple commands
- Splits the Server APIs into internal and external APIs, served on different ports
- Node usage metrics, provides detailed metrics by tenant, user, or template
- Significant bug and stability fixes to provisioner and server
Release 0.9.8 – Multi-tenancy themed:
- Multi-tenancy introduced
- Added the ability for tenant admins to manage their own plugin resources, such as scripts and chef cookbooks
- Added AWS and GCE providers
Release 0.9.7 – Extensibility themed:
- Expanded service dependencies
- Added the ability for users to add, start, stop, restart, and configure services on an existing cluster
- Added pluggable cluster callbacks, allowing custom logic before and after cluster operations
Interested in helping to improve Coopr? We welcome all contributions, whether in filing detailed bug reports, submitting pull requests for code changes and improvements, or by asking questions and assisting others on the mailing list.
We have a simple pull-based development model with a consensus-building phase, similar to Apache’s voting process. If you’d like to help make Coopr better by adding new features, enhancing existing features, or fixing bugs, here’s how to do it:
- If you are planning a large change or contribution, discuss your plans on the coopr-dev mailing list first. This will help us understand your needs and guide your solution in a way that best fits the project.
- Fork Coopr into your own GitHub repository.
- Create a topic branch with an appropriate name.
- Work on the code to your heart’s content.
- Once you’re satisfied, create a pull request from your GitHub repo (it’s helpful if you fill in all of the description fields).
- After we review and accept your request, we’ll commit your code to the repository.
Thanks for helping to improve Coopr!