Chuckwagon


Chuckwagon is a Scala/sbt AWS Lambda Toolkit. It makes creating and maintaining Continuous Delivery pipelines typesafe and declarative.

Chuckwagon provides all of the following features from within a standard sbt project

The documentation introduces all of these features incrementally in a series of guides so that you can learn how they compose with each other to produce the most powerful pipelines,

All of the above and other optional features are further documented in the extensive Reference section.

Chuckwagon is maintained by @caoilte_oconnor - you may remember him from such inspirational youtube videos as "live-coding introduction to Chuckwagon for colleagues at ITV",



Getting Started


This guide will walk you through the steps required to create a Helloworld AWS Lambda using the Chuckwagon Library and then upload it to AWS using the Chuckwagon sbt plugin. We will begin by configuring your build.

Configuring the Build

In order to create your AWS Lambda you will need to have AWS credentials. For this introduction those credentials will need to grant read and write access to EC2, IAM and Lambda. By default Chuckwagon will use the AWS SDK default credentials lookup process to find your credentials.

One easy way to provide your credentials is to setup your ~/.aws/credentials file as below, with the appropriate settings filled in for <LABELLED_SECTIONS>. 

[default]
region = <YOUR_REGION>
aws_access_key_id=<YOUR_ACCESS_KEY>
aws_secret_access_key=<YOUR_SECRET_KEY>
To install the Chuckwagon sbt plugin add the following line to the project/plugins.sbt file in your Lambda project. 
addSbtPlugin("com.itv.chuckwagon" % "sbt-chuckwagon" % "0.1.0")
Next add the following configuration to your Lambda project's build.sbt file with appropriate settings filled in for <LABELLED_SECTIONS>. 
scalaVersion := "2.12.1"
libraryDependencies ++= "com.itv.chuckwagon" %% "chuckwagon-jvm" % "0.1.0"
enablePlugins(ChuckwagonPublishPlugin)
chuckRegion := "<AN_AWS_REGION_EG_-_eu-west-1>"
chuckPublishConfig := chuckPublishConfigBuilder
  .withName("<THE_NAME_YOU_WANT_FOR_YOUR_LAMBDA>")
  .withHandler("Helloworld::handleRequest")
  .withMemorySizeInMB(192)
  .withTimeout("5 seconds")
  .withStagingBucketName("<THE_S3_BUCKET_WHERE_CHUCKWAGON_WILL_UPLOAD_YOUR_CODE")
  .withCodeFile(assembly)
You now have the minimum build configuration required for the rest of this Getting Started Guide. You can start sbt and open the project in your favourite IDE.

Helloworld Function

Create /src/main/scala/Helloworld.scala and paste the following code into it, 
import com.amazonaws.services.lambda.runtime.Context
import com.itv.chuckwagon.lambda._
import io.circe.generic.auto._

case class Response(response: String)

class Helloworld extends Handler[Unit, Response] {
  def handler(query: Unit, context: Context): Response = {
    Response(s"Hello World!")
  }
}

This is the simplest possible code for creating an AWS Lambda using Chuckwagon. It extends a Generic Handler that is basically the same as the Underscore Scala and AWS Lambda Blueprints. It takes no arguments but does return a Case Class that Chuckwagon turns into the following JSON Response payload 

{
  "response" : "Hello World!"
}
Note how the configuration .withHandler("Helloworld::handleRequest") in the build.sbt references an AWS Lambda Class and method name that is nearly the same as the Helloworld Class we created. The only difference is that the method configured to be invoked by the build.sbt is the underlying Library method handleRequest which will turn the raw request into the Case Class (or in this case Unit object) that we expect.

Creating a Lambda

We are finally ready to try out out our Helloworld AWS Lambda with the following Task,
chuckPublishSnapshot - Create/Update Lambda

The first time you run this task it will do the following,

  1. Compile the Helloworld.scala file.
  2. Create a fat JAR out of it and all of its dependencies using the sbt-assembly plugin.
  3. Upload your Assembly JAR to the S3 Bucket that you specified in the chuckPublishConfig.
  4. Create an IAM Role with the appropriate permissions for running the Lambda.
  5. Create a Lambda configured to use
    • the runtime configuration that you provided in chuckPublishConfig
    • a copy of the Assembly JAR in the S3 Bucket
    • the IAM Role with appropriate permissions for running the Lambda
  6. Print the ARN (Amazon Resource Name) that will be the uniquely addressable location of the newly created AWS Lambda.
Run the task. It should print something like, 
[info] Chuckwagon: Just Published Snapshot as 'arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:<LAMBDA_NAME>'
You can test your Helloworld AWS Lambda by running the following sbt Task,
chuckInvoke - Execute latest Snapshot Code/Configuration and print response
It will use the ARN for the newly created AWS Lambda and should print something like, 
[info] Chuckwagon: <LAMBDA_NAME>: About to invoke Lambda
[info] Chuckwagon: <LAMBDA_NAME>: Result of running Lambda '{"response":"Hello World!"}'

There's nothing too surprising about this output and even if you update your Lambda to do something more useful, running it from sbt isn't very practical for every day use. This task is, however, a great way of demonstrating the Library and Plugin that we shall continue to use in later Guides.

Updating a Lambda

Update your Helloworld.scala file to say something else and then run chuckPublishSnapshot. This time the task will do the following,
  1. Recompile the Helloworld.scala file with your new string.
  2. Create a new Assembly JAR and over-write the previous JAR in S3 with it.
  3. Check that the IAM Role still exists and still has the appropriate permissions. Recreate it or modify it if necessary.
  4. Update the existing Lambda to use
    • the runtime configuration that you provided in chuckPublishConfig
    • a copy of the updated Assembly JAR in S3
    • the appropriate IAM Role
Running chuckInvokeLambda now will print the new string that you updated Helloworld.scala with. You can safely run chuckPublishSnapshot as many times as you want because it is idempotent (rather like a REST PUT). It will also always make sure that the created or updated Lambda exactly matches the contents of chuckPublishConfig.

This explanation of chuckPublishSnapshot concludes the Getting Started Guide but it barely scratches the surface of what is possible in AWS Lambda with Chuckwagon. Keep reading through the Guides to discover where you can take Chuckwagon next.

Guides


The Getting Started Guide introduced enough of Chuckwagon to create/update your first AWS Lambda. The following Guides cover more complete examples of how you might manage the more sophisticated setups required to configure/deploy your AWS Lambda in larger organisations. Additional concepts introduced include managed release processes, gated tests and multiple accounts.

Deployment Pipelines


Chuckwagon provides the building blocks for incorporating your entire AWS Lambda deployment pipeline into the same Scala project that builds it. This guide will introduce those building blocks and show you how to assemble them into a very basic release/deployment pipeline. It picks up straight after the Getting Started guide.

Versioning

The chuckPublishSnapshot Task is a great technology demonstrator but has no place in the deployment pipeline or production environment for a reliable system. Introducing a requirement for it is equivalent to writing a program that depends on the HEAD reference of another git repository or the SNAPSHOT version of a Java/Scala Library, ie a system that it is impossible to reliably recreate. If the rest of your system refers to a Lambda via the ARN returned by chuckPublishSnapshot you can never be certain what code will be executed.

In order to have a fixed reference to code and configuration on a Lambda that can be guaranteed never to change we need to publish a version of that Lambda. That can be achieved with the Task,

chuckPublish - Create/Update Lambda and create an immutable uniquely versioned copy

This Task will do everything that chuckPublishSnapshot did, but will also carry out the non-idempotent operation of making a readonly versioned copy of the Lambda which can be uniquely referred to via its own ARN. This means two ARNs get created/updated as a result of running chuckPublish,

The VERSION created will be 1 the first time chuckPublish is called, then 2 etc. At any time you can see the list of currently published AWS Lambdas from within the sbt shell by running,

chuckCurrentlyPublished - The currently published versions of this Lambda (if Lambda exists)

You can easily satisfy yourself that Versioned Lambdas are different to each other by making a few changes to your Helloworld Function from the Getting Started Guide and publishing it. The chuckInvoke task takes a number as an optional argument which it will use to run a specific version of your Published AWS Lambda.

The downside to relying solely on the Versioned Lambda is having to update every place that refers to it every time there is a new version. Updating lots of things is fine when it only changes every few months, but is un-manageable when it could change many times in a single day. In order to build a pipeline that treats every publish we do as a candidate for Production we need the ability to associate a specific Lambda version with an Environment.

Environments

The chuckPublishTo Input Task takes the chuckPublish Task and adds a tiny amount more functionality on the end. In addition to creating a versioned copy of your updated Lambda it makes sure that the new version is promoted to an environment (eg qa).

However, before you can publish to an environment you need to define one. Here is an example of how to define your Chuckwagon environments in sbt. (Remember to reload sbt after adding this sbt Setting.)

chuckEnvironments := Set[String]("qa", "prd") - Define qa and prd as valid Chuckwagon environments (referred to as aliases in the AWS Console).

Please note that defining the chuckEnvironments sbt Setting does not automatically lead to them being created as aliases in AWS. Chuckwagon will wait until environments are used before creating them.

With valid Chuckwagon environments defined in sbt it now becomes possible to use the chuckPublishTo Input Task

chuckPublishTo <ENVIRONMENT> - Create/Update Lambda, then create an immutable uniquely versioned copy of it and promote that to <ENVIRONMENT>

chuckPublishTo is the most useful variant of the Publish tasks as it returns an ARN that can safely be referred to from other parts of your infrastructure, but which is associated with a specific version of Lambda configuration and code. In total that makes three ARNs which get created/updated as a result of running chuckPublishTo,

Just as with specific versions, you can try out Lambdas in a specific environment using the chuckInvoke task, which in addition to taking version numbers as input arguments also takes environments.

Promotion

It would be possible to use chuckPublishTo to publish to each environment in turn, but this would mean a different version of the lambda gets deployed to QA than gets deployed to production. This is undesirable. What we want is an Input Task that can promote an existing version from one environment to another
chuckPromote <FROM_ENVIRONMENT> <TO_ENVIRONMENT> - Promote the Lambda Version currently in <FROM_ENVIRONMENT> to also be referenced by <TO_ENVIRONMENT>

It is important to note that although we talk about 'promoting' the Lambda all we are physically doing in AWS is editing the destination alias to point at the same Lambda version as the source alias. After the promotion, the code/configuration that gets run in the two environments is exactly the same. The only difference is the ARN used to call the Lambda. This means that if you want your Lambda to know which environment it is being executed in you need to inspect the ARN that is passed to it during execution.

sbt-release

Chuckwagon doesn't need to provide a mechanism for building your actual pipeline, because sbt already has a brilliant plugin called sbt-release. This plugin allows you to define a release process in sbt out of any existing sbt Tasks, Input Tasks or Commands. Consider the following custom release process, 
releaseProcess := Seq[ReleaseStep](
    releaseStepTask(clean),
    releaseStepTask(test in Test),
    releaseStepInputTask(chuckPublishTo, " qa"),
    releaseStepTask(test in IntegrationTest),
    releaseStepInputTask(chuckPromote, " qa prd").
    releaseStepTask(chuckCleanUp)
)

This release process uses the Tasks we introduced earlier in this guide to build a deployment pipeline that,

  1. Runs the normal unit tests for your project (your Continuous Integration)
  2. Builds a fat JAR and deploys it to qa
  3. runs the Integration Tests (which in this very simple pipeline are expected to execute tests against the Lambda version in the qa environment via the associated ARN)
  4. If the integration tests pass then promote the version in qa to prd
  5. chuckCleanUp is a Task we haven't previously introduced which deletes any versions no longer referenced by an environment (ie the version that was previously in prd).

This very simple pipeline demonstrates how you can very easily use Chuckwagon to run your entire deployment pipeline for an AWS Lambda project. In the Multiple AWS Accounts guide we consider a more realistic scenario where your organisation insists on doing Production Deployments using a different AWS Account.

Environment Configuration


Several times in the Deployment Pipelines Guide we mentioned that even though you can deploy a Lambda to more than one environment it still only exists in one place. This is because what you are actually doing is pointing two or more AWS Aliases at the same Lambda. This is an efficient feature because it means that Amazon doesn't have to create multiple copies of the JAR file / Lambda Configuration metadata that isn't allowed to change in any case. It does, however, highlight a challenge in AWS. How do you apply environment specific configuration to an AWS Lambda?

Dead Ends

Traditionally when configuring a Service you deploy an environmental configuration file along side it on the box it is deployed to. This configuration file contains things like the location of that service's database in the associated environment. There is no built in way to associate an environmental configuration file with an AWS Lambda.

AWS Lambda provides a feature called 'Environment variables' that do at first glance look suitable for this purpose. However, AWS Lambda environment variables cannot be configured per environment. They appropriate the term environment from shell scripting, where variables can be passed to scripts by setting them as global constants on the shell. AWS Lambda doesn't run in such a shell/environment. Once you version a Lambda its environment variables cannot be changed. They are intended to be used to save time changing configuration that would otherwise require a code deployment, but this is a very different thing to the flexibility of environment variables in a shell scripting environment.

Consequently, since code deployments are very fast and accessing AWS Lambda environment variables is necessarily cumbersome they are not obviously ideal for any use. It is possible that they would be useful for encrypted fields. Chuckwagon does not yet support such functionality.

Environment Detection

The only way to manage Environment specific variables in AWS Lambda is to take advantage of the fact that when your Lambda is executed it knows which ARN was used to invoke it. The environment is the final section of the ARN. For example,

arn:aws:lambda:eu-west-1:123456789012:function:demo:qa - QA arn:aws:lambda:eu-west-1:123456789012:function:demo:prd - PRODUCTION
The ARN is passed to the Lambda Handler in the Context and so needs to be detected each time that the Lambda executes.

Deriving Configuration

You could use the knowledge of which environment you are in to lookup configuration on an external system (for example Consul). This might well make sense for configuration produced very frequently by automation or sensitive configuration that needs to be encrypted (especially if your infrastructure team already has such a setup). However, for most configuration the ideal way to store it is as typed classes inside your own code base. Given that it should be possible to push a Lambda from check-in to Production in a couple of minutes, even the time to change configuration is negligible.

The Chuckwagon Library makes configuration stored inside the Lambda codebase simple to lookup on a per environment basis. Consider the following changes to the Helloworld.scala Handler introduced in the Getting Started guide, 

import com.amazonaws.services.lambda.runtime.Context
import com.itv.chuckwagon.lambda._
import io.circe.generic.auto._

case class Response(response: String)

case class HelloConfig(env: String, someServiceLocation: String) extends LambdaConfigForEnv

object Helloworld {
  val configs = LambdaConfig(
    HelloConfig(env = "qa", someServiceLocation = "testLocation"),
    HelloConfig(env = "prd", someServiceLocation = "prdLocation")
  )
}

class Helloworld extends Handler[Unit, Response] {
  def handler(query: Unit, context: Context): Response = {

    val config = Helloworld.configs.configFor(context)

    Response(s"Hello World from ${config.env}!")
  }
}
This code now does a number of interesting things in addition to printing 'Hello World', This example code is really all you need in order to add per environment configuration to Lambdas deployed into an Environment. It is extremely simple, but also extremely powerful.

Multiple AWS Accounts


Many organisations using AWS take the very sensible precaution of putting non-production environments in an entirely different AWS Account. This protects the Production Account from fat fingered development but has the side-effect of making a single deployment pipeline very difficult. The usual compromise is to have two deployment pipelines, one in the development environment and one in the production environment. The hand-off between the two environments may or may not be automated.

Chuckwagon supports the two deployment pipeline approach by providing a second plugin tailored to use in the Production Account. It is configured very similarly to the ChuckwagonPublishPlugin introduced in Getting Started and that you should continue to use in your Development Account. The new plugin is called ChuckwagonCopyPlugin and supports copying a Lambda from a different account and re-publishing it to an environment in the current account.

To use it something like the following complete configuration is required in your build.sbt, 

enablePlugins(ChuckwagonCopyPlugin)
chuckRegion := "<AN_AWS_REGION_EG_-_eu-west-1>"
chuckCopyConfig := chuckCopyConfigBuilder
  .withName("<THE_NAME_YOU_WANT_FOR_YOUR_LAMBDA>")
  .withStagingBucketName("<THE_S3_BUCKET_WHERE_CHUCKWAGON_WILL_UPLOAD_YOUR_CODE")
  .withAssumableDevAccountRoleARN(
    "arn:aws:iam::<DEV_ACCOUNT_ID>:role/<ASSUMABLE_DEV_ACCOUNT_ROLE_ID>")

Notice that chuckRegion, withName and withStagingBucketName are configured in the same way as they were for the ChuckwagonPublishPlugin. Many other fields are missing because they will be taken from the source Lambda being copied. The only additional field in this example is AssumableDevAccountRoleARN. This Role ARN must be configured on the Development AWS Account, so that when Chuckwagon is run on the Production AWS Account it can use that role to copy the Lambda.

Permissions

Making the IAM Role AssumableDevAccountRoleARN in the Development Account available to the Production Account requires granting specially configured Cross AWS Account privileges. It also requires giving a Role in the Production Account access permissions.

The next two sub-sections will explain the two steps in detail, but for background I thoroughly recommend reading the AWS tutorial on Delegating Access Across AWS Accounts using IAM Roles.

Development Changes

In your Development Account you need to create a Role (<ASSUMABLE_DEV_ACCOUNT_ROLE_ID> - the same ID used in the ChuckwagonCopyPlugin config earlier) with the following permissions policy, 
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "lambda:GetFunction"
    ],
    "Resource": "arn:aws:lambda:*:*:*"
  }]
}

and the following trust relationship 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<PRODUCTION_ACCOUNT_ID>:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
This will make it possible to configure any Role in your Production Account to be able to temporarily become this Role in your Development Account.

Production Changes

In your Production Account you need to make sure that whatever Role you use to run Chuckwagon is assigned the following additional policy, 
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::<DEVELOPMENT_ACCOUNT_ID>:role/<ASSUMABLE_DEV_ACCOUNT_ROLE_ID>"
  }
}

Copying

In order to make use of the new configuration you have used in this Guide you need to use a new Input Task in ChuckwagonCopyPlugin that is very similar to the chuckPromote Input Task introduced in Deployment Pipelines,
chuckCopyFromOtherAccountTo <FROM_ACCOUNT_ENVIRONMENT> <TO_ENVIRONMENT> - Copy Lambda from the environment of another account into the <TO_ENVIRONMENT> of this Account.

If you have correctly configured IAM Roles then this Task should download the Lambda (and all its configuration) from your Development Account and Publish it as a new Lambda (with the same configuration) in your Production Account. This is as close as we can get to promoting the same Lambda between two accounts. The only (unavoidable) difference is that the Lambda will have a different Version in Production.

Multi Module Releases

For Development Deployment Pipelines we encouraged you to use the sbt-release plugin. However, since this plugin can only define one release process per normal project a little more work needs to be done if you want to have a release process in your Production Account as well. In order to support two release processes we will move to having a multi-module build. This has other advantages too, as you will see.

Consider the following skeleton outline of a multi-module sbt project 

lazy val commonSettings = Seq(
  chuckRegion := "<LAMBDA_REGION>"
)
val lambdaName = "<LAMBDA_NAME>"
lazy val `<LAMBDA_NAME>-service` = project
lazy val `<LAMBDA_NAME>-qa-tests` = project
lazy val `<LAMBDA_NAME>-prd-tests` = project
lazy val `<LAMBDA_NAME>-dev-pipeline` = project
  .enablePlugins(ChuckwagonPublishPlugin)
    .settings(
      commonSettings ++
        Seq(
          chuckEnvironments := Set[String]("qa"),
          chuckPublishConfig := chuckPublishConfigBuilder
            .withName(lambdaName)
            .withHandler("<HANDLER_CLASS>::<HANDLER_METHOD>")
            .withMemorySizeInMB(192)
            .withTimeout("5 seconds")
            .withStagingBucketName("<DEV_S3_BUCKET>")
            .withCodeFile(assembly in `<LAMBDA_NAME>-service`),
          releaseProcess := Seq[ReleaseStep](
            releaseStepTask(clean in `<LAMBDA_NAME>-service`),
            releaseStepTask(test in Test in `<LAMBDA_NAME>-service`),
            releaseStepInputTask(chuckPublishTo, " qa"),
            releaseStepTask(test in `<LAMBDA_NAME>-qa-tests`),
            releaseStepTask(chuckCleanUp)
          )
        )
    )
lazy val `<LAMBDA_NAME>-prd-pipeline` = project
  .enablePlugins(ChuckwagonCopyPlugin)
    .settings(
      commonSettings ++
        Seq(
          chuckEnvironments := Set[String]("prd"),
          chuckPublishConfig := chuckCopyConfigBuilder
            .withName(lambdaName)
            .withStagingBucketName("<PRD_S3_BUCKET>")
            .withAssumableDevAccountRoleARN(
              "<ASSUMABLE_DEV_ACCOUNT_ROLE_ARN>"),
          releaseProcess := Seq[ReleaseStep](
            releaseStepInputTask(chuckCopyFromOtherAccountTo, " qa prd"),
            releaseStepTask(test in `<LAMBDA_NAME>-prd-tests`),
            releaseStepTask(chuckCleanUp)
          )
        )
    )

Managing a pipeline in this fashion has the following interesting properties,

Running release steps in a sub-module is still a one liner from the command-line, but the format is slightly counter-intuitive and bears elucidating here,

sbt "project <LAMBDA_NAME>-dev-pipeline" release - Execute the dev pipeline
sbt "project <LAMBDA_NAME>-prd-pipeline" release - Execute the prd pipeline

Sadly no such one-liner exists for the sbt shell (no build tool is perfect).

Managed Infrastructure


For all but the most trivial of use-cases you will want to deploy your Lambdas into an AWS Account that contains other infrastructure (eg databases, EC2 instances). It is beyond the scope of Chuckwagon to help you manage this infrastructure and there are already very good tools for helping you do so (eg Terraform and CloudFormation). Chuckwagon still needs to help you interface with that infrastructure however. This guide will walk you through some of the ways in which Chuckwagon can help you.

Managed IAM Roles

Every AWS Lambda requires an IAM Role be set that it can use to execute as. By default Chuckwagon will create and manage a basic IAM Role for your Lambda that grants the basic permissions required, however there are downsides to this approach.

If these limitations are blockers for your organisation you can choose to create your IAM Role elsewhere and simply set the ARN on your publish or copy config builders as so, 

chuckPublishConfig := chuckPublishConfigBuilder
  .withRoleARN("<ROLE_WITH_LAMBDA_EXECUTE_ARN>")

chuckCopyConfig := chuckCopyConfigBuilder
  .withRoleARN("<ROLE_WITH_LAMBDA_EXECUTE_ARN>")

VPCs

It is optionally possible to deploy an AWS Lambda into a VPC using both the ChuckwagonPublishPlugin and the ChuckwagonCopyPlugin. Chuckwagon supports two ways of doing this.

Using Ids

The simplest way is to specify all of the Ids for the associated VPC resources. For example, 
chuckPublishConfig := chuckPublishConfigBuilder
  .withVpc(
    chuckVpcUsingIdsBuilder
      .withVpcId("vpc-12345678")
      .withSubnetIds("subnet-a2345678", "subnet-b2345678")
      .withSecurityGroupIds("sg-12345678")

Using Filters

Alternatively, in order to reduce the coupling between your Lambda and your infrastructure you can reference the VPC resources via tags instead of Ids. Consider this example, 
chuckPublishConfig := chuckPublishConfigBuilder
  .withVpc(
    chuckVpcUsingFiltersBuilder
      .withVpcLookupFilters("tag:Name" -> "prd-vpc")
      .withSubnetsLookupFilters("tag:Name" -> "private-subnet*")
      .withSecurityGroupsLookupFilters("group-name" -> "default")

Each filter (multiple filters are allowed) is a tuple matching the AWS Filter Format. The VPC filter must return exactly one VPC, but the Subnets and SecurityGroups filters can return zero or more.

Reference


The Guides introduce the minimum subset of Chuckwagon features required to build a Lambda pipeline, but there are many more optional features. You should find every feature fully documented in this section.

Plugins


ChuckwagonBasePlugin

This plugin is a dependency of both ChuckwagonPublishPlugin and ChuckwagonCopyPlugin. It contains Settings, Tasks and helper functions that are useful whether you are publishing an AWS Lambda from source or copying one from another Account.

Here are the base Settings available to both the ChuckwagonPublishPlugin and the ChuckwagonCopyPlugin (in addition to their own).

SettingExampleDescription
chuckRegioneu-west-1The AWS Region within which to create/update the AWS Lambda
chuckEnvironmentsSet[String]("blue-qa", "qa")(Optional) The Environments into which the the AWS Lambda should be published, copied and/or promoted (known as Aliases in the AWS Console)
chuckAWSCredentialsProvidernew MyCustomAWSCredentialsProvider()(Optional) The credentials provider to use for AWS interactions. By default this uses the DefaultAWSCredentialsProviderChain. You can override it with any AWSCredentialsProvider
chuckSDKFreeCompilerWIP(Optional) For testing pipelines against mock versions of the AWS SDK

Here are the base Tasks available to both the ChuckwagonPublishPlugin and the ChuckwagonCopyPlugin (in addition to their own).

TaskDescription
chuckCurrentEnvironmentsThe Environments (AWS Aliases) currently configured (if it exists)
chuckCurrentlyPublishedThe currently published versions of this AWS Lambda (if it exists)
chuckPromotePromote the AWS Lambda Version in the first Environment (AWS Alias) to the second. Environments must exist in chuckEnvironments Setting, but associated Aliases will be created in AWS if they are missing. eg "chuckPromote blue-qa qa"
chuckCleanUpRemove all AWS Lambda Versions not deployed to an Environment (AWS Alias) and all Environments not defined in chuckEnvironments Setting
chuckSetLambdaTriggerSchedule AWS Lambda to be invoked based on a cron expression eg 'chuckSetLambdaTrigger qa "rate(1 minute)"'
chuckRemoveLambdaTriggerRemove Scheduled execution associated with AWS Lambda
chuckInvokeInvoke the AWS Lambda (if it exists). SNAPSHOT if no arguments passed. Otherwise either the version or the environment passed as a parameter. Sends no arguments but does print output. (If you have configured multiple Lambdas then you will also need to specify which Lambda you want to run, eg 'chuckInvoke myFirstLambda qa'.)

Here are the base helper functions available to both the ChuckwagonPublishPlugin and the ChuckwagonCopyPlugin (in addition to their own).

chuckVpcUsingIdsBuilder

Starts a fluent builder for a VpcConfigUsingIdsLookup (which extends the VpcConfigLookup required by both chuckPublishConfigBuilder and chuckCopyConfigBuilder for their withVpc builder methods.

Here is an example (which is useless alone but which could be dropped into either plugin). 

chuckVpcUsingIdsBuilder
  .withVpcId("vpc-12345678")
  .withSubnetIds("subnet-a2345678", "subnet-b2345678")
  .withSecurityGroupIds("sg-12345678")

Here are some notes on all of the parameters available to chuckVpcUsingIdsBuilder

ParameterNotes
VpcIdThe string for a VPC Id
SubnetIdsOne or more Subnet Ids
SecurityGroupIdsOne or more Security Group Ids

chuckVpcUsingFiltersBuilder

Starts a fluent builder for a VpcConfigUsingFiltersLookup (which extends the VpcConfigLookup required by both chuckPublishConfigBuilder and chuckCopyConfigBuilder for their withVpc builder methods.

Here is an example (which is useless alone but which could be dropped into either plugin). 

chuckVpcUsingFiltersBuilder
  .withVpcLookupFilters("tag:Name" -> "prd-vpc")
  .withSubnetsLookupFilters("tag:Name" -> "private-subnet*")
  .withSecurityGroupsLookupFilters("group-name" -> "default")

Here are some notes on all of the parameters available to chuckVpcUsingFiltersBuilder

ParameterNotes
VpcLookupFiltersOne or more tuples representing AWS Filters that can select a single AWS VPC
SubnetsLookupFiltersOne or more tuples representing AWS Filters that can select zero or more AWS Subnets
SecurityGroupsLookupFiltersOne or more tuples representing AWS Filters that can select zero or more Security Groups

ChuckwagonPublishPlugin

This plugin allows you to build, configure and publish AWS Lambdas. It has only one setting, chuckPublishConfig which you configure using a fluent builder started by chuckPublishConfigBuilder. Here is an example that sets every required and optional parameter for creating a single AWS Lambda. 

chuckPublishConfig := chuckPublishConfigBuilder
  .withName("myFirstLambda")
  .withHandler("com.itv.MyHandler::handler")
  .withMemorySizeInMB(256)
  .withTimeout("5 seconds")
  .withStagingBucketName("dev-staging")
  .withStagingBucketKeyPrefix("forChuckwagon")
  .withRoleARN(
    "arn:aws:iam::123456789012:role/dev_lambda_execute")
  .withVpc(chuckVpcUsingFiltersBuilder
    .withVpcLookupFilters("tag:Name" -> "qa-vpc")
    .withSubnetsLookupFilters("tag:Name" -> "private-subnet*")
    .withSecurityGroupsLookupFilters("group-name" -> "default"))
  .withDeadLetterARN("arn:aws:sqs:eu-west-1:444455556666:dlq")
  .withCodeFile(assembly)

Here is an example which creates multiple AWS Lambdas from the same code file (it doesn't set any optional parameters). 

chuckPublishConfig := chuckPublishConfigBuilder
  .withNamesToHandlers(
    "myFirstLambda" -> "com.itv.MyHandlerOne::handler",
    "myOtherLambda" -> "com.itv.MyHandlerTwo::handler",
  )
  .withMemorySizeInMB(256)
  .withTimeout("5 seconds")
  .withStagingBucketName("dev-staging")
  .withCodeFile(assembly)

Here are some notes on all of the parameters available to chuckPublishConfigBuilder

ParameterNotes
NameThe name to be used creating/updating the AWS Lambda. If you set this you must set 'Handler' and cannot set 'NamesToHandlers'.
HandlerThe fully qualified class and method of the Handler. If you set this you must set 'Name' and cannot set 'NamesToHandlers'.
NamesToHandlersOne or more (ie varargs) comma separated (String, String) Tuples of Names to Handlers. An AWS Lambda will be created for each name with all of the same settings except for the Handler. If you set this you cannot set 'Name' or 'Handler'
MemorySizeInMBMust be between 128 and 1536 MBs
TimeoutMust be between 1 and 300 seconds
StagingBucketNameBucket that fat JAR will be uploaded to for Lambda create/update
StagingBucketKeyPrefix(Optional) Key prefix that will be used when fat JAR is uploaded to S3
RoleARN(Optional) Pre-existing ARN that AWS Lambda will be configured to execute using
Vpc(Optional) specify the VPC properties to configure the AWS Lambda with using either 'chuckVpcUsingIdsBuilder' or 'chuckVpcUsingFiltersBuilder'
DeadLetterARN(Optional) specify the ARN of an SQS or SNS endpoint where messages that failed Lambda invocation to be delivered
CodeFileA Task[File] that produces a fat JAR suitable for creating an AWS Lambda with.

Here are the Tasks available to the ChuckwagonPublishPlugin

TaskDescription
chuckPublishSnapshotCompile/Create/Update $LATEST AWS Lambda according to chuckPublishConfig Setting (Not recommended for real world usage)
chuckPublishAs chuckPublishSnapshot except additionally create a (Numbered) Lambda Version immutable copy of $LATEST (Not recommended for real world usage)
chuckPublishToAs chuckPublish except additionally assign the Lambda Version to the environment passed as an input parameter (must be one of environments defined in chuckEnvironments)

ChuckwagonCopyPlugin

This plugin allows you to copy AWS Lambdas from another Account. It has only one setting, chuckCopyConfig which you configure using a fluent builder started by chuckCopyConfigBuilder. Here is an example that sets every required and optional parameter. 

chuckPublishConfig := chuckCopyConfigBuilder
  .withName("myFirstLambda")
  .withStagingBucketName("prd-staging")
  .withStagingBucketKeyPrefix("forChuckwagon")
  .withRoleARN(
    "arn:aws:iam::0987654321098:role/prd_lambda_execute")
  .withVpc(chuckVpcUsingFiltersBuilder
    .withVpcLookupFilters("tag:Name" -> "prd-vpc")
    .withSubnetsLookupFilters("tag:Name" -> "private-subnet*")
    .withSecurityGroupsLookupFilters("group-name" -> "default"))
  .withAssumableDevAccountRoleARN(
    "arn:aws:iam::123456789012:role/dev_lambda_get_granted_to_prd")

Here is an example which copies multiple AWS Lambdas another account (it doesn't set any optional parameters). (This feature is broken in v0.1.0 because of #1.) 

chuckPublishConfig := chuckCopyConfigBuilder
  .withNames("myFirstLambda", "myOtherLambda")
  .withStagingBucketName("prd-staging")
  .withAssumableDevAccountRoleARN(
    "arn:aws:iam::123456789012:role/dev_lambda_get_granted_to_prd")

Here are some notes on all of the parameters available to chuckCopyConfigBuilder

ParameterNotes
NameThe name of the Lambda to be copied from the other Account. If you set this you cannot set 'Names'.
Names(This feature is broken in 0.1.0 because of #1.) One or more (ie varargs) names of Lambdas to be copied from the other Account. An AWS Lambda will be created for each name with all of the same settings except for the Handler. If you set this you cannot set 'Name'
NameThe name to be used creating/updating the AWS Lambda
StagingBucketNameBucket that fat JAR will be uploaded to for Lambda create/update
StagingBucketKeyPrefix(Optional) Key prefix that will be used when fat JAR is uploaded to S3
RoleARN(Optional) Pre-existing ARN that AWS Lambda will be configured to execute using
Vpc(Optional) specify the VPC properties to configure the AWS Lambda with using either 'chuckVpcUsingIdsBuilder' or 'chuckVpcUsingFiltersBuilder'
AssumableDevAccountRoleARNThe ARN of a Role in the Development Account that can be used to download the Lambda

Here are the Tasks available to the ChuckwagonCopyPlugin

TaskDescription
chuckCopyFromOtherAccountToCopy an AWS Lambda from another account and publish it to the Environment expressed as an input parameter in this Account (using chuckPublishConfig Setting)

Library


The Chuckwagon Library is an optional layer on top of the AWS Lambda Library. It currently enables the following,

You can use it by adding the following dependency to your build.sbt 

libraryDependencies ++= "com.itv.chuckwagon" %% "chuckwagon-jvm" % "0.1.0"

IAM Roles


This section won't tell you how to configure IAM Roles for Chuckwagon but it does describe when and why specific permissions are required so that you can justify the creation of the required roles within your organisation.

Basic Actions

ActionsNotes
s3:ListBuckets
s3:PutObject
s3:PutObjectAcl
lambda:ListVersionsByFunction
lambda:CreateFunction
lambda:GetFunction
lambda:GetFunctionConfiguration
lambda:UpdateFunctionCode
lambda:UpdateFunctionConfiguration*All Required* for any operation of the sbt Plugin
lambda:PublishVersion*All Recommended* Required to Publish Versions (ie chuckPublish)
lambda:CreateAlias
lambda:GetAlias
lambda:ListAliases
lambda:UpdateAlias*All Recommended* Required to Create/Manage Environments (ie chuckPublishTo)
lambda:DeleteFunction
lambda:DeleteAlias*All Recommended* Required to run chuckCleanup Task
ec2:DescribeSecurityGroups
events:DescribeSubnets
events:DescribeVpcs*All Optional* Required if Lambda VPC Configured with Tag Lookups
events:PutRule
events:PutTargets
lambda:AddPermission
lambda:GetPolicy
lambda:UpdateEventSourceMapping*All Optional* Required for chuckSetLambdaTrigger
events:DeleteRule
events:RemoveTargets
lambda:RemovePermission*All Optional* Required for chuckRemoveLambdaTrigger
lambda:InvokeFunction*Optional* Required for chuckInvoke
iam:**Not Recommended* Required if 'RoleARN' on chuckPublishConfigBuilder/chuckCopyConfigBuilder not set and Chuckwagon is being relied upon to create IAM Role.

Executing Actions

ActionsNotes
logs:CreateLogGroup
logs:CreateLogStream
logs:PutDestination
logs:PutLogEvents*All Recommended* Required if using Cloudwatch Logging
cloudwatch:PutMetricData*Recommended* Required if using Cloudwatch Metrics
ec2:CreateNetworkInterface
ec2:DescribeNetworkInterfaces
ec2:DeleteNetworkInterface*All Optional* Required if running within a VPC

Copying Actions

The following Actions are required to be configured in your Source Account

ActionsNotes
lambda:GetFunction*Required* Must also grant access to Destination Account

The following Actions are required to be configured in your Destination Account

ActionsNotes
sts:AssumeRole*Required* Must specify the Resource in your Source Account

Release Notes


0.1.0 Release


Roadmap


Version 0.1.0 Release contains everything that I currently need to deploy AWS Lambdas in my workplace. My focus for future releases is currently things which are peripheral to my initial vision or would simply make the project easier to run. However, there may be features which the community would find useful and which I have overlooked. I would welcome the opportunity to adjust my current plans to implement ideas which would be a good fit with the project.

0.1.1 Plans


0.2.0 Plans


Later 0.x Releases


Acknowledgments


I'd like to thank the following,
Fork me on GitHub