How to add dependencies to AWS lambda

The process of adding dependencies to an AWS Lambda consists of two steps. First, we have to install the dependencies in the source code directory. Later, we have to package our lambda function into a zip file that also contains all of the dependency files.

As an example, I will install the Python Scrapy dependency because, in addition to pip packages, it also requires some Linux dependencies.

Installing dependencies

There is one little problem with installing Lambda dependencies. Lambdas run on a special Linux distribution made by AWS, and we need the dependencies compiled for that particular distribution.

Because of that, I am going to prepare a Docker image in which I will install all of the required packages and then copy them to the local directory that contains the lambda function.

First, I must create a Dockerfile.

FROM amazonlinux

RUN yum update -y && \
    yum install -y python3 pip3 libxml2 libxslt && \
    pip3 install -t /deps_source lxml scrapy

CMD cp -r /deps_source/* /deps

When I build the Docker image using that file, it will install all of the required packages and store them in a directory inside the image.

1
docker build - t lambdadeps

The only thing left to do when I run the image is copying the already installed dependencies to a directory that was mounted as a Docker volume.

1
docker run -v $(pwd)/deps:/deps lambdadeps:latest

That will copy the Python dependencies for Amazon Linux into my local directory.

Packaging the lambda function

Now, I can write a Make script that packages the lambda function and its dependencies. Note that, I have the dependencies in the deps directory, and the lambda function is in the test_lambda.py file.

build:
    docker run -v $(pwd)/deps:/deps lambdadeps:latest
    cd deps && zip -r9 function.zip .
    mv deps/function.zip function.zip
    zip -g function.zip test_lambda.py

You MUST use tabs for indentation in the Makefile. If your IDE replaces tabs with spaces, it will not work! If you copy-paste my code, you have to replace four spaces with a tab.

When the Makefile is ready, I can run the make build command to package the lambda and all its dependencies into a zip file.

After that, I have to deploy the function on AWS. The process depends on the deployment of choice, and it is not in the scope of this article.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you watch programming live streams, check out my YouTube channel.
You can also follow me on Twitter: @mikulskibartosz

If you want to hire me, send me a message on LinkedIn or Twitter.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craftsmanship Poznan & Poznan Scala User Group