Paramiko is a powerful Python library that enables you to set up SSH sessions with remote hosts, execute commands and receive their output, as well as a bunch of other things.
Lambda functions is one of AWS extremely usable services. It allows you to execute code-on-demand. You don't need any host or infrastructure preparation of any kind to use a Lambda function. You just deploy it and execute it.
Bundling the package using pip
In one of my labs, I needed my Lambda function to execute a command that retrieves the top five running processes on an EC2 instance of mine. To do this, I had to use the Paramiko package in my Python code. Since Lambda does not have Paramiko by default in the Python environment that it uses, I had to bundle the library and all its dependencies in the zip file that gets uploaded to AWS Lambda console. For detailed instructions on how to package and deploy your Lambda function, you can follow the this link: .com/lambda/latest/dg/with-s3-example-deployment-pkg.html
So, to do that I had to use a command like the following
pip install paramiko -t .
Having tested the code locally and ensured that it works without issues, I had to compress the contents of the directory and upload it to the Lambda console as follows:
zip -r package.zip *
The function throws an error that explains nothing
When I try to execute the function, it throws the following error: unable to load module main.
If you've worked with Lambda functions before, you'd know that most of the time, this error means that you did not specify the correct file name that has your code. For example, if you placed your code in a file called “main.py” and the entry point function id called “lambda_handler”, then you should configure the handler part of the Lambda function configuration to be
main.lambda_handler as shown:
But, still even when you correctly set the handler part of the function, it throws the same error.
It's not your fault
Actually, there is nothing wrong with your code, neither with the Lambda function configuration. It turns out that the problem is that you have used
pip on either a Windows, a macOS, or - in some rare cases - a non-RedHat machine. Let's explain this further.
Behind the scenes, Lambda functions get executed on instances (alright, containers) that run Amazon Linux. Amazon Linux is a custom Linux flavor based on Red Hat and offered by Amazon to be used in their EC2 instances (along with other OS types). When you installed Paramiko on yout local machine, pip automatically grabbed the version that was compiled for your specific OS environment. So, it might not work when deployed to AWS Lambda console.
What are my options?
There is a number of options that you can use depending on your resources:
- Use an EC2 instance that is already based on Amazon Linux or some other Red Hat-based flavor like Centos or Fedora and run
pip install paramiko -t .. Package the directory and deploy it.
- Use a virtual machine on a platform like Virtual Box or VMWare, install a Red Hat variant. Use pip and deploy the package as described in step 1.
- Use a tool like Vagrant and start a vm based on an Red Hat image. Use pip and follow the same instructions on this machine.
- Use Amazon Linux official Docker image, create a container an run pip inside it.
I opted to use that last option as it was the fastest and easiest in my case. Let's see how:
- You will have to install Docker on your machine. Just Google how to install Docker and you'll find plenty of resources out there.
- Once installed, and inside the directory where you placed your code, run the following command:
sudo docker run -it --name alinux -v $(pwd):/opt/lambda/ amazonlinux /bin/bash
What this command does it that it will attempt to start a container based on Amazon Linux, it will name it
alinux for future reference, it will mount the current directory to a
/opt/lambda inside the container, and finally it will drop you inside a bash shell inside the container.
3. Once inside, you can run pip to install paramiko and any other dependencies (make sure that you place your code file somewhere outside the directory first):
cd /opt/lambda yum -y install python-pip mv main.py .. rm -rf * pip install paramiko -t . mv ../main.py . exit
- Now, you have a directory that has Paramiko, with all its dependencies ready to work on AWS Lambda's environment.
Just zip the directory contents and upload it to the Lambda as you did before. It should work without any issues.
The only downside for this approach is that you may not be able to test this code locally because the library version is now built for a different platform. A simple workaround for this would be creating a second directory that has the same dependencies but using
pip on the machine and not the one from the container. This will make sure that this code will work locally. You can use the first version for Lambda deployment.
I hope I saved someones's day.