Generic Python script-runner using Dockerized Jenkins-pipeline

Jenkins+Docker Python logo

The challange

During a project of providing Jenkins-pipeline scripts - along with Docker solution - for a Tikal-customer, I’ve noticed a pattern for many of the jobs I need to provide:

  • Run the job inside an isolated (Docker-based) Python environment (with common list of Python modules installed).
  • Code checkout from a GIT repository (and branch).
  • Install dedicated Python modules based on a list which resides in a text file (usually requirements.txt file) which is part of the GIT repository.
  • Run a Python script, which is also resides in the GIT repository.

I’ve suggested and provided a generic job that also supports other needs:

  • The upstream job (which activates the generic one) should provide its parameters and/or environment variables as environment variables to the generic job
  • The job should be wrapped with security credentials (such as AWS, Google and more).
  • The job must run inside an isolated Docker environment.
  • The job should be implemented using Jenkins pipeline.

The first implementation were provided internally to the customer, but I’ve shared it also in tikal-advanced-pipeline shared library with some adjustments which makes it more community-oriented.

The solution

The solution is basically made of 2 files: Dockerfile and Jenkinsfile.

The Dockerfile provides an isolated Python environment with common modules installed and additional Docker-client installed as well (because many of the Python scripts I needed to run creates Docker images as products, therefore needed to activated Docker build commands).

Here’s the Dockerfile content:

FROM python:2.7

# install some generic Python modules
RUN pip install ansible==2.3
RUN pip install boto==2.46.1
RUN pip install credstash==1.13.2
RUN pip install fasteners==0.14.1
RUN pip install futures==3.0.5
RUN pip install pyfscache==0.9.12
RUN pip install PyYAML==3.12
RUN pip install hvac==0.2.17
RUN pip install awscli==1.11.70
RUN pip install docker==2.5.1

# install docker client
RUN apt-get update && \
    apt-get install -y apt-transport-https ca-certificates curl software-properties-common && \
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu trusty stable" && \
    apt-get update && \
    apt-get install -y docker-ce

And here is the full Jenkinsfile content:

pipeline
{
    parameters
    {
        string(name: 'REPOSITORY_URL', defaultValue: '', description: '<BR><font color=RED>*</font> URL to GIT repository')
        string(name: 'GIT_REPO_CRED', defaultValue: '', description: '<BR><font color=RED>*</font> Credential name of GIT user')
        string(name: 'BRANCH', defaultValue: 'master', description: '<BR>Name of branch (default: master)')
        string(name: 'REQUIREMENTS_FILE', description: '<BR>Relative path to the requirements file')
        string(name: 'PYTHON_SCRIPT_FILE', description: '<BR><font color=RED>*</font> Relative path to the python script')
        text(name: 'GROOVY_SCRIPT', defaultValue: '', description: '<BR>Insert a groovy script text to run.<BR>e.g.:<br>env.PARAM1=&quot;value1&quot;<br>env.PARAM2=&quot;value2&quot;')
    }

    options
    {
        buildDiscarder(logRotator(numToKeepStr: '100', daysToKeepStr: '45'))
        ansiColor('xterm')
        timestamps()
    }

    agent
    {
        dockerfile
        {
            filename 'examples/pipelines/TAP_generic_python_runner/Dockerfile'
            args "-u root -v /var/run/docker.sock:/var/run/docker.sock"
        }
    }

    stages
    {
        stage('Setup')
        {
            steps
            {
                script
                {
                    def script = PYTHON_SCRIPT_FILE.tokenize('/')[-1]
                    currentBuild.displayName = "#${BUILD_ID} | ${script}"
                }
                checkout([
                        $class           : 'GitSCM', branches: [[name: BRANCH]],
                        userRemoteConfigs: [[url: "${REPOSITORY_URL}", credentialsId: "${GIT_REPO_CRED}"]]
                ])
            }
        }

        stage('Install requirements')
        {
            steps
            {
                sh '''
                    if [ "x${REQUIREMENTS_FILE}" != "x" ] && [ -f ${REQUIREMENTS_FILE} ]; then 
                        pip install -r ${REQUIREMENTS_FILE}; 
                    fi
                '''
            }
        }
        stage('Run script')
        {
            steps
            {
                script
                {
                    writeFile encoding: 'UTF-8',file: './variables.groovy', text: GROOVY_SCRIPT
                    load './variables.groovy'
                    
                    sh "python ./${PYTHON_SCRIPT_FILE}"
                }
            }
        }
    }
}

Topics about the solution

Inside the Jenkinsfile I’ve implemented the above requirements using some Jenkins-solutions, which I’ll explain in details now.

Creates an isolated Docker environment for running the Python script

    agent
    {
        dockerfile
        {
            filename 'examples/pipelines/TAP_generic_python_runner/Dockerfile'
            args "-u root -v /var/run/docker.sock:/var/run/docker.sock"
        }
    }

The pipeline job runs inside a Docker container which built using the above Dockerfile. Jenkins pipeline plugin take care of building the Docker image and running (and deleting at the end) the Docker container.

I’ve also run the container as root user and also mount it into the Jenkins server Docker daemon.

Wrap the generic pipeline the parameters of the upstream job, credentials and other environment needs

script
                {
                    writeFile encoding: 'UTF-8',file: './variables.groovy', text: GROOVY_SCRIPT
                    load './variables.groovy'
                }

The user of the pipeline can provide a groovy script as a text parameter which can contain code for passing the upsteam job parameters (e.g. env.PARAM1=$PARAM1) and such to the generic Python script runner.

Actived a list of Python modules to install from a text file inside the GIT repository

                sh '''
                    if [ "x${REQUIREMENTS_FILE}" != "x" ] && [ -f ${REQUIREMENTS_FILE} ]; then 
                        pip install -r ${REQUIREMENTS_FILE}; 
                    fi
                '''

This code checks whether the provided requirements file exist or that the parameter is not empty, and installs the provided modules list using the ‘pip install’ command.

Usage example

In tikal-advanced-pipeline shared library I’ve also included a set of script with matching Python modules requirement file for testing the generic pipeline.

It can be run using the following parameters: job parameters

And here’s how the pipeline job page looks like after running the job few times: job snapshot

Enjoy!

DevOps Fullstack Tech Leader

DevOps Group