By: Joe Marshall
Seed apps are great.
They can be the test-bed for new devops features, mini onboarding exercises, or just “batteries-included” starter kits for greenfield applications.
Especially in the web application (and Python) world, almost everything comes with extra considerations – testing, linting, containerization – wouldn’t it be great if we could make a Python seed that came with all of that baked in?
What a beautiful, productive world that would be (skip straight to the github repo to go there now).
Structure
Let’s start with a simple filesystem layout.
| /src /tests |
Conventional wisdom for Python projects is to use Jean-Paul Calderon’s excellent advice in “Filesystem of a Python Project”.
This is great, but Jean-Paul’s advice is aimed primarily at Python Package creates / maintainers. We’re envisioning this as a seed for a simple Python web app or script, which means we’d prefer things like using src as our source code directory (as opposed to something project-specific).
Keep in mind too that Calderone’s article was written in 2007, when Docker (2010) was just a blink in Solomon Hyke’s eye. Using the srcpattern gives us a clear, semantic way of bundling our code to be copied over as part of the Dockerization process. And since we’re going to be writing this app in Python 3.X, we don’t have to worry about keeping empty, placeholder __init__.py files.
The advice generally ages well though. If you find yourself needing a collection of scripts or executables, for example, adding a scripts or bin directory will give you an easy-to-understand location that other devs jumping onto the project (or you in a year) will grok instantly.
Let’s also add an app.py under /src as some sample Python application code to test. We’ll keep it to a classic:
| print(“Hello World!”) |
Housekeeping
Though we’ve settled on a _basic_ directory structure there are still some essential files we should add, like a `README.md` and `setup.py`.We’ll need the `setup.py` file for our test runner, tox.
We can create a really basic one with just a few lines:
| from distutils.core import setup setup( name=’Seed Project’, version=’0.1′, packages=[‘src’,], ) |
Of course the project’s `name` should be changed to reflect the application you’re building, but a placeholder works fine too
Let’s also add a `.gitignore` so we can exclude things from versioning – like cached testing info – we don’t particularly care about project-to-project
| /src /tests. gitignore setup.py README.md |
Dependency Management
What dependencies could an empty project have? We haven’t written any application code – what sort of dependencies do we need to account for?
Nothing yet. But we can still structure our dependency files for the eventuality that, you know, we’ll put something in this someday
| /src /tests .gitignore setup.py README.md dev-requirements.txt requirements.txt |
We split our `requirements.txt` files into two because we can imagine different use cases for different environments – our Dockerfile / Dockerized version of the app, for example, won’t need any of our testing code, just the raw production application. By keeping our testing dependencies out of what will be the production image, we can make deploying it that much faster.
I prefer using the `dev-requirements.txt` and `requirements.txt` naming convention, as opposed to having a `requirements` directory or some other nested structure, so I can keep things flat and Pythonic.
Linting
If it’s not already a part of your development process, baking linting into everything you do can be a powerful resource and teaching tool. Incorporating linting into your own red-green-refactor cycle can nudge you towards adopting style patterns that are less bug-prone, easier to read, and faster, without the apocalyptic potential of a massive global refactor. Lint _as_ you develop instead of *after* you develop, so you can anticipate and course-correct idiosyncratic style decisions – linting, like testing, is best adopted at the beginning of a project, instead of grafted on to the finished product.
We’re going to use `
Let’s update our `dev-requirements.txt` to include our new dependency `pylint==2.1.1` and install it with `pip install -r dev-requirements.txt`.
With `pylint` installed, we can generate a `pylintrc` configuration file with a simple command.
| pylint –generate-rcfile > .pylintrc |
| /src /tests .gitignore setup.py README.md dev-requirements.txt requirements.txt |
Testing
For testing we’re going to use `pytest` (you could also use `nose` here or whatever framework you’re most comfortable with) as our testing library and `tox` as our test runner. `tox` was designed to test a Python app against multiple different Python versions for maximum interoperability, making it a bit overkill for our purposes, but it’s still a great handle for external CI/CD systems to execute our tests (and linting) in the proper virtual context. Let’s add `tox==3.2.1` and `pytest==3.7.3` to our `dev-requirements.txt` and do another `pip install -r dev-requirements.txt`.
Remember that `/tests` directory we set up forever ago? Let’s go ahead and add a sample “hello world”-type placeholder for `pytest` . Create a `test_example.py` file inside of `/tests` and add this code:
| def test_example(): assert True |
Pretty simple – we want an always-true test case for evaluating our setup, and this gives us just that.
Now for setting up tox, we just need to create a `tox.ini` file. Here’s what ours will look like:
| [tox] minversion = 2.0 envlist = py36 skipsdist = True [testenv] commands = {envbindir}/pytest {posargs} deps = -r{toxinidir}/requirements.txt -r{toxinidir}/dev-requirements.txt recreate = False passenv = * [testenv:lint] basepython=python3.6 deps = -r{toxinidir}/requirements.txt -r{toxinidir}/dev-requirements.txt commands=pylint src |
Going through the `tox` documentation, our choices should be pretty clear. We’re selecting just the `py36` environment because we’re only interested in running tests against the Python version we’re actually using. We have the `[testenv]` section, which lays out the actual commands for executing a test (installing all of our requirements, running the `
Containerization
The final piece of our puzzle is integrating this setup with [Docker](https://www.docker.com/products/docker-desktop), to make it portable. Docker is of course excellent for devops and setting up structured, deterministic deploy pipelines. But even as a simple dependency management system similar to `virtualenv`, Docker can be a great solution (if a little overkill). There’s a certain joy to working on a project on your own system, collaborating with a colleague who has a different machine, and deploying to a third operating system for the deploy – where everything works every step of the way because of the common Docker foundation.
Let’s look at our `Dockerfile` and explain some of the choices we’ve made along the way. Add this `Dockerfile` to your project’s root directory.
| FROM python:3.6-alpine COPY requirements.txt ./ RUN pip install -r requirements.txt Add /src /app WORKDIR /app CMD [“python”, “app.py”] |
It doesn’t come much simpler: We’re using the `alpine` version of Python 3.6 to try and keep our base image slim (you can read more about using the alpine versions as base images and their performance benefits on [Nick Janetakis’ blog](https://nickjanetakis.com/blog/alpine-based-docker-images-make-a-difference-in-real-world-apps)); then copying over our production `requirements.txt` file and installing our pip dependencies, before we add our `src` code to the Docker image’s `/app` directory and declare it our working directory. Docker’s `CMD` functionality allows us to specify the command we want to run when the image is executed, in this case bootstrapping our `app.py`.
Validation
Let’s take a look at our final project directory.
| /src /tests .gitignore Dockerfile setup.py README.md dev-requirements.txt requirements.txt |
Now to see what this gets us, verifying the CI/CD tools we’ve put into place.
| tox -e lint“` |
Running the lint command you might see it spit out a failing exit code like this.
But don’t panic! There’s no error in our implementation – `tox` is doing exactly what we want it to do in this situation, returning an error for a failing style. If we correct the offending rule by adding a newline to the end of the `app.py` file and run it again…
Success! Let’s move on to testing.
| tox |
We can see tox build a Python 3.6 environment, installing our dependencies and executing our (one) test – just like it should. Unsurprisingly, our test passes.
In order to test our Dockerfile, let’s build and tag an image. We’re calling ours “seed-app” but in principle you can name your Docker image anything.
| docker build -t seed-app . |
When that’s been successfully built, we can run the image with the following command:
| docker run -it seed-app |
We’re using the `-t` flag just as we did with our `build` command to make sure we run the image we just tagged as “seed-app”, while the `-i` (interactive) flag makes sure the Docker image STDOUT gets piped back to our terminal. And what gets piped back to our terminal when we run this image?
| Hello world! |
Hello Python seed!
Conclusion
Seed apps are useful for the purpose they fulfill – sometimes you just want to spin up a Python applet with all the goodies without gutting an open source app or hand-coding the boilerplate each time.
They can also be useful as a PoC and teaching tool, showing the simplest ways in which different services can be glued together.
_______________________________
Joe Marshall
Joe is a web application engineer, independent security researcher, and writer. His first book, Hands-On Bug Hunting for Penetration Testers, is out now.
If you’re interested in receiving updates about the book and related events, sign up for the newsletter below. Or order it now.
