Building a Python 3.6 Seed App with Docker, Tox, and Pylint

Building a Python 3.6 Seed App with Docker, Tox, and Pylint

By: Joe Marshall

Seed apps are great.

They can be the test-bed for new devops features, mini onboarding exercises, or just “batteries-included” starter kits for greenfield applications.

Especially in the web application (and Python) world, almost everything comes with extra considerations – testing, linting, containerization – wouldn’t it be great if we could make a Python seed that came with all of that baked in?

What a beautiful, productive world that would be (skip straight to the github repo to go there now).

Structure

Let’s start with a simple filesystem layout.

/src
/tests

Conventional wisdom for Python projects is to use Jean-Paul Calderon’s excellent advice in “Filesystem of a Python Project”.

This is great, but Jean-Paul’s advice is aimed primarily at Python Package creates / maintainers. We’re envisioning this as a seed for a simple Python web app or script, which means we’d prefer things like using src as our source code directory (as opposed to something project-specific).

Keep in mind too that Calderone’s article was written in 2007, when Docker (2010) was just a blink in Solomon Hyke’s eye. Using the srcpattern gives us a clear, semantic way of bundling our code to be copied over as part of the Dockerization process. And since we’re going to be writing this app in Python 3.X, we don’t have to worry about keeping empty, placeholder __init__.py files.

The advice generally ages well though. If you find yourself needing a collection of scripts or executables, for example, adding a scripts or bin directory will give you an easy-to-understand location that other devs jumping onto the project (or you in a year) will grok instantly.

Let’s also add an app.py under /src as some sample Python application code to test. We’ll keep it to a classic:

print(“Hello World!”)

Housekeeping

Though we’ve settled on a _basic_ directory structure there are still some essential files we should add, like a `README.md` and `setup.py`.We’ll need the `setup.py` file for our test runner, tox.

We can create a really basic one with just a few lines:

from distutils.core import setup
setup(
        name=’Seed Project’,
        version=’0.1′,
        packages=[‘src’,],
)

Of course the project’s `name` should be changed to reflect the application you’re building, but a placeholder works fine too.

Let’s also add a `.gitignore` so we can exclude things from versioning – like cached testing info – we don’t particularly care about project-to-project.

Here’s what the root project directory looks like now.

/src
/tests.
gitignore
setup.py
README.md

Dependency Management

What dependencies could an empty project have? We haven’t written any application code – what sort of dependencies do we need to account for?

Nothing yet. But we can still structure our dependency files for the eventuality that, you know, we’ll put something in this someday.

Let’s update our current repo.

/src
/tests
.gitignore
setup.py
README.md
dev-requirements.txt
requirements.txt

We split our `requirements.txt` files into two because we can imagine different use cases for different environments – our Dockerfile / Dockerized version of the app, for example, won’t need any of our testing code, just the raw production application. By keeping our testing dependencies out of what will be the production image, we can make deploying it that much faster.

I prefer using the `dev-requirements.txt` and `requirements.txt` naming convention, as opposed to having a `requirements` directory or some other nested structure, so I can keep things flat and Pythonic.

Linting

If it’s not already a part of your development process, baking linting into everything you do can be a powerful resource and teaching tool. Incorporating linting into your own red-green-refactor cycle can nudge you towards adopting style patterns that are less bug-prone, easier to read, and faster, without the apocalyptic potential of a massive global refactor. Lint _as_ you develop instead of *after* you develop, so you can anticipate and course-correct idiosyncratic style decisions – linting, like testing, is best adopted at the beginning of a project, instead of grafted on to the finished product.

We’re going to use `pylint` as our Python linting solution. I’ve chosen `pylint` because it’s easy to use, employs a simple configuration system, and has granular message control for suppressing unwanted rules.

Let’s update our `dev-requirements.txt` to include our new dependency `pylint==2.1.1` and install it with `pip install -r dev-requirements.txt`.

With `pylint` installed, we can generate a `pylintrc` configuration file with a simple command.
pylint –generate-rcfile > .pylintrc

pylint –generate-rcfile > .pylintrcAnd this will create a `.pylintrc` file you can edit. `pylint` provides you the option of reading from a user-level instead of a project-level configuration (e.g. `~/.pylintrc`) but it’s best to keep the linting configuration file in the project – even if it weren’t strictly necessary for CI/CD integration.

Here’s an updated look at our project root.

/src
/tests
.gitignore
setup.py
README.md
dev-requirements.txt
requirements.txt

Testing

For testing we’re going to use `pytest` (you could also use `nose` here or whatever framework you’re most comfortable with) as our testing library and `tox` as our test runner. `tox` was designed to test a Python app against multiple different Python versions for maximum interoperability, making it a bit overkill for our purposes, but it’s still a great handle for external CI/CD systems to execute our tests (and linting) in the proper virtual context. Let’s add `tox==3.2.1` and `pytest==3.7.3` to our `dev-requirements.txt` and do another `pip install -r dev-requirements.txt`.

Remember that `/tests` directory we set up forever ago? Let’s go ahead and add a sample “hello world”-type placeholder for `pytest` . Create a `test_example.py` file inside of `/tests` and add this code:

def test_example():
assert True

Pretty simple – we want an always-true test case for evaluating our setup, and this gives us just that.

Now for setting up tox, we just need to create a `tox.ini` file. Here’s what ours will look like:

[tox]
minversion = 2.0
envlist = py36
skipsdist = True

[testenv]
commands = {envbindir}/pytest {posargs}
deps = -r{toxinidir}/requirements.txt
             -r{toxinidir}/dev-requirements.txt
recreate = False
passenv = *
[testenv:lint]
basepython=python3.6
deps = -r{toxinidir}/requirements.txt
             -r{toxinidir}/dev-requirements.txt
commands=pylint src

Going through the `tox` documentation, our choices should be pretty clear. We’re selecting just the `py36` environment because we’re only interested in running tests against the Python version we’re actually using. We have the `[testenv]` section, which lays out the actual commands for executing a test (installing all of our requirements, running the `pytest` command), and the `[testenv:lint]` section, which allows us to kick off our linting regimen via `tox -e lint`. Using `pylint` this way allows us to return an error code to CI/CD systems when we have style mistakes.

You also might notice that our `tox.ini` file uses the `pylint src` command to make sure that we’re only linting our application code and not our configuration or setup files.

Containerization

The final piece of our puzzle is integrating this setup with [Docker](https://www.docker.com/products/docker-desktop), to make it portable. Docker is of course excellent for devops and setting up structured, deterministic deploy pipelines. But even as a simple dependency management system similar to `virtualenv`, Docker can be a great solution (if a little overkill). There’s a certain joy to working on a project on your own system, collaborating with a colleague who has a different machine, and deploying to a third operating system for the deploy – where everything works every step of the way because of the common Docker foundation.

Let’s look at our `Dockerfile` and explain some of the choices we’ve made along the way. Add this `Dockerfile` to your project’s root directory.

FROM python:3.6-alpine

COPY requirements.txt ./

RUN pip install -r requirements.txt

Add /src /app

WORKDIR /app

CMD [“python”, “app.py”]

It doesn’t come much simpler: We’re using the `alpine` version of Python 3.6 to try and keep our base image slim (you can read more about using the alpine versions as base images and their performance benefits on [Nick Janetakis’ blog](https://nickjanetakis.com/blog/alpine-based-docker-images-make-a-difference-in-real-world-apps)); then copying over our production `requirements.txt` file and installing our pip dependencies, before we add our `src` code to the Docker image’s `/app` directory and declare it our working directory. Docker’s `CMD` functionality allows us to specify the command we want to run when the image is executed, in this case bootstrapping our `app.py`.

Validation

Let’s take a look at our final project directory.

/src
/tests
.gitignore
Dockerfile
setup.py
README.md
dev-requirements.txt
requirements.txt

Now to see what this gets us, verifying the CI/CD tools we’ve put into place.

tox -e lint“`

Running the lint command you might see it spit out a failing exit code like this.

But don’t panic! There’s no error in our implementation – `tox` is doing exactly what we want it to do in this situation, returning an error for a failing style. If we correct the offending rule by adding a newline to the end of the `app.py` file and run it again…

Success! Let’s move on to testing.

tox

We can see tox build a Python 3.6 environment, installing our dependencies and executing our (one) test – just like it should. Unsurprisingly, our test passes.

In order to test our Dockerfile, let’s build and tag an image. We’re calling ours “seed-app” but in principle you can name your Docker image anything.

docker build -t seed-app .

When that’s been successfully built, we can run the image with the following command:

docker run -it seed-app

We’re using the `-t` flag just as we did with our `build` command to make sure we run the image we just tagged as “seed-app”, while the `-i` (interactive) flag makes sure the Docker image STDOUT gets piped back to our terminal. And what gets piped back to our terminal when we run this image?

Hello world!

Hello Python seed!

Conclusion

Seed apps are useful for the purpose they fulfill – sometimes you just want to spin up a Python applet with all the goodies without gutting an open source app or hand-coding the boilerplate each time.

They can also be useful as a PoC and teaching tool, showing the simplest ways in which different services can be glued together.

Hopefully this post has accomplished some small version of that, and shown how a Python 3.6 app can be easily integrated into a modern, containerized, tooling setup – one that’s ready from the first commit.

_______________________________

Joe Marshall

Joe is a web application engineer, independent security researcher, and writer. His first book, Hands-On Bug Hunting for Penetration Testers, is out now.

If you’re interested in receiving updates about the book and related events, sign up for the newsletter below. Or order it now.

2018 Python Software Foundation Recurring Giving Campaign

2018 Python Software Foundation Recurring Giving Campaign

Help us in supporting the Python Software Foundation with their annual end-of-year fundraising drive.

Every year the Python Software Foundation launches an end-of-year fundraising drive to help build a sustainable community of supporters.

Their goal is to raise $30,000!

You can help by signing up to give monthly or if you’re already a supporting member (Thank You!!), by checking the box to renew your membership automatically.

The drive begins October 22 and concludes November 21, 2018.

Your donations have IMPACT

Over $118,543 was awarded in financial aid to 143 PyCon attendees in 2018.
$240,000 has been paid in grants from January through September 2018 to recipients in 45 different countries.

Some examples of how your donation dollars are spent:

  • $35,000 was provided to the Python Core Development Sprint in 2018
  • $20,000 helps support 11 Python conferences, impacting over  3,000 people globally
  • $20,000 helps support ~25 hands-on workshops impacting over 700 people
  • $10,000 supports user group meetup fees for 150 groups for 6 months
  • $5,000 provides hosting for 3-4 PSF community web services for a year(e.g. Mail.python.org, uspycon.org, wiki.python.org)
  • $1,000 helps supports 2 regional conferences, impacting over 500 people
  • $50 supports a Python meetup group for 3 months

This work can’t be done without the generous financial support that people like you provide.

It’s easy to donate – 

  • Click here to make a one-time or recurring donation.
  • If you’re an existing PSF Supporting Member and would like to make your support recurring, log into your account and click “Auto-Renewal”.
  • If you’re an existing Supporting Member and would like to support the PSF beyond your $99 annual fee, use this donation page to select an additional one-time or monthly donation.
  • If you’d like to join the PSF, click here to become a Supporting Member and optionally make a one-time donation.

 More details on contributing can be found on the 2018 PSF Recurring Giving Campaign page.

Thank you to everyone who has contributed to our past fundraisers! Your support is what makes the PSF possible and is greatly appreciated by the Python community.

Please consider contributing to the Recurring Giving Campaign & help them reach their goal of $30K.

The PSF is a non-profit organization entirely supported by its sponsors, members & the public.

Preparing for a “Silicon Valley Style” coding interview.

By: Dan Bader – http://bit.ly/2xbr0oE

Participating in a “Silicon Valley style” coding interview can feel scary as heck.n

Unlike other professionals, it seems to be okay for software developers to expect to get humiliated at a job interview.

“What, you can’t code up a recursive descent parser on a whiteboard in 7.5 minutes? How DARE YOU even apply for this job!”

Yeah, it’s one of the things that sucks about our industry—

Personally, I believe that 80-90% of the questions that you get asked during the typical coding interview have nothing to do with your real performance on the job.

But unfortunately, these interviews aren’t going to go away over night.

If you want a well-paid job as a software developer, you’re likely going to encounter some coding quiz as part of your interviewing experience.

For the foreseeable future, interviewers are going to keep squeezing you through the same processes and will keep asking you those same questions…

And if you’re like me, there’s a pretty slim chance you’ll pass an interview like that without some serious prep work—either to learn the right skills or to refresh your memory.

Alright, that all sounds pretty glum, no?

But here’s what you need to realize, interviewing is a skill you can learn like any other.

It’s something you get better at with practice.

It’s true—

Just remember that all that prep work needs time. So be sure to plan ahead with ample of time to get enough study days in before your “big day.”

If I had a coding interview coming up in 1-2 months, here’s a rough outline of what I’d do to prepare:

Step 1:

Buy the following two books: “Elements of Programming Interviews (Python Ed.)” and “Cracking the Coding Interview.”

Step 2:

Buy a whiteboard and some markers. Put the whiteboard on an actual wall, and make sure you get a board with a decent size. This is where 90% of your prep work will happen over the next few weeks.

Step 3:

Every day, stand in front of your whiteboard and work on at least one problem from the books listed in Step 1. Talk out loud about what you’re doing, and snap a photo of the board when you’re done. Set a 30-minute timer for each problem to put some pressure on yourself. If you can’t solve a problem, pick up the book and go through all the motions with the solution in front of you. Rinse and repeat.

The closer you get to your interview date the more you want to practice—ramp it up to around 5 problems per day in the last two weeks before your interview.

Just repeat after me: “Interviewing is a learned skill.”

The more “reps” you can get on each problem, the better your chances of getting a job offer will be.

Happy Pythoning!

Dan Bader is a self-professed Python “Nut” and is passionate about helping Python developers take their coding skills to the next level. His articles, videos, and tutorials have reached over half a million developers around the world. 

 

PythonTek’s Commitment to the Victims of Hurricane Harvey

Like everyone else in the US and Texas we sit and watch with heavy hearts the devastation that Hurrican Harvey has brought to Texas.

Hurricane Harvey’s punishing rains, fierce winds, and raging floods have left a wake of devastation that Texans have never seen before. Families have been forced to evacuate their homes, not knowing whether they’ll still be standing when they return and businesses have been completely destroyed leaving owners wondering how long it will take to rebuild.

We have seen tremendous acts of selflessness by people helping rescue people from flooded houses and apartment buildings, providing food and shelter for those displaced, and making financial donations.

In order do our part, PythonTek is pledging to donate 10% of their revenue from September 1st – December 31st to the following relief funds: The American Red Cross, Houston Relief Fund, and Heart to Heart International.

We will also be encouraging our clients to make matching contributions to one of the above charities.

Our thoughts and prayers are with all those affected through out the state of Texas.

Dave Hitt – Founder

 

Jim McGrath Joins PythonTek

We are pleased to announce and welcome the addition of Jim McGrath as Director of Strategy and Innovation to our team.

Jim brings over 30 years of IT executive leadership for both small and large enterprise companies in the Austin area and has a track record for building high performing teams and organizations.

Jim is not only deeply technical in all aspects of IT but a recognized leader and mentor to his clients and all levels of talent.

Jim’s strong winning attitude and personality are assets that will enhance PythonTek’s ability to meet the needs of our customers as we continue to provide quality service to Python community.

In joining PythonTek Jim says, “I am excited about this new opportunity to build a best in class company specifically dedicated to the Python community”.

Please join us in welcoming Jim to the PythonTek team. He can be reached at jmcgrath@pythontek.com.

 

Niche IT Recruiting Firm Focuses on Python Community

AUSTIN, Texas–(BUSINESS WIRE)– PythonTek, a specialized recruitment firm focused on the Python community, today announced it has launched operations serving both Python developers seeking jobs and companies looking for Python talent. Rising in popularity, Python is now the fourth most requested programming skill according to the TIOBE Index for May 2017.

“Traditional recruiting firms try to be everything to everybody. PythonTek is a unique IT recruiting firm specializing only in the Python community,” said David Hitt, PythonTek founder and Director of Client Engagement. “By specializing in Python talent, we understand the challenges companies face in attracting and retaining staff, are able to build a more focused pipeline of candidates and help reduce our clients’ hiring costs.”

PythonTek is based in Austin, Texas, a state with the second highest employment for software developers. Texas employs 60,550 software developers with an annual mean wage range of $101,720 – $129,440, as reported in the May 2016 employment statistics from the U.S. Department of Labor.

To assess depth of knowledge, PythonTek incorporates a technical screening to assess a candidate’s familiarity with common Python frameworks. Screening is done by top Python leaders in both the public and private sectors to ensure the best fit between candidates and hiring companies.

“Our mission is to provide our clients unrivaled customer service, honesty, and transparency. Our candidates can expect access to opportunities that will advance their careers while working on cutting-edge products that are positively impacting today’s world,” said Hitt.

About PythonTek

PythonTek is a niche IT recruiting firm that focuses only on the Python community. Based in Austin, Texas, the company works to provide the best match for both job candidates and companies hiring Python developers. Company founders have more than 40 years of recruiting and IT consulting experience with small, medium and large enterprise companies.

Learn more: 

http://www.pythontek.com

Twitter: https://twitter.com/pythontek1

LinkedIn: https://www.linkedin.com/company/pythontek

Facebook: https://www.facebook.com/pythontek/

 

Why Python?

By Dr. Rob Flemming

PyDr.thon is an easy-to-learn yet deceptively powerful programming language. Its clear concise readable code, large standard library, and built-in testing framework make it ideal for rapid software development. While the long-standing industry anecdote that one python programmer can do in two months what two C++ programmers can do in one year may stretch the truth, it is certain that Python shaves hours, days, or even months off development projects depending on their size and complexity. What’s more, the small program size and readability of code facilitates the upkeep and maintenance of Python code.

When comparing Python to other languages it is not hard to find individual areas where they shine brighter: C++ is faster, R has better visualization, PERL can be even more concise. However, it is the unique combination of powerful features that attracts programmers and keeps the Python one of the most popular and required languages by clients, especially in the areas of web development, software development, data analysis and scientific computing.

The ease with which the basics of Python programming are picked up combined with a healthy job market for Python programmers tempts many of varying skill to promote themselves online as Python Developers.

When hiring for python projects or full-time employees it is important to incorporate a technical interviewer to assess the candidates’ depth of knowledge and familiarity with common Python frameworks. Ideally, a small test project would be assigned as well to evaluate both programming and communication skills.

Candidates’ contributions to code repositories and user communities should also be considered. The combination of these approaches along with a well-crafted job description will ensure that the best candidate is attracted and chosen.

Dr. Rob Flemming is the Co-Founder of hoytNIVA, a global technology, and human capital consulting firm. Dr. Flemming is a graduate of MIT, earned his Doctorate in Applied Mathematics from Northwestern University and has served in senior-level DoD Cryptology and private client leadership positions for over 20-years.

Terms of Use
Privacy Policy