The Blog

Posts from 2015

Oct 2

Getting Started With Flask

By David Czarnecki

Over the last year and a half, I’ve been working with Flask on a large project. Flask is a “microframework for Python based on Werkzeug, Jinja 2 and good intentions.” I figured it would be a good time to distill down some of the knowledge gained in taking Flask through rapid development and test iterations.

Boilerplate

I wanted to choose a starting point with Flask development that would have sensible defaults and an application structure that I felt could evolve in a reasonable way as the application grew in scope from its initial prototype. Looking at a number of Flask template projects on GitHub, I finally settled on Flask Empty. Flask Empty has changed a bit since I first used it as you can now use cookiecutter to point to the Flask Empty repository on GitHub, give your project a name, and it will setup the basic Flask project structure for you.

Flask Empty will setup a number of default environments for you to start using. I changed their default environment names from dev and prod to development and production and added a test environment. All of that should be pretty familiar, if but slightly different in your preferred naming convention.

For the most part, all of our requirements were in the common requirements file. All other environments merely included the common requirements file using -r common.pip. In development we included requirements such as chai, freezegun, and honcho that we’d use only in testing the application.

Libraries

As far as libraries used in this application, they’re pretty standard for a JSON API using MySQL with a small number of asynchronous tasks run via Celery. Of note:

  • Flask-SQLAlchemy - For interaction with MySQL
  • SQLAlchemy-Utils - A collection of useful additions to SQLAchemy
  • Flask-Migrate - “SQLAlchemy database migrations for Flask applications using Alembic”
  • Celery - “Asynchronous task queue/job queue based on distributed message passing”
  • requests - A better HTTP library than Python’s standard urllib2
  • marshmallow - “ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes”

Until starting on this project, I was unaware of Python’s support for extra requirements for a project. For example, in our requirements file, we have the following:

requests[security]==2.7.0

This will pull in extra security-related requirements for the requests library. You can see where this is defined in the requests setup.py here.

Blueprints

As I knew this application would grow to provide a number of distinct services with some shared common code, being able to support a more modular application layout with Flask Blueprints made sense. Since this was a JSON API application, there was no need for views, so the layout and contents of each blueprint application differs slightly from the boilerplate in Flask Empty. I changed ours to be as follows:


/blueprint
    __init__.py
    application.py
    models.py
    serializers.py
    tasks.py
  • __init__.py imports the app from the application.py file:

from .application import app
  • application.py contains all of the API methods
  • models.py has all of the SQLAlchemy models used by the API
  • serializers.py contains any serializers appropriate for transforming models to be serialized to JSON
  • tasks.py has any specific asynchronous celery tasks used in the blueprint

Zero Downtime Deploys

As I’ve had extensive experience running our Ruby applications under unicorn, I used gunicorn to run this Python application. We can achieve zero downtime deploys with gunicorn by sending a USR2 signal to the master process and a QUIT to the old master process once the new master and workers have been spun up. I put together a gist detailing Zero downtime deploys with gunicorn last year.

The important part of that gist is the pre_fork function in the gunicorn configuration file.


def pre_fork(server, worker):
    old_pid_file = '{}/pids/unicorn.pid.oldbin'.format(os.getcwd())

    if os.path.isfile(old_pid_file):
        with open(old_pid_file, 'r') as pid_contents:
            try:
                old_pid = int(pid_contents.read())
                if old_pid != server.pid:
                    os.kill(old_pid, signal.SIGQUIT)
            except Exception as err:
                pass

pre_fork=pre_fork

You might also want to include the setproctitle library in your requirements so that you can use the proc_name configuration option in gunicorn for better process discovery.

For example:


someuser 13617     1  0 Sep25 ?        00:01:15 gunicorn: master [project-name]
someuser 13635 13617  0 Sep25 ?        00:01:23 gunicorn: worker [project-name]
someuser 13637 13617  0 Sep25 ?        00:01:54 gunicorn: worker [project-name]
someuser 13639 13617  0 Sep25 ?        00:01:53 gunicorn: worker [project-name]

Transactional Testing with SQLAlchemy

After the project had been in development for about 6 months (maybe longer), it was clear that one bottleneck in our development process was in testing. Our test suite was approaching 10 minutes to run the entire suite. We were able to run individual test files, but we would have to run the full test suite before pushing our changes up to ensure we didn’t break any of the larger application. Being able to run individual test files was a band-aid, not a long-term solution. In the end, we setup our project to use transactional testing with the database. This meant we were not dropping and then creating all the database tables each test. Coming from a Ruby and Rails background where this is standard practice, I had to set this up to work with the particulars of our test suite.

There’s a page in the SQLAlchemy documentation that describes this situation, Joining a Session into an External Transaction (such as for test suites). A number of posts I found useful when developing the particulars for our environment are:

The last post there contained the useful tidbit of knowledge to know that if we have an API call that will perform a rollback of the current transaction, that we need to start a nested transaction within the test code. For example:


    def test_something(self):
        # setup code to create some data in the DB
        db.session.begin_nested()
        response = self.client.post(
            '/blueprint/',
            data=json.dumps(dict(key=value)),
            headers=client_key_headers())
        # If the API call results in a rollback, the data
        # in the DB before the begin_nested() call will
        # still be there.
        #
        # All of the database data is cleared at the
        # end of the test.

By migrating to using transactional testing, I was able to bring the time to run our full test suite from 10 minutes to around 30 seconds.


(project-name)@orth ➜  project-name python manage.py -c Test test
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 488 tests in 34.686s

OK

And for that band-aid for running a single test file, should you need it:


class Test(Command):

    """
    Run tests
    """

    start_discovery_dir = "tests"

    def get_options(self):
        return [
            Option('--start_discover', '-s', dest='start_discovery',
                   help='Directory to look for tests in',
                   default=self.start_discovery_dir),
            Option('--pattern', '-p', dest='pattern',
                   help='Pattern to search for features',
                   default=None)
        ]

    def run(self, start_discovery, pattern):
        import unittest2

        if os.path.exists(start_discovery):
            argv = [config.project_name, "discover"]
            argv += ["-s", start_discovery]

            if pattern:
                pattern = pattern if pattern.endswith(".py") else pattern + ".py"
                print(" *** Filtering by '%s'" % pattern)
                argv += ["-p", pattern]



            unittest2.main(argv=argv)
        else:
            print("Directory '%s' was not found in project root." %
                  start_discovery)

Random Tidbits

I did notice that logging of international characters was throwing errors for us, so I had to add the following to the boilerplate code that is setup by Flask Empty:


def configure_logger(app, config):
    import sys
    reload(sys)
    sys.setdefaultencoding('utf-8')
    # Other logging setup

For naming our internal decorators, the following convention was chosen:


def decorator_name(api_function):
    @wraps(api_function)
    def decorated_decorator_name_function(*args, **kwargs):
        # Do some useful work
        return api_function(*args, **kwargs)
    return decorated_decorator_name_function

This made it easier for debugging exceptions that we log into Sentry when we had multiple decorators being used around a single API call.

Conclusion

This is a very high-level dump of some things I learned using Flask for the past year and change. I might distill this into a more concrete sample application. However, I think there are still some useful bits of knowledge for you to jumpstart your own development with Flask.

Mar 21

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

handlebars.rb

handlebars.rb is a set of Ruby bindings for Handlebars.js. One of our engineers, Matt Wilson, is now a maintainer of the project.

Contributor(s): Matt Wilson (GitHub, Twitter)

leaderboard

leaderboard allows you to build leaderboards using Redis. The leaderboard (Ruby) and leaderboard-python libraries were updated this week to fix an issue with the TieRankingLeaderboard class, to allow for member data to be passed to the change_score_for(...) method, and to add an :include_missing option in leaderboard request options to change whether or not to include missing members in the result.

Contributor(s): David Czarnecki (GitHub, Twitter)

nginx Google OAuth

nginx-google-oauth is a Lua module to add Google OAuth to nginx. We released version 1.1.0 this week which integrates a pull request for security and usability enhancements.

Contributor(s): Eric Schwimmer (GitHub)

stache

stache is our Rails 3.x and Rails 4.x compatible Mustache/Handlebars Template Handler, with support for partials and a couple extra niceties to make sharing the raw templates with client-side javascript a little easier. The 1.1.1 release includes gugfixes from many new contributors and some expanded test coverage!

Contributor(s): Matt Wilson (GitHub, Twitter)

Jan 31

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

chai

chai provides a very easy to use api for mocking/stubbing your python objects, patterned after the Mocha library for Ruby. This week we released chai 1.1.0, which adds support for PyPy.

Contributor(s): Aaron Westendorf (GitHub, Twitter)