The Blog

Oct 2

Getting Started With Flask

By David Czarnecki

Over the last year and a half, I’ve been working with Flask on a large project. Flask is a “microframework for Python based on Werkzeug, Jinja 2 and good intentions.” I figured it would be a good time to distill down some of the knowledge gained in taking Flask through rapid development and test iterations.

Boilerplate

I wanted to choose a starting point with Flask development that would have sensible defaults and an application structure that I felt could evolve in a reasonable way as the application grew in scope from its initial prototype. Looking at a number of Flask template projects on GitHub, I finally settled on Flask Empty. Flask Empty has changed a bit since I first used it as you can now use cookiecutter to point to the Flask Empty repository on GitHub, give your project a name, and it will setup the basic Flask project structure for you.

Flask Empty will setup a number of default environments for you to start using. I changed their default environment names from dev and prod to development and production and added a test environment. All of that should be pretty familiar, if but slightly different in your preferred naming convention.

For the most part, all of our requirements were in the common requirements file. All other environments merely included the common requirements file using -r common.pip. In development we included requirements such as chai, freezegun, and honcho that we’d use only in testing the application.

Libraries

As far as libraries used in this application, they’re pretty standard for a JSON API using MySQL with a small number of asynchronous tasks run via Celery. Of note:

  • Flask-SQLAlchemy - For interaction with MySQL
  • SQLAlchemy-Utils - A collection of useful additions to SQLAchemy
  • Flask-Migrate - “SQLAlchemy database migrations for Flask applications using Alembic”
  • Celery - “Asynchronous task queue/job queue based on distributed message passing”
  • requests - A better HTTP library than Python’s standard urllib2
  • marshmallow - “ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes”

Until starting on this project, I was unaware of Python’s support for extra requirements for a project. For example, in our requirements file, we have the following:

requests[security]==2.7.0

This will pull in extra security-related requirements for the requests library. You can see where this is defined in the requests setup.py here.

Blueprints

As I knew this application would grow to provide a number of distinct services with some shared common code, being able to support a more modular application layout with Flask Blueprints made sense. Since this was a JSON API application, there was no need for views, so the layout and contents of each blueprint application differs slightly from the boilerplate in Flask Empty. I changed ours to be as follows:


/blueprint
    __init__.py
    application.py
    models.py
    serializers.py
    tasks.py
  • __init__.py imports the app from the application.py file:

from .application import app
  • application.py contains all of the API methods
  • models.py has all of the SQLAlchemy models used by the API
  • serializers.py contains any serializers appropriate for transforming models to be serialized to JSON
  • tasks.py has any specific asynchronous celery tasks used in the blueprint

Zero Downtime Deploys

As I’ve had extensive experience running our Ruby applications under unicorn, I used gunicorn to run this Python application. We can achieve zero downtime deploys with gunicorn by sending a USR2 signal to the master process and a QUIT to the old master process once the new master and workers have been spun up. I put together a gist detailing Zero downtime deploys with gunicorn last year.

The important part of that gist is the pre_fork function in the gunicorn configuration file.


def pre_fork(server, worker):
    old_pid_file = '{}/pids/unicorn.pid.oldbin'.format(os.getcwd())

    if os.path.isfile(old_pid_file):
        with open(old_pid_file, 'r') as pid_contents:
            try:
                old_pid = int(pid_contents.read())
                if old_pid != server.pid:
                    os.kill(old_pid, signal.SIGQUIT)
            except Exception as err:
                pass

pre_fork=pre_fork

You might also want to include the setproctitle library in your requirements so that you can use the proc_name configuration option in gunicorn for better process discovery.

For example:


someuser 13617     1  0 Sep25 ?        00:01:15 gunicorn: master [project-name]
someuser 13635 13617  0 Sep25 ?        00:01:23 gunicorn: worker [project-name]
someuser 13637 13617  0 Sep25 ?        00:01:54 gunicorn: worker [project-name]
someuser 13639 13617  0 Sep25 ?        00:01:53 gunicorn: worker [project-name]

Transactional Testing with SQLAlchemy

After the project had been in development for about 6 months (maybe longer), it was clear that one bottleneck in our development process was in testing. Our test suite was approaching 10 minutes to run the entire suite. We were able to run individual test files, but we would have to run the full test suite before pushing our changes up to ensure we didn’t break any of the larger application. Being able to run individual test files was a band-aid, not a long-term solution. In the end, we setup our project to use transactional testing with the database. This meant we were not dropping and then creating all the database tables each test. Coming from a Ruby and Rails background where this is standard practice, I had to set this up to work with the particulars of our test suite.

There’s a page in the SQLAlchemy documentation that describes this situation, Joining a Session into an External Transaction (such as for test suites). A number of posts I found useful when developing the particulars for our environment are:

The last post there contained the useful tidbit of knowledge to know that if we have an API call that will perform a rollback of the current transaction, that we need to start a nested transaction within the test code. For example:


    def test_something(self):
        # setup code to create some data in the DB
        db.session.begin_nested()
        response = self.client.post(
            '/blueprint/',
            data=json.dumps(dict(key=value)),
            headers=client_key_headers())
        # If the API call results in a rollback, the data
        # in the DB before the begin_nested() call will
        # still be there.
        #
        # All of the database data is cleared at the
        # end of the test.

By migrating to using transactional testing, I was able to bring the time to run our full test suite from 10 minutes to around 30 seconds.


(project-name)@orth ➜  project-name python manage.py -c Test test
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 488 tests in 34.686s

OK

And for that band-aid for running a single test file, should you need it:


class Test(Command):

    """
    Run tests
    """

    start_discovery_dir = "tests"

    def get_options(self):
        return [
            Option('--start_discover', '-s', dest='start_discovery',
                   help='Directory to look for tests in',
                   default=self.start_discovery_dir),
            Option('--pattern', '-p', dest='pattern',
                   help='Pattern to search for features',
                   default=None)
        ]

    def run(self, start_discovery, pattern):
        import unittest2

        if os.path.exists(start_discovery):
            argv = [config.project_name, "discover"]
            argv += ["-s", start_discovery]

            if pattern:
                pattern = pattern if pattern.endswith(".py") else pattern + ".py"
                print(" *** Filtering by '%s'" % pattern)
                argv += ["-p", pattern]



            unittest2.main(argv=argv)
        else:
            print("Directory '%s' was not found in project root." %
                  start_discovery)

Random Tidbits

I did notice that logging of international characters was throwing errors for us, so I had to add the following to the boilerplate code that is setup by Flask Empty:


def configure_logger(app, config):
    import sys
    reload(sys)
    sys.setdefaultencoding('utf-8')
    # Other logging setup

For naming our internal decorators, the following convention was chosen:


def decorator_name(api_function):
    @wraps(api_function)
    def decorated_decorator_name_function(*args, **kwargs):
        # Do some useful work
        return api_function(*args, **kwargs)
    return decorated_decorator_name_function

This made it easier for debugging exceptions that we log into Sentry when we had multiple decorators being used around a single API call.

Conclusion

This is a very high-level dump of some things I learned using Flask for the past year and change. I might distill this into a more concrete sample application. However, I think there are still some useful bits of knowledge for you to jumpstart your own development with Flask.

Mar 21

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

handlebars.rb

handlebars.rb is a set of Ruby bindings for Handlebars.js. One of our engineers, Matt Wilson, is now a maintainer of the project.

Contributor(s): Matt Wilson (GitHub, Twitter)

leaderboard

leaderboard allows you to build leaderboards using Redis. The leaderboard (Ruby) and leaderboard-python libraries were updated this week to fix an issue with the TieRankingLeaderboard class, to allow for member data to be passed to the change_score_for(...) method, and to add an :include_missing option in leaderboard request options to change whether or not to include missing members in the result.

Contributor(s): David Czarnecki (GitHub, Twitter)

nginx Google OAuth

nginx-google-oauth is a Lua module to add Google OAuth to nginx. We released version 1.1.0 this week which integrates a pull request for security and usability enhancements.

Contributor(s): Eric Schwimmer (GitHub)

stache

stache is our Rails 3.x and Rails 4.x compatible Mustache/Handlebars Template Handler, with support for partials and a couple extra niceties to make sharing the raw templates with client-side javascript a little easier. The 1.1.1 release includes gugfixes from many new contributors and some expanded test coverage!

Contributor(s): Matt Wilson (GitHub, Twitter)

Jan 31

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

chai

chai provides a very easy to use api for mocking/stubbing your python objects, patterned after the Mocha library for Ruby. This week we released chai 1.1.0, which adds support for PyPy.

Contributor(s): Aaron Westendorf (GitHub, Twitter)

Dec 12

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

globalize

globalize builds on the I18n API in Ruby on Rails to add model translations to ActiveRecord models. A pull request to fix a problem with dup and instance variables was recently integrated.

Contributor(s): Matthew Wilson (GitHub, Twitter)

Dec 5

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

chai

chai provides a very easy to use api for mocking/stubbing your python objects, patterned after the Mocha library for Ruby. This week we released chai 1.0.1. chai 1.0.1 fixes tests and unbound method support in Python 3.

Contributor(s): Graylin Kim (GitHub) and Pierre-Yves Chibon (GitHub)

haigha

haigha is our simple to use client library for interacting with AMQP brokers. In 0.7.2, we fixed installation under Windows.

Contributor(s): Anton Khlynovskiy (GitHub)

Nov 14

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

leaderboard

leaderboard allows you to build leaderboards using Redis. The leaderboard (Ruby), leaderboard-python and leaderboard-javascript libraries all saw updates this week. All 3 updates were to add support for the change_score_for(...) method in the TieRankingLeaderboard class.

Contributor(s): David Czarnecki (GitHub, Twitter)

Oct 17

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

chai

chai provides a very easy to use api for mocking/stubbing your python objects, patterned after the Mocha library for Ruby. This week we released chai 1.0.0. chai 1.0.0 added spies, a variation of an expectation that asserts a call has happened.

Contributor(s): Aaron Westendorf (GitHub, Twitter)

sidekiq-failures

sidekiq-failures “keeps track of Sidekiq failed jobs and adds a tab to the Web UI to let you browse them.” We submitted a pull request to add the paging links to the bottom of a page of failures.

Contributor(s): David Czarnecki (GitHub, Twitter)

stache

stache is our Rails 3.x and Rails 4.x compatible Mustache/Handlebars Template Handler, with support for partials and a couple extra niceties to make sharing the raw templates with client-side javascript a little easier. In the stache 1.1.0 release, we accepted a pull request to cache compiled templates between requests. You may see a 200-300% boost in rendering performance.

Contributor(s): Markus Harmsen (GitHub.

Aug 15

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

leaderboard

leaderboard allows you to build leaderboards using Redis. The leaderboard (Ruby), leaderboard-python and leaderboard-coffeescript libraries all saw updates a few weeks ago. In leaderboard 3.7.0, we added TieRankingLeaderboard and CompetitionRankingLeaderboard classes to handle ties in leaderboards. Specifics on ranking between those 2 different leaderboards are described in the alternate leaderboard types documentation. This same functionality was ported to leaderboard-python 3.0.0 and leaderboard-coffeescript 1.8.0.

Contributor(s): David Czarnecki (GitHub, Twitter)

Jul 28

Hackathon 15

By Sam Toews

Thursday, July 24 - Friday, July 25 was Agora’s first ever Hydra themed Hackathon.

Kafka Scripts

Andrew implemented a simple utility allowing users to view filtered subsets of Hydra Studio API traffic (request response pairs). Filtering is done by request time window, environment, and API key as well as request processing time and response HTTP status code. This tool can be used to tail traffic as it occurs as well as view historical traffic via a unified interface. It’s a simple program that is around 100 lines of Python reading messages from a Kafka topic.

Oculus Unity Demo + Hydra

John worked on getting familiar with Agora’s Hydra Studio features as well as our Unity SDK. Looking through the Unity store there’s a lot of services similar to ours integrated into the Unity demo projects to help developers get an idea of how to use them. Following this pattern he took the 2D example project and started integrating Hydra features into it.

Hydra UI for Achievements

Nick worked on making a customizable yet easy to use achievement system for the Hydra SDK for Unity3d. It was made with Unity’s Editor Script, so users can adjust all kinds of settings (Display time, color, trophy/picture) without looking at any code. Proof of concept as to what tools for unity can be made once Hydra Studio comes out of beta.

Tournament Seeding Gem UI

Matt worked on a plain-ruby library for managing and tracking multi-round tournaments. It knows about two types of tournament structures right now (random pairing and swiss pairing), with the promise of more to come, via bracket-tree and a couple swiss variations. You can customize various things about the tournament and also track player metadata, so it should be pretty easy to serialize data to and from.

Next steps are a more formal data serialization layer and cleaning up some known edge cases (mostly around byes) in the current code, then wrapping bracket_tree to handle elimination style tourneys.

It’s not open source yet, but it will be once he has cleaned up the codebase.

Text Web Game (MUD)

David A. worked on a text-based, RPG style, game with Hydra integration. You can play it here.

It is written in Dart, and compiled to javascript.

It uses the following Hydra features:

  • Match Templates: Store metadata about how to generate maps
  • Matches: Store map data in the match
  • Profiles: Save basic stats
  • Achievements: Display achievements for the logged in user

Lua for OAuth

Aaron finished extracting last hack-a-thon hacks into a legit project, adding most of the features and documentation a user would need to add Google OAuth to their nginx hosts, available here.

Tie Handling in Leaderboard Libraries

David Cz. worked on tie handling in the Ruby, Python and JavaScript leaderboard libraries. Two types of tie handling are now supported, Tie Ranking and Competition Ranking. In Tie Ranking, you define a leaderboard where members with the same score are given the same rank. In Competition Ranking, you define a leaderboard where members with the same score will have the same rank, and then a gap is left in the ranking numbers. The documentation for each of the libraries goes into more detail on what the tie rankings look like.

Confirm with Reveal Bug

Jack fixed a bug with Agora’s open-source “confirm-with-reveal” JavaScript plugin that stopped it from working with Rails' unobtrusive JavaScript jQuery plugin.

See here for details/commit.

He also made some progress toward extending that plugin to allow users to provide an existing popup already on the page as the confirmation dialog, rather than using one generated by the plugin.

HTML/CSS

Brett used the day to learn the basics of HTML and CSS using codeacademy. He made a really simple HTML file and a simple CSS file to show off a little bit of what he learned.

Jun 13

Game Face

By David Czarnecki

“Game Face” will be our weekly round-up of our internal and external open source work here at Agora Games. Internal open source refers to our public projects that you can find over at our Agora Games GitHub account. External open source work refers to projects that we contribute to in off-hours and may or may not have anything to do with video games because we’re swell folks like that. Pretty simple right? Here goes…

I was going to use the excuse that we’re still playing a bit of catch up since migrating our blog, but I’ve just been lazy :P

activity_feed

activity_feed is a Ruby library for creating activity feeds or timelines backed by Redis. In the 3.1.0 release, we added a trim_to_size(user_id, size, aggregate = ActivityFeed.aggregate) method to trim activity feed to a certain number of items.

Contributor(s): David Czarnecki (GitHub, Twitter)

chai

chai provides a very easy to use api for mocking/stubbing your python objects, patterned after the Mocha library for Ruby. As of 0.4.8, released this past week, we addressed PEP8 compliance as well as fixing an issue to not pollute the global module namespace.

Contributor(s): Aaron Westendorf (GitHub, Twitter)