-What it takes to build a successful online gaming community
The odds are against a business that is trying to build a branded online community (not unlike an online gaming community). According to a study performed by Deloitte and Beeline Labs, 35% of branded online communities have less than 100 members, and only 9% have more than 10,000 members. These businesses are investing heavily in these online communities, with 43% of the online communities operating with an annual budget of more than $50,000, and 6% with more than $1 million per year. These businesses realize that a successful online community has an incredibly high ROI, but the chances of failure are also high.
Agora Games, Inc. has successfully developed over a dozen successful online gaming communities, with total community members exceeding 1.2 million. Our most recent gaming communities have hundreds of thousands of community members. While complex software engineering is needed to efficiently transfer terabytes of in-game data to and from the web, we believe that half of the success of an online gaming community is due to the community design and management of the site.
Community design focuses on the creative process of “getting into gamers’ heads” to design web and in-game features that add value to the online community and video game. Before an engineer begins programming or a graphic designer starts drawing, our community designers develop a deep understanding of how the game will be played, and who will play it. The design process creates a framework for the online community experience that will help software engineers visualize how features should be implemented.
Community management includes forum and content moderation, hosting and promotion of tournaments, email communication, YouTube video production, and many other proactive actions. Community managers are on the front-lines interacting with community members. The community managers and volunteer community members are integral to building a successful online community because they are the game fanatics that jumpstart and guide the community through its lifecycle.
According to the Deloitte online community study, 34% of the communities only have part-time community managers, and only 16% have more than 6 full-time community managers. Directly after the launch of an online community, Agora dedicates multiple community managers exclusively to a community, which may scale-back over time. However, community managers are always needed for moderating the site, creating new tournaments, disseminating communications, and reporting community activity back to clients.
Developing a successful online gaming community requires so much more than just the engineering of the underlying software. Half of the effort should be expended on software development, and the other half should focus on the creative process and engaging early community adopters. Too many companies think “if we build it, they will come.” Gamers are especially sophisticated, and desire quirky sites that connect with their identity. The end goal is to have an active site with engaged community members that extends game life and enhances game play.
First (technical) post!
AgoraGames has been doing community sites for several years now, which means that some of them are pretty old and some of them have particularly unruly codebases, especially one such site. This site (which shall remain nameless to protect the guilty) ran perfectly fine at launch, but over the course of a year-and-a-half has slowly gone from speed-demon to slug-stuck-in-molasses.
(I apologize for the lack of accompanying images, etc. as I worked through this… next time, I’ll keep posterity in mind as I work through an interesting problem!)
I’m the System Administrator here, so it’s my job to make sure that everything stays running smoothly from the OS’s standpoint; thanks to our religious use of monit, the cause of the site’s slowness was pretty obvious: the mongrels running the site would balloon to consuming a gigabyte of memory within minutes, forcing monit to kill-and-restart the servers. This is a Bad Thing ™.
Finding and fixing memory leaks in Rails isn’t the easiest task in the world (as other bloggers have been apt to mention), but in this case, the path to fixing it was pretty straightforward.
First task was to set up a separate bank of mongrels for the forums: since the site is so old, it’s the only part of the site that gets much traffic these days. I told NGINX to proxy all requests starting with /forums to those. It worked just fine, and the forums were immediately quite quick. From looking at monit, it was pretty apparent that the forum mongrels didn’t have the memory leak, since they ran without problems.
So, to start digging into the main portion of the site, I booted up a local server in production mode and curled the homepage. It took a whopping 222 seconds (and, from looking at top, consumed 800MB of memory!). Looking at the debug logs, virtually all of that time was spend doing two queries: loading forum posts with associations, and loading tournaments with associations. Sounds a bit sketchy, no?
The home page’s controller was laid out very logically, so mapping the SQL queries in the log to method invokations in the controller was simple. There were two methods that were killing us, and they both looked something like this:
class ForumPost < ActiveRecord::Base ... def self.most_recent find(:first, :order => "forum_posts.created_at desc", :include => [:account, :forum_topic, :forum]) end ... end
Apparently, the find is loading up the (tens of thousands) of forum posts, with all the associated objects! Ruby doesn’t play nice when you have thousands of object in memory, and loading up hundreds of thousands of objects really makes things go south. Also makes sense why it didn’t become a problem for a while – it’s no big deal to load 100 object into memory (during testing and launch); but after a year and the site’s forum starts to fill…
But why? Isn’t this doing a :first? Shouldn’t it only be loading one object with associations? My guess is that Rails is loading all objects (with their associations) into an array and shifting off the first element. This isn’t smart behavior, but would explain what we’re seeing. (It’s also worth noting that this is using a fairly old version of Rails; I believe this silliness is fixed in more recent versions.)
Fixing this was actually trivial. Slap a :limit => 1 on there, or replace the :include with a :join. (Or, in the particular case above, remove the :include entirely. It’s not being used for anything relevant.)
I grepped the codebase for any other :includes, and fixed the ones that were broken. The site blazes along now and doesn’t leak, just like it did in its glory days.
Moral of the story: don’t use includes lightly. Make sure you know what your finds are actually doing. In our case, it translated into memory leaks, but it could just as easily translate into high database load, which is just as bad.
This past Thursday, we hosted Rock To A Cure in partnership with Bombers Burrito Bar. The verdict, a total success. The final count of money raised was over $1,400! It was night which will be remembered - or be a blurry vision for those who had one too many Agora Gamers - by all. While the dinner crowd chowed down some burritos all of us were busy hyping up the crowd, raising awareness for the Juvenile Diabetes Research Foundation (JDRF), and just being amiable hosts. Once the dinner crowd was finished with their burritos they were still hungry, hungry for Guitar Hero. Everyone started joining in to play Guitar Hero and Nicole was practically drowning in raffles. Luckily CZ was there to give a hand and rescue Nicole from the raffle abyss.
If things weren’t crazy enough already Mike DelPrete decided to “kick things up a notch” by suggesting some good ol' fashion arm wrestling. The burly Dave Loya took on contestants from the floor and went undefeated until the last round where one of the members of the crowd brought him down hard.
The event went well past 10PM ending close to 11PM where everyone was eagerly awaiting the raffle of the XBox 360 which was taken home by a fine gentleman who was there the entire night and bought over $100 in raffle tickets! Definitely well deserved. With Rock To A Cure being such a success this is only the beginning to many more charity events to come.