Applications go through various stages of development. The shining moments for an application are usually in the early stages. There isn’t much complexity, there is plenty of time left until ‘launch day’, and developers are usually relaxed when coding. Eventually every application suffers from poor feature requests, bad design, and rushed code. The beautiful grassy plain of peace your application used to be is turned into a bustling city.
Boston is a city in Massachusetts that was founded a whole buncha years ago. Back in the day, the guys pushing wagons around, and mules laden with baggage didn’t care about paths that they plodded into the ground. All they cared about was getting their goods to where ever they needed to be. If there was a ginormous rock in the way, they could just path around it. After a while, cars started getting pretty cool. Cars need roads, so all the roads were built right on top of those paths.
At that time, it was easier to just build a road on top of the path. No one knew the windy mess of wagon-path converted roads would cause tons of lost tourists and traffic accidents today. It might have been better to just nuke the ginormous rock out of the way, and make the road straight. Unfortunately, that ginormous rock is some type of historical monument now, so it’s not possible anymore.
Anything is possible in software (with limit to it’s own scope), the only constraint is time. Time is the key element here, it’s something you can’t get back. Every decision that is made in software has an effect on how long something else will take. Every second saved by hacking around something today, is going to cost you two seconds tomorrow.
“Tomorrow is so far away though, and I need this to work now." When the pressure is on, and time is short, this is a common approach to decision making. It’s in these essential minutes during an applications life span, that hacks start to leak into every crevice. To make things worse, instead of refactoring the hacks out, time is spent building more new features. Eventually those crevices become a major fault line, and it causes changes to the code base to take three times as long. By this time, new developers have no chance of contributing, and the existing developers hate working on the application because everything they do causes an earthquake.
A major problem with software development is the existing solution.
Things change, standards form, upgrades happen. The best idea you ever had yesterday, can probably be done better today. Don’t let your legacy code stink take over, get rid of it, refactor it, and upgrade it. There is always a better way to approach a problem in software and in most every case the quickest solution is the worst solution.
That’s right, myself and three others flew over to Anaheim, California to the Anaheim Convention Center for Blizzard’s 3rd Annual BlizzCon! This year Blizzard featured all three of its games at the Con: World of Warcraft, Starcraft and Diablo. I decided that despite the embarrassment which I might endure, I would dress up as a Blood Elf, featured in Blizzard’s World of Warcraft: The Burning Crusade. I was a bit nervous at first. While approaching the convention center I saw no one dressed up in costume in sight, and there I was, sporting these ridiculous elf ears. Once I entered the convention center I was astounded. About 40% of the 15,000 people who showed up for the convention were dressed up in the most elaborate outfits. From bulked up Horde warlocks to Alliance elves skimpy bikinis, it was all there. Our goodie bags were amazing too! Filled to the brim with fun swag, including a blow up Frostmourne, keychains, Starcraft “Zerg Creep”, Diablo III stress balls, BlizzCon hand sanitizer, mints, Warcraft trading card game starter kit, bottle opener, n00b tissues, paper WoW masks, Starcraft II beta key, polar bear mount for WoW, mouse pads, and these nifty Blizzard Digipass Authenticator by Vasco!
Tickets to Blizzcon were about $100, which was pretty good considering everything at the Con other than food was free. They had free pictures with a background of the game of your choice. You can view more of these photo’s at www.politeinpublic.com Someone actually proposed in one of the photos - and don’t worry, she said yes!
During many of the developer panels - where they explained the new game World of Warcraft: Wrath of the Lich King - I was texting our lead QA pro Dave Loya of all of the awesome new details for the new unveiled class, The Death Knight. The Con also had several computer set-ups for participants to play Starcraft II, World of Warcraft: Wrath of the Lich King and Diablo III. I ended up demoing Diablo III, and it truly looks fantastic! They have some more work to do on the game, but they did unveil a new class as well, the Wizard!
The Con itself was a sight to see. There were meet and greets galore! Participants had the opportunity to meet with Blizzard employees and obtain free swag such as professional pictures, figure prints, trading cards and so much more! We all waited in line for about two hours to get plush Murocs, which Jinx was all sold out by 6 p.m the first evening. I tried to get pictures of everyone that were dressed up, I even found a few fellow Blood Elves! My crew was very recognizable due to the fact that we were all wearing red matching Tabards. They are a exact replica of our guild tabards in game which are The Bloodrocutioners. Chelsea (Designer, 1st Playable Productions) made all of the Tabards worn at the Con all on her own with some sewing help from our other friends who came along for the trip. The painstaking work totally paid off as people were shouting “Look! Guildies!” and snapping pictures in our faces. I think we averaged about 150 photos taken of us by random people per day. By 10p.m. both nights I was seeing spots. I couldn’t believe that if you were dressed up, you were photographed like a celebrity. I even had one kid ask us to sign his Warcraft book with our names, realm name and guild name. There were also many gals paid by Blizzard to model official WoW costumes! Also, My crew and myself were interviewed my two different websites!
The final night had a great concert by Level 80 Elite Tauren Chieftain, which is Blizzards metal band. Even the CEO was in the band! The Orchestra concluded the evening with powerful music from Warcraft, Starcraft and Diablo. The theme music from World of Warcraft I must say, did give me goosebumps. I encourage anyone who plays any of the Blizzard games to check out the next Con in 2009, because I know I will be there! It was an amazing time and so surreal with the ambience of gaming and fantasy combined into two fun filled days which was BlizzCon 2008!
Our VP of Sales, Peter Ryan is famous! Well, maybe not, but we’re proud of his post on Gamasutra. Check it out here:
Sometimes, things go wrong.
Sometimes, things go horribly, horribly wrong, and there’s nothing you can do to stop the slow-motion train wreck unfolding before your very eyes. (Like that one senior prom where you spilled your dinner all over your date’s dress. On the upside, you didn’t need that $200 anyway.) When things go that awry, there’s no way to fix it. (She never did return your phone calls, did she?) The only thing you can do with a mistake like that is to make sure it never, ever happens again.
For most of the sites we produce, we use MySQL as a backend. While it’s extremely rare for the MySQL server to catastrophically fail, it happens, and the more MySQL servers you have, the more likely it is to happen at least once. To keep this from causing problems for us, we replicate our databases (that is, maintain a live copy of them). This provides us with two benefits:
So, let me walk you through setting up a database slave from scratch.
I am starting with two identical machines that are running identical instances of MySQL. The first of these, Dragon, is going to be the master. The second of these, Unicorn, is going to be the slave. (So I name my servers after mythical beasts. What’s wrong with that?) I am assuming a reasonable level of knowledge with the UNIX shell. If you lack familiarity, you really should spend some time learning. It will make you a better programmer, system administrator, friend, spouse, and human being.
1. First, you should have the following items set in your MySQL configuration (/etc/mysql/my.cnf). If you don’t, add them: they can go anywhere under the [mysqld] heading.
server-id = 1 log-bin = /var/lib/mysql/binlog/mysql-bin expire_logs_days = 1 max_binlog_size = 100M relay_log = /var/lib/mysql/binlog/mysqld-relay-bin relay_log_index = /var/lib/mysql/binlog/mysqld-relay-bin.index
The server-id should be some unique number, not shared with any other machine on your network. Replication occurs by copying around binary log files; you should set expirelogsdays so that you don’t end up with a huge pileup of data clogging up your disk drives (as they will eventually eat up your hard disk). We set binary logs to expire after 1 day, since our disks have a tendency to fill up on us quickly (for example, one of our Guitar Hero master databases has a 6GB binary log directory, even though it expires data after a single day!) Be advised that if you are worried about your slaves falling more than a day behind, you should keep logs around for a greater duration. (However, from our experience, if your slaves fall more than a day behind, you have bigger problems.) Finally, you can put datadir, relay_log, and relay_log_index wherever you please; above are just where we use. (Make sure, however, that they exist and are writable by the mysql user!)
You should restart the server after changing the settings. On Debian and Ubuntu, you can do this via “/etc/init.d/mysql restart”.
2. Your slave should have the same options defined as above in its MySQL configuration. The server-id option, however, must be different from the master (in our case, setting it to “2” will do). All the other options can be whatever you please.
Also restart this server when it’s set as you like.
3. You should grant permission for the slave to read data from the master. This can be done from the MySQL console on the master (mysql -h dragon -u root). Assuming the user name “foo” and the password “bar”:
mysql> GRANT REPLICATION SLAVE ON \*.\* TO 'foo'@'unicorn' IDENTIFIED BY 'bar';
You can, of course, restrict which databases and tables you grant permissions with. We don’t in this particular case.
4. You now need to dump all of the data on the master, copy it over to the slave, and import it there; this is necessary even if your databases are empty, as it will initialize the slave regardless.
On the master (this will require your MySQL root password):
mysqldump -uroot -p --all-databases --single-transaction --master-data=1 | gzip >master.sql.gz
If you don’t have a root password (which is generally a bad idea, but fine if your MySQL server is on a private network), you should omit the
You can copy this file over to the slave server using
scp master.sql.gz unicorn:
Finally, import the dump on the slave:
zcat master.sql.gz | mysql -uroot
5. Tell the slave to mirror the master.
mysql> CHANGE MASTER TO MASTER_HOST='dragon', MASTER_USER='foo', MASTER_PASSWORD='bar';
6. Start the slave!
mysql> START SLAVE;
You can now monitor the progress of your slave using
SHOW SLAVE STATUS. (If you use a
\G instead of a semicolon to terminate the line, it will print out the data in a much more readable format.)
mysql> SHOW SLAVE STATUS\G
There could be a number of errors, which are beyond the scope of fixing here. (In my case, we ran into issues with binary logs and the binary log index files.) These can generally be troubleshooted (troubleshot?) by reading the output of syslog (
tail -f /var/log/syslog) and fixing the problem one piece at a time; it helps to have a lot of UNIX experience, though.) If all goes well, though,
SHOW SLAVE STATUS will have lines something like the following:
Slave_IO_State: Waiting for master to send event Master_Host: dragon Master_Port: 3306 Slave_IO_Running: Yes Slave_SQL_Running: Yes Seconds_Behind_Master: 12887
This indicates that the slave is running and in the process of catching up to the master, and in time will be synchronized with it.
And, that’s (hopefully!) all there is to it! Now, with your new slave, you can go forth and conquer the world! (Or, less ambitiously, cover your backside when you accidentally delete something.)
Obligatory CTO Note: Make sure you keep, and regularly verify your daily backups too!
-What it takes to build a successful online gaming community
The odds are against a business that is trying to build a branded online community (not unlike an online gaming community). According to a study performed by Deloitte and Beeline Labs, 35% of branded online communities have less than 100 members, and only 9% have more than 10,000 members. These businesses are investing heavily in these online communities, with 43% of the online communities operating with an annual budget of more than $50,000, and 6% with more than $1 million per year. These businesses realize that a successful online community has an incredibly high ROI, but the chances of failure are also high.
Agora Games, Inc. has successfully developed over a dozen successful online gaming communities, with total community members exceeding 1.2 million. Our most recent gaming communities have hundreds of thousands of community members. While complex software engineering is needed to efficiently transfer terabytes of in-game data to and from the web, we believe that half of the success of an online gaming community is due to the community design and management of the site.
Community design focuses on the creative process of “getting into gamers’ heads” to design web and in-game features that add value to the online community and video game. Before an engineer begins programming or a graphic designer starts drawing, our community designers develop a deep understanding of how the game will be played, and who will play it. The design process creates a framework for the online community experience that will help software engineers visualize how features should be implemented.
Community management includes forum and content moderation, hosting and promotion of tournaments, email communication, YouTube video production, and many other proactive actions. Community managers are on the front-lines interacting with community members. The community managers and volunteer community members are integral to building a successful online community because they are the game fanatics that jumpstart and guide the community through its lifecycle.
According to the Deloitte online community study, 34% of the communities only have part-time community managers, and only 16% have more than 6 full-time community managers. Directly after the launch of an online community, Agora dedicates multiple community managers exclusively to a community, which may scale-back over time. However, community managers are always needed for moderating the site, creating new tournaments, disseminating communications, and reporting community activity back to clients.
Developing a successful online gaming community requires so much more than just the engineering of the underlying software. Half of the effort should be expended on software development, and the other half should focus on the creative process and engaging early community adopters. Too many companies think “if we build it, they will come.” Gamers are especially sophisticated, and desire quirky sites that connect with their identity. The end goal is to have an active site with engaged community members that extends game life and enhances game play.
First (technical) post!
AgoraGames has been doing community sites for several years now, which means that some of them are pretty old and some of them have particularly unruly codebases, especially one such site. This site (which shall remain nameless to protect the guilty) ran perfectly fine at launch, but over the course of a year-and-a-half has slowly gone from speed-demon to slug-stuck-in-molasses.
(I apologize for the lack of accompanying images, etc. as I worked through this… next time, I’ll keep posterity in mind as I work through an interesting problem!)
I’m the System Administrator here, so it’s my job to make sure that everything stays running smoothly from the OS’s standpoint; thanks to our religious use of monit, the cause of the site’s slowness was pretty obvious: the mongrels running the site would balloon to consuming a gigabyte of memory within minutes, forcing monit to kill-and-restart the servers. This is a Bad Thing ™.
Finding and fixing memory leaks in Rails isn’t the easiest task in the world (as other bloggers have been apt to mention), but in this case, the path to fixing it was pretty straightforward.
First task was to set up a separate bank of mongrels for the forums: since the site is so old, it’s the only part of the site that gets much traffic these days. I told NGINX to proxy all requests starting with /forums to those. It worked just fine, and the forums were immediately quite quick. From looking at monit, it was pretty apparent that the forum mongrels didn’t have the memory leak, since they ran without problems.
So, to start digging into the main portion of the site, I booted up a local server in production mode and curled the homepage. It took a whopping 222 seconds (and, from looking at top, consumed 800MB of memory!). Looking at the debug logs, virtually all of that time was spend doing two queries: loading forum posts with associations, and loading tournaments with associations. Sounds a bit sketchy, no?
The home page’s controller was laid out very logically, so mapping the SQL queries in the log to method invokations in the controller was simple. There were two methods that were killing us, and they both looked something like this:
class ForumPost < ActiveRecord::Base ... def self.most_recent find(:first, :order => "forum_posts.created_at desc", :include => [:account, :forum_topic, :forum]) end ... end
Apparently, the find is loading up the (tens of thousands) of forum posts, with all the associated objects! Ruby doesn’t play nice when you have thousands of object in memory, and loading up hundreds of thousands of objects really makes things go south. Also makes sense why it didn’t become a problem for a while – it’s no big deal to load 100 object into memory (during testing and launch); but after a year and the site’s forum starts to fill…
But why? Isn’t this doing a :first? Shouldn’t it only be loading one object with associations? My guess is that Rails is loading all objects (with their associations) into an array and shifting off the first element. This isn’t smart behavior, but would explain what we’re seeing. (It’s also worth noting that this is using a fairly old version of Rails; I believe this silliness is fixed in more recent versions.)
Fixing this was actually trivial. Slap a :limit => 1 on there, or replace the :include with a :join. (Or, in the particular case above, remove the :include entirely. It’s not being used for anything relevant.)
I grepped the codebase for any other :includes, and fixed the ones that were broken. The site blazes along now and doesn’t leak, just like it did in its glory days.
Moral of the story: don’t use includes lightly. Make sure you know what your finds are actually doing. In our case, it translated into memory leaks, but it could just as easily translate into high database load, which is just as bad.
This past Thursday, we hosted Rock To A Cure in partnership with Bombers Burrito Bar. The verdict, a total success. The final count of money raised was over $1,400! It was night which will be remembered - or be a blurry vision for those who had one too many Agora Gamers - by all. While the dinner crowd chowed down some burritos all of us were busy hyping up the crowd, raising awareness for the Juvenile Diabetes Research Foundation (JDRF), and just being amiable hosts. Once the dinner crowd was finished with their burritos they were still hungry, hungry for Guitar Hero. Everyone started joining in to play Guitar Hero and Nicole was practically drowning in raffles. Luckily CZ was there to give a hand and rescue Nicole from the raffle abyss.
If things weren’t crazy enough already Mike DelPrete decided to “kick things up a notch” by suggesting some good ol' fashion arm wrestling. The burly Dave Loya took on contestants from the floor and went undefeated until the last round where one of the members of the crowd brought him down hard.
The event went well past 10PM ending close to 11PM where everyone was eagerly awaiting the raffle of the XBox 360 which was taken home by a fine gentleman who was there the entire night and bought over $100 in raffle tickets! Definitely well deserved. With Rock To A Cure being such a success this is only the beginning to many more charity events to come.
Today, August 26th marks the birthday of David “My Last Name Is Too Long” CZ. Happy Birthday CZ, hope it’s a good one!