The Magic of Cadence

Production for any form of development, regardless of method, is focused on addressing a series of solvable problems. Every problem has a solution, although it may not be the solution you want or expect. It is important to remember this, as it is easy to become caught up in a particular way of addressing problems.

When I entered the production discipline, it was still under the guise of developing a product for an end-user, versus software as a service (SaaS). By that point, I’d worked on projects that used a waterfall approach to scheduling and projects that applied a Scrum-like methodology to development. Both approaches presumed that we had a concrete start, defined progress points within the schedule for review, the possibility of iteration time, bug fixing, and a projected release date. Even when we shifted to a Scrum-like environment, we could still maintain several baseline expectations regarding the product and our approach.

This isn’t to say that it was ever easy. We were always addressing new problems and inventing new ways to leverage our tech. It’s simply that, for most games, we had a defined start, a series of expectations for the middle, and — whether or not things went well — an eventual end. Once the game was released to the wild, it might have received support the form of a patch or a DLC update, but the need for continuous development ended.

This changed for me the first time I encountered live cadence.

Feeling the Tempo on the Frontier

Before I dive deeply into this, I do want to note that I experienced a version of cadence with a couple of Playdom titles (ESPNU College Town and Social City), prior to Zynga’s acquisition of Buzz Monkey Software. The first truly successful cadence structure that I encountered, however, was with Zynga’s FrontierVille.

FrontierVille was a very popular invest-and-express title under the “Ville” brand. “Invest-and-express” — which I’ve also heard referred to as decoration management or dollhouse — is a game genre, particularly popular in the free-to-play space. As I was told, the name comes from the following statement: “Players invest their time and money in order to express themselves on the game board.”

FrontierVille pushed a feature out to the public approximately every week. At first, this schedule seemed impossible to me. Playdom’s releases were on a more casual basis. How did live games do this? That was, indeed, a question Zynga had asked themselves. In fact, they’d attempted a few different cadence approaches before settling on a winning strategy for FrontierVille.

We had four teams (pods) on the project. Each pod was a mini development team, complete with a producer, a product manager (PM), 1-3 engineers, 1-3 designers, 1-2 technical artists, and a QA point of contact. The project as a whole had an engineering lead, a senior producer, and an art director. We also had a community manager, a customer service manager, and a lead QA manager assigned to the project. The pods coordinated regularly at various levels throughout the week, and the entire team had a daily leads meeting where we could get reports from community, customer service, product management, and production.

Every morning, during dailies, we saw the game’s schedule — a giant interactive Excel file, which was later converted to a Google Sheet — projected on a shared screen. A single pod’s efforts could be broken into three weeks:

• Week 1: Design specification, art asset list, dev costing, and narrative development
• Week 2: Feature development, PM tuning, initial playthrough, and test plan development
• Week 3: QA, structured playthrough, triage, signoff, and release

An idealized version of this structure might look like this in the schedule for a single pod:

We would often share core resources, however, from the analyst pool, from central content, from QA, and from other groups. As a result, the schedule displayed how every release interlocked with the ones that came before and after. Additionally, some pods were staggered, so as to ensure a degree of variety in the release schedule. As a result, the schedule looked something like this:

 

To those of us who were new to the process, this seemed somewhat daunting. Thankfully, Zynga’s senior management understood this was a lot of information to process. To ensure our success, each producer worked with a partner producer for a business quarter, in order to help us understand both how our pod worked and how the pods had to work together.

The features under development were largely pulled from a backlog. The backlog contained wishlist items, design features that hadn’t yet found a monetization hook, or features that were deemed too expensive to develop. PM would plot out a viable development path with production, using the backlog and PM reports on the live game. Once the high level goals were defined, the development leads would be looped in, leading to the creation of the design specs for the upcoming cadence cycle.

Despite how well it worked, it wasn’t perfect. The schedule itself allowed very little room for error. The ratio of “must have” items for any given release often exceeded 60% of the work in a cadence cycle. (This ratio is very important for properly managing risk. I’ll bring it up again when I talk about costing.) But the group norms and roles were very well-defined, and this type of development was very limited in scope. As such, our risk of failure was fairly well mitigated.

After a few releases, everyone had picked up on the pace of cadence and could move effectively within that structure. Because we moved quickly and because we used a few management tools like a daily meeting and a backlog, the terms “Scrum” and “Agile” were thrown around to describe our process. It became clear, though, that everyone had a different functional definition of those terms, resulting in a measurable linguistic drift over the course of development.

Still, our success kept us from solving minor problems like that. (What is language, really?) FrontierVille went on to make just over a half a billion dollars over its lifetime. That certainly says something.

Well, at the very least it says someone (certainly not me!) made a lot of money. But it was nice to know that the game was popular among players and that our process, even with its flaws, worked.

After a year on FrontierVille, I was picked to work on CastleVille Legends, the first mobile invest-and-express game developed by Zynga. The development phase was familiar to development phases I’d encountered before, even though half of our development team was off-site. When we transitioned to cadence, however, things changed. The web-based model didn’t work. It couldn’t work.

Have Fun Storming the Castle

Mobile games, in many ways, have more in common with console development than web-based Flash games. This should be obvious. To those of us with console development backgrounds, it was obvious. After all, unlike web-based games, we had to worry about things like:

• QA for different executables of the same build, which were then tested on an array of different devices for each OS
• Ensuring the *.apk and *.ipa did not exceed 100MB in size
• Allowing players to transfer play across wholly different mobile devices
• Localization and localization QA
• Submission to an external party (Apple) for approval prior to release in the App Store
• Marketing updates and storefront setup in the App Store and Play Store

One of our first production exercises was to identify the new critical paths in this process. Apple, for example, has a submission process that can take up to 10 business days from submission to approval. We couldn’t simply push a change or a hotfix to the players on a whim. We developed an average approval case, based on our corporate relationship with Apple, with provisions to support the maximum case. This gave us our first new production limit for the mobile cadence model.

We also didn’t have pods. Where FrontierVille could support approximately 50 people during live cadence, CastleVille Legends was going to have a cadence team of approximately 25 people. Most of those people were going to be engineers. Art was reduced to two resources for the entire project. Even with an outsource model for asset creation, pod-style production simply wouldn’t work in this context.

We attempted to push for the development of 3 overlaid development schedules that would occur over a 7-week cycle, versus the structure of 4-5 guaranteed release per month. It looked something like this:

The issue that became immediately apparent was that QA did not have the time to adequately test any one release before the next release required its test plans. Additionally, this style of compressed development created a strain on the artists and the PMs. The art director and UI/UX artist could not adjust or implement assets quickly enough for any given release, and the PMs were unable to provide adequate tuning or analyses for any given event.

We had continued to operate under the assumption that we had already solved these problems. Our past success had kept us from truly examining the actual problem that presented itself. Clearly, mobile cadence could not be like web cadence.

As an organization, we were also flummoxed by the fact that simply stating we were “Agile” didn’t automatically and magically solve our scheduling problems. It is certainly possible to make adjustments and incorporate pivots in an Agile context, but Agile is merely a set of values and principles that can be used to produce a product. Agile has a meaning in a production context, and it’s not simply “moving quickly and nimbly.” A term cannot save a team. For that matter, neither can a process. The process must serve the team, not the other way around.

It was a hard lesson, but we ultimately accepted that we had to reassess our development limitations. After looking at old test plans and burndown charts, we determined with QA that it would take them an average of 10 days to adequately test a month’s worth of content. During that time, they couldn’t be tasked with QA for the previous or upcoming feature. The content also had to remain within the normal capacity of the development team. That is, if the dev team was going to have to spend extra hours to build the feature, the QA team would likely need extra hours in order to adequately test it. This was our second true production limit.

We also set hard limits on capacity for dev, design, art, and PM. By identifying what we could actually, repeatedly do, we could get down to the business of finding the right tempo for our cadence.

Ultimately, we came up with a 6-week overlapping schedule for each individual feature. On its own, it looked something like this:

Overlapped with other features, it looked like this:

Once the process was nailed down, we were able to consistently bank on solid releases, resulting in a steady stream of content for the players and a predictable level of engagement. As we mastered the flow of this cadence schedule, we were able to iterate upon it, creating even more engaging content and providing the illusion of more frequent updates without the burden of going through a formal submission process.

Words Mean Things

What was bothersome, though, was that I kept hearing “Agile” and “Scrum” throughout this process. The more I learned about both, the more I realized that what we were doing didn’t really fit either description well.

Our development was feature-driven. We worked from a backlog. We had daily meetings. We had a disciplined release schedule. But our development was to the cadence pace we set, and our internal control flow didn’t support a traditional Scrum master or product owner framework.

Ultimately, I realized that our cadence cycle was very close to the Dynamic Systems Development Method (DSDM). DSDM is an Agile project delivery framework that works by fixing cost, quality, and time parameters prior to development.

The core principles of DSDM are:

• Focus on the business need (in our case, generally defined by PM)
• Deliver on time, all the time
• Collaborate internally
• Never compromise quality (a bad release can mean a dead product)
• Build incrementally from firm foundations
• Develop iteratively
• Communicate continuously and clearly
• Demonstrate control at a production/project management level

From those principles come a variety of production tools that can help producers balance prototyping, testing, and configuration management against project priorities, team resources, and time available. Because of the control granted at the production level — control, not authority — any process or velocity failures are more directly owned by production.

All of this lined up with our actual process. Once I understood what we were doing and could put real terms to that process, it became far easier to refine and direct our efforts. This led to improved communication and greater overall success for the team. It also helped me to manage and coordinate with other partner teams that used different organizational methods, either consciously or unconsciously.

Language does matter, it turns out. It’s much easier to get around in life if you know which words to use and how to use them.

And now… you have a fundamental understanding of what cadence development is for free-to-play games on both web and mobile devices! I know. You’re excited. Who wouldn’t be? Just promise me you’ll use your newfound powers for good. Well… for mostly good.

I’ll get into costing and prioritization next. Until then!