HandmadeCon 2016 - Technical Direction at Blizzard

Introduction to Game Development Insights

Overview of the Discussion

• The speaker introduces the topic of game development, focusing on structural aspects such as hiring processes and management within large game companies.

• Emphasizes the importance of team dynamics in addition to code functionality, highlighting that effective teamwork is crucial for project success.

Guest Introduction: Marco Kugler

• Marco Kugler from Blizzard is invited to share insights about studio functions and address common questions regarding game development practices.

Marco Kugler's Background in Game Programming

Early Career and Education

• Marco shares his journey starting as a hobbyist interested in games, leading him to study computer science in Germany with limited formal game curriculum.

• He mentions self-taught knowledge through online communities like FlipCode, which helped him learn about game programming.

First Professional Experience

• After completing his degree, he applied for various positions and was hired by Crytek to work on their first console project for Xbox.

• Discusses how he initially joined as a gameplay programmer but quickly transitioned into lead roles due to team changes during project developments.

Roles and Responsibilities at Crytek

Understanding Job Titles

• Marco explains the fluidity of job titles in small companies like Crytek, where roles can evolve rapidly based on project needs.

Distinction Between Roles

• He clarifies the difference between tech director and lead programmer roles at Crytek, emphasizing collaboration between game teams and R&D teams focused on engine development.

Transitioning to Blizzard Entertainment

Moving Away from First-Person Shooters

• After working on Far Cry, Marco sought new challenges outside of first-person shooters and pursued opportunities at Blizzard while being an avid player of World of Warcraft.

Game Development Insights at Blizzard

Modernizing Game Mechanics

• Discussion on the potential for modernizing game elements without completely overhauling them, focusing on remastering aspects that could enhance player experience.

• Mention of "low-hanging fruit" improvements that can be made to the game's zoning and mechanics, indicating areas where small changes could yield significant benefits.

Team Structure and Dynamics

• Overview of the team structure at Blizzard, emphasizing a semi-autonomous studio model where different teams focus on various projects like RTS games, Overwatch, and Hearthstone.

• Explanation of how programming teams are organized within Blizzard, highlighting shared engineering groups that work across multiple projects to improve efficiency.

Evolution of Collaboration

• Insight into the shift towards more standardized technology sharing among teams over recent years to address scalability issues in development processes.

• Discussion about knowledge exchange between teams being limited due to production schedules but evolving towards better collaboration.

Role of Tech Director

• Description of responsibilities as a tech director for World of Warcraft, including overseeing internal tools, live service operations, and infrastructure management.

• Clarification on gameplay programming roles within the team structure and their significance in maintaining game quality.

Historical Context of Game Development

• Reflection on the original development team size for World of Warcraft (10 engineers), noting how this was relatively small yet effective for an MMO launch.

Understanding Life Services in Game Development

Overview of Life Services

• The concept of life services in gaming refers to the ongoing management and maintenance of a game while it is live, including monitoring for exploits like duplication glitches.

• An emergency team exists to analyze server performance and health, addressing issues such as crashes or bottlenecks in the code through strategic fixes.

Server Team Structure

• The server team includes reliability engineers who troubleshoot potential problems before they escalate, deploying hotfixes as necessary.

• Database programmers work with both off-the-shelf and custom databases, specifically using Oracle for managing game data.

Database Management

• There are two types of databases: static (world structure) and persistent (player data), which include items and spells.

Tools and Engine Teams

• A tools team develops various utilities for content creation, including localization tools.

• The graphics engine has evolved over time with multiple backends (OpenGL, Metal, DX9/DX11), adapting to technological advancements.

Support for Legacy Systems

• Supporting older systems like Windows XP remains challenging due to reliance on outdated third-party libraries; even a small percentage of users can represent significant numbers in a large player base.

User Interface Development

• A specialized UI team manages the extensive user interface needs of an MMO while also supporting community-driven add-ons.

Infrastructure Management

• A systems group focuses on infrastructure aspects such as build systems and content delivery methods that allow partial installations while streaming additional content.

Shared Technology Initiatives

Game Development Insights and Team Structure

Automation in Game Development

• The Overwatch team added UDP support to a project initiated by the WoW team, showcasing collaboration across teams.

• An automation group was integrated into the WoW team to enhance communication and streamline processes, improving overall efficiency.

• Automated systems are employed for performance measurements during content creation, monitoring both game building and running phases.

Performance Testing Strategies

• Performance measurement tools create heat maps by placing cameras around game maps to analyze data effectively, aiding content creators in managing their work.

• Early attention to performance is crucial; new features must be balanced with existing hardware specifications to avoid future complications.

Stress Testing Methodologies

• Stress testing simulates user clients on servers, spawning hundreds of thousands of them to identify breaking points and ensure server stability under load.

Team Management Dynamics

• The engineering team for World of Warcraft consists of 65 engineers, with an emphasis on maintaining small teams for effective management.

• Ideally, each manager should oversee 5 to 7 people; however, some teams have grown larger due to structural changes within the organization.

Career Development in Game Industry

• Collaboration between engineering managers and leads helps address career growth concerns for engineers while balancing workload responsibilities.

• Understanding the hiring process is essential for newcomers; candidates often come from various backgrounds or previous gaming experience.

Application Process Overview

• Candidates typically submit resumes through formal channels; those with prior gaming experience may find it easier to navigate this process.

Interview Process Insights

Phone Screen Overview

• The phone screen is conducted by actual engineers, typically the hiring manager and engineering manager, rather than HR or recruiters.

• Recruiters serve as touchpoints for candidates, answering questions about the interview process, company policies, and relocation details.

Technical Interview Structure

• A programming test is administered during the interview; it includes various C and C++ programming questions.

• Candidates may face challenges such as using basic text editors without development environments to complete coding tasks.

Evaluation Criteria

• The focus of the test is not solely on whether code runs correctly but on initiating a conversation about problem-solving approaches and trade-offs.

• Candidates are encouraged to sketch out solutions without worrying about syntax errors like missing semicolons.

Candidate Assessment

• Each phone screen may vary based on the position being filled; however, there are fixed elements involving general computer knowledge questions.

• Interviewers aim to understand candidates' experiences better by discussing their resumes in detail and exploring their motivations for applying.

Test Complexity and Expectations

• While variations exist in tests for different levels of experience, even entry-level candidates should be able to complete them with adequate time management.

Game Development Insights and Internship Processes

Runtime Considerations in Game Development

• Emphasizes the importance of runtime considerations alongside correctness in game development, highlighting the trade-offs involved.

• Notes that many new developers learn through platforms like Unity and Unreal, potentially missing foundational knowledge necessary for creating games.

Interview Preparation for Game Developers

• Discusses the necessity of understanding performance differences in coding practices, such as why certain methods are faster or slower than others.

• Mentions that interviews may include specific technical questions related to engine positions, including 3D math and animation challenges.

Internship Programs and Application Process

• Describes a popular internship program where candidates submit resumes through university relations, leading to a competitive selection process.

• Highlights that interns often return after their internships, with opportunities for job offers post-completion to ensure a smooth transition into full-time roles.

University Relations Program Explained

• Clarifies the role of the University Relations program in connecting local universities with companies for internship placements.

• Acknowledges uncertainty about the specifics of how far this program extends beyond local institutions.

Job Application Insights for New Graduates

• Compares intern application processes to full-time positions, noting fewer applications are typically reviewed at once.

• Stresses the value of showcasing personal projects on resumes, indicating that completed work demonstrates initiative and preparedness.

Importance of Project Experience

• Suggests that having tangible project experience is crucial; it can be anything from games to libraries demonstrating practical skills.

• Raises questions about whether HR understands what engineering teams look for in applicants' resumes regarding project experience.

Enhancing Resume Visibility

• Discusses how familiarity between recruiters and hiring teams can improve candidate selection based on past feedback loops regarding desirable resume traits.

How to Stand Out in the Game Industry Hiring Process

Importance of Easy Access to Work Samples

• Candidates should provide easy access to their work, such as a GitHub link, allowing potential employers to view relevant code samples.

The Changing Landscape of the Game Industry

• The speaker reflects on how the game industry has evolved, noting that past experiences may not be applicable for new entrants today.

Crafting a Meaningful Cover Letter

• A well-crafted cover letter is essential; it should convey personal motivations and reasons for wanting to join a specific company like Blizzard.

• Personalizing applications increases the likelihood of catching an employer's attention during screening processes.

Expectations for New Hires on Day One

• New hires can expect structured onboarding activities, including HR meetings covering benefits and company values.

Onboarding Activities at Blizzard

• Blizzard organizes scavenger hunts for new employees to familiarize them with the campus and its amenities.

• The first day includes introductions to team members and executives, fostering a welcoming environment.

Technical Setup and Initial Tasks

• By day two, new hires typically have their workstations set up with necessary software pre-installed, minimizing technical delays.

Understanding Game Development Processes at Blizzard

The Role of Detail and Objects in Game Development

• Discussion on the importance of understanding game terminology, specifically the difference between "detail" and "map objects." The detail refers to procedural vegetation, while map objects can include various structures like buildings or chairs.

Setting Up a Local Server

• Explanation of the necessity for developers to familiarize themselves with console commands and local server setups. Many developers run their own servers to connect with the game environment.

Code Review Process

• Overview of the code review process at Blizzard, emphasizing that every code check-in undergoes a review. Transitioning from in-person to asynchronous reviews is noted as a response to high interruption rates.

Structure of Code Reviews

• Description of who conducts code reviews; typically led by task assigners or leads who guide new employees through their first reviews, explaining best practices and expectations.

Subject Matter Experts

• Identification of senior engineers as targets for code reviews based on specific subsystems. New hires are encouraged to consult these experts when dealing with complex tasks.

Managing Complexity in Development

• Insight into how new team members are gradually introduced to more complex systems without overwhelming them. Emphasis on avoiding mission-critical tasks initially.

Handling Merge Conflicts

• Discussion about merge conflicts that arise during development due to simultaneous work across branches. These situations serve as teaching moments for new developers.

Qualities of an Ideal Team Member

• Inquiry into what makes a good team member at Blizzard, focusing on personality traits and skills that contribute positively to team dynamics.

Knowledge Gaps for New Hires

Effective Communication and Collaboration in Programming

Importance of Asking Questions

• New programmers often hesitate to ask questions due to fear of appearing incompetent, which can hinder their learning and problem-solving.

• It's essential for team leads to recognize when team members are struggling and encourage open communication to address issues.

Team Dynamics and Interaction Styles

• A positive work environment is fostered by individuals who remain calm when tasks aren't completed as expected, focusing on the positives instead of assigning blame.

• First-time engineers may encounter territorial behavior among peers; ideally, collaboration should be prioritized over competition.

Proactive Problem-Solving

• Engineers should take initiative in identifying problems rather than waiting for others to notice them, promoting a proactive culture within teams.

• Effective communication between engineers and designers is crucial; misunderstandings can arise if tasks are not clearly defined or updated regularly.

The Role of Game Knowledge

• Familiarity with the game being developed enhances communication among team members, allowing for more effective discussions about design elements.

• While playing the game isn't mandatory, it significantly aids understanding of terminology and context during project discussions.

Ideal Engineer Characteristics

• An ideal engineer can manage complex tasks independently while collaborating effectively with producers and other stakeholders for feedback before coding begins.

What Skills Are Essential for New Game Developers?

Understanding Team Dynamics and Project Experience

• The importance of collaboration in large codebases is emphasized, highlighting the need for new developers to coexist without disrupting others' work.

• Candidates with limited job experience may lack practical knowledge; education should better prepare them for real-world engineering challenges.

• Project experience is crucial; understanding game development processes and collaboration with artists/designers can significantly enhance a developer's effectiveness.

Challenges Faced by Younger Developers

• Younger developers often exhibit a "know-it-all" attitude, which can lead to conflicts when they push for advanced programming features without considering team dynamics.

• The introduction of complex C++ features (like lambdas) sparked debate within teams about their appropriateness and necessity in projects.

Navigating Code Quality and Best Practices

• It's essential that all team members understand new language features before implementation to maintain code quality across the board.

• Many developers may not be familiar with best practices surrounding newer programming features, leading to potential pitfalls in project execution.

Learning from Experience

• Developers who have faced challenges in past projects are more likely to appreciate caution when adopting new technologies or methodologies.

• Empathy towards experienced colleagues' concerns is vital; younger developers may struggle to grasp the implications of certain coding decisions without prior exposure.

Key Programming Concepts for Success

• A strong foundation in memory management and resource allocation is critical; understanding these concepts helps prevent performance issues down the line.

• Developers should recognize that small mistakes can accumulate ("death by a thousand paper cuts"), emphasizing the need for meticulous coding practices.

Decision-Making Skills in Development

• Solid fundamentals are necessary, such as knowing how garbage collection works versus manual memory management.

Game Development Insights: Employee Perspectives

Understanding the Non-Code Aspects of Game Development

• The discussion highlights the importance of understanding the employee side of game development, emphasizing that there may be aspects not covered in technical discussions.

Variability in Engineering Roles

• Different engineering positions require distinct skill sets; for instance, UI teams focus on C++ and Lua, while tools teams need familiarity with various toolkits.

Domain-Specific Knowledge Requirements

• Game development roles can be highly domain-specific. For example, working on "World of Warcraft" (WoW) necessitates knowledge about databases like Oracle due to its unique requirements.

Long-Term Assumptions in Game Lifecycle

• The conversation shifts to long-term assumptions made during WoW's 16-year lifecycle, noting that many current issues stem from outdated decisions made years ago.

• An example is given regarding past practices where servers would go offline weekly for maintenance, which contrasts sharply with modern expectations for zero downtime operations.

Technical Debt and Modernization Efforts

• A significant portion of engineering time (approximately 25%) is dedicated to addressing technical debt to modernize systems and improve user experience without disrupting ongoing projects.

• Specific examples include updating graphical elements like water textures to enhance realism while ensuring compatibility with existing art assets and legacy systems.

Challenges with Legacy Systems

• The discussion touches on challenges posed by legacy systems such as memory management issues related to 32-bit architecture, which frequently leads to crashes in WoW due to player add-ons consuming resources.

• Players often run into memory limitations because they use diverse add-ons within their process space, complicating memory management further.

Compatibility Issues Across Operating Systems

• There are complications when players install different versions of Windows; for instance, some users unknowingly run a 32-bit client on a capable 64-bit machine due to installation errors or misunderstandings about system capabilities.

Addressing Cheating and Technical Limitations

Challenges in Content Creation and Hot Fixing

Modernizing the Art Pipeline

• The current art pipeline for content creation is described as "horrible," with efforts underway to modernize it. This involves cautious additions of new features late in the process, leading to special cases that complicate development.

Managing Complexity in Game Design

• Designers use spells as basic execution units to trigger events in the game world. These spells can affect large areas and involve complex scripting, which can lead to an unmanageable number of unique solutions for similar problems.

The Role of Good Design

• Effective designers often push boundaries using limited tools, resulting in intricate designs that may complicate systems but also enhance creativity and innovation within the game.

Understanding Hot Fixing

• Hot fixing is a critical aspect of game development at Blizzard. It allows for quick updates without restarting servers or disrupting gameplay, including binary hot fixes (client-side executable changes) and data hot fixes (server-side adjustments).

Challenges with Shared Databases

• Developers work on a shared database, which can lead to conflicts when multiple team members make changes simultaneously. This necessitates careful management to avoid overwriting each other's work during testing.

Transitioning Towards Local Workflows

Understanding Content Creation and Testing in Game Development

The Culture of System Rewriting

• The speaker discusses the cautious approach to changing systems, emphasizing the risk of breaking existing functionalities. They highlight a culture focused on massive system rewrites.

• Automated testing is not heavily relied upon; instead, they utilize Player Test Realms (PTR) for early content testing, allowing real players to engage with new features and provide feedback.

Modernizing Workflows

• There is significant work being done to modernize the content creation pipeline and art pipeline, aiming for a more intuitive "what you see is what you get" workflow.

• Previously, making changes required running complex processes (e.g., map generation), but there’s a push towards real-time updates where changes can be instantly reflected in the client.

Hotfixing and Local Changes