21 Feb 2009

N3’s Game Database Structure

Here’s some info on the game database structure used by N3 and N2’s Mangalore, which might be helpful for level editor tools.

“Game data” basically means “everything except graphics, audio or movie stuff”.

All Nebula game-data lives in 2 SQLite database files:

  • home:export/db/static.db4: This contains all “static” data which doesn’t change over the course of a game session.
  • home:export/db/game.db4: This contains all the data which may change during a game session.

Those files can be inspected and modified with the SQLiteSpy tool, but since some of the contained data are blobs it may be a good idea to access them with custom tools (N3’s db addon offers wrapper classes for this, but of course it’s also possible to use sqlite directly to access the database).

Both database files are normally populated by various exporter tools in our inhouse-asset-pipeline.

The game.db4 file is important for N3’s standardized NewGame/Continue/Save/Load feature:

  • on New Game, game.db4 will be copied to the user: directory of the application, changes during game play will be committed back to this copy (usually when a level is left)
  • on Continue Game, N3 will simply open the existing game.db4 file under “user:”
  • on Save Game, a copy of the game.db4 file under “user:” will be performed into the save-game directory
  • on Load Game, the game.db4 in “user:” will be overwritten with a copy from the save-game directory

The main chunk of data in both database files are game entity related. A game entity is completely described by a collection of key/value pairs called attributes, which live in a single row of a database table.

Application layer subsystems may store additional data in the database by managing their own tables in both databases. Usually, save-game relevant data lives in the game.db4 database, while read-only data lives in the static.db4 database file. Those custom-tables are usually read/written by derived Manager::OnLoad() and Manager::OnSave() methods.

The static.db4 database contains at least the following tables:

  • _Attributes: This table contains a description of all attribute types used in other database tables. This lets an application load a database in a meaningful way even if the attributes haven’t been defined in its C++ code. The columns of _Attribute are:
    • AttrName (string, primary, unique, indexed): the attributes name, doh
    • AttrType (string): the type of the attribute, this is how the different types are stored in the database:
      • int: stored as SQL type INTEGER
      • bool: stored as SQL type INTEGER
      • float: stored as SQL type REAL
      • string: stored as SQL type TEXT
      • vector3: a float[3] stored as SQL type BLOB
      • vector4: a float[4] stored as SQL type BLOB (a raw Math::float4 written to the database)
      • matrix44: a float[16] stored as SQL type BLOB (a raw Math::matrix44 written to the database)
      • guid: a ubyte[16] stored as SQL type BLOB (result of Util::Guid::AsBinary())
      • blob: can be used to efficiently store “anything else” in the database, hardly used though, Drakensang uses this to store the fog-of-war data for instance
    • AttrReadWrite (bool), AttrDynamic (bool): these two columns are currently not used in any meaningful way
  • _Categories: Contains a description of every entity category which shows up in the database. This is basically the lookup-table for the CategoryManager. The _Categories table has the following columns:
    • CategoryName (string, primary, unique, indexed): The name of the category (e.g. Camera, NPC, Actor, Monster, etc…). Some category names are hard-coded and standardized across games:
      • Light: describes level light sources
      • _Environment: describes “environment objects”, these are simple graphics objects which don’t have game-play functionality
      • Levels: a meta-table which contains one row for every level in the game
    • IsVirtualCategory (bool, indexed): a virtual category only has a ‘template table’ but no ‘instance table’, virtual categories can be used to provide read-only data (lookup tables, etc) to the game application without having to write a custom exporter tool
    • IsSpecialCategory (bool, indexed): only the above mentioned hard-coded categories are marked as “special”
    • CategoryTemplateTable (string): The name of the database table with entity templates, a template table contains “blueprints” for entities with their initial attribute values. The method FactoryManager::CreateEntityByTemplate() is used to instantiate a new entity from such a template. Template tables live in the static.db4 database.
    • CategoryInstanceTable (string): The name of the instance table (only if this is not a virtual category). Instance tables live in the game.db4 database file and contain one row per actual game entity.
  • Template tables: Template tables contain blueprint for entities which are created through the FactoryManager::CreateEntityByTemplate() method. A row in the template table contains the starting attribute values for a new entity of that type. A template table must contain at least the Id column, which must be (string, primary, unique, indexed).

The game.db4 database has the following tables:

  • _Attributes: this has the exact same structure as the _Attributes table in static.db4
  • _Globals: this is a simple global variable storage managed by the GlobalAttrsManager.
  • Instance tables: the instance tables as listed in the _Categories table of static.db4. Every game entity in the entire game has a row in one of the instance tables. The general structure of an instance table is the same as the associated template table, but with the following additional columns:
    • Guid (guid, primary, unique, indexed): a unique identifier for the entity
    • _ID (string): a human-readable unique identifier (only within the level)
    • _Level (string, indexed): the level where this entity currently resides (may change during game)
    • _Layers (string): The layers this entity is a member of. Entity layers can be used to “overlay” a specific group of entities at level load time. This can be used for day/night, intact/destroyed versions of a level without duplicating the entire level. An empty _Layers attribute means that this entity is always loaded into a level.
    • Transform (matrix44): the current world space transform of the entity

The _Instance_Levels instance table is special in that it doesn’t have the above mentioned hardcoded attributes. Instead it contains one row per level in the game with the following attributes:

  • Id (string, primary, unique, indexed): a unique-id string of the level
  • Name (string, indexed): the human-readable name of the level
  • StartLevel (bool): true if this is the start level which should be loaded on New Game, false for all other levels
  • _Layers (string): the currently active entity layers
  • Center (vector3): the midpoint of the level in world space
  • Extents (vector3): the extents vector of the level (Center and Extents are used to describe the bounding box of the level)
  • NavMesh, NavMeshTransform, NoGoArea, NoGoAreaTransform: used by the navigation subsystem (not currently implemented in Nebula3)

Most entity-related database accesses are wrapped by the CategoryManager, which currently caches most of the data in memory (all template tables, and all entity instance data of the current level). This may change in the future when we generally run SQLite through the in-memory mode. If the in-memory database accesses are fast enough we might drop the caching (which isn’t as good as I’d like because of the huge amount of memory allocations going on during reading from the database).

an
Labels: