Entities and Models

A datastore entity has a key and a set of properties. An application uses the datastore API to define data models, and create instances of those models to be stored as entities. Models provide a common structure to the entities created by the API, and can define rules for validating property values.

• Datastore Entities
• The Model Interface
• Expando Models
• Properties and Types
  • Strings, Long Strings and Blobs
  • Lists
  • References
• Property_Names

Datastore Entities

A data object in the App Engine datastore is known as an entity. An entity has one or more properties, named values of one of several supported data types.

Each entity also has a key that uniquely identifies the entity. The simplest key has a kind and a unique numeric ID provided by the datastore. The ID can also be a string provided by the application. For more information on keys, see Keys and Entity Groups.

An application can fetch an entity from the datastore by using its key, or by performing a query that matches the entity's properties. A query can also match an "ancestor" in the key; see Keys and Entity Groups. A query can return zero or more entities, and can return the results sorted by property values. A query can also limit the number of results returned by the datastore to conserve memory and run time.

Unlike relational databases, the App Engine datastore does not require that all entities of a given kind have the same properties. The application can specify and enforce its data models using the model API.

The Model Interface

An application describes the kinds of data it uses with models. A model is a Python class that inherits from the Model class. The model class defines a new kind of datastore entity and the properties the kind is expected to take.

Model properties are defined using class attributes on the model class. Each class attribute is an instance of a subclass of the Property class, usually one of the provided property classes. A property instance holds configuration for the property, such as whether or not the property is required for the instance to be valid, or a default value to use for the instance if none is provided.

from google.appengine.ext import db

class Pet(db.Model):
  name = db.StringProperty(required=True)
  type = db.StringProperty(required=True, choices=set(["cat", "dog", "bird"]))
  birthdate = db.DateProperty()
  weight_in_pounds = db.IntegerProperty()
  spayed_or_neutered = db.BooleanProperty()
  owner = db.UserProperty(required=True)

An entity of one of the defined entity kinds is represented in the API by an instance of the corresponding model class. The application can create a new entity by calling the constructor of the class. The application accesses and manipulates properties of the entity using attributes of the instance. The model instance constructor accepts initial values for properties as keyword arguments.

from google.appengine.api import users

pet = Pet(name="Fluffy",
          type="cat",
          owner=users.get_current_user())
pet.weight_in_pounds = 24

Note: The attributes of the model class are configuration for the model properties, whose values are Property instances. The attributes of the model instance are the actual property values, whose values are of the type accepted by the Property class.

The Model class uses the Property instances to validate values assigned to the model instance attributes. Property value validation occurs when a model instance is first constructed, and when an instance attribute is assigned a new value. This ensures that a property can never have an invalid value.

Because validation occurs when the instance is constructed, any property that is configured to be required must be initialized in the constructor. In this example, name, type and owner are all required values, so their initial values are specified in the constructor. weight_in_pounds is not required by the model, so it starts out unassigned, then is assigned a value later.

An instance of a model created using the constructor does not exist in the datastore until it is "put" for the first time. See Creating, Getting and Deleting Data.

Note: As with all Python class attributes, model property configuration is initialized when the script or module is first imported. Because App Engine caches imported modules between requests, module configuration may be initialized during a request for one user, and re-used during a request for another. Do not initialize model property configuration, such as default values, with data specific to the request or the current user. See App Caching for more information.

Expando Models

A model defined using the Model class establishes a fixed set of properties that every instance of the class must have (perhaps with default values). This is a useful way to model data objects, but the datastore does not require that every entity of a given kind have the same set of properties.

Sometimes it is useful for an entity to have properties that aren't necessarily like the properties of other entities of the same kind. Such an entity is represented in the datastore API by an "expando" model. An expando model class subclasses the Expando superclass. Any value assigned to an attribute of an instance of an expando model becomes a property of the datastore entity, using the name of the attribute. These properties are known as dynamic properties. Properties defined using Property class instances in class attributes are fixed properties.

An expando model can have both fixed and dynamic properties. The model class simply sets class attributes with Property configuration objects for the fixed properties. The application creates dynamic properties when it assigns them values.

class Person(db.Expando):
  first_name = db.StringProperty()
  last_name = db.StringProperty()
  hobbies = db.StringListProperty()

p = Person(first_name="Albert", last_name="Johnson")
p.hobbies = ["chess", "travel"]

p.chess_elo_rating = 1350

p.travel_countries_visited = ["Spain", "Italy", "USA", "Brazil"]
p.travel_trip_count = 13

Because dynamic properties do not have model property definitions, dynamic properties are not validated. Any dynamic property can have a value of any of the datastore base types, including None. Two entities of the same kind can have different types of values for the same dynamic property, and one can leave a property unset that the other sets.

Unlike fixed properties, dynamic properties need not exist. A dynamic property with a value of None is different from a non-existent dynamic property. If an expando model instance does not have an attribute for a property, the corresponding data entity does not have that property. You can delete a dynamic property by deleting the attribute.

del p.chess_elo_rating

A query that uses a dynamic property in a filter will only return entities whose value for the property is of the same type as the value used in the query. Similarly, the query will only return entities with that property set.

p1 = Person()
p1.favorite = 42
p1.put()

p2 = Person()
p2.favorite = "blue"
p2.put()

p3 = Person()
p3.put()

people = db.GqlQuery("SELECT * FROM Person WHERE favorite < :1", 50)
# people has p1, but not p2 or p3

people = db.GqlQuery("SELECT * FROM Person WHERE favorite > :1", 50)
# people has no results

The Expando class is a subclass of the Model class, and inherits all of its methods.

Properties and Types

The datastore supports a fixed set of value types for entity properties, including Unicode strings, integers, floating point numbers, dates, entity keys, byte strings (blobs), and various GData types. Each of the datastore value types has a corresponding Property class provided by the google.appengine.ext.db module.

Types and Property Classes describes all of the supported value types and their corresponding Property classes. Several special value types are described below.

Strings, Long Strings and Blobs

The datastore supports two value types for storing text: short text strings shorter than 500 bytes, and long text strings longer than 500 bytes. Short strings are indexed and can be used in query filter conditions and sort orders. Long strings are not indexed and cannot be used in filter conditions or sort orders.

A short string value can be either a unicode value or a str value. If the value is a str, an encoding of 'ascii' is assumed. To specify a different encoding for a str value, you can convert it to a unicode value with the unicode() type constructor, which takes the str and the name of the encoding as arguments. Short strings can be modelled using the StringProperty class.

class MyModel(db.Model):
  string = db.StringProperty()

obj = MyModel()

# Python Unicode literal syntax fully describes characters in a text string.
obj.string = u"kittens"

# unicode() converts a byte string to a Unicode value using the named codec.
obj.string = unicode("kittens", "latin-1")

# A byte string is assumed to be text encoded as ASCII (the 'ascii' codec).
obj.string = "kittens"

# Short string properties can be used in query filters.
results = db.GqlQuery("SELECT * FROM MyModel WHERE string = :1", u"kittens")

A long string value is represented by a db.Text instance. Its constructor takes either a unicode value, or a str value and optionally the name of the encoding used in the str. Long strings can be modelled using the TextProperty class.

class MyModel(db.Model):
  text = db.TextProperty()

obj = MyModel()

# Text() can take a Unicode value.
obj.text = db.Text(u"lots of kittens")

# Text() can take a byte string and the name of an encoding.
obj.text = db.Text("lots of kittens", "latin-1")

# If no encoding is specified, a byte string is assumed to be ASCII text.
obj.text = db.Text("lots of kittens")

# Text properties can store large values.
obj.text = db.Text(open("a_tale_of_two_cities.txt").read(), "utf-8")

The datastore also supports a type for non-text byte strings, or "blobs." As with long text strings, blobs are not indexed, and cannot be used in query filters or sort orders. A Blob instance represents a string of bytes, and takes a str value as an argument to its constructor. Blobs are modelled using the BlobProperty class.

class MyModel(db.Model):
  blob = db.BlobProperty()

obj = MyModel()

obj.blob = db.Blob(open("image.png").read())

Lists

A property can have multiple values, represented in the datastore API as a Python list. The list can contain values of any of the value types supported by the datastore. Order is preserved, so when entities are returned by queries and get(), list properties will have values in the same order as when they were stored.

The ListProperty class models a list, and enforces that all values in the list are of a given type. For convenience, the library also provides StringListProperty, equivalent to ListProperty(basestring).

class MyModel(db.Model):
  numbers = db.ListProperty(long)

obj = MyModel()
obj.numbers = [2, 4, 6, 8, 10]

obj.numbers = ["hello"]  # ERROR: MyModel.numbers must be a list of longs.

A query filter on a list property tests the given value against members of the list. The condition is true if at least one member of the list meets the condition.

# Get all entities where numbers contains a 6.
results = db.GqlQuery("SELECT * FROM MyModel WHERE numbers = 6")

# Get all entities where numbers contains at least one element less than 10.
results = db.GqlQuery("SELECT * FROM MyModel WHERE numbers < 10")

Query filters only operate on the list members. There is no way to test two lists for similarity in a query filter.

Internally, the datastore represents a list property value as multiple values for the property. If a list property value is the empty list, then the property has no representation in the datastore. The datastore API treats this situation differently for static properties (with ListProperty) and dynamic properties:

• A static ListProperty can be assigned the empty list as a value. The property does not exist in the datastore, but the model instance behaves as if the value is the empty list. A static ListProperty cannot have a value of None.
• A dynamic property with a list value cannot be assigned an empty list value. However, it can have a value of None, and can be deleted (using del).

The ListProperty model tests that a value added to the list is of the correct type, and throws a BadValueError if it isn't. Because the datastore converts str values to unicode values for storage using the ascii codec, and returns such values as Python unicode values, ListProperty(str) doesn't work: It accepts str values from the application, but when the entity is loaded back into a model instance, the values have been converted to unicode and no longer match, resulting in a BadValueError. The solution is to use ListProperty(basestring), which accepts both str and unicode, or use the shortcut StringListProperty(). For storing non-text byte strings, use db.Blob values.

List properties interact in unusual ways with sort orders. See Sort Orders and List Properties for details.

References

A property value can contain the key of another entity. The value is a Key instance.

The ReferenceProperty class models a key value, and enforces that all values refer to entities of a given kind. For convenience, the library also provides SelfReferenceProperty, equivalent to a ReferenceProperty that refers to the same kind as the entity with the property.

Assigning a model instance to a ReferenceProperty property automatically uses its key as the value.

class FirstModel(db.Model):
  prop = db.IntegerProperty()

class SecondModel(db.Model):
  reference = db.ReferenceProperty(FirstModel)

obj1 = FirstModel()
obj1.prop = 42
obj1.put()

obj2 = SecondModel()

# A reference value is the key of another entity.
obj2.reference = obj1.key()

# Assigning a model instance to a property uses the entity's key as the value.
obj2.reference = obj1
obj2.put()

A ReferenceProperty property value can be used as if it were the model instance of the referenced entity. If the referenced entity is not in memory, using the property as an instance automatically fetches the entity from the datastore.

obj2.reference.prop = 999
obj2.reference.put()

results = db.GqlQuery("SELECT * FROM SecondModel")
another_obj = results.fetch(1)[0]
v = another_obj.reference.prop

When an entity whose key is the value of a reference property is deleted, the reference property does not change. A reference property value can be a key that is no longer valid. If an application expects that a reference could be invalid, it can test for the existence of the object using an if statement:

obj1 = obj2.reference

if not obj1:
  # Referenced entity was deleted.

ReferenceProperty has another handy feature: back-references. When a model has a ReferenceProperty to another model, each referenced entity gets a property whose value is a Query that returns all of the entities of the first model that refer to it.

# To fetch and iterate over every SecondModel entity that refers to the
# FirstModel instance obj1:
for obj in obj1.secondmodel_set:
  # ...

The name of the back-reference property defaults to modelname_set (with the name of the model class in lowercase letters, and "_set" added to the end), and can be adjusted using the collection_name argument to the ReferenceProperty constructor.

If you have multiple ReferenceProperty values that refer to the same model class, the default construction of the back-reference property will raise an error:

class FirstModel(db.Model):
  prop = db.IntegerProperty()

# This class raises a DuplicatePropertyError with the message
# "Class Firstmodel already has property secondmodel_set"
class SecondModel(db.Model):
  reference_one = db.ReferenceProperty(FirstModel)
  reference_two = db.ReferenceProperty(FirstModel)

To avoid this error, you must explicitly set the collection_name argument:

class FirstModel(db.Model):
  prop = db.IntegerProperty()

# This class runs fine
class SecondModel(db.Model):
  reference_one = db.ReferenceProperty(FirstModel,
      collection_name="secondmodel_reference_one_set")
  reference_two = db.ReferenceProperty(FirstModel,
      collection_name="secondmodel_reference_two_set") 

Automatic referencing and dereferencing of model instances, type checking and back-references are only available using the ReferenceProperty model property class. Keys stored as values of Expando dynamic properties or ListProperty values do not have these features.

Property Names

The datastore reserves all property names begining and ending with two underscore characters (__*__). An application cannot create a property with such a name.

In the Python API, attributes of model instances whose names begin with an underscore (_) are ignored, and are not saved to the datastore entity. This allows you to store values on the model instance for temporary internal use without affecting the data saved with the entity.

Because the Python API uses attributes of model instances as property names by default, none of the attributes already in use by instance methods can be used directly as property attribute names. Similarly, none of the names used by the keyword arguments of model constructors can be used as property attribute names. See the list of reserved property names.

The datastore itself allows these names. If an application needs a datastore entity to have a property with a name similar to a word reserved in the Python API, the application can use a fixed property and pass the name argument to the Property class constructor. See the Property class constructor.

class MyModel(db.Model):
  obj_key = db.StringProperty(name="key")