gaetk2.datastore module

gaetk2.datastore tries to codify a common set of expectations and usages for gaetk2.

Inherit from gaetk2.datastore.gaetkModel instead of ndb.Model to get some added functionality. The rationale there is that e common interface and thus admin- and programmer-time is more important than savings on space and and processing time. To we add possible unneded database fields. You can remove them on a case by case basis in derivered classes.

  • query_iterator() - helps to iterate over big query results
  • get_or_insert_if_new() helps you to see if a new Entity was created.
  • copy_entity() - can write an entity with a different key to the datastore
  • update_obj() - basically implements conditional put()
  • reload_obj() - forces an object to be re-read from disk
  • apply_to_all_entities() - iterates over a table executing a function (“mapper”)

Data Model Conventions

  • url
  • created_at, updated_at
  • name, nicename
  • designator

Module contents

gaetk2.datastore.query_iterator(query, limit=50)[source]

Iterates over a datastore query while avoiding timeouts via a cursor.

Especially helpful for usage in backend-jobs.

gaetk2.datastore.copy_entity(e, **extra_args)[source]

Copy ndb entity but change values in kwargs.

Usage::
b = copy_entity(a, id=’new_id_here’) b.put()
gaetk2.datastore.get_or_insert_if_new(cls, id, **kwds)[source]

Like ndb.get_or_insert()` but returns (entity, new).

This allows you to see if something has been created or if there was an already existing entity:

>>> get_or_insert_if_new(Model, 'newid')
(<instance>, True)
>>> get_or_insert_if_new(Model, 'newid')
(<instance>, False)
gaetk2.datastore.write_on_change2(obj, data)[source]

Apply new data to an entity and write to datastore if anything changed.

This should save you money since reads are 3 times cheaper than writes. It also helps you do leave not given attributes unchanged.

Usage:

instance = ndb.Model...get()
dirty = write_on_change2(instance, ...,
  dict(id=123, amout_open=500, score=5, ...)
gaetk2.datastore.update_obj(obj, **kwargs)[source]

More modern Interface to write_on_change2().

gaetk2.datastore.reload_obj(obj)[source]

Returns a reloaded Entity from disk.

gaetk2.datastore.apply_to_all_entities(func, model, batch_size=0, num_updated=0, num_processed=0, cursor=None)[source]

Appliy a certain task all entities of model.

It scans every entity in the datastore for the model, exectues func(entity) on it and re-saves it if func trturns true. Tries to keep updated_at and updated_by unchanged.

Example

def _fixup_MyModel_updatefunc(obj):
if obj.wert_eur is not None:
obj.wert_eur = int(obj.wert_eur) return True

return False

def fixup_MyModel():
apply_to_all_entities(_fixup_app_angebotspos_updatefunc, MyModel)

# or

def execute(_now):
datastore.apply_to_all_entities(
_fixup_bestandsbuch_updatefunc, ic_bestandsbuch.ic_BestandsbuchEintrag)
def _fixup_bestandsbuch_updatefunc(obj):

changed = False # Attribute, die es als string und text in der datebnbank gibt normalisieren for attrname in ‘’‘ausloeser vorhergehender_bestandsbucheintrag info’‘’.split():

if getattr(obj, attrname, None) is not None:
setattr(obj, attrname, unicode(getattr(obj, attrname))) changed = True

return changed