gaetk2.tools Package

Thees package contains functionality mostly used intenally.

Todo

  • ids.py
  • hujson2.py
  • http.py
  • config.py
  • auth0tools.py
  • sentry.py
  • structured.py

gaetk2.tools.caching - smart caching

Caching on Google App Engine makes your application faster and cheaper. While for key.get() operations ndp provides caching for you, queries are nwver cached by the datastore infrastructure.

After years of experimentation we come to the conclusion that you should always use some time-based cache invalidation. This will result in “eventual consitency” even if you do not get your cache invalidation strategy perfectly right.

We provide lru_cache() with a default TTL of 12 hours. It does local instance memory caching and is an extension of functools from Python 3.3.

lru_cache_memcache() is an extension using a two-level strategy: content which is not found in the local instance cache is pulled from the shared memcache. Cache entries are not shared between different versions of your application.

It is suggested, that you use a relatively small maxsize with lru_cache_memcache() to save on instance memory.

gaetk2.tools.caching.lru_cache(maxsize=64, typed=False, ttl=43200)[source]

Least-recently-used cache decorator.

Parameters:
  • maxsize (int or None) – if None, the LRU features are disabled and the cache can grow without bound.
  • typed (boolean) – if True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results.
  • ttl (int or None) – if set, cache entries are only served for ttl seconds.

Arguments to the cached function must be hashable.

View the cache statistics named tuple (hits, misses, maxsize, currsize) with f.cache_info(). Clear the cache and statistics with f.cache_clear(). Access the underlying function with f.__wrapped__.

See: http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used

Example

@lru_cache(maxsize=6)
def _fetchbrands():
    query = mk_models.mk_Brand.query()
    return [brand.name for brand in query.iter() if not brand.deleted]
class gaetk2.tools.caching.lru_cache_memcache(maxsize=8, typed=False, ttl=43200)[source]

Use lru_cache() with memcache as an fallback.

Arguments are the same as lru_cache().

Example

@lru_cache_memcache(ttl=3600)
def _fetchbrands():
    query = mk_models.mk_Brand.query()
    return [brand.name for brand in query.iter() if not brand.deleted]

gaetk2.tools.datetools

Todo

  • Explain what gaetk2.tools.datetools is for
gaetk2.tools.datetools.tertial(date)[source]

Wandelt ein Date oder Datetime-Objekt in einen Tertial-String

gaetk2.tools.datetools.rfc3339_date(date=None)[source]

Formates a datetime object according to RfC 3339.

gaetk2.tools.datetools.rfc3339_date_parse(date)[source]

Parses an RfC 3339 timestamp into a datetime object.

gaetk2.tools.datetools.convert_to_date(date)[source]

Converts argument into a date object.

Assumes argument to be a RfC 3339 coded date or a date(time) object.

gaetk2.tools.datetools.convert_to_datetime(date)[source]

Converts argument into a datetime object.

Assumes argument to be a RfC 3339 coded date or a date(time) object.

gaetk2.tools.datetools.rfc2616_date(date=None)[source]

Formates a datetime object according to RfC 2616.

RfC 2616 is a subset of RFC 1123 date. Weekday and month names for HTTP date/time formatting; always English!

gaetk2.tools.datetools.rfc2616_date_parse(data)[source]

Parses an RfC 2616/2822 timestapm into a datetime object.

gaetk2.tools.unicode - string handling

This are functions which help to handle data from a pre-Unicode world. Much of this code is acient and has no use in a worl where JSON and XML ensure somewhat clean encoding. But still there are so many places where you are allowd to send only ASCII subsets.

Data Cleanup

  • de_noise() - removed Unicode Characters which normally have no place in buiseness documents (eg street names). This includes Emojii but also protected spaces unusual quotation marks etc. This data is usually included dut to cut and paste errors. Read source to see what is replaced.
  • de_umlaut() - converts data to plain ASCII while converting german Umlauts to something reasonable.
  • de_utf8() - “repair” wrongly decoded UTF-8.

Number Conversion

num_encode() and num_decode() convert arbitrary long numbers to strings and back again. Works nice for datastore IDs. Uses base 62 (lowwer and upper letters and numbers) to get a compact representation.

num_encode_uppercase() uses base36 which is less compact but case insensitive.

You can use these functions to getsomewhat easy to tipe compact datastore ids:

class SomeEntity(ndb.Model):
    nr = ndb.ComputedProperty(lambda num_encode(self: self.key.id()) if self.key.id() else '?')

Module contents

gaetk2.tools.unicode.de_utf8(data)[source]

This is meant to help with utf-8 data appearing where unicode should apperar.

gaetk2.tools.unicode.de_umlaut(data)[source]

Converts a text to ASCII acting smart about Umlauts.

>>> de_umlaut('1 Über Hügel saß René äöüÄÖÜß')
'1 Ueber Huegel sass Rene aeoeueAeOeUess'
gaetk2.tools.unicode.de_noise(data)[source]

Removes all stuff which should not appear in normal Western Text.

>>> de_noise(u'»Susie`s Giga\Super-Markt®¿«')
u">Susie's Giga/Super-Markt(R)?<"
>>> de_noise(u"ümlaut eins:")
u'\xfcmlaut eins:'
>>> de_noise(u'«A» {C} ¿D? „E“ ›F‹')
u'<A> (C) ?D? "E" >F<'
>>> de_noise(u'`A´')
u"'A'"
>>> de_noise(u'«😎» Umlaute kann doctest !gut {®} ¿👩‍👩‍👧‍👦? „👨‍❤️‍💋‍👨“ ›🎅🏻🎅🏼🎅🏽🎅🏾🎅🏿‹')
u'<> Umlaute kann doctest !gut ((R)) ?? "" ><'
>>> de_noise(u'DE37  330 5 13 50 0 010  4414  22')
u'DE37330513500010441422'
gaetk2.tools.unicode.slugify(value)[source]

Converts a string to be usable in URLs without excaping.

Normalizes string, converts to lowercase, removes non-alpha characters, and converts spaces to hyphens.

Inspired by Django’s “django/template/defaultfilters.py”.

gaetk2.tools.unicode.num_encode(n)[source]

Convert an integer to an base62 encoded string.

gaetk2.tools.unicode.num_decode(s)[source]

Convert the result of num_encode() back to an integer.

gaetk2.tools.unicode.num_encode_uppercase(n)[source]

Convert an integer to an base36 (unly uppercase and numbers) encoded string.

gaetk2.tools.structured_xls package

Todo

  • Explain what gaetk2.tools.structured_xls is for
class gaetk2.tools.structured_xls.XLSwriter(output=None, sheetname='This Sheet')[source]

csv - Module compatible Interface to generate excel files.

… but you have to call save() oder getvalue() to generate the final XLS file.

Parameters:
  • output (file or None) – optional File-Like Object for save() to export to.
  • sheetname (str) – optional Name of the single Worksheet we export to.

Uses the deprecated xlwt.

Usage:

xlswriter = XLSwriter()
xlswriter.writerow(['foo', 1, 2])
xlswriter.writerow(['bar', 3, datetime.date.today()])
xlswriter.save(open('test.xls')
writerow(row)[source]

Eine Zeile schreiben. Row ist eine Liste von Werten.

save(fd=None)[source]

Write rendered XLS file to fd or self.output.

getvalue()[source]

Returns rendered XLS file as a StringIO().