gaetk2.tools Package

Thees package contains functionality mostly used intenally.

Contents

• gaetk2.tools Package
  • gaetk2.tools.caching - smart caching
  • gaetk2.tools.datetools
  • gaetk2.tools.unicode - string handling
    • Data Cleanup
    • Number Conversion
    • Module contents
  • gaetk2.tools.structured_xls package

Todo

• ids.py
• hujson2.py
• http.py
• config.py
• auth0tools.py
• sentry.py
• structured.py

gaetk2.tools.caching - smart caching

Caching on Google App Engine makes your application faster and cheaper. While for key.get() operations ndp provides caching for you, queries are nwver cached by the datastore infrastructure.

After years of experimentation we come to the conclusion that you should always use some time-based cache invalidation. This will result in “eventual consitency” even if you do not get your cache invalidation strategy perfectly right.

We provide lru_cache() with a default TTL of 12 hours. It does local instance memory caching and is an extension of functools from Python 3.3.

lru_cache_memcache() is an extension using a two-level strategy: content which is not found in the local instance cache is pulled from the shared memcache. Cache entries are not shared between different versions of your application.

It is suggested, that you use a relatively small maxsize with lru_cache_memcache() to save on instance memory.

gaetk2.tools.caching.lru_cache(maxsize=64, typed=False, ttl=43200)

Least-recently-used cache decorator.

Parameters:

• maxsize (int or None) – if None, the LRU features are disabled and the cache can grow without bound.
• typed (boolean) – if True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results.
• ttl (int or None) – if set, cache entries are only served for ttl seconds.

Arguments to the cached function must be hashable.

View the cache statistics named tuple (hits, misses, maxsize, currsize) with f.cache_info(). Clear the cache and statistics with f.cache_clear(). Access the underlying function with f.__wrapped__.

See: http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used

Example

@lru_cache(maxsize=6)
def _fetchbrands():
    query = mk_models.mk_Brand.query()
    return [brand.name for brand in query.iter() if not brand.deleted]

class gaetk2.tools.caching.lru_cache_memcache(maxsize=8, typed=False, ttl=43200)

Use lru_cache() with memcache as an fallback.

Arguments are the same as lru_cache().

Example

@lru_cache_memcache(ttl=3600)
def _fetchbrands():
    query = mk_models.mk_Brand.query()
    return [brand.name for brand in query.iter() if not brand.deleted]

gaetk2.tools.datetools

Todo

• Explain what gaetk2.tools.datetools is for

gaetk2.tools.datetools.tertial(date)

Wandelt ein Date oder Datetime-Objekt in einen Tertial-String

gaetk2.tools.datetools.rfc3339_date(date=None)

Formates a datetime object according to RfC 3339.

gaetk2.tools.datetools.rfc3339_date_parse(date)

Parses an RfC 3339 timestamp into a datetime object.

gaetk2.tools.datetools.convert_to_date(date)

Converts argument into a date object.

Assumes argument to be a RfC 3339 coded date or a date(time) object.

gaetk2.tools.datetools.convert_to_datetime(date)

Converts argument into a datetime object.

Assumes argument to be a RfC 3339 coded date or a date(time) object.

gaetk2.tools.datetools.rfc2616_date(date=None)

Formates a datetime object according to RfC 2616.

RfC 2616 is a subset of RFC 1123 date. Weekday and month names for HTTP date/time formatting; always English!

gaetk2.tools.datetools.rfc2616_date_parse(data)

Parses an RfC 2616/2822 timestapm into a datetime object.

gaetk2.tools.datetools.date_trunc(trtype, timestamp)

Truncate date or datetime object. Truncated object of the given type.

This function is inspired by date_trunc from PostgreSQL, see http://www.postgresql.org/docs/8.1/static/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC

Supported types are year, quarter, month, week, day, hour, minute, second.

>>> date_trunc('week', datetime.datetime(1974, 8, 21))
datetime.datetime(1974, 8, 19, 0, 0)
>>> date_trunc('week', datetime.date(1973, 8, 8))
datetime.date(1973, 8, 6)

gaetk2.tools.datetools.get_tertial(date)

Calculates the tertial

>>> get_tertial(datetime.date(2015, 1, 9))
1
>>> get_tertial(datetime.datetime(2015, 2, 19))
1
>>> get_tertial(datetime.date(2015, 3, 9))
1
>>> get_tertial(datetime.datetime(2015, 4, 20))
1
>>> get_tertial(datetime.datetime(2015, 5, 4))
2
>>> get_tertial(datetime.datetime(2015, 6, 11))
2
>>> get_tertial(datetime.datetime(2015, 7, 22))
2
>>> get_tertial(datetime.date(2015, 8, 3))
2
>>> get_tertial(datetime.date(2015, 9, 23))
3
>>> get_tertial(datetime.datetime(2015, 10, 24))
3
>>> get_tertial(datetime.date(2015, 11, 11))
3
>>> get_tertial(datetime.datetime(2015, 12, 6))
3

gaetk2.tools.datetools.get_quarter(date)

Calculates the quarter

>>> get_quarter(datetime.date(2015, 1, 9))
1
>>> get_quarter(datetime.datetime(2015, 2, 19))
1
>>> get_quarter(datetime.date(2015, 3, 9))
1
>>> get_quarter(datetime.datetime(2015, 4, 20))
2
>>> get_quarter(datetime.datetime(2015, 5, 4))
2
>>> get_quarter(datetime.datetime(2015, 6, 11))
2
>>> get_quarter(datetime.datetime(2015, 7, 22))
3
>>> get_quarter(datetime.date(2015, 8, 3))
3
>>> get_quarter(datetime.date(2015, 9, 23))
3
>>> get_quarter(datetime.datetime(2015, 10, 24))
4
>>> get_quarter(datetime.date(2015, 11, 11))
4
>>> get_quarter(datetime.datetime(2015, 12, 6))
4

gaetk2.tools.datetools.get_yearspan(date)

Gibt den ersten und letzten Tag des Jahres zurück in dem date liegt

>>> get_yearspan(datetime.date(1980, 5, 4))
(datetime.date(1980, 1, 1), datetime.date(1980, 12, 31))
>>> get_yearspan(datetime.date(1986, 3, 11))
(datetime.date(1986, 1, 1), datetime.date(1986, 12, 31))

gaetk2.tools.datetools.get_tertialspan(date)

Gibt den ersten und den letzten Tag des Tertials zurück in dem date liegt

>>> get_tertialspan(datetime.date(1978, 9, 23))
(datetime.date(1978, 9, 1), datetime.date(1978, 12, 31))

gaetk2.tools.datetools.get_quarterspan(date)

Gibt den ersten und den letzten Tag des Quartals zurück in dem date liegt

>>> get_quarterspan(datetime.date(1978, 6, 12))
(datetime.date(1978, 4, 1), datetime.date(1978, 6, 30))

gaetk2.tools.datetools.get_monthspan(date)

Gibt den ersten und letzten Tag des Monats zurück in dem date liegt

>>> get_monthspan(datetime.date(1980, 5, 4))
(datetime.date(1980, 5, 1), datetime.date(1980, 5, 31))

gaetk2.tools.datetools.get_weekspan(date)

Gibt den ersten und den letzten Tag der Woche, in der date liegt, zurück.

Dabei ist Montag der erste Tag der woche und Sonntag der letzte.

>>> get_weekspan(datetime.date(2011, 3, 23))
(datetime.date(2011, 3, 21), datetime.date(2011, 3, 27))

gaetk2.tools.datetools.get_timespan(period, date)

Get given timespan for date

Convenience function as a wrapper for the other get_*span functions

gaetk2.tools.datetools.tertial_add(date, tertials)

Add number of tertials to date.

>>> date = datetime.date(1982, 11, 7)
>>> tertial_add(date, -1)
datetime.date(1982, 5, 1)
>>> tertial_add(date, 0)
datetime.date(1982, 9, 1)
>>> tertial_add(date, 1)
datetime.date(1983, 1, 1)
>>> tertial_add(date, 2)
datetime.date(1983, 5, 1)
>>> tertial_add(date, 3)
datetime.date(1983, 9, 1)
>>> tertial_add(date, 4)
datetime.date(1984, 1, 1)

>>> date = datetime.datetime(1982, 11, 7)
>>> tertial_add(date, 4)
datetime.datetime(1984, 1, 1)

gaetk2.tools.datetools.month_add(date, months)

Add number of months to date.

>>> import datetime
>>> date = datetime.date(1986, 3, 9)
>>> month_add(date, -12)
datetime.date(1985, 3, 9)
>>> month_add(date, -1)
datetime.date(1986, 2, 9)
>>> month_add(date, 0)
datetime.date(1986, 3, 9)
>>> month_add(date, 3)

>>> date = datetime.datetime(1986, 3, 9)
>>> month_add(date, 12)
datetime.datetime(1987, 3, 9)

gaetk2.tools.datetools.year_add(date, years)

Add number of years to date.

>>> import datetime
>>> year_add(datetime.datetime(2016, 2, 29), 1)
datetime.date(2017, 2, 28)
>>> year_add(datetime.date(2016, 2, 29), 1)
datetime.date(2017, 2, 28)
>>> year_add(datetime.date(2015, 2, 28), 1)
datetime.date(2016, 2, 28)
>>> year_add(datetime.date(2017, 2, 28, -1)
datetime.date(2016, 2, 28)
>>> year_add(datetime.datetime(2016, 2, 29), -1)
datetime.datetime(2015, 2, 28)

gaetk2.tools.datetools.add_to_day(day, offset)

Returns the date n days before or after day.

gaetk2.tools.datetools.easter(year)

Returns the day of Easter sunday for ‘year’.

This function only works betweeen 1900 and 2099

gaetk2.tools.datetools.easter_related_holidays(year)

Returns a list of holidays which are related to easter for ‘year’.

gaetk2.tools.datetools.holidays_german(start, end)

Returns a list of dates between start and end that are holidays.

gaetk2.tools.datetools.workdays(start, end)

Calculates the number of working days (Mo-Fr) between two given dates.

Whereas the workdays are calculated siilar to Python slice notation: [start : end[ Example: >>> workdays(datetime.date(2007, 1, 26), datetime.date(2007, 1, 27)) # Fr - Sa 1 >>> workdays(datetime.date(2007, 1, 28), datetime.date(2007, 1, 29)) # Su - Mo 0

gaetk2.tools.datetools.workdays_german(start, end)

Calculates the number of working days between two given dates while considering german holidays.

gaetk2.tools.datetools.is_workday_german(day)

Checks if a day is a workday in germany (NRW).

>>> is_workday_german(datetime.date(2007, 1, 1))
False
>>> is_workday_german(datetime.date(2007, 1, 2))
True

gaetk2.tools.datetools.next_workday_german(startday)

Returns the next workday after startday.

>>> next_workday_german(datetime.date(2006, 12, 29))
datetime.date(2007, 1, 2)

gaetk2.tools.datetools.previous_workday_german(startday)

Returns the workday before startday.

>>> previous_workday_german(datetime.date(2007, 1, 2))
datetime.date(2006, 12, 29)

gaetk2.tools.datetools.add_workdays_german(startday, count)

Adds <count> workdays to <startday>.

gaetk2.tools.unicode - string handling

This are functions which help to handle data from a pre-Unicode world. Much of this code is acient and has no use in a worl where JSON and XML ensure somewhat clean encoding. But still there are so many places where you are allowd to send only ASCII subsets.

Data Cleanup

• de_noise() - removed Unicode Characters which normally have no place in buiseness documents (eg street names). This includes Emojii but also protected spaces unusual quotation marks etc. This data is usually included dut to cut and paste errors. Read source to see what is replaced.
• de_umlaut() - converts data to plain ASCII while converting german Umlauts to something reasonable.
• de_utf8() - “repair” wrongly decoded UTF-8.

Number Conversion

num_encode() and num_decode() convert arbitrary long numbers to strings and back again. Works nice for datastore IDs. Uses base 62 (lowwer and upper letters and numbers) to get a compact representation.

num_encode_uppercase() uses base36 which is less compact but case insensitive.

You can use these functions to getsomewhat easy to tipe compact datastore ids:

class SomeEntity(ndb.Model):
    nr = ndb.ComputedProperty(lambda num_encode(self: self.key.id()) if self.key.id() else '?')

Module contents

gaetk2.tools.unicode.de_utf8(data)

This is meant to help with utf-8 data appearing where unicode should apperar.

gaetk2.tools.unicode.de_umlaut(data)

Converts a text to ASCII acting smart about Umlauts.

>>> de_umlaut('1 Über Hügel saß René äöüÄÖÜß')
'1 Ueber Huegel sass Rene aeoeueAeOeUess'

gaetk2.tools.unicode.de_noise(data)

Removes all stuff which should not appear in normal Western Text.

>>> de_noise(u'»Susie`s Giga\Super-Markt®¿«')
u">Susie's Giga/Super-Markt(R)?<"
>>> de_noise(u"ümlaut eins:
")
u'\xfcmlaut eins:'
>>> de_noise(u'«A» {C} ¿D? „E“ ›F‹')
u'<A> (C) ?D? "E" >F<'
>>> de_noise(u'`A´')
u"'A'"
>>> de_noise(u'«😎» Umlaute kann doctest !gut {®} ¿👩‍👩‍👧‍👦? „👨‍❤️‍💋‍👨“ ›🎅🏻🎅🏼🎅🏽🎅🏾🎅🏿‹')
u'<> Umlaute kann doctest !gut ((R)) ?? "" ><'
>>> de_noise(u'DE37  330 5 13 50 0 010  4414  22')
u'DE37330513500010441422'

gaetk2.tools.unicode.slugify(value)

Converts a string to be usable in URLs without excaping.

Normalizes string, converts to lowercase, removes non-alpha characters, and converts spaces to hyphens.

Inspired by Django’s “django/template/defaultfilters.py”.

gaetk2.tools.unicode.num_encode(n)

Convert an integer to an base62 encoded string.

gaetk2.tools.unicode.num_decode(s)

Convert the result of num_encode() back to an integer.

gaetk2.tools.unicode.num_encode_uppercase(n)

Convert an integer to an base36 (unly uppercase and numbers) encoded string.

gaetk2.tools.structured_xls package

Todo

• Explain what gaetk2.tools.structured_xls is for

class gaetk2.tools.structured_xls.XLSwriter(output=None, sheetname='This Sheet')

csv - Module compatible Interface to generate excel files.

… but you have to call save() oder getvalue() to generate the final XLS file.

Parameters:

• output (file or None) – optional File-Like Object for save() to export to.
• sheetname (str) – optional Name of the single Worksheet we export to.

Uses the deprecated xlwt.

Usage:

xlswriter = XLSwriter()
xlswriter.writerow(['foo', 1, 2])
xlswriter.writerow(['bar', 3, datetime.date.today()])
xlswriter.save(open('test.xls')

writerow(row)

Eine Zeile schreiben. Row ist eine Liste von Werten.

save(fd=None)

Write rendered XLS file to fd or self.output.

getvalue()

Returns rendered XLS file as a StringIO().