what memcache stores?

Now after I learned how to store one-to-many collections with AppEngine datastore, I want to use Memcache API to eliminate DB hits between requests. Quick test shows it would not work out of the box:

class C(object):
 def __init__(self):
   self.field='field'

c=C()
c.attr='attr'
c.__dict__['dict']='dict'
print dir(c)
memcache.add('c', c)
c=memcache.get("c")
print dir(c)

I would expect this code to output the same list of public attributes before and after, but it does not happen. We can see that only c.field was saved, while c.attr and c.dict are missing. Need to investigate memcache internals to understand why.

Read More......

AppEngine Datastore and memcache

I miss Hibernate collections. In the following code I access the collection a thousand times:

class Master(db.Model):
  pass

class Detail(db.Model):
  master=db.ReferenceProperty(Master)

m=Master()
m.put()
d=Detail(master=m)
d.put()

for i in range(1000):
  for tmp_d in m.detail_set:
    pass

The above code takes a few second to execute. The reason is Datastore fetches the collection from the storage every time, and in Hibernate the collection would be fetched from the database only once until the end of the session. Oops, no sessions with Datastore. So Datastore developers were right when they opted to fetch collection every time - they don't know when the details change.

This is the reason Master cannot be put in memcache effectively: it would be stored without the Details. Master.detail_set holds only the definition of the query needed to get the details. So I'm thinking of a way I could decorate ReferenceProperty to make one-to-many relations suitable for the memcache. So big object trees will be read from Datastore once and then accessible in a fast way.

Read More......