Google App Engine Datastore and Search API integration

It seems that there are quite a few situations where you want to store data in the Datastore and register it in the Index of the Search API. However, if you are not careful about transactions, it is possible that even though it is stored in the Datastore, it is not registered in the Search API Index (hereinafter, Index means the Search API Index). It ends up.

Therefore, let us consider a method of registering data in the Datastore and Index (almost) at the same time.

How to use _post_put_hook (consistency is not guaranteed)
Simple way to use @ ndb.transacational (consistency guaranteed)
How to use Task Queue (consistency guaranteed)
Conclusion

** 1 method is not recommended **. However, there are some examples of using method 1 (one was submitted as an answer on Stack Overflow, and Ferris 2.2 uses method 1), so I'm introducing it for the time being.

1. How to use _post_put_hook

** Not recommended ** unless you have a specific reason.

class User(ndb.Model):
    name = ndb.StringProperty()
    
    def _post_put_hook(self, future):
        result = future.get_result()

#Register in Search API Index ...

post_put_hook is called after User is put. If put fails, an error will be sent in the future.get_result () part, so it will not be registered in Index. However, there is a possibility that the registration of User will fail even though the registration of User is successful.

In other words, this method does not guarantee the consistency of Datastore and Index. It's okay if consistency doesn't have to be guaranteed, but it should be avoided otherwise.

2. How to simply use @ ndb.transacational

user = User(name='Yohei')

@ndb.transactional
def put():
    user.put()
    doc = search.Document(
        doc_id = person.key.urlsafe(),
        fields=[
            search.TextField(name='name', value=user.name),
        ],)
    index.put(doc)
    
put()

This guarantees consistency. However, you have to be careful in the following examples.

user = User(name='Yohei')

@ndb.transactional
def put():
    user.put()
    doc = search.Document(
        doc_id = person.key.urlsafe(),
        fields=[
            search.TextField(name='name', value=user.name),
        ],)
    index.put(doc)
    # do something
    ...
    
put()

If an error occurs in the do something part here, it may not be registered in the Datastore, but it may be registered in the Index.

3. How to use Task Queue

** Task Queue can be processed transactional. [^ 2-1] ** Therefore, if you add the task "Register Search API to Index" in Task Queue, consistency can be guaranteed.

user = User(name='Yohei')
user2 = User(name='Yohei2')

@ndb.transactional(xg=True)
def put():
    user.put()
    user2.put()
    taskqueue.add(
        url='/put_to_index',
        params={
            'key': user.key.urlsafe(),
            'name': user.name},
        transactional=True,)
    taskqueue.add(
        url='/put_to_index',
        params={
            'key': user2.key.urlsafe(),
            'name': user2.name},
        transactional=True,)
    # do something
    ...
    
put()

This guarantees consistency, even if do something causes an error. As in the example, it is possible to register to two indexes at the same time. Note that you can only stack up to 5 transactional Task Queues. It should be avoided in situations where there are many writes at the same time.

4. Conclusion

Use 2 or 3 methods. Use properly according to the situation.