Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C4-23 Fix Mirror Health #125

Merged
merged 6 commits into from
Feb 12, 2020
Merged

C4-23 Fix Mirror Health #125

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cb45768
C4-23 sketch TTL cache for esstorage
willronchetti Feb 7, 2020
1962ae0
C4-23 sol'n 1 cache and dont recompute
willronchetti Feb 7, 2020
f0ed95b
C4-23 small fixes, implement ttl reset
willronchetti Feb 7, 2020
e36de67
C4-23 cache mirror_health, recompute if error
willronchetti Feb 12, 2020
9214133
C4-23remove unused import
willronchetti Feb 12, 2020
a81bee3
C4-23 clean up logic in init
willronchetti Feb 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions src/snovault/elasticsearch/esstorage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,11 @@ def __init__(self, registry):
self.registry = registry
self.es = registry[ELASTIC_SEARCH]
self.index = get_namespaced_index(registry, '*')
self.mirror_env = self.registry.settings.get('mirror.env.name', None)
self.mirror_health = None
if self.mirror_env: # mirror info does not change. Cache it
self.mirror_health = ff_utils.get_health_page(ff_env=self.mirror_env)


def _one(self, search):
# execute search and return a model if there is one hit
Expand Down Expand Up @@ -250,24 +255,25 @@ def purge_uuid(self, rid, item_type, max_sid=None):
self.registry[INDEXER].find_and_queue_secondary_items(set([rid]), set(), sid=max_sid)

# if configured, delete the item from the mirrored ES as well
if self.registry.settings.get('mirror.env.name'):
if self.mirror_health:
mirror_env = self.registry.settings['mirror.env.name']
use_aws_auth = self.registry.settings.get('elasticsearch.aws_auth')
mirror_health = ff_utils.get_health_page(ff_env=mirror_env)
log.info('PURGE: attempting to purge %s from mirror storage %s' % (rid, mirror_env))
# if we could not get mirror health, bail here
if 'error' in mirror_health:
if 'error' in self.mirror_health:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So in this case you have a dictionary with an 'error' key in it? e.g.,

{"error": "unable to do something"}

Is that right? it'd be nice to have some kind of note saying what it was looking for.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes in fact we do - this actually tripped me up the first time I put this code up. See ff_utils.get_health_page (https://github.com/4dn-dcic/utils/blob/master/dcicutils/ff_utils.py#L867)

log.error('PURGE: Tried to purge %s from mirror storage but couldn\'t get health page. Is staging up?' % rid)
log.error('PURGE: Mirror health error: %s' % mirror_health)
log.error('PURGE: Mirror health error: %s' % self.mirror_health)
log.error('PURGE: Recomputing mirror health...')
self.mirror_health = ff_utils.get_health_page(ff_env=self.mirror_env)
return
# make sure use_aws_auth is bool
if not isinstance(use_aws_auth, bool):
use_aws_auth = True if use_aws_auth == 'true' else False
mirror_client = es_utils.create_es_client(mirror_health['elasticsearch'],
mirror_client = es_utils.create_es_client(self.mirror_health['elasticsearch'],
use_aws_auth=use_aws_auth)
try:
# assume index is <namespace> + item_type
mirror_index = mirror_health.get('namespace', '') + item_type
mirror_index = self.mirror_health.get('namespace', '') + item_type
mirror_client.delete(id=rid, index=mirror_index, doc_type=item_type)
except elasticsearch.exceptions.NotFoundError:
# Case: Not yet indexed
Expand Down