Repository navigation
Transparently handle new elected Hyperspace master #84
Description
Activity
if any one have a chance to try the following updates kashirin-alex@0780b30 will be great to know the status!
https://github.com/kashirin-alex/hypertable/tree/0.9.8.14-rc
Tested only on a 3 servers cluster, (require more tests)
Hyperspace Master elected on event (at start)
if master falls new master elected, log is:1535028153 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:479) Received DB_EVENT_REP_CONNECT_BROKEN event
1535028153 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:505) Received DB_EVENT_REP_MASTER_FAILURE event
1535028158 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:485) Received DB_EVENT_REP_CONNECT_TRY_FAILED event
1535028163 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:485) Received DB_EVENT_REP_CONNECT_TRY_FAILED event
1535028168 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:485) Received DB_EVENT_REP_CONNECT_TRY_FAILED event
1535028170 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:411) BDB ERROR: BDB3534 Not enough votes to elect: recvd 1 of 2 from 3 sites
1535028170 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:492) Received DB_EVENT_REP_ELECTION_FAILED event
1535028173 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:485) Received DB_EVENT_REP_CONNECT_TRY_FAILED event
1535028178 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:485) Received DB_EVENT_REP_CONNECT_TRY_FAILED event
1535028181 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:439) Received DB_EVENT_REP_NEWMASTER event
1535028181 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:451) New master elected: 159.69.115.129
1535028182 INFO htHyperspace : (/root/builds/sources/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:471) Received DB_EVENT_REP_STARTUPDONE event
`
Hello,
The re-election of new Hyperspace master as discussed at (http://www.hypertable.com/documentation/administrator_guide/machine_failure/#hyperspace) does not include transparent auto re-election.
I think the best way around it is to further develop the method String get_current_master() of Hyperspace/BerkeleyDbFilesystem.h and to have it toward a "get_safe_current_master" with the ability to work with a state of other replicas on interval and re-elect a new Hyperspace master when latest is no longer operative.
AN EDIT: best way it is the events handling of void BerkeleyDbFilesystem::db_event_callback procedures.
and there should be another cmd "decommission" to the hyperspace command interpreter and do a dbenv.rep_sync(0) when possible.
Further, It is to have the configuration file applied on run-time, by extending the PropertiesPtr's get methods between the lines 280-320 (https://github.com/kashirin-alex/hypertable/blob/master/src/cc/Common/Properties.h#L287) on TTL or file modify date to load the new configuration name requested (if not all)
Part of the subjective, there is a new configuration option "Hypertable.RangeServer.Location.AutoReInitiate=bool" defaults to false
simple process of if true for auto re-initiate,
(https://github.com/kashirin-alex/hypertable/blob/master/src/cc/Hypertable/RangeServer/RangeServer.cc#L323)
clears current m_location and file
(https://github.com/kashirin-alex/hypertable/blob/master/src/cc/Hypertable/RangeServer/LocationInitializer.cc#L107)
and continues the same way as a new RangeServer came up.
So far no issue raised up and when the RS is marked removed, the log would include "Auto re-initiated location, removed location N" without failure on cluster/RangeServer start.
Will be great to know if there are better plans for the Hyperspace master auto re-election.
Thank You,
Kashirin Alex