Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12943 Consistent Reads from Standby Node
  3. HDFS-13522

HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Changes will need to occur to the router to support the new observer node.

      One such change will be to make the router understand the observer state, e.g. FederationNamenodeServiceState.

      This patch captures the state of all namespaces in the routers and propagates it to clients. A follow up patch will change router behavior to direct requests to the observer.

       

       

      Attachments

        1. ShortTerm-Routers+Observer.png
          155 kB
          CR Hota
        2. Router+Observer RPC clogging.png
          101 kB
          CR Hota
        3. RBF_ Observer support.pdf
          298 kB
          CR Hota
        4. rbf_observer_reads - third_party_reads.pdf
          56 kB
          Simbarashe Dzinamarira
        5. rbf_observer_reads - read_own_writes.pdf
          34 kB
          Simbarashe Dzinamarira
        6. observer_reads_in_rbf_proposal_simbadzina_v2.pdf
          320 kB
          Simbarashe Dzinamarira
        7. observer_reads_in_rbf_proposal_simbadzina_v1.pdf
          171 kB
          Simbarashe Dzinamarira
        8. HDFS-13522.002.patch
          93 kB
          zhuobin zheng
        9. HDFS-13522.001.patch
          52 kB
          Surendra Singh Lilhore
        10. HDFS-13522_WIP.patch
          80 kB
          Hemanth Boyina

        Issue Links

        causes

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16826 [RBF SBN] ConnectionManager should advance the client stateId for every request

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16831 [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16837 [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespace

        • Major - Major loss of function.
        • Resolved
        Dependent

        Task - A task that needs to be done. HDFS-16589 RBF: Support cluster level SBN read switch in Router-Based Federation

        • Major - Major loss of function.
        • Open
        incorporates

        Bug - A problem which impairs or prevents the functions of the product. HDFS-14887 RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable

        • Major - Major loss of function.
        • Resolved
        is related to

        Sub-task - The sub-task of the issue HDFS-13286 Add haadmin commands to transition between standby and observer

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16826 [RBF SBN] ConnectionManager should advance the client stateId for every request

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16831 [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16834 PoolAlignmentContext should not max poolLocalStateId with sharedGlobalStateId when sending requests to the namenode.

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16837 [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespace

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-16827 [RBF SBN] RouterStateIdContext shouldn't update the ResponseState if client doesn't use ObserverReadProxyProvider

        • Major - Major loss of function.
        • Open

        Task - A task that needs to be done. HDFS-16889 Backport JIRAs related to RBF SBN to branch-3.3

        • Major - Major loss of function.
        • Open
        relates to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-16821 Fix regression in HDFS-13522 that enables observer reads by default.

        • Major - Major loss of function.
        • Resolved
        requires

        New Feature - A new feature of the product, which has yet to be developed. HADOOP-18345 Enhance client protocol to propagate last seen state IDs for multiple nameservices.

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18406 Adds alignment context to call path for creating RPC proxy with multiple connections per user.

        • Major - Major loss of function.
        • Resolved
        split to

        Sub-task - The sub-task of the issue HDFS-16767 RBF: Support observer node from Router-Based Federation

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #3005

        Web Link GitHub Pull Request #4127

        Web Link GitHub Pull Request #4311

        Web Link GitHub Pull Request #4441

        Web Link GitHub Pull Request #4523

        Web Link GitHub Pull Request #4883

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            simbadzina Simbarashe Dzinamarira
            xkrogen Erik Krogen
            Votes:
            2 Vote for this issue
            Watchers:
            28 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20h 50m
                20h 50m

                Slack

                  Issue deployment