Jump to: navigation, search

V4/Manage replication topology

Name: V4/Manage replication topology
Ticket: #4302
Target version: 4.3.0
Author: LKrispen
Incomplete.png Pending review
Test plan: V4/Manage_replication_topology/Test_plan
Last updated: 2017-01-9 by Akasurde

Overview

The goal of this new feature is to provide a consistent information about the replication topology on each server. It is achieved by maintaining the replication topology information in the shared tree.

Use Cases

  • Provide all replication configuration information consistent across all deployed servers on all servers, eg to easily visualize the replication topology by contacting only one server
  • Do sanity checks on replication configuration, denying modifications which would break replication topology or issue warnings.
  • Use the information in the shared tree to trigger changes to the replication configuration in the corresponding servers, this means to allow to completely control replication configuration with modifications of entries in the shared tree

Example 1

From an IPA master (A) server two replicas (B,C) have been created. Now there exist the following replication connections C <--> A <--> B To connect B <--> C a connection element B-C can be added on any server, it will be replicated to all other servers. The topology plugin on each server handles this update and checks if it is the starting point of a connection and adds a replication agreement to the endpoint. So Server B will create a replication agreement B-->C and, C will create an agreement C-->B.

Example 2

If the list of replicated attributes should be changed, the entry containing the default list of attributes can be changed on one server and it will be applied to all replication agreements on all servers in the topology.

Design

The feature is implemented as a Directory Server plugin. It handles changes to the replication configuration in the server specific configuration entries in cn=config and to the configuration entries in one or multiple shared (replicated) databases. First the plugin configuration is defined, then the layout and schema of the shared configuration information. In the plugin operation the detailed activities of the plugin for different operations are described.

Plugin configuration

Dependencies

multimaster replication plugin

The topology plugin has a strong relation to the multiMasterReplication plugin, it controls attributes and entries used by the replication plugin , but the settings done via the topology plugins should have priority over settings in cn=config. But to manage the topology in the shared tree it also relies on replication to be working, it therefore defines the MMR plugin as a required plugin.

bind dn group

The plugin will create replication agreements from segments. This should happen without directly editing the replica object of the receiving server to add a new bindDN (ldaprincipal of sending server). Directory server has a feature to use groups in the shared tree to contain authorized bindDn: bindDNgroup. These have to be defined and populated with the ldapprincipals of new replicas. The group will be replicated and a replcation agreement between any two servers can be created without further editing of bind permissions.

Support domain level control

The plugin will support the use of the domain level defined in "Domain Levels" in environment with different IPA versions. It will only become active if the global domain level is >= 1 So the plugin will wait until the domain level is set and then perform its initialization operations. There are three different scenarios when a change of the domain level can be detected The plugin provides a callback for rootdse searches, so that any admin or management tool can monitor the state. It will publish the domain_level, if the plugin is activated and (only informational) the plugin version

This is an example of a base search:

ldapsearch -h <hostname> -x  -b "" -s base
....
ipatopologypluginversion: 1.0
ipatopologyismanaged: off
ipadomainlevel: 0
Check at startup

When the directory server starts, the plugin init and start functions are executed, they will read the domain level attribute and act accordingly

Check for modify operation

If an admin or tool changes the domain level the plugin detects this change and performs initialization tasks if the domain level is >= 1

Check after online initializatition

A change of the domain level can also be the effect of an online initialization of a replica from a server where the domain level is set. The domain level entry is part of the realm database, of this database is reinitialized, the replica has now the same domain level as the master from which initialization originated. The plugin has to verify if the domain level has changed or if the topology configuration in the new version of the database has changed. To detect this change the plugin registers a statechange function and will be notified if the backend comes online after initialization. Behaviour is the same as after startup.

Startup delay

In the startup phase the init/start functions for all plugins will be called, but plugins only will be activated once ALL plugins have been initialized. If the topology plugin would do any changes to the shared tree, eg generate a segment from an agreement this would be applied to the local database, but will not be written into the changelog and replicated to other servers since the multimaster replication plugin is not yet active and no csns are assigned. The solution is to do the basic startup operation in the plugin init but for the rest of the startup operations queue a function in the event queue. This queue will be started after all the plugins have been activated. In principle the topology startup function could be called without delay, but tests have shown that the plugin could run into one of the deadlock scenarios at startup, so a delay can be defined for the execution of this function (and can be set to 0 once the deadlock problems are resolved).

A similar situation exists when a replica is being initalized from an other master, the backend goes offline before the init and online when init is complete. The replication plugin, the topolgy plugin and some other plugins register a function to be notified by state changes. But the order of these notifications is not defined, so the topology plugin could be notified before the replication plugin and a potential update to the shared tree would not be replicated. Therefor also when a backend gets online, the delay is applied before performing operations to the shared tree.

Configuration attributes

The plugin needs to know where in the shared tree the configuration information resides, which databases should be controlled and some other basic information. This is provided in the plugin entry in cn=config.

The plugin configuration attribute type names are of the form nsslapd-topo-plugin-<config attr name>.

Defined config attributes
Option Name Type Description
shared-config-base configuration base in shared tree DN This is the starting point for the replication topology in the shared tree, the information for specific databases and server connections are represented in entries below this base. In the first implementation there will be a single configuration area for all suffixes, it can later be extended to be multivalued and distributed over the managed suffixes.
shared-replica-root suffixes controlled by the plugin DN multivalued attribute listing the replicated suffixes which should be managed by the topology plugin
plugin hostname The segments defining the replication topology use a "leftNode" and "rightNode" attribute to define the endpoints (nodes) of an edge in the replication topology graph. The plugin instance needs to be able to decide if a segment applies to the instance it is running in and requires knowledge of it own hostname. To avoid an additional place where a hostname has to be configured, the plugin hostname is derived from the servername use in DS configuration.
Potential config attributes
Option Name Type Description
shared-binddngroup bind dn group off Directory server has a feature to specify a group to define which users are allowed to update a replica, this feature is used to be able to create new replication agreements without having to edit the replica object to add new binddns, it can be managed in the shared tree.

The topology plugin uses this feature indirectly to be able to generate and add replication agreements from topology segments without having to update the receiving server. The only direct use of the binddn group is when a server is removed from the topology and the principal is removed from the group. Therefor the dn of the binddngroup must be available as a plugin configuration parameter.

NOTE: the managed suffixes are also define by the suffix entries in the shared configuration and is redundant, could be deprecated.

Example

dn: cn=IPA Toplogy Configuration,cn=plugins,cn=config
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject
cn: IPA Toplogy Configuration
nsslapd-pluginPath: /home/ludwig/freeipa/myplugin/libipatopo-plugin
nsslapd-pluginInitfunc: ipa_topo_init
nsslapd-pluginType: object
nsslapd-pluginEnabled: on
nsslapd-topo-plugin-shared-config-base: cn=topology,cn=etc,dc=example,dc=com
nsslapd-topo-plugin-shared-replica-root: dc=example,dc=com
nsslapd-topo-plugin-shared-bindngroup: cn=replication managers,cn=etc,cn=idm,dc=example,dc=com
nsslapd-pluginId: ipa-topology-plugin
nsslapd-pluginVersion: 0.1
nsslapd-pluginVendor: freeipa
nsslapd-pluginDescription: ipa-topology-plugin

Shared configuration layout

The information about the replication topology information is contained in subtree of the shared tree, eg

cn=topology,cn=etc,SUFFIX

but can be configured to be in any other place. The topology information consists of nodes (currently only masters), segments (connections between servers) and properties of the segments (configuration parameters).

The information about the nodes is already available in cn=masters,cn=etc,SUFFIX cn=topology is a container for holding default parameters and segment information for the topology of the managed suffices.

A level below are the top entries for segments of a specific replicated database.

cn=replica xxx,cn=topology,<shared-config-base>

If the topology for different databases should be identical, a replicaConfig object can reference another replicaConfig object as a master config. The top entry for a replica can also maintain default values for managed replication configuration attributes

At the next level are entries defining the segments, each segment is defined by the nodes it connects and the type of connectivity. The nodes are noted as "left" and "right", properties for connections originating at the "left" node are noted as "left-<attribute-name>", same for "right"

cn=aToB, cn=replica xxx,cn=topology,<shared-config-base>

In the next sections the schema of the configuration objects is defined

shared configuration: container

The topology plugin can be used to manage different backends an in each backend define connection segments. There is a need for a top level container as starting point for the configuration in the shared tree. No specific attributes required.

dn: cn=topology,dc=example,dc=com
objectClass: nsContainer
objectClass: top
cn: topology

shared configuration: database

There is one top entry for each database for which replication configuration should be controlled via configuration information and the topology plugin. It is defined by the below specified objectclass and attributes

cn=<replicaroot>

the entry exists for each managed database, the naming attribute could be derived or identical to the replicaroot, but this is not required.

objectclass: ipaReplicationTopologyConfig
ipaReplTopoConfigRoot: < dn >

The dn of the suffix, identical ro ReplicaRoot in the plugin config and in the replica object

ipaReplTopoConfigMaster: < dn >

Refererence to another TopologyConfig entry, if present for this replica the same segments and configuration settings will be used as in the master topology.

ipsReplTopoConfig<ManagedAttribute>: <managedattribute value>

here all default settings can be defined, so every replication agreement would use eg the same set of replicated attributes

shared configuration: segment

The segment is the core object describing the connection between two nodes, it will be defined by the below objectclass and attributes

cn: <uniquename defining the segment>

The segment is defined by the two nodes, so the naming component could be generated from the hostnames, but this could get quite long and become unreadable, the required information about the nodes is contained in the entry so there is no need to have it in the naming attribute. So the naming component should be supplied by the admin creating the segment. An alternative would be to generally use an ipauniqueId as naming component for segments, but the number of segments is limited and could be well managed by admins using meaningful names.

objectclass: ipaReplicationTopologySegment

Objectclass describing the segment. Required attributes are LeftNode,RightNode, SegmentDirection

ipaReplSegmentLeftNode: < dn > # reference to cn=masters
ipaReplSegmentRightNode: < dn > # reference to cn=masters

the attribute could contain a FQDN, but the hostnames are also in the entries in cn=masters, so a reference ensures consistency

ipaReplSegmentDirection: [both | left-right | right-left | none ]

"none" could indicate the there definitely should be no connection, the absence of a segment only indicates that the segment has not been created. "none" is not yet supported, one-directional segments ("left-right","right-left") are only created when initially a segment is created from a replication agreement, the creation of one directional segments via ipa commands is not yet suported.

<ManagedAttrName>[;left|;right]: < attrvalue >

to be used if default attribute values eg for replicated attributes should be overwritten. Probably not used/needed for most of them, but eg it could be used to temporarily disable a specific replication agreement. These attributes have a 1-1 mapping to the attributes in a replication agreement. So the same attributetypes can be used in the tiopology configuration. If there shold be different settings for the different directions then subtypes ";left" or ";right" can be used. So we have the following hierachy of configuration parameters with decreasing precedenc:

managedattr;left or managedattr;right in segment entry
managedattr in replica enty
default topology plugin value
default replication plugin value

shared configuration: example

dn: cn=topology,dc=example,dc=com
│   ├── objectclass: nsContainer
│
├── cn=replica example,cn=topology,dc=example,dc=com
│   │   ├── objectClass: ipaReplTopoConf
│   │   └── ipaReplTopoConfRoot: dc=example,dc=com
│   │
│   ├── cn=111-to-102,cn=replica example,cn=topology,dc=example,dc=com
│   │   ├── objectClass: ipaReplTopoSegment
│   │   ├── ipaReplTopoSegmentDirection: both
│   │   ├── ipaReplTopoSegmentLeftNode: vm-111.idm.lab.eng.brq.redhat.com
│   │   ├── ipaReplTopoSegmentRightNode: vm-102.idm.lab.eng.brq.redhat.com
│   │   ├── nsds5ReplicaEnabled;right: on
│   │   ├── nsds5ReplicaEnabled;left: off
│   │   ├── nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn 
│   │   │                         krblastsuccessfulauth krblastfailedauth krbloginfailedcount
│   │   └── nsds5ReplicaStripAttrs;left: modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp
│   │
│   └── cn=111-to-191,cn=replica example,cn=topology,dc=example,dc=com
│       ├── objectClass: ipaReplTopoSegment
│       ├── ipaReplTopoSegmentDirection: left-to-right
│       ├── ipaReplTopoSegmentLeftNode: vm-111.idm.lab.eng.brq.redhat.com
│       ├── ipaReplTopoSegmentRightNode: vm-191.idm.lab.eng.brq.redhat.com
│       └── ....
│
└── cn=replica ipaca,cn=topology,dc=example,dc=com
   │   ├── objectClass: ipaReplTopoConf
   │   └── ipaReplTopoConfRoot: o=ipaca
   │
   └── cn=111-to-191,cn=replica example,cn=topology,dc=example,dc=com
       ├── ipaReplTopoSegmentDirection: both
       ├── ipaReplTopoSegmentLeftNode: vm-111.idm.lab.eng.brq.redhat.com
       ├── ipaReplTopoSegmentRightNode: vm-191.idm.lab.eng.brq.redhat.com
       └── ....

Managed servers

The topology plugin, when activated controls the replication configuration, but the scope of control is restricted to "managed servers". This means that connections between managed servers can only be managed by changing the the topology segments for a managed suffix. For connections to "unmanaged" servers, direct creation of replication agreements in cn=config is possible.

The identification of managed servers is done with an additional objectclass in the server entries "cn=hostxxxx,cn=masters,.....".

objectclass: ipaReplTopoManagedServer

This objectclass has a multivalued attribute, defining the suffix(es) for which this server is managed by the topology plugin

ipaReplTopoManagedSuffix: dc=idm,dc=lab,dc=eng,dc-brq,dc=redhat,dc=com

Managed attributes

The topology plugin has two tasks regarding specific replication configuration attributes (managed attributes):

1] for modifications of the attribute in the shared config it will trigger an update of the corresponding attribute in cn=config

2] Any updates for the managed attributes in cn=config will be rejected.

The following table lists the configuration attributes for replication agreements, how they are currently used and how they should be managed by the plugin.

Used: this means the attribute is activly set and modified by IPA installation or tools

Manage: the attribute should be availabe (l) or managable (m) by the topology plugin

Default: should there be a default value in the shared tree

Config: should this attribute be modifiable by a modify operation of the cn=config entry

Attribute Used Manage Default Config
nsds5ReplicaRoot y l - n
nsds5ReplicaHost y l - n
nsds5ReplicaPort y l - n
nsds5ReplicaBindDN y* y* - n
nsds5ReplicaCredentials y* y* - n
nsds5ReplicaBindMethod y l - n
nsds5ReplicatedAttributeList y m y n
nsds5ReplicatedAttributeListTotal y m y n
nsds5ReplicaStripAttrs y m y n
nsds5ReplicaUpdateSchedule n n - ?
nsds5BeginReplicaRefresh n ? - y
nsds5ReplicaTimeout n n - n
nsds5ReplicaBusyWaitTime n n - n
nsds5ReplicaSessionPauseTime n n - n
nsds5ReplicaEnabled n m y y
* IPA usually does not use bindDN and credentials, but in the initial phase when setting up a connection and the Kerberos principals have not yet been deployed these are used. 

The managed attributes can be defined in the ipaReplicationTopologyConfig entry or in the individual segments

connectivity management

Since all operations to add or remove a replication agreement are now applied to the configuration in the shared tree and the actual changes in the connectivity are triggered by the topology plugin it is now possible to prevent any configuration where connectivity is broken.

connectivity check before removing a segment

Task: Check if nodes in a topology are still connected after the removal of a segment

Assumption: the topology is fully connected. If not, the removal of the segment does not make connectivity worse.

Algorithm: Remove the segment and find a path connecting the two nodes (f,t) in taining graph

Let N be the set of nodes ni in the topology. N = {n; n server in topology}
Let S be the set of segments defined in the topology. .s is a triple (l,r,d) , l,r ε N, d is direction d ε{left->right, right->left,both} 
Let the fanout of a node F(n,S) be the nodes reachable from n by segments in S
Let R be a set of reachable nodes
Let V be the set of visited nodes.
Check if a connection f-->t exist in S' = S \ {(f,t,*)}
0. R = F(f, S'), V={}
1. if R == {} exit; not connected
2  choose r  ε  R
3  if r == t break; connected
4  V = V u {r}
5  R = R u (F(r,S') \ V)
6  goto 1.

connectivity check for topology

The topology plugin will prevent to break the topology by removing segments (agreements), which would disconnect the topology. But the plugin will become active only after the domain level is set, so it can be useful to check the connectivity of a given deployment. Apart from the basic check if all servers are directly or indirectly connected to each other, this can be extended to detect the deegree of connectivity, eg if there are single points of failure.

plugin operations

basic operations

marking agreements

The topology plugin is authoritative for all replication connections between managed servers of the topology. But such connections can exist before the domain level is raised and they are transformed into connection segments. Also replication agreements from managed servers to unmanaged servers may exist. In addition situations where teh database or the config in dse.ldif is restored from a backup need to be handled. To handle this agreements are "marked" when they are put under control of the topology plugin. Marking agreements could be achieved by different methods. One merthod could be to use a naming convention and rename the agreement to distinguish it from "ordinary" agreements, but this approach fails as modrdn operations are not (yet) supported for the cn=config tree. An other method is to use a marker attribute in the replication agreement. To avoid the extension of core schema for replication a new objectclass is defined which can be added to the replication agreement entry. It will be added to the schema when the topology plugin is deployed. This obejectclass has only one optional attribute for additional information, eg if the agreement did exist and is now controlled by the plugin or if the agreement was created by the plugin

objectclass: ipaReplTopoManagedAgreement
ipaReplTopoManagedAgreementState: managed agreement - generated by topology plugin
merging segments

When segments are created from replication agreements, these segments are created on directional "left-right", since it is not required or known if the other direction exists. But for every addition of a segment, it is checked if the opposite direction already exists - and the two segments are merged. To avoid replication conflicts or multiple merge operations, the merging is only done on one of the endpoints of the merged segments, and there is an arbitrary priority on which one of these two servers it is done. So merging will happen only once and on a specific server only.

handling replica removal

For the topology plugin the removal of a replica is the removal of a managed server, the removal of a host from cn=masters. Before the host is removed at least one segment connecting this host must remain, its removal would be rejected. To allow a clean removal of a replica the plugin implements the following actions: If a replica/master is removed, all segments connecting this server will be removed and with the segment removal also the corresponding replication agreements will be removed. The ldap principal of the removed servers will be removed from the allowed binddns in a replica object and as a member of the bindDN group.

NOTE:There is one problem, the removal of a segment triggers the removal of the corresponding replication agreement, and this  introduces a race condition of removing the agreement and replicating the deletion of the segment to other servers. This cannot fully e resolved, but the plugin operates in a way that the remaining topology is consistent and clean
To achieve this 
- the deletion of a segment is never done on the removed server, but always on the other endpoint of the connection. So this removal is always replicated to all other remaining servers in the topology
- even if the removal of the segment is not replicated to the removed server  and its replication agreements persist, it will not be able to replicate into the remaining topology since its credentials have been removed.
NOTE 2: At the moment there is no check that the removal of replica does not disconnect the replica, eg if in the topology A <--> B <--> C the replica B would be removed A and C would be disconnected. This should be handled in the admin tools for removing a replica

startup

The startup function reads the plugin configuration and sets up its environment

If the plugin is activated it checks initial consisteny of cn=config and shared configuration area. This core startup function will be called in three potential places: - startup of directory server - raising of domain level so that the pluging gets activated - notification that a backend comes online after an online initialization and the domain level is set

check if topology container exist
    read all replication configuration entries in cn=config
    for each configured replication root
          read all shared configuration segments
                if a replication agreement exists in cn=config and as a segment
                       compare agreement parameters and update replagreement (shared tree config is authoritative)
                if  segment exists, but no repl agreement
                       create repl agreement
                if repl agrement exist, but no segment
                      if agreement is marked
                          delete agreement
                      else
                           create one directional segment from agreement

preop

The modifications attempts of replication agreements and attributes in cn=config, which are managed by the plugin should be captured in the pre-operation phase, no need to apply the modifications and later undo it. The updates from the shared tree also trigger modify operations for these entries, but these should pass. This can be handled by generally allowing only internal operations by the topology plugin itself to modify specific replication configuration in cn=config. If a modify operation tries to modify in one operation a managed attribute and an unmanaged attribute the action is to reject the complete operation and not only to strip the change to the managed attribute. A client would not see that its modify operation only partially had been accepted. If a modification is rejected, an appropriate error mesage should be returned This applies to ADD and DEL operations as well, once the plugin is active replication agreements for managed replicas can only be added or deleted by the toplogy plugin

The attempt to add an existing segment or the attempt to delete a segment, which would disconnect the topology have to detected and rejected in the preop plugin

if internal op (by topology plugin)
      return success
 if modified entry  in cn=config and is managed object
      return error (unwilling to perform)
 if entry is topology segment
     if ADD operation check existence of segment
     if DEL operation check if it would break connectivity

postop

In the postop phase successful modifications of the shared config area are captured and applied to cn=config

if entry in cn=config
    return
if entry is domainlevel entry 
     get domain level and compare to plugin version
     if plugin is activated 
          execute startup 
if entry is topology container
  initialize plugin data
if entry is configRoot
    for all segments check and update modified attributes in cn=config
if entry is segment
         if segment contains plugin host as left or right node
             ADD/DEL/MOD replication agreement to other node
          if segment properties are changed apply changes
          if segment is added and one directional 
                if segment for opposite direction exists
                      merge segments
                            update one segment to bidirectional
                            remove other segment

managing multiple suffixes

For each replicated suffix in a set of servers it can be configured if it is to be managed by the topolgy plugin and to which subset of servers this should be applied. Each suffix has to be added as a managed suffix to all participating servers in their server entry below cn=masters.

If a suffix, which was not managed, should become a managed suffix, it has to be added to the topology configuration in the shared tree. This means a suffix entry has to added to the topology container and for each master which should be managed for this suffix the master antry has to be updated and the managedSuffix attribute for this suffix has to be added.

If replication agreements for a newly managed suffix already exist, they will be converted to segments and controlled by the plugin.

An alternative method is to use the existing replication configuration of an other suffix as a template and reference this in the ipaReplTopoConf object

Feature Management

UI

Feature management in the UI: TBD

CLI

'ipa subcommands

In the current version replication connections are managed with two command line utilities: ipa-replica-manage and ipa-csreplica-manage. When the topology plugin is deployed these commands will no longer work since the directky modify replication configuration in cn=config - which will be rejected.

The topology plugin provides a set of ipa subcommands to view, verify and manage the topology in the shared tree.

ipa topologysuffix-show # display all managed hosts and segments
ipa topologysuffix-verify # check connectivity, missing connections, redundant connections
ipa toplogysuffix-add # let plugin control other than default suffix
ipa topology-configure # if needed to set/change default configurations 
ipa topologysegment-add # create a new segment to be transformed to one or two replication agreement(s)
ipa topologysegment-del # delete segment and corresponding replication agreements
ipa topologysegment-mod # modify a segment and corresponding replication agreements

ipa-replica-install

If a new replica is added to the topology this is done with ipa-replica-install. Replication agreements are added and the new replica is initialized online. This could also be managed via the topology plugin. The only difference is that it cannot be managed by contacting one server only, since the topology information must be added to a server in the existing topology and to be replicated to the new replica replica credentials have to be set on the new server in the cn=config area, even if it is temporary.

Example Scenario (Test)

An example scenario showing all the features involves three servers: A, B, C

Step 1: Install IPA master on A

Step 2: Install replica from A on B

Domainlevel is not set, no segments are created

Step 3: Add domainlevel entry, domainlevel >pluginversion

On A a segment A->B will be created
On A replication agreement to B gets marked
Same procedure on B,
When created segments are replicated they will be merged into one
     bidirectional segment

Step 4: Install replica C from A

As the domainlevel is set on A and B, after initializing C from A it is set on C
Segments A->C and C->A are generated and merged

Step 5: Try to generate a replication agreement B->C

It will be rejected as B and C are managed by the plugin now

Step 6: Try to delete segment A<-->C

It will be rejected as the server C would become disconnected 

Step 7: Add segment B<-->C

The segment is added and replication agreements B->C and C->B are generated

Step 8: Try to delete segment A<-->C

deletion now succeeds, the replication agreements A->C and C->A are deleted

Step 9: Remove host A from the topology

the segments and agreements connecting A will be removed

Upgrades

Any impact on updates and upgrades?