Skip to content

DOCSP-29504-many-to-one-clusters #161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
d7b60e7
DOCSP-29504-many-to-one-clusters
Sep 18, 2023
9ee0798
DOCSP-29504-many-to-one-clusters
Sep 18, 2023
4eacfcb
DOCSP-29504-many-to-one-clusters
Sep 18, 2023
e75c5c1
DOCSP-29504-many-to-one-clusters
Sep 18, 2023
b9f7fb0
DOCSP-29504-many-to-one-clusters
Sep 19, 2023
0312da9
DOCSP-29504-many-to-one-clusters
Sep 19, 2023
7b0c9cf
DOCSP-29504-many-to-one-clusters
Sep 19, 2023
2bb9bf8
DOCSP-29504-many-to-one-clusters
Sep 19, 2023
d28ec9f
DOCSP-29504-many-to-one-clusters
Sep 19, 2023
a198bf3
DOCSP-29504-many-to-one-clusters
Sep 19, 2023
2955064
DOCSP-29504-many-to-one-clusters
Sep 19, 2023
42ca77b
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
cc3c685
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
6afb425
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
227df64
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
e8bf512
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
f8d52e4
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
7a5f8d5
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
cb10e8c
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
6c36a09
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
0999eb5
DOCSP-29504-many-to-one-clusters
Sep 20, 2023
1236b40
DOCSP-29504-many-to-one-clusters
Sep 21, 2023
2ee16ab
DOCSP-29504-many-to-one-clusters
Sep 21, 2023
c8d2c30
DOCSP-29504-many-to-one-clusters
Sep 21, 2023
ff93a48
DOCSP-29504-many-to-one-clusters
Sep 21, 2023
8378d75
DOCSP-29504-many-to-one-clusters
Sep 22, 2023
39587bc
DOCSP-29504-many-to-one-clusters
Sep 22, 2023
5c61971
DOCSP-29504-many-to-one-clusters
Sep 22, 2023
d7cc789
DOCSP-29504-many-to-one-clusters
Sep 22, 2023
76f9ecd
DOCSP-29504-many-to-one-clusters
Sep 27, 2023
c3d6e42
DOCSP-29504-many-to-one-clusters
Nov 8, 2023
ae403c6
DOCSP-29504-many-to-one-clusters
Nov 8, 2023
2dcf791
DOCSP-29504-many-to-one-clusters
Nov 8, 2023
6ab48fe
DOCSP-29504-many-to-one-clusters
Nov 29, 2023
883c02b
DOCSP-29504-many-to-one-clusters
Dec 6, 2023
5df8b18
DOCSP-29504-many-to-one-clusters
May 7, 2024
f5c583a
DOCSP-29504-many-to-one-clusters
May 7, 2024
34cd153
DOCSP-29504-many-to-one-clusters
May 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions source/includes/many-with-one-cluster.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Starting in ``mongosync-beta`` 1.8, you can sync multiple source clusters
simultaneously with a destination cluster. For example, you can
consolidate data from many small clusters into a central cluster.
4 changes: 4 additions & 0 deletions source/includes/migrationName-description.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Starting in ``mongosync-beta`` 1.8, you can set a migration name for a
sync operation. For example, you can set a migration name to identify
each sync operation from multiple source clusters into a destination
cluster.
18 changes: 18 additions & 0 deletions source/includes/opts/migrationName.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
.. reference/configuration.txt
.. reference/mongosync.txt

Starting in ``mongosync-beta`` 1.8, sets a migration name for a sync
operation. For example, you can set a migration name to identify each
sync operation from multiple source clusters into one destination
cluster.

The ``migrationName`` string can contain up to 44 alphanumeric
and underscore characters. ``migrationName`` is appended to the string
``"mongosync_internal_"`` to set the migration metadata database name.

If you set ``migrationName`` to
``"cluster_27000_to_cluster_35000_sync"``, the resulting ``mongosync``
metadata database name is
``"mongosync_internal_cluster_27000_to_cluster_35000_sync"``.

For a complete example, see :ref:`c2c-quickstart-many-with-one`.
2 changes: 1 addition & 1 deletion source/index.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Cluster-to-Cluster Sync
.. default-domain:: mongodb

{+c2c-product-name+} provides continuous data synchronization or a
one-time data migration between two MongoDB clusters. You can enable
one-time data migration between MongoDB clusters. You can enable
{+c2c-product-name+} with the :ref:`mongosync <c2c-mongosync>` utility.

``mongosync`` can continuously synchronize data between two clusters.
Expand Down
34 changes: 34 additions & 0 deletions source/quickstart.txt
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,8 @@ Initialization Notes
<c2c-api-start>` command. "cluster0" and "cluster1" are just labels,
either cluster can be ``cluster0`` or ``cluster1``.

.. _c2c-quickstart-synchronize:

Synchronize Data Between Clusters
---------------------------------

Expand Down Expand Up @@ -305,3 +307,35 @@ Synchronization Notes

- To estimate the size of ``oplog`` needed for initial synchronization,
see :ref:`c2c-oplog-sizing`.

.. _c2c-quickstart-many-with-one:

Sync Multiple Source Clusters Simultaneously with a Destination Cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. include:: /includes/many-with-one-cluster.rst

For limitations, see :ref:`c2c-limitations_multiple_sync`.

The following examples connect source clusters running on port ``27000``
and ``27001`` with a destination cluster running on port ``35000``. The
commands also set the optional :option:`--migrationName` string to
describe the operations.

.. code-block:: shell

./bin/mongosync \
--cluster0 "mongodb://localhost:27000" \
--cluster1 "mongodb://localhost:35000" \
--migrationName "cluster_27000_to_cluster_35000_sync"

./bin/mongosync \
--cluster0 "mongodb://localhost:27001" \
--cluster1 "mongodb://localhost:35000" \
--migrationName "cluster_27001_to_cluster_35000_sync"

.. include:: /includes/many-with-one-cluster.rst

To start the sync operation between the source clusters and the
destination cluster, see the earlier section
:ref:`c2c-quickstart-synchronize`.
9 changes: 9 additions & 0 deletions source/reference/configuration.txt
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,15 @@ Options
To set the ``logPath`` setting from the command line, see the
:option:`--logPath` option.

.. setting:: migrationName

*Type*: string

.. include:: /includes/opts/migrationName.rst

To set the ``migrationName`` setting from the command line, see the
:option:`--migrationName` option.

.. setting:: port

*Type*: integer
Expand Down
31 changes: 31 additions & 0 deletions source/reference/limitations.txt
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,37 @@ General Limitations
- ``mongosync`` must read from the source cluster using the
:readmode:`primary` read preference.

.. _c2c-limitations_multiple_sync:

Multiple Source Sync Limitations
--------------------------------

.. include:: /includes/many-with-one-cluster.rst

A namespace is a ``database_name.collection_name`` combination. You can
only sync namespaces that don't conflict.

For example, consider this scenario:

- Two source clusters S1 and S2.
- A destination cluster D.
- Databases named ``inventory`` and ``sales`` on both S1 and S2.
- Collections named ``products``, ``orderLines``, ``orderStatus``, and
``orders`` on both S1 and S2.
- You can sync both of these combinations:

- ``inventory.products`` and ``sales.orderStatus`` on S1 with D.
- ``inventory.orderLines`` and ``sales.orders`` on S2 with D.

- You cannot sync both of these combinations because they conflict:

- ``inventory.products`` and ``inventory.orderLines`` on S1 with D. If
D is initially empty, you can sync S1 with D. ``inventory.products``
and ``inventory.orderLines`` are copied from S1 to D.
- ``inventory.products`` and ``inventory.orderLines`` on S2 with D.
You cannot sync S2 with D because ``inventory.products`` and
``inventory.orderLines`` conflict with the namespaces already on D
from the scenario in the previous point.

MongoDB Community Edition
-------------------------
Expand Down
7 changes: 7 additions & 0 deletions source/reference/mongosync.txt
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,13 @@ Global Options
.. note::

.. include:: /includes/fact-log-rotation-usr1-signal

.. option:: --migrationName <name>

.. include:: /includes/opts/migrationName.rst

To set the ``--migrationName`` option from a configuration file,
see the :setting:`migrationName` setting.

.. option:: --port

Expand Down
16 changes: 15 additions & 1 deletion source/release-notes/1.8.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,22 @@ Release Notes for mongosync 1.8

**Upcoming**

New Features:
Beta Features:

Synchronize Data Between Multiple Source Clusters Simultaneously
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. include:: /includes/many-with-one-cluster.rst

Set a Migration Name
~~~~~~~~~~~~~~~~~~~~

.. include:: /includes/migrationName-description.rst

To set a migration name, see:

- :setting:`migrationName` configuration file setting
- :option:`--migrationName` command line option

Minimum Supported Version
-------------------------
Expand Down