From 1498105b38f209624f5bbdfb16d3aca61d5f4e40 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Fri, 12 Jul 2024 13:48:43 -0400 Subject: [PATCH 01/14] DOCSP-41105 --- source/reference.txt | 6 +- source/reference/about-mongosync.txt | 131 +++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 source/reference/about-mongosync.txt diff --git a/source/reference.txt b/source/reference.txt index c2f432e0f..c00a2d7df 100644 --- a/source/reference.txt +++ b/source/reference.txt @@ -2,8 +2,11 @@ Reference ========= +:ref:`about-mongosync` + The primary process used in ``mongosync``. + :ref:`c2c-mongosync` - The primary process used in {+c2c-product-name+}. + The options and behavior {+c2c-product-name+}. :ref:`c2c-config` Configure :program:`mongosync` instances at startup using a @@ -51,6 +54,7 @@ Reference .. toctree:: :titlesonly: + /reference/about-mongosync /reference/mongosync /reference/configuration /reference/api diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt new file mode 100644 index 000000000..e709d30ba --- /dev/null +++ b/source/reference/about-mongosync.txt @@ -0,0 +1,131 @@ +.. _about-mongosync: + +======================== +About ``mongosync`` +======================== + +The ``mongosync`` binary is the primary process used in {+c2c-product-name+}. +``mongosync`` migrates data from one cluster to another and can keep the +clusters in continuous sync. You can use ``mongosync`` to create dedicated +analytics, development, or testing clusters that mirror your production +environment. + +``mongosync`` enters different states depending on the requests it receives. +The current ``mongosync`` state determines which API operations you can run. +For more information on ``mongosync`` states, see +:ref:`mongosync States `. + +.. note:: + + Most state transitions happen as the result of an API call. The exceptions + are: + + - The transition from ``COMMITTING`` to ``COMMITTED`` + - The transition from ``REVERSING`` to ``RUNNING`` + +:ref:`IDLE ` State +---------------------------------- + +When ``mongosync`` connects to the source and destination clusters, it is in the +``IDLE`` state. + +:ref:`RUNNING ` State +------------------------------------- + +Call the :ref:`start ` endpoint +to begin syncing, which starts the ``RUNNING`` state. + +During the ``RUNNING`` state, the sync process is currently running. If you +start a new sync, ``mongosync`` syncs data to the destination cluster. When you +make subsequent writes to the source cluster, ``mongosync`` applies them to the +destination cluster. + +For every collection on the source, ``mongosync`` creates a corresponding +collection on the destination. + +``mongosync`` builds special indexes by making modifications on the destination +cluster. For example, ``mongosync`` relaxes uniqueness constraints when syncing +unique indexes. It also sets the expireAfterSeconds value to MAX_INT when +syncing TTL indexes to avoid premature expirations. ``mongosync`` changes these +index values back to their source cluster values during the ``COMMIT`` state. + +Collection Copy +~~~~~~~~~~~~~~~ + +Collection Copy is the initial full data transfer during the ``RUNNING`` state +and copies each collection from the source cluster to the destination cluster. +``mongosync`` performs collection copy by splitting each source collection into +partitions, gathering all partitions from all collections into one list, and +copying all partitions to their corresponding destination collections. + +Change Event Application +~~~~~~~~~~~~~~~~~~~~~~~~ + +Change Event Application (CEA) is an ongoing process that continuously updates +the destination cluster when there are changes to the source cluster. If there +are multiple ``mongosync`` instances, all the ``mongosync`` instances finish +Collection Copy before entering CEA. + +During CEA, ``mongosync`` receives change stream events from the source cluster, +converts these events into writes, and applies these writes on the destination +cluster. CEA ensures both CRUD (Create, Read, Update, Delete) and DDL +(Data Definition Language) operations are handled accurately. + +:ref:`PAUSED ` State +--------------------------------------- + +During the ``PAUSED`` state, the sync process is paused. To pause your sync +process, call the :ref:`pause ` endpoint. To use the ``pause`` +endpoint, ``mongosync`` must be in the ``RUNNING`` state. + +If you plan to pause synchronization for an extended period of time, increase +the size of the replica set oplog in the source cluster. To learn more, see +Frequently Asked Questions. + +To resume a paused synchronization session, call the +:ref:`resume ` endpoint. To use the ``resume`` endpoint, +``mongosync`` must be in the ``PAUSED`` state. + +:ref:`COMMITTING ` State +----------------------------------------------- + +During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the +source cluster and the destination cluster. To start the ``COMMITTING`` state, +call the :ref:`commit ` endpoint, which stops continuous sync +between clusters and restores collection characteristics that ``mongosync`` +temporarily alters during synchronization. To use the ``commit`` endpoint, +``mongosync`` must be in the ``RUNNING`` state. + +Once you call the ``commit`` endpoint, ``mongosync`` halts any writes on the +source cluster and retrieves the timestamp of the most recent operation in the +source cluster. + +If there are no errors, you see a success message and ``mongosync`` exits CEA by +processing events up until the commit timestamp. + +Once CEA is over, ``mongosync`` corrects indexes and resolves document conflicts. +If write blocking is enabled, ``mongosync`` turns write operations back on. + +:ref:`COMMITTED ` State +--------------------------------------------- + +``mongosync`` updates its state to ``COMMITTED`` once the ``COMMITTING`` state +is over. The ``COMMITTED`` state indicates the cutover process is finished, and +you can direct writes to the destination cluster. + +:ref:`REVERSING ` State +--------------------------------------------- + +During the ``REVERSING`` state, ``mongosync`` copies metadata from the +destination cluster to the source cluster. Then, ``mongosync`` swaps the source +and destination clusters and resumes applying change events. + +To start the ``REVERSING`` state, call the :ref:`reverse ` +endpoint, which syncs writes from the destination cluster to the source cluster, +including any writes that occur on the destination cluster after ``mongosync`` +turns write operations back on. To use the ``reverse`` endpoint, ``mongosync`` +must be in the ``COMMITTED`` state. + +If ``REVERSING`` is successful, ``mongosync`` enters the ``RUNNING`` state. The +synchronization continues in the reverse direction from the original sync job. + From f6c77b9d99aaa2ef398eece3090b40179a218c7b Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 10:07:11 -0400 Subject: [PATCH 02/14] DOCSP-41105 links --- source/reference/about-mongosync.txt | 25 +++++++++++++------------ source/reference/api/commit.txt | 6 +++++- source/reference/mongosync.txt | 2 ++ 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index e709d30ba..587c379a3 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -43,11 +43,10 @@ destination cluster. For every collection on the source, ``mongosync`` creates a corresponding collection on the destination. -``mongosync`` builds special indexes by making modifications on the destination -cluster. For example, ``mongosync`` relaxes uniqueness constraints when syncing -unique indexes. It also sets the expireAfterSeconds value to MAX_INT when -syncing TTL indexes to avoid premature expirations. ``mongosync`` changes these -index values back to their source cluster values during the ``COMMIT`` state. +``mongosync`` temporarily alters some collection characteristics during +the ``RUNNING`` state and changes these back to their source +cluster characteristics during the ``COMMIT`` state. For more information, see +:ref:`commit alters `. Collection Copy ~~~~~~~~~~~~~~~ @@ -68,8 +67,8 @@ Collection Copy before entering CEA. During CEA, ``mongosync`` receives change stream events from the source cluster, converts these events into writes, and applies these writes on the destination -cluster. CEA ensures both CRUD (Create, Read, Update, Delete) and DDL -(Data Definition Language) operations are handled accurately. +cluster. CEA ensures both :ref:`CRUD ` and +:ref:`DDL ` operations are handled accurately. :ref:`PAUSED ` State --------------------------------------- @@ -92,9 +91,10 @@ To resume a paused synchronization session, call the During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the source cluster and the destination cluster. To start the ``COMMITTING`` state, call the :ref:`commit ` endpoint, which stops continuous sync -between clusters and restores collection characteristics that ``mongosync`` -temporarily alters during synchronization. To use the ``commit`` endpoint, -``mongosync`` must be in the ``RUNNING`` state. +between clusters and +:ref:`restores collection characteristics ` +that ``mongosync`` temporarily alters during synchronization. To use the +``commit`` endpoint, ``mongosync`` must be in the ``RUNNING`` state. Once you call the ``commit`` endpoint, ``mongosync`` halts any writes on the source cluster and retrieves the timestamp of the most recent operation in the @@ -103,8 +103,9 @@ source cluster. If there are no errors, you see a success message and ``mongosync`` exits CEA by processing events up until the commit timestamp. -Once CEA is over, ``mongosync`` corrects indexes and resolves document conflicts. -If write blocking is enabled, ``mongosync`` turns write operations back on. +Once CEA is over, ``mongosync`` corrects indexes and resolves document +conflicts. If write blocking is enabled, ``mongosync`` turns write operations +back on. :ref:`COMMITTED ` State --------------------------------------------- diff --git a/source/reference/api/commit.txt b/source/reference/api/commit.txt index 5fbf63db6..35ba1090e 100644 --- a/source/reference/api/commit.txt +++ b/source/reference/api/commit.txt @@ -16,7 +16,11 @@ Description ----------- Finalizes the sync between the source cluster and the destination -cluster and stops continuous sync between clusters. ``commit`` also +cluster and stops continuous sync between clusters. + +.. _c2c-commit-alters: + +``commit`` also restores collection characteristics that ``mongosync`` temporarily alters during synchronization. diff --git a/source/reference/mongosync.txt b/source/reference/mongosync.txt index cd050b5f9..7a2cb6421 100644 --- a/source/reference/mongosync.txt +++ b/source/reference/mongosync.txt @@ -206,6 +206,8 @@ Even if documents move between source shards during replication, {+c2c-product-name+} maintains the :term:`eventual consistency` guarantee on the destination cluster. +.. _c2c-mongosync-ddl: + Data Definition Language (DDL) Operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 7a9fe4707463ea0eae13ce47e28b9eb4718b1396 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 10:55:22 -0400 Subject: [PATCH 03/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 85 ++++++++++++++-------------- 1 file changed, 42 insertions(+), 43 deletions(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index 587c379a3..98631e0c7 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -5,7 +5,7 @@ About ``mongosync`` ======================== The ``mongosync`` binary is the primary process used in {+c2c-product-name+}. -``mongosync`` migrates data from one cluster to another and can keep the +``mongosync`` migrates data from one cluster to another and keeps the clusters in continuous sync. You can use ``mongosync`` to create dedicated analytics, development, or testing clusters that mirror your production environment. @@ -30,15 +30,13 @@ When ``mongosync`` connects to the source and destination clusters, it is in the ``IDLE`` state. :ref:`RUNNING ` State -------------------------------------- +----------------------------------------- -Call the :ref:`start ` endpoint -to begin syncing, which starts the ``RUNNING`` state. +Call the :ref:`start ` endpoint to begin the ``RUNNING`` state. -During the ``RUNNING`` state, the sync process is currently running. If you -start a new sync, ``mongosync`` syncs data to the destination cluster. When you -make subsequent writes to the source cluster, ``mongosync`` applies them to the -destination cluster. +During the ``RUNNING`` state, if you start a new sync, ``mongosync`` syncs data +to the destination cluster. If you make subsequent writes to the source cluster, +``mongosync`` applies them to the destination cluster. For every collection on the source, ``mongosync`` creates a corresponding collection on the destination. @@ -46,55 +44,57 @@ collection on the destination. ``mongosync`` temporarily alters some collection characteristics during the ``RUNNING`` state and changes these back to their source cluster characteristics during the ``COMMIT`` state. For more information, see -:ref:`commit alters `. +:ref:`Commit Alters `. Collection Copy ~~~~~~~~~~~~~~~ -Collection Copy is the initial full data transfer during the ``RUNNING`` state -and copies each collection from the source cluster to the destination cluster. -``mongosync`` performs collection copy by splitting each source collection into -partitions, gathering all partitions from all collections into one list, and -copying all partitions to their corresponding destination collections. +Collection copy is the initial full data transfer that copies each collection +from the source cluster to the destination cluster. ``mongosync`` performs +collection copy by splitting each source collection into partitions, gathering +all partitions from all collections into one list, and copying all partitions +to their corresponding destination collections. Change Event Application ~~~~~~~~~~~~~~~~~~~~~~~~ -Change Event Application (CEA) is an ongoing process that continuously updates +Change event application (CEA) is an ongoing process that continuously updates the destination cluster when there are changes to the source cluster. If there are multiple ``mongosync`` instances, all the ``mongosync`` instances finish -Collection Copy before entering CEA. +collection copy before entering CEA. -During CEA, ``mongosync`` receives change stream events from the source cluster, -converts these events into writes, and applies these writes on the destination -cluster. CEA ensures both :ref:`CRUD ` and +During CEA, ``mongosync`` receives :ref:`change stream ` events +from the source cluster, converts these events into writes, and applies these +writes on the destination cluster. CEA ensures both :ref:`CRUD ` and :ref:`DDL ` operations are handled accurately. :ref:`PAUSED ` State --------------------------------------- -During the ``PAUSED`` state, the sync process is paused. To pause your sync -process, call the :ref:`pause ` endpoint. To use the ``pause`` -endpoint, ``mongosync`` must be in the ``RUNNING`` state. +Call the :ref:`pause ` endpoint to begin the ``PAUSED``state. +To use the ``pause`` endpoint, ``mongosync`` must be in the ``RUNNING`` state. + +During the ``PAUSED`` state, the sync process is paused. If you plan to pause synchronization for an extended period of time, increase -the size of the replica set oplog in the source cluster. To learn more, see -Frequently Asked Questions. +the size of the replica set :ref:`` in the source +cluster. -To resume a paused synchronization session, call the -:ref:`resume ` endpoint. To use the ``resume`` endpoint, -``mongosync`` must be in the ``PAUSED`` state. +To resume a paused synchronization session and return to the ``RUNNING`` state, +call the :ref:`resume ` endpoint. To use the ``resume`` +endpoint, ``mongosync`` must be in the ``PAUSED`` state. :ref:`COMMITTING ` State ----------------------------------------------- +Call the :ref:`commit ` endpoint to begin the ``COMMITTING`` +state. To use the ``commit`` endpoint, ``mongosync`` must be in the ``RUNNING`` +state. + During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the -source cluster and the destination cluster. To start the ``COMMITTING`` state, -call the :ref:`commit ` endpoint, which stops continuous sync -between clusters and -:ref:`restores collection characteristics ` -that ``mongosync`` temporarily alters during synchronization. To use the -``commit`` endpoint, ``mongosync`` must be in the ``RUNNING`` state. +source cluster and the destination cluster by stopping continuous sync +between them and restoring collection characteristics that ``mongosync`` +temporarily alters during synchronization. Once you call the ``commit`` endpoint, ``mongosync`` halts any writes on the source cluster and retrieves the timestamp of the most recent operation in the @@ -111,21 +111,20 @@ back on. --------------------------------------------- ``mongosync`` updates its state to ``COMMITTED`` once the ``COMMITTING`` state -is over. The ``COMMITTED`` state indicates the cutover process is finished, and -you can direct writes to the destination cluster. +is over. The ``COMMITTED`` state indicates the +:ref:`cutover process ` is finished, and you can direct +writes to the destination cluster. :ref:`REVERSING ` State --------------------------------------------- -During the ``REVERSING`` state, ``mongosync`` copies metadata from the -destination cluster to the source cluster. Then, ``mongosync`` swaps the source -and destination clusters and resumes applying change events. +Call the :ref:`reverse ` endpoint to start the ``REVERSING`` +state. To use the ``reverse`` endpoint, ``mongosync`` must be in the +``COMMITTED`` state. -To start the ``REVERSING`` state, call the :ref:`reverse ` -endpoint, which syncs writes from the destination cluster to the source cluster, -including any writes that occur on the destination cluster after ``mongosync`` -turns write operations back on. To use the ``reverse`` endpoint, ``mongosync`` -must be in the ``COMMITTED`` state. +During the ``REVERSING`` state, ``mongosync`` copies data from the +destination cluster to the source cluster. Then, ``mongosync`` swaps the source +and destination clusters and resumes applying change events. If ``REVERSING`` is successful, ``mongosync`` enters the ``RUNNING`` state. The synchronization continues in the reverse direction from the original sync job. From 50e8297c6beab179064a26fc991ec5ef6e4caba3 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 10:57:22 -0400 Subject: [PATCH 04/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index 98631e0c7..9f99f1a0f 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -71,7 +71,7 @@ writes on the destination cluster. CEA ensures both :ref:`CRUD ` and :ref:`PAUSED ` State --------------------------------------- -Call the :ref:`pause ` endpoint to begin the ``PAUSED``state. +Call the :ref:`pause ` endpoint to begin the ``PAUSED`` state. To use the ``pause`` endpoint, ``mongosync`` must be in the ``RUNNING`` state. During the ``PAUSED`` state, the sync process is paused. From d146f2114942b3cc9598480606d99689a0b0ac13 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 11:02:15 -0400 Subject: [PATCH 05/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index 9f99f1a0f..d2955fddb 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -77,7 +77,7 @@ To use the ``pause`` endpoint, ``mongosync`` must be in the ``RUNNING`` state. During the ``PAUSED`` state, the sync process is paused. If you plan to pause synchronization for an extended period of time, increase -the size of the replica set :ref:`` in the source +the size of the replica set :ref:`oplog ` in the source cluster. To resume a paused synchronization session and return to the ``RUNNING`` state, From dd4d0d611c64e10daa4b7e16ed13ff7f4b458be1 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 12:00:03 -0400 Subject: [PATCH 06/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 80 ++++++++++++++-------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index d2955fddb..accfcfaf3 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -23,20 +23,28 @@ For more information on ``mongosync`` states, see - The transition from ``COMMITTING`` to ``COMMITTED`` - The transition from ``REVERSING`` to ``RUNNING`` -:ref:`IDLE ` State ----------------------------------- +How ``mongosync`` Works +------------------------ + +Initial Connection +~~~~~~~~~~~~~~~~~~~ + +``mongosync`` must create an initial connection to the source and destination +clusters before it can start to sync data. For more information, see +:ref:`c2c-connecting`. When ``mongosync`` connects to the source and destination clusters, it is in the -``IDLE`` state. +:ref:`IDLE ` state. -:ref:`RUNNING ` State ------------------------------------------ +During Sync +~~~~~~~~~~~~ -Call the :ref:`start ` endpoint to begin the ``RUNNING`` state. +The :ref:`start ` endpoint begins the +:ref:`RUNNING ` state. -During the ``RUNNING`` state, if you start a new sync, ``mongosync`` syncs data -to the destination cluster. If you make subsequent writes to the source cluster, -``mongosync`` applies them to the destination cluster. +If you start a new sync, ``mongosync`` syncs data to the destination cluster. +If you make subsequent writes to the source cluster, ``mongosync`` applies them +to the destination cluster. For every collection on the source, ``mongosync`` creates a corresponding collection on the destination. @@ -47,7 +55,7 @@ cluster characteristics during the ``COMMIT`` state. For more information, see :ref:`Commit Alters `. Collection Copy -~~~~~~~~~~~~~~~ +''''''''''''''' Collection copy is the initial full data transfer that copies each collection from the source cluster to the destination cluster. ``mongosync`` performs @@ -56,7 +64,7 @@ all partitions from all collections into one list, and copying all partitions to their corresponding destination collections. Change Event Application -~~~~~~~~~~~~~~~~~~~~~~~~ +''''''''''''''''''''''''' Change event application (CEA) is an ongoing process that continuously updates the destination cluster when there are changes to the source cluster. If there @@ -68,28 +76,25 @@ from the source cluster, converts these events into writes, and applies these writes on the destination cluster. CEA ensures both :ref:`CRUD ` and :ref:`DDL ` operations are handled accurately. -:ref:`PAUSED ` State ---------------------------------------- +Pausing Sync +~~~~~~~~~~~~~ -Call the :ref:`pause ` endpoint to begin the ``PAUSED`` state. -To use the ``pause`` endpoint, ``mongosync`` must be in the ``RUNNING`` state. +The :ref:`pause ` endpoint begins the +:ref:`PAUSED ` state. To use the ``pause`` endpoint, +``mongosync`` must be in the ``RUNNING`` state. During the ``PAUSED`` state, the sync process is paused. -If you plan to pause synchronization for an extended period of time, increase -the size of the replica set :ref:`oplog ` in the source -cluster. - -To resume a paused synchronization session and return to the ``RUNNING`` state, -call the :ref:`resume ` endpoint. To use the ``resume`` -endpoint, ``mongosync`` must be in the ``PAUSED`` state. +The :ref:`resume ` endpoint resumes a paused synchronization +session and returns ``mongosync`` to the ``RUNNING`` state. To use the +``resume`` endpoint, ``mongosync`` must be in the ``PAUSED`` state. -:ref:`COMMITTING ` State ------------------------------------------------ +Committing Sync +~~~~~~~~~~~~~~~~ -Call the :ref:`commit ` endpoint to begin the ``COMMITTING`` -state. To use the ``commit`` endpoint, ``mongosync`` must be in the ``RUNNING`` -state. +The :ref:`commit ` endpoint begins the +:ref:`COMMITTING `state. To use the ``commit`` endpoint, +``mongosync`` must be in the ``RUNNING`` state. During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the source cluster and the destination cluster by stopping continuous sync @@ -101,26 +106,21 @@ source cluster and retrieves the timestamp of the most recent operation in the source cluster. If there are no errors, you see a success message and ``mongosync`` exits CEA by -processing events up until the commit timestamp. - -Once CEA is over, ``mongosync`` corrects indexes and resolves document -conflicts. If write blocking is enabled, ``mongosync`` turns write operations -back on. - -:ref:`COMMITTED ` State ---------------------------------------------- +processing events up until the commit timestamp. Then, ``mongosync`` corrects +indexes and resolves document conflicts. If write blocking is enabled, +``mongosync`` turns write operations back on. ``mongosync`` updates its state to ``COMMITTED`` once the ``COMMITTING`` state is over. The ``COMMITTED`` state indicates the :ref:`cutover process ` is finished, and you can direct writes to the destination cluster. -:ref:`REVERSING ` State ---------------------------------------------- +Reversing Sync +~~~~~~~~~~~~~~~ -Call the :ref:`reverse ` endpoint to start the ``REVERSING`` -state. To use the ``reverse`` endpoint, ``mongosync`` must be in the -``COMMITTED`` state. +The :ref:`reverse ` endpoint starts the +:ref:`REVERSING ` state. To use the ``reverse`` endpoint, +``mongosync`` must be in the ``COMMITTED`` state. During the ``REVERSING`` state, ``mongosync`` copies data from the destination cluster to the source cluster. Then, ``mongosync`` swaps the source From 3239f2b1cd1e7cc338a76c2c4d57f5c0a1fabbf5 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 12:23:16 -0400 Subject: [PATCH 07/14] DOCSP-41105 --- source/index.txt | 2 ++ source/reference/about-mongosync.txt | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/source/index.txt b/source/index.txt index 90fc25181..b126c07e0 100644 --- a/source/index.txt +++ b/source/index.txt @@ -21,6 +21,8 @@ and data residency compliance. In addition to continuous data synchronization, ``mongosync`` can also facilitate a one time data migration between clusters. +For a overview of how ``mongosync`` works, see :ref:`about-mongosync`. + To get started with ``mongosync``, refer to the :ref:`Quick Start Guide `. For more detailed information, refer to the :ref:`c2c-install` or :ref:`c2c-connecting` page that best fits your diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index accfcfaf3..2638d45c2 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -93,7 +93,7 @@ Committing Sync ~~~~~~~~~~~~~~~~ The :ref:`commit ` endpoint begins the -:ref:`COMMITTING `state. To use the ``commit`` endpoint, +:ref:`COMMITTING ` state. To use the ``commit`` endpoint, ``mongosync`` must be in the ``RUNNING`` state. During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the From 31ea24d30badddb14785eab363093f7459d02636 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 12:28:32 -0400 Subject: [PATCH 08/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index 2638d45c2..9b7292fd5 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -110,8 +110,8 @@ processing events up until the commit timestamp. Then, ``mongosync`` corrects indexes and resolves document conflicts. If write blocking is enabled, ``mongosync`` turns write operations back on. -``mongosync`` updates its state to ``COMMITTED`` once the ``COMMITTING`` state -is over. The ``COMMITTED`` state indicates the +``mongosync`` updates its state to :ref:`COMMITTED ` once +the ``COMMITTING`` state is over. The ``COMMITTED`` state indicates the :ref:`cutover process ` is finished, and you can direct writes to the destination cluster. From e1875d1497b018be651b5840d668020a6fcabd15 Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 14:33:30 -0400 Subject: [PATCH 09/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 77 ++++++++++++--------------- source/reference/mongosync-states.txt | 8 +++ 2 files changed, 42 insertions(+), 43 deletions(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index 9b7292fd5..8055116ab 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -1,8 +1,8 @@ .. _about-mongosync: -======================== +==================== About ``mongosync`` -======================== +==================== The ``mongosync`` binary is the primary process used in {+c2c-product-name+}. ``mongosync`` migrates data from one cluster to another and keeps the @@ -10,41 +10,34 @@ clusters in continuous sync. You can use ``mongosync`` to create dedicated analytics, development, or testing clusters that mirror your production environment. -``mongosync`` enters different states depending on the requests it receives. -The current ``mongosync`` state determines which API operations you can run. +``mongosync`` keeps track of its current actions through states. ``mongosync`` +enters different states depending on the requests it receives. The current +``mongosync`` state determines which API operations you can run. For more information on ``mongosync`` states, see :ref:`mongosync States `. -.. note:: - - Most state transitions happen as the result of an API call. The exceptions - are: - - - The transition from ``COMMITTING`` to ``COMMITTED`` - - The transition from ``REVERSING`` to ``RUNNING`` - How ``mongosync`` Works ------------------------- +----------------------- Initial Connection -~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~ ``mongosync`` must create an initial connection to the source and destination clusters before it can start to sync data. For more information, see :ref:`c2c-connecting`. -When ``mongosync`` connects to the source and destination clusters, it is in the -:ref:`IDLE ` state. +When ``mongosync`` connects to the source and destination clusters, the +synchronization process is in the :ref:`IDLE ` state. During Sync -~~~~~~~~~~~~ +~~~~~~~~~~~ The :ref:`start ` endpoint begins the :ref:`RUNNING ` state. -If you start a new sync, ``mongosync`` syncs data to the destination cluster. -If you make subsequent writes to the source cluster, ``mongosync`` applies them -to the destination cluster. +During the sync, ``mongosync`` syncs data from the source cluster to the +destination cluster. If you make subsequent writes to the source cluster, +``mongosync`` applies the operations to the destination cluster. For every collection on the source, ``mongosync`` creates a corresponding collection on the destination. @@ -59,53 +52,50 @@ Collection Copy Collection copy is the initial full data transfer that copies each collection from the source cluster to the destination cluster. ``mongosync`` performs -collection copy by splitting each source collection into partitions, gathering -all partitions from all collections into one list, and copying all partitions -to their corresponding destination collections. +collection copy by splitting each source collection into partitions and copying +all the partitions to their corresponding destination collections. Change Event Application -''''''''''''''''''''''''' +'''''''''''''''''''''''' -Change event application (CEA) is an ongoing process that continuously updates -the destination cluster when there are changes to the source cluster. If there +The change event application (CEA) process continuously updates +the destination cluster when you make changes to the source cluster. If there are multiple ``mongosync`` instances, all the ``mongosync`` instances finish -collection copy before entering CEA. +collection copy before starting the CEA process. During CEA, ``mongosync`` receives :ref:`change stream ` events from the source cluster, converts these events into writes, and applies these -writes on the destination cluster. CEA ensures both :ref:`CRUD ` and -:ref:`DDL ` operations are handled accurately. +writes on the destination cluster. CEA ensures that :ref:`CRUD ` and +:ref:`DDL ` events from the source cluster are accurately +applied to the destination cluster. Pausing Sync -~~~~~~~~~~~~~ +~~~~~~~~~~~~ The :ref:`pause ` endpoint begins the -:ref:`PAUSED ` state. To use the ``pause`` endpoint, -``mongosync`` must be in the ``RUNNING`` state. +:ref:`PAUSED ` state. During the ``PAUSED`` state, the sync process is paused. The :ref:`resume ` endpoint resumes a paused synchronization -session and returns ``mongosync`` to the ``RUNNING`` state. To use the -``resume`` endpoint, ``mongosync`` must be in the ``PAUSED`` state. +session and returns ``mongosync`` to the ``RUNNING`` state. -Committing Sync -~~~~~~~~~~~~~~~~ +Finalizing Sync +~~~~~~~~~~~~~~~ The :ref:`commit ` endpoint begins the -:ref:`COMMITTING ` state. To use the ``commit`` endpoint, -``mongosync`` must be in the ``RUNNING`` state. +:ref:`COMMITTING ` state. During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the source cluster and the destination cluster by stopping continuous sync -between them and restoring collection characteristics that ``mongosync`` +between them. It also restores collection characteristics that ``mongosync`` temporarily alters during synchronization. Once you call the ``commit`` endpoint, ``mongosync`` halts any writes on the source cluster and retrieves the timestamp of the most recent operation in the source cluster. -If there are no errors, you see a success message and ``mongosync`` exits CEA by +If there are no errors, ``mongosync`` sends a success message and exits CEA by processing events up until the commit timestamp. Then, ``mongosync`` corrects indexes and resolves document conflicts. If write blocking is enabled, ``mongosync`` turns write operations back on. @@ -116,7 +106,7 @@ the ``COMMITTING`` state is over. The ``COMMITTED`` state indicates the writes to the destination cluster. Reversing Sync -~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~ The :ref:`reverse ` endpoint starts the :ref:`REVERSING ` state. To use the ``reverse`` endpoint, @@ -126,6 +116,7 @@ During the ``REVERSING`` state, ``mongosync`` copies data from the destination cluster to the source cluster. Then, ``mongosync`` swaps the source and destination clusters and resumes applying change events. -If ``REVERSING`` is successful, ``mongosync`` enters the ``RUNNING`` state. The -synchronization continues in the reverse direction from the original sync job. +If the reverse sync is successful, ``mongosync`` enters the ``RUNNING`` state. +The synchronization continues in the reverse direction from the original +sync job. diff --git a/source/reference/mongosync-states.txt b/source/reference/mongosync-states.txt index 0b69d69f9..8aeb9a4b1 100644 --- a/source/reference/mongosync-states.txt +++ b/source/reference/mongosync-states.txt @@ -17,6 +17,14 @@ receives. ``mongosync`` can only be in a single state at a given time. The current ``mongosync`` states determines which API operations you can run. +.. note:: + + Most state transitions happen as the result of an API call. The exceptions + are: + + - The transition from ``COMMITTING`` to ``COMMITTED`` + - The transition from ``REVERSING`` to ``RUNNING`` + This page describes ``mongosync`` states. View the Current State From 5ae28d3aba293cc4ed7d4164ba98682e3c171f8b Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 14:35:54 -0400 Subject: [PATCH 10/14] DOCSP-41105 --- source/index.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/index.txt b/source/index.txt index b126c07e0..2bc69c961 100644 --- a/source/index.txt +++ b/source/index.txt @@ -21,7 +21,7 @@ and data residency compliance. In addition to continuous data synchronization, ``mongosync`` can also facilitate a one time data migration between clusters. -For a overview of how ``mongosync`` works, see :ref:`about-mongosync`. +For an overview of how ``mongosync`` works, see :ref:`about-mongosync`. To get started with ``mongosync``, refer to the :ref:`Quick Start Guide `. For more detailed information, refer to the From 282d494eeba12d4cdc1b665f2c18744f9c087c6d Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Mon, 15 Jul 2024 17:11:05 -0400 Subject: [PATCH 11/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 77 +++++++++++++++------------- 1 file changed, 40 insertions(+), 37 deletions(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index 8055116ab..3587bc259 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -5,16 +5,15 @@ About ``mongosync`` ==================== The ``mongosync`` binary is the primary process used in {+c2c-product-name+}. -``mongosync`` migrates data from one cluster to another and keeps the -clusters in continuous sync. You can use ``mongosync`` to create dedicated -analytics, development, or testing clusters that mirror your production -environment. +``mongosync`` migrates data from a source cluster to a destination cluster and +keeps the clusters in continuous sync. You can use ``mongosync`` to create +dedicated analytics, development, or testing clusters that mirror your +production environment. -``mongosync`` keeps track of its current actions through states. ``mongosync`` -enters different states depending on the requests it receives. The current -``mongosync`` state determines which API operations you can run. -For more information on ``mongosync`` states, see -:ref:`mongosync States `. +``mongosync`` keeps track of its current actions through +:ref:`states `. ``mongosync`` enters different states depending on +the requests it receives. The current ``mongosync`` state determines which API +operations you can run. How ``mongosync`` Works ----------------------- @@ -27,17 +26,17 @@ clusters before it can start to sync data. For more information, see :ref:`c2c-connecting`. When ``mongosync`` connects to the source and destination clusters, the -synchronization process is in the :ref:`IDLE ` state. +synchronization process enters the :ref:`IDLE ` state. During Sync ~~~~~~~~~~~ -The :ref:`start ` endpoint begins the -:ref:`RUNNING ` state. - -During the sync, ``mongosync`` syncs data from the source cluster to the -destination cluster. If you make subsequent writes to the source cluster, -``mongosync`` applies the operations to the destination cluster. +During the :ref:`RUNNING ` state, ``mongosync`` syncs data +from the source cluster to the destination cluster through the +:ref:`collection copy ` process. If you make subsequent +writes to the source cluster, ``mongosync`` applies the operations to the +destination cluster through the :ref:`change event application ` +process. For every collection on the source, ``mongosync`` creates a corresponding collection on the destination. @@ -47,50 +46,54 @@ the ``RUNNING`` state and changes these back to their source cluster characteristics during the ``COMMIT`` state. For more information, see :ref:`Commit Alters `. +The :ref:`start ` endpoint begins the ``RUNNING`` state. + +.. _c2c-collection-copy: + Collection Copy ''''''''''''''' -Collection copy is the initial full data transfer that copies each collection -from the source cluster to the destination cluster. ``mongosync`` performs -collection copy by splitting each source collection into partitions and copying -all the partitions to their corresponding destination collections. +The collection copy process is the initial full data transfer that copies each +collection from the source cluster to the destination cluster. ``mongosync`` +performs collection copy by splitting each source collection into partitions +and copying all the partitions to their corresponding destination collections. + +.. _c2c-cea: Change Event Application '''''''''''''''''''''''' The change event application (CEA) process continuously updates the destination cluster when you make changes to the source cluster. If there -are multiple ``mongosync`` instances, all the ``mongosync`` instances finish +are multiple ``mongosync`` instances, all instances finish collection copy before starting the CEA process. During CEA, ``mongosync`` receives :ref:`change stream ` events from the source cluster, converts these events into writes, and applies these -writes on the destination cluster. CEA ensures that :ref:`CRUD ` and -:ref:`DDL ` events from the source cluster are accurately -applied to the destination cluster. +writes on the destination cluster. Pausing Sync ~~~~~~~~~~~~ +During the ``PAUSED`` state, the sync process is paused. + The :ref:`pause ` endpoint begins the :ref:`PAUSED ` state. -During the ``PAUSED`` state, the sync process is paused. - The :ref:`resume ` endpoint resumes a paused synchronization session and returns ``mongosync`` to the ``RUNNING`` state. Finalizing Sync ~~~~~~~~~~~~~~~ -The :ref:`commit ` endpoint begins the -:ref:`COMMITTING ` state. - During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the source cluster and the destination cluster by stopping continuous sync between them. It also restores collection characteristics that ``mongosync`` temporarily alters during synchronization. +The :ref:`commit ` endpoint begins the +:ref:`COMMITTING ` state. + Once you call the ``commit`` endpoint, ``mongosync`` halts any writes on the source cluster and retrieves the timestamp of the most recent operation in the source cluster. @@ -100,22 +103,22 @@ processing events up until the commit timestamp. Then, ``mongosync`` corrects indexes and resolves document conflicts. If write blocking is enabled, ``mongosync`` turns write operations back on. -``mongosync`` updates its state to :ref:`COMMITTED ` once -the ``COMMITTING`` state is over. The ``COMMITTED`` state indicates the -:ref:`cutover process ` is finished, and you can direct -writes to the destination cluster. +When the above process is completed, ``mongosync`` enters the +:ref:`COMMITTED ` state. The ``COMMITTED`` state indicates +that the :ref:`cutover process ` is finished, and you can +direct writes to the destination cluster. Reversing Sync ~~~~~~~~~~~~~~ -The :ref:`reverse ` endpoint starts the -:ref:`REVERSING ` state. To use the ``reverse`` endpoint, -``mongosync`` must be in the ``COMMITTED`` state. - During the ``REVERSING`` state, ``mongosync`` copies data from the destination cluster to the source cluster. Then, ``mongosync`` swaps the source and destination clusters and resumes applying change events. +The :ref:`reverse ` endpoint starts the +:ref:`REVERSING ` state. To use the ``reverse`` endpoint, +``mongosync`` must be in the ``COMMITTED`` state. + If the reverse sync is successful, ``mongosync`` enters the ``RUNNING`` state. The synchronization continues in the reverse direction from the original sync job. From 89e6c2f9bb128a22132677fe597e3f5861bfbab9 Mon Sep 17 00:00:00 2001 From: sonderdonk-mdb Date: Wed, 10 Jul 2024 09:52:43 -0400 Subject: [PATCH 12/14] DOCS-41102 added code snippets cutover page (#341) * DOCS-41102 * DOCSP-41102 added to quickstart * DOCSP-41102 --- source/quickstart.txt | 9 ++ source/reference/cutover-process.txt | 124 +++++++++++++++++++++++++-- 2 files changed, 126 insertions(+), 7 deletions(-) diff --git a/source/quickstart.txt b/source/quickstart.txt index 2483c123f..7d3662fc4 100644 --- a/source/quickstart.txt +++ b/source/quickstart.txt @@ -247,6 +247,15 @@ documentation provides details on using the following endpoints: * - :ref:`c2c-api-reverse` - Reverses the direction of a committed sync operation. +Finalize Cutover Process +------------------------ + +You can finalize a migration and transfer your application +workload from the source to the destination cluster using the +``mongosync`` cutover process. + +For more information, see :ref:`c2c-cutover-process`. + One-Time Sync ------------- diff --git a/source/reference/cutover-process.txt b/source/reference/cutover-process.txt index e4a39575f..c84ec547a 100644 --- a/source/reference/cutover-process.txt +++ b/source/reference/cutover-process.txt @@ -36,12 +36,29 @@ Steps Ensure that the ``mongosync`` process status indicates the following values: + - ``canCommit`` is ``true``. + - ``lagTimeSeconds`` is small (near ``0``). If ``lagTimeSeconds`` isn't close to ``0`` when the cutover starts, cutover might take a long time. - - ``canCommit`` is ``true``. + The following example returns the status of the synchronization process. + + Request + ~~~~~~~ + + .. literalinclude:: /includes/api/requests/progress.sh + :language: shell + + Response + ~~~~~~~~ + + .. literalinclude:: /includes/api/responses/progress.json + :language: json + :copyable: false + :emphasize-lines: 5, 8 + .. step:: Stop any write operations to the synced collections on the source. @@ -50,8 +67,16 @@ Steps the entire source cluster during the commit (step 4) for you. - If you didn't start ``mongosync`` with ``enableUserWriteBlocking``, ensure that writes are disabled. - For example, run the ``setUserWriteBlockMode`` command on the - source cluster. + For example, run the :dbcommand:`setUserWriteBlockMode` command on the + source cluster: + + .. code-block:: javascript + + db.adminCommand( { + setUserWriteBlockMode: 1, + global: true + } ) + - If ``mongosync`` uses filtered sync, it's not necessary to disable writes to the entire source cluster. But you must ensure that write operations are stopped for the collections included @@ -63,12 +88,24 @@ Steps migration, you must issue a commit request for each ``mongosync`` instance. + Request + ~~~~~~~~ + + .. literalinclude:: /includes/api/requests/commit.sh + :language: shell + + Response + ~~~~~~~~ + + .. literalinclude:: /includes/api/responses/success.json + :language: json + :copyable: false + .. note:: - Ensure that the ``mongosync`` process response for the - ``commit`` request indicates that the ``mongosync`` state is - ``COMMITTING`` or ``COMMITTED``, which is the correct - ``mongosync`` state when you send a ``commit`` request. + After you submit a ``commit`` request, call the ``progress`` endpoint + to ensure that the ``mongosync`` state is ``COMMITTING`` or + ``COMMITTED``. .. step:: Wait until you can perform writes on the destination cluster. @@ -76,6 +113,42 @@ Steps ``true``. If ``canWrite`` is ``false``, wait until ``progress`` shows ``canWrite`` is ``true``. + Request + ~~~~~~~ + + .. literalinclude:: /includes/api/requests/progress.sh + :language: shell + + Response + ~~~~~~~~ + + .. code-block:: json + :emphasize-lines: 6 + :copyable: false + + { + "progress": + { + "state":"COMMITTED", + "canCommit":true, + "canWrite":true, + "info":"change event application", + "lagTimeSeconds":0, + "collectionCopy": + { + "estimatedTotalBytes":694, + "estimatedCopiedBytes":694 + }, + "directionMapping": + { + "Source":"cluster0: localhost:27017", + "Destination":"cluster1: localhost:27018" + } + }, + "success": true + } + + .. step:: Verify data transfer. Verify the successful sync of data from the source to the @@ -102,6 +175,43 @@ Steps ``mongosync`` state is ``COMMITTED``, the cutover process is complete. + Request + ~~~~~~~ + + .. literalinclude:: /includes/api/requests/progress.sh + :language: shell + + Response + ~~~~~~~~ + + .. code-block:: json + :emphasize-lines: 4 + :copyable: false + + { + "progress": + { + "state":"COMMITTED", + "canCommit":true, + "canWrite":false, + "info":"change event application", + "lagTimeSeconds":0, + "collectionCopy": + { + "estimatedTotalBytes":694, + "estimatedCopiedBytes":694 + }, + "directionMapping": + { + "Source":"cluster0: localhost:27017", + "Destination":"cluster1: localhost:27018" + } + }, + "success": true + } + + + Learn More ---------- From d8261932fb6652a66fd94eba7ed764df85b0f8f4 Mon Sep 17 00:00:00 2001 From: Alison Huh <112565127+ajhuh-mdb@users.noreply.github.com> Date: Mon, 15 Jul 2024 11:23:34 -0400 Subject: [PATCH 13/14] DOCSP-40138-Mongosync-Delay-After-Resume-Restart (#343) * DOCSP-40138-Mongosync-Delay-After-Resume-Restart * add 1.8 to toc * * * * * JA feedback * add retry bit --- source/faq.txt | 2 ++ source/includes/fact-restart-resume-delay.rst | 6 ++++++ source/reference/api/progress.txt | 2 ++ source/reference/api/resume.txt | 7 ++++--- source/reference/mongosync-states.txt | 2 ++ source/release-notes.txt | 1 + source/release-notes/1.7.txt | 4 ++++ 7 files changed, 21 insertions(+), 3 deletions(-) create mode 100644 source/includes/fact-restart-resume-delay.rst diff --git a/source/faq.txt b/source/faq.txt index 3becd22d0..dd6c15db7 100644 --- a/source/faq.txt +++ b/source/faq.txt @@ -102,6 +102,8 @@ available again, restart the ``mongosync`` process with the same parameters. ``mongosync`` resumes the operation from where it stopped when ``mongosync`` became unavailable. +.. include:: /includes/fact-restart-resume-delay.rst + Can the source or destination be a replica set with arbiters? ------------------------------------------------------------- diff --git a/source/includes/fact-restart-resume-delay.rst b/source/includes/fact-restart-resume-delay.rst new file mode 100644 index 000000000..ddf0a0ff1 --- /dev/null +++ b/source/includes/fact-restart-resume-delay.rst @@ -0,0 +1,6 @@ +.. note:: + + Starting in ``mongosync`` 1.7.3, ``mongosync`` can take at least two minutes + to respond when you resume or restart a sync operation. During this time, + any calls to the :ref:`c2c-api-progress` endpoint might fail. If a + ``progress`` call fails, it is safe to retry. diff --git a/source/reference/api/progress.txt b/source/reference/api/progress.txt index 8e53cdc7d..00406f5db 100644 --- a/source/reference/api/progress.txt +++ b/source/reference/api/progress.txt @@ -64,6 +64,8 @@ Behavior - The endpoint does not auto-refresh. To get updated status, call the ``progress`` endpoint again. + + .. include:: /includes/fact-restart-resume-delay.rst Endpoint Protection ~~~~~~~~~~~~~~~~~~~ diff --git a/source/reference/api/resume.txt b/source/reference/api/resume.txt index 334c0b0fd..32e63177a 100644 --- a/source/reference/api/resume.txt +++ b/source/reference/api/resume.txt @@ -71,12 +71,13 @@ Response Behavior -------- +- When you send a ``resume`` request, ``mongosync`` might take at least two + minutes before resuming the migration. To see if the migration resumed, you + can call the :ref:`c2c-api-progress` endpoint. + - If the ``resume`` request is successful, ``mongosync`` enters the ``RUNNING`` state. -- When you send a ``resume`` request, there may be a delay of up to a - few minutes before ``mongosync`` resumes the migration. - Endpoint Protection ~~~~~~~~~~~~~~~~~~~ diff --git a/source/reference/mongosync-states.txt b/source/reference/mongosync-states.txt index 8aeb9a4b1..bff611711 100644 --- a/source/reference/mongosync-states.txt +++ b/source/reference/mongosync-states.txt @@ -34,6 +34,8 @@ To view the current state of ``mongosync``, use the :ref:`/progress `. endpoint. The :ref:`/progress ` endpoint returns the state in the ``state`` field. +.. include:: /includes/fact-restart-resume-delay.rst + .. _c2c-states-descriptions: State Descriptions diff --git a/source/release-notes.txt b/source/release-notes.txt index 5047706f5..4ffade3d7 100644 --- a/source/release-notes.txt +++ b/source/release-notes.txt @@ -38,6 +38,7 @@ Previous Rapid Releases .. toctree:: :titlesonly: + /release-notes/1.8 /release-notes/1.7 /release-notes/1.6 /release-notes/1.5 diff --git a/source/release-notes/1.7.txt b/source/release-notes/1.7.txt index 34113e303..83c88ca65 100644 --- a/source/release-notes/1.7.txt +++ b/source/release-notes/1.7.txt @@ -70,6 +70,10 @@ Issues Fixed: events when mongosync is interrupted and resumed. This is an unlikely edge case. +Other Changes: + +- Adds a two minute delay when restarting and resuming sync operations. + Limitations: - You must stop the balancer on both the source and destination sharded From 9c5442533898bf28de4d08ee99ba1aa1b80dbd4f Mon Sep 17 00:00:00 2001 From: Sarah Onderdonk Date: Tue, 16 Jul 2024 16:31:20 -0400 Subject: [PATCH 14/14] DOCSP-41105 --- source/reference/about-mongosync.txt | 59 ++++++++++++++++------------ 1 file changed, 33 insertions(+), 26 deletions(-) diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt index 3587bc259..840ded992 100644 --- a/source/reference/about-mongosync.txt +++ b/source/reference/about-mongosync.txt @@ -4,6 +4,14 @@ About ``mongosync`` ==================== +.. default-domain:: mongodb + +.. contents:: On this page + :local: + :backlinks: none + :depth: 1 + :class: singlecol + The ``mongosync`` binary is the primary process used in {+c2c-product-name+}. ``mongosync`` migrates data from a source cluster to a destination cluster and keeps the clusters in continuous sync. You can use ``mongosync`` to create @@ -18,11 +26,11 @@ operations you can run. How ``mongosync`` Works ----------------------- -Initial Connection -~~~~~~~~~~~~~~~~~~ +Connect Source and Destination Clusters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -``mongosync`` must create an initial connection to the source and destination -clusters before it can start to sync data. For more information, see +You must create an initial connection to the source and destination +clusters before ``mongosync`` can start to sync data. For more information, see :ref:`c2c-connecting`. When ``mongosync`` connects to the source and destination clusters, the @@ -31,8 +39,11 @@ synchronization process enters the :ref:`IDLE ` state. During Sync ~~~~~~~~~~~ -During the :ref:`RUNNING ` state, ``mongosync`` syncs data -from the source cluster to the destination cluster through the +To start the synchronization process, call the :ref:`start ` +endpoint. + +The ``start`` endpoint begins the ``RUNNING`` state, which is when ``mongosync`` +syncs data from the source cluster to the destination cluster through the :ref:`collection copy ` process. If you make subsequent writes to the source cluster, ``mongosync`` applies the operations to the destination cluster through the :ref:`change event application ` @@ -46,8 +57,6 @@ the ``RUNNING`` state and changes these back to their source cluster characteristics during the ``COMMIT`` state. For more information, see :ref:`Commit Alters `. -The :ref:`start ` endpoint begins the ``RUNNING`` state. - .. _c2c-collection-copy: Collection Copy @@ -75,24 +84,22 @@ writes on the destination cluster. Pausing Sync ~~~~~~~~~~~~ -During the ``PAUSED`` state, the sync process is paused. +To pause the sync process and enter the :ref:`PAUSED ` state, +call the :ref:`pause ` endpoint. -The :ref:`pause ` endpoint begins the -:ref:`PAUSED ` state. - -The :ref:`resume ` endpoint resumes a paused synchronization -session and returns ``mongosync`` to the ``RUNNING`` state. +To resume a paused synchronization session and return ``mongosync`` to the +``RUNNING`` state, call the :ref:`resume ` endpoint. Finalizing Sync ~~~~~~~~~~~~~~~ -During the ``COMMITTING`` state, ``mongosync`` finalizes the sync between the -source cluster and the destination cluster by stopping continuous sync -between them. It also restores collection characteristics that ``mongosync`` -temporarily alters during synchronization. +To finalize the sync between the source and destination clusters, +call the :ref:`commit ` endpoint. -The :ref:`commit ` endpoint begins the -:ref:`COMMITTING ` state. +The ``commit`` endpoint starts the :ref:`COMMITTING ` +state. ``mongosync`` stops continuous sync between the source and desitnation +clusters. It also restores collection characteristics that ``mongosync`` +temporarily alters during synchronization. Once you call the ``commit`` endpoint, ``mongosync`` halts any writes on the source cluster and retrieves the timestamp of the most recent operation in the @@ -111,13 +118,13 @@ direct writes to the destination cluster. Reversing Sync ~~~~~~~~~~~~~~ -During the ``REVERSING`` state, ``mongosync`` copies data from the -destination cluster to the source cluster. Then, ``mongosync`` swaps the source -and destination clusters and resumes applying change events. +To reverse the direction of a committed sync operation, call the +:ref:`reverse ` endpoint. -The :ref:`reverse ` endpoint starts the -:ref:`REVERSING ` state. To use the ``reverse`` endpoint, -``mongosync`` must be in the ``COMMITTED`` state. +The reverse endpoint starts the :ref:`REVERSING ` state. +``mongosync`` copies data from the destination cluster to the source cluster. +Then, ``mongosync`` swaps the source and destination clusters and resumes +applying change events. If the reverse sync is successful, ``mongosync`` enters the ``RUNNING`` state. The synchronization continues in the reverse direction from the original