Doc for upgrading to v3.6 from v3.5

[shivamgcodes]

Created the draft doc for upgrading to etcd version 3.6 from etcd version 3.5

Did not yet put the deprecated flags (if any)
Fixes #963

[k8s-ci-robot]

Hi @shivamgcodes. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ivanvc]

/ok-to-test

[ivanvc]

Thanks for your pull request, @shivamgcodes! I would like to suggest iterating on the progress of this document. Other pull requests are failing due to the file not existing yet, and with every release candidate release that we've been doing, we've had to wipe out the link to this page.

So, to speed up the review and the merge, I think we should trim this pull request down to the basics (i.e., like what I proposed in #966).

Did not yet put the deprecated flags (if any)

Please refer initially to https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.6.md#deprecations (and etcd-io/etcd#19492).

Thanks again :)

[shivamgcodes]

Sorry, I'm a bit confused. Should I trim down the PR to only include the core changes as outlined in #966 and then add the deprecated flags according to the linked changelog? Or is there something else you'd like me to adjust?
Thanks for clarifying!

[shivamgcodes]

fixed the linting issues.

[jmhbnz]

Thanks for the work on this @shivamgcodes, I think it's a great start that we can iterate on once merged to finalise the details for each section.

[jmhbnz]

Think we need to refresh this given the recent work on downgrade support. We can do this as a follow-up.

[shivamgcodes]

Thanks for pointing this out. I’ll look into it and follow up.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jmhbnz, shivamgcodes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jmhbnz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ahrtr]

I see the title still has "draft", please remove the "draft" if it's ready to review. Please also squash the commits.

• We need to document the flags changes between 3.5 and 3.6. Note that some flags have been removed in 3.6, so users should ensure the flags are not used before upgrading to 3.6. Refer to [CHANGELOG-3.6] List all the removed flags etcd#19492 (comment)
• Users need to ensure the v2store doesn't contain any non-membership data. Usually if --enable-v2 isn't configured or set to false, then usually the v2store doesn't contain any custom content. Users can double check using command etcdutl check v2store. Refer to [3.5] support custom content check offline in v2store etcd#19113
• Double check the metrics change, not sure if there are any metrics being removed in 3.6?
• anything else? cc @fuweid @ivanvc @siyuanfoundation @serathius

[shivamgcodes]

Got it, I'm on it. I have exams until March 8th, so I'll need until around March 10th to complete this.

[siyuanfoundation]

• anything else? cc @fuweid @ivanvc @siyuanfoundation @serathius

I think that's all. I'll follow up with the embed.Config breaking changes.

[ajaysundark]

This should be 3.6?

And this should capture the breaking changes introduced in v3.6. For example, there has been many flag migrations happened in v3.6, we need to call out that information here.

[ajaysundark]

you could refer: #965

[ajaysundark]

Is this still relevant? If we feel this is still a concern, I think this could also include the migration guide here suggesting how etcd-v2 user could migrate data to etcd-v3 -- https://etcd.io/docs/v3.6/tutorials/how-to-migrate/?

[ajaysundark]

/cc @ahrtr @siyuanfoundation

[shivamgcodes]

I have changed the entire section, according to changes suggested by #967 (review), and also incorporated the link in the updated doc.
Thanks for the suggestion

[ajaysundark]

nit: I think we could make this clear as "Before upgrading your etcd cluster, please create a snapshot backup of your etcd cluster. . If you need to downgrade the cluster to 3.5 after a complete upgrade, you can use this snapshot to restore an etcd instance to its 3.5 state."

[shivamgcodes]

i have made the changes,
removed - "Please download the snapshot backup to make downgrading the cluster possible even after it has been completely upgraded."
and the resultant doc contains -
"Before upgrading your etcd cluster, please create a snapshot backup of your etcd cluster. . If you need to downgrade the cluster to 3.5 after a complete upgrade, you can use this snapshot to restore an etcd instance to its 3.5 state.
If all members have been upgraded to v3.6, the cluster will be upgraded to v3.6, and downgrade from this completed state is not possible. If any single member is still v3.5, however, the cluster and its operations remains "v3.5", and it is possible from this mixed cluster state to return to using a v3.5 etcd binary on all members."

Thanks for the suggestions

[ajaysundark]

-etcd-old --name ${name} \
+etcd-new --name ${name} \
  --data-dir /path/to/${name}.etcd \
..

instead?

[shivamgcodes]

yup, sounds like an improvement, i have made the changes.

However further in the same command i specify some more names - --initial-cluster s1=http://localhost:2380,s2=http://localhost:22380,s3=http://localhost:32380 \

Should I change this as well, or is it fine as it is ?

[ajaysundark]

@ahrtr / @siyuanfoundation -- do you think, upgrade guide could benefit by adding "If the server to be stopped is the leader, you can avoid some downtime by move-leader to another server before stopping this server." step here as well?

[ahrtr]

"If the server to be stopped is the leader, you can avoid some downtime by move-leader to another server before stopping this server."

Yes, it's nice to have, but not mandatory. It's also better to upgrade the leader last (similar to https://github.com/ahrtr/etcd-defrag), otherwise you will need to move-leader multiple times. But again it isn't mandatory.

[jberkus]

Aside from the comments, we also need a "deprecated flags" section. You can get that from the mirror downgrade PR: #965

[jberkus]

Weight here is wrong, it causes this page to appear out of order in the TOC:

preview

[shivamgcodes]

Fixed it, Thanks for the suggestion.

[jberkus]

Someone upgrading from etcd v2 to v3.6 is going to be an extremely rare event; v2.3 has been EOL for more than 5 years. Given that, the note as written is alarming and confusing. See below for a suggestion for a note that give a very short warning but making it clear who it affects. This warning also belongs in a different section of the instructions, see below.

[jberkus]

Again, let's make this clear that this only affects users who have run etcd v2. This is also where we should put the warning about initializing v3 data.

Suggested change

	#### Limitations
	Note: If the cluster only has v3 data and no v2 data, it is not subject to this limitation.
	#### Limitations on Clusters with v2 Data
	If this cluster has been in use since etcd v2, there are some additional requirements.
	First, if you are upgrading stepwise from etcd v2 to etcd v3.6, you need to take care that the database is initialized with v3 data. See the [v2 data upgrade issue](https://github.com/etcd-io/etcd/issues/9480) for more details.

[shivamgcodes]

I have incorporated the suggested change, thanks.