Breaking Changes

• Docker images are now being published to consensys/teku. The pegasys/teku images will continue to be updated for the next few releases but please update your configuration to use consensys/teku.
• Support for the v3 database format has been removed. This is an old format that was replaced prior to the Medalla testnet launching so all users should already be on more recent versions.

Additions and Improvements

• External signing API now includes all data required for the external signer to independently compute the signing root
• New standard REST API endpoints:
  • GET /eth/v1/beacon/headers - currently the parent_root option is not supported
  • GET /eth/v1/debug/states/{state_id}
  • GET /eth/v1/config/spec
• Validator client now supports basic authentication when the credentials are specified as part of the --beacon-node-api-endpoint option
• Validator client now has simple handling for 429 Too Many Request responses from the beacon node
• Added --p2p-subscribe-all-subnets-enabled option to the beacon node to automatically create persistent subscriptions to all attestation subnets
• Early access: support for starting sync from an arbitrary point in the chain with --Xws-initial-state=<path-to-ssz-encoded-state> --Xws-initial-block=<path-to-ssz-encoded-block>. As this is early access, the CLI options are subject to change.

Bug Fixes

• Fixed a long stall when the chain finalizes after a long period of non-finalization in archive mode
• --config-file option can now be used with subcommands and specified either before or after the subcommand on the command line
• Fixed issue where validator committee indices were not string formatted in the standard REST API responses
• Requests for head, justified and finalized blocks would incorrectly return a 404 response if the slot was empty
• Fixed issue where the slot parameter for the /eth/v1/validator/aggregate_attestation endpoint was ignored and the validator client always set it to 0 when making requests.
• Fixed issue where the /eth/v1/validator/attestation_data endpoint did not wrap the response in the data element as expected
• Fixed an issue where a state from the previous epoch was used when calculating proposer duties
• Performed block gossip validation checks which do not require the state prior to loading it to improve performance

Additions and Improvements

• Teku now supports running the validator client as a separate process via the teku validator-client subcommand. The in-process validator client is still supported.

  IMPORTANT: To avoid being slashed, ensure validator keys are only passed as arguments to either the beacon node or the validator client but never both at the same time.

• Migrates to a new directory structure for data storage to provide clear separation between beacon node and validator files. Existing databases will be automatically migrated.

• Improved management of states to reduce on-heap memory usage during long periods of non-finalization.

• Head update and chain reorg events are no longer published during sync. A single coalesced event is published when sync completes.

• Reduced database growth rate during periods of non-finalization. State snapshots are only stored every two epochs instead of each epoch. This can be further adjusted with --Xhot-state-persistence-frequency.

• New standard REST API endpoints:

  • /eth/v1/beacon/headers/{block_id}
  • /eth/v1/beacon/states/:state_id/committees/
  • /eth/v1/beacon/states/:state_id/committees/:epoch
  • /eth/v1/beacon/states/:state_id/finality_checkpoints
  • /eth/v1/beacon/states/:state_id/validators
  • /eth/v1/beacon/states/:state_id/validator_balances
  • /eth/v1/validator/aggregate_attestation
  • /eth/v1/validator/aggregate_and_proofs
  • /eth/v1/validator/beacon_committee_subscriptions
  • /eth/v1/validator/attestation_data
  • /eth/v1/beacon/pool/attestations
  • /eth/v1/beacon/states/:state_id/root
  • /eth/v1/beacon/blocks
  • /eth/v1/validator/blocks/{slot}

• /eth/v1/validator/duties/attester/:epoch now includes the committees_at_slot value.

• /eth/v1/validator/duties/attester/:epoch now supports using the POST method to specify validator IDs, avoiding URL length limits.

• The beacon node now automatically manages persistent subnet subscriptions based on the requests to /eth/v1/validator/beacon_committee_subscriptions endpoint

• Log to the console if the ETH1 node's chain ID doesn't match the expected value from the ETH2 network definition. This alerts users if the ETH1 node is sync'd to the wrong ETH1 network. Thanks to systemfreund.

• Performance tracker reports now include information on duties which could not be performed due to an error.

• Added support for v1.0.0-rc.0 of the beacon chain spec.

• Added a beacon_attestation_pool_size metric to report the number of attestations held in the attestation pool.

• StrictNoSign requirements are now enforced on gossip messages. Messages with from, signature or seqno fields will be rejected.

• Removed support for onyx and altona options to the --network command line option. These networks used an older, incompatible version of the beacon chain spec and have been shut down.

• The --config-file option can now be used to provide options for all subcommands.

• Added support for checking the weak subjectivity period and checkpoint. Violations are currently reported to the log file only while the remaining details and infrastructure for handling the weak subjectivity period are finalized.

Bug Fixes

• Fixed "Unhandled error while processes req/response" NullPointerException.
• Fixed issue where performance tracker logs counted attestations twice.
• Force external signer connections to use HTTP1.1. HTTP2 connections were regularly failing in Azure.
• Fixed incompatibility with Lighthouse caused by sending an unexpected RESET after both sides of a stream were closed.
• Fixed issue where REST API reported a 500 internal server error when syncing instead of 503.
• Fixed issue where REST APIs returning validator information incorrectly used the effective balance instead of the actual balance.

Upcoming Breaking Changes

• REST API endpoints are being migrated to the standard API. Deprecated endpoints will be removed in a future release. Current replacements:
  • GET /eth/v1/beacon/states/{state_id}/validators/{validator_id} replaced by POST /eth/v1/beacon/states/{state_id}/validators/{validator_id}
  • /network/enr replaced by /eth/v1/node/identity
  • /network/listen_addresses replaced by /eth/v1/node/identity
  • /network/peer_id replaced by /eth/v1/node/identity
  • /network/peers replaced by /eth/v1/node/peers
  • /network/peer_count replaced by /eth/v1/node/peers
  • /network/listen_port replaced by /eth/v1/node/identity
  • /node/fork replaced by /eth/v1/beacon/states/{state_id}/fork
  • /node/genesis_time replaced by /eth/v1/beacon/genesis
  • /node/syncing replaced by /eth/v1/node/syncing
  • /node/version replaced by /eth/v1/node/version
  • GET /validator/attestation replaced by /eth/v1/validator/attestation_data
  • POST /validator/attestation replaced by /eth/v1/beacon/pool/attestations
  • GET /validator/block replaced by /eth/v1/validator/blocks/{slot}
  • POST /validator/block replaced by /eth/v1/beacon/blocks
  • /validator/aggregate_attestation replaced by /eth/v1/validator/aggregate_attestation
  • POST /validator/duties replaced by POST /eth/v1/validator/duties/attester/{epoch} and GET /eth/v1/validator/duties/proposer/{epoch}
  • /validator/aggregate_and_proofs replaced by /eth/v1/validator/aggregate_and_proofs
  • /validator/beacon_committee_subscription replaced by /eth/v1/validator/beacon_committee_subscriptions
  • /validator/persistent_subnets_subscription deprecated. The beacon node now automatically establishes persistent subnet subscriptions based on calls to /eth/v1/validator/beacon_committee_subscriptions
• --validators-key-files and --validators-key-password-files have been replaced by --validator-keys. The old arguments still work but will be removed in a future release.
• Validator subcommands for generating and registering validators are now deprecated and will be removed in a future release to encourage the use of the Eth2 Launchpad, which is the most secure way of generating keys and sending deposits.

Additions and Improvements

• Added zinken network genesis state so an ETH1 endpoint is no longer required when connecting to Zinken.
• Deprecated the validator subcommands (which were mainly for testing) used for generating and registering validators.
• Removed the option to configure snappy compression. It is enabled by default.

Bug Fixes

• Fixed an issue where some of the newer validator options weren't being applied in the stand alone validator client.
• Fixed an issue where some of the blocks produced by the stand alone validator client was being dropped.
• Fixed an issue where the create block API coudn't create blocks after more than one epoch of empty slots.
• Fixed a DOS vector where queues for blocks and attestation that are not yet ready for processing could be exploited.

0.12.9

Additions and Improvements

• Added zinken network definition. As the genesis state is not yet known, an ETH1 endpoint must be specified when connecting to the zinken testnet
• Added the option to output validator performance over time. Service can be enabled by using --validators-performance-tracking-enabled
• Implemented caching of beacon block roots to improve block import and thus sync speed
• Support symlinked keystore files
• Updated to spec version 0.12.3

Bug Fixes

• Fixed issues where slot calculation and Store time management led to underflow errors
• Fixed issue discovered with the remote validator where the websocket publishing backed up due to slow readers

Additions and Improvements

• Added spadina network genesis state so an ETH1 endpoint is no longer required when connecting to Spadina.

Bug Fixes

• Fixed issue where topped-up up deposits did not lead to activated validators.

Additions and Improvements

• Added spadina network definition. As the genesis state is not yet known, an ETH1 endpoint must be specified when connecting to the spadina testnet
• New REST APIs
  • /eth/v1/validator/duties/attester/:epoch - gets attester duties for the given epoch
  • /eth/v1/validator/duties/proposer/:epoch - gets block proposer duties for the given epoch
  • Deprecated POST /validator/duties, as the new standard endpoints are now implemented
  • eth/v1/beacon/genesis - retrieves details of the chain's genesis
  • Deprecated the previous genesis endpoint /node/genesis_time
  • /eth/v1/beacon/states/:state_id/validators/:validator_id - gets validator from state by id
  • /eth/v1/beacon/states/{state_id}/fork - gets Fork object for requested state
  • Deprecated the previous fork endpoint /node/fork
  • /eth/v1/events - subscribes to beacon node events
• Implemented validator keystore file locking to prohibit another process using the same keys and getting slashed
• Updated slashing protection interchange format version to v.4
• Upgraded Jblst version to 0.2.0 which adds ARMv8 arch support
• Implemented sending goodbye message to peers on shutdown
• Reduced reorg noise during sync
• Updated metrics library from Besu to latest version
• Better handle P2P target peer bounds

Bug Fixes

• Fixed debug-tools db subcommands to support writing UInt64 as YAML
• Prevented fork choice head from going backwards when updating the chain

Additions and Improvements

• Added support for the slashing protection interchange format via the teku slashing-protection import and teku slashing-protection export subcommands
• New REST APIs
  • /eth/v1/node/peers - lists information about the currently connected peers
  • /eth/v1/node/peers/:peer_id - list information about a specific peer
  • /eth/v1/node/health - return the node health status via HTTP status codes. Useful for load balancers
  • /eth/v1/node/syncing - describe the node's current sync status
  • /v1/node/version has been moved to /eth/v1/node/version and /v1/node/identity to /eth/v1/node/identity matching changes in the standard API spec
• Gossip messages produced by Teku no longer set the from, signature or seqNo fields
• Enabled Gossipsub flood publishing to reduce propagation times for attestations and blocks produced locally
• Generated P2P private keys and Discovery v5 sequence numbers are now persisted across restarts
• Implemented unicode normalization process for validator keystore passwords
• Progress messages are now logged when loading large number of validator keys at startup
• The default network is now Medalla instead of Altona
• Avoid recalculating validator duties for reorgs that are not long enough to change the scheduling
• Validator duties are now calculated as soon as the genesis state is known instead of waiting for the actual genesis time.
• Added additional validation for deposit events received from the ETH1 node to flag when the ETH1 node has malfunctioned and missed some deposit log events
• Exit with a clear message when the ETH1 service is unable to start. Note that this only applies when the existing deposit data is invalid. Teku will continue retrying if the ETH1 node is not currently available.
• Operations (e.g. attestations, slashings etc) included in blocks are now readded to the pending pool if a reorg causes them to no longer be in the canonical chain
• Removed support for generating unencrypted keystores
• Discv5 now caches the hash of the local node to reduce load caused by significant numbers of incoming discovery messages
• Early access support for running the validator node independently of the beacon node (see #2683 for details). Please note this is not yet a recommended configuration and the CLI options and APIs used are subject to change.

Bug Fixes

• Gossip messages with null from, signature or seqNo fields are now rebroadcast with the fields still null instead of replaced by default values
• Added validation to ensure that the uncompressed length of Gossip messages is within the possible ranges for a valid message for each gossip message type
• Fixed validation of compressed gossip message length which may incorrectly reject messages
• Fixed unhandled exception reported when an attestation received over gossip had an invalid checkpoint, specifying a block root from after the specified epoch.
• Improved validation of remote peer status
• Fix "AbstractRouter internal error on message control" errors when messages are received from peers before the outbound connection has been fully established
• Fixed errors logged when the ETH1 chain is shorter than the configured follow distance
• Explicitly fsync the RocksDB write ahead log files to disk on shutdown
• Fixed issue where environment variables named TEKU_VERSION or TEKU_HELP would be incorrectly interpreted as specifying the --version and --help arguments, preventing Teku from starting
• Avoid performing duplicate tasks to regenerate states in a particular corner case when the task can be rebased to start from the output of an already scheduled task
• Improve performance of cacheable task queue used for state regeneration
• Suppressed ClosedChannelException errors in logs

Bug Fixes

• Fix race condition when a block and its parents are received at around the same time which could cause the node to fall out of sync until it reverted to syncing mode to catch up
• Fix issue where attestations from blocks could be processed prior to the block they target being available resulting in ProtoNode: Delta to be subtracted is greater than node weight errors
• Return a non-zero exit code from validator register subcommand when the user does not confirm the transaction

Upcoming Breaking Changes

• REST API endpoints will be updated to match emerging standards in a future release.
• --validators-key-files and --validators-key-password-files have been replaced by --validator-keys. The old arguments still work but will be removed in a future release.

Additions and Improvements

• Includes a significant number of bug fixes and performance improvements as a result of the recent issues on the Medalla testnet. See #2596 for a full list of related issues.
• Support loading an entire directory of validator keys using --validator-keys=<keyDir>:<passDir>. Individual keystore and password files can also be specified using this new argument.
• Major reduction in CPU and memory usage during periods of non-finalization by intelligently queuing and combining requests for beacon states and checkpoint states
• Fixed slow startup times during long periods of non-finalization. Non-finalized states are now periodically persisted to disk to avoid needing to replay large numbers of blocks to regenerate state.
• Reduced sync times during long periods of non-finalization by searching for a more recent common ancestor than the finalized checkpoint
• Added explicit UInt64 overflow and underflow protection
• Local signing is now multithreaded to better utilise CPU when running large numbers of validators
• Improved sync performance by continuing to update the target peer's status during the sync process
• Removed support for SecIO. Only the NOISE handshake is now supported.
• Provide a more userfriendly error message and exit if the P2P network port is already in use
• Added new metrics
  • validator_attestation_publication_delay reports a histogram showing real time between when validations were due and when they were published to the network

Bug Fixes

• Fixed issue where attestations were created for the wrong head because fork choice had not yet been run. Results in a significant improvement in attestation inclusion rate.
• Fixed issue where invalid blocks were produced because they included attestations from forks that had different attestation committees
• Fixed issue where block production may be scheduled for the wrong slot due to calculating duties one epoch ahead
• Fixed issue where sync could appear to stall because fork choice data wasn't being updated during sync
• Fixed race condition when updating fork choice votes which could lead to ProtoNode: Delta to be subtracted is greater than node weight errors
• Reduce RejectedExecutionException noise in logs when Teku is unable to keep up with incoming gossip messages. Other performance improvements should also improve the ability to keep up.
• Fixed cases where states were regenerated without first checking if a cached version was available
• Fixed excessive memory usage in discovery when parsing an invalid RLP message
• Fixed issue where maximum cache sizes could be exceeded resulting in excessive memory usage
• Be more lenient in detecting peers that excessively throttle requests for blocks to better handle long stretches of empty slots
• Fixed error when validating gossiped attestations that point to old blocks
• Fixed netty thread blocked error messages from metrics by avoiding contention on key locks while retrieving metrics
• Fixed issue where sockets were left open when using an external signer

Prebuilt binaries are available from https://bintray.com/consensys/pegasys-repo/teku/0.12.3#files

Breaking Changes

• Removed --validators-unencrypted-key-files option. This was only intended for early interop testing. Keys should be loaded from encrypted keystores.

Additions and Improvements

• Add basic built-in slashing protection. Note that this only a last line of defence against bugs in the beacon node and will not prevent slashing if validator keys are run in multiple processes simultaneously.
• Validator duty logging is now enabled by default. It can be disabled with --log-include-validator-duties-enabled=false
• Add updated Medalla bootnodes
• Updated to be compliant with beacon chain spec 0.12.2
• Prioritise more recent attestations when creating blocks as they pay higher rewards
• Refuse to start if the existing database is from a different network to the current configuration
• Added rate limiting for remote peers based on both number of blocks requested and total number of requests made
• Discovery now requires confirmation from multiple peers before updating the external IP reported by the node
• Improved interoperability with other clients: seqno field is now optional for libp2p messages
• REST API updates:
  • Added genesis validator root to the /node/fork REST API
  • Added /validator/aggregate_attestation
  • Added /validator/persistent_subnets_subscription
  • Added /validator/beacon_committee_subscription
  • Added /validator/aggregate_and_proofs
  • Added /node/pending_attestation_count
• Report the current peer count in "time to genesis"
• Added UInt64 overflow and underflow detection
• Improved performance of list shuffling operations
• Snappy compression is now enabled by default for custom networks. It can be disabled with --p2p-snappy-enabled=false

Bug Fixes

• Fixed vector for DOS attack caused by not throttling libp2p response rate. (See libp2p/jvm-libp2p#127 and ethereum/public-attacknets#7 for futher details)
• Fixed issue that delayed publication of created attestations by a slot
• Fixed "Invalid attestation: Signature is invalid" errors caused by incorrect caching of committee selections (see #2501 for further details)
• Fixed issues where validators failed to perform duties because the node incorrectly returned to syncing state
• Fixed --logging option to accept lowercase debug option. Renamed the debug subcommand to avoid the naming conflict
• Avoid lock contention when reading in-memory storage metrics
• Reduced memory usage when loading large numbers of scrypt encoded keystores
• Increased read timeout for ETH1 requests to avoid repeatedly timing out when the ETH1 node is slow
• Reduced noise in logs from ClosedChannelException when a peer unexpected disconnects
• Fixed IllegalArgumentException when RPC response code was greater than 127
• Fixed IllegalArgumentException when unexpectedly short discovery messages were received
• Fixed very frequenet InternalErrorException when a peer disconnected during initial libp2p handshake
• Fixed crash during shutdown caused by metrics accessing RocksDB after it was closed
• Restricted the maximum epoch value accepted by REST API to ensure it can be converted to a slot without overflowing uint64
• Fixed help text for --p2p-discovery-bootnodes