Merge pull request #246 from hirosystems/fix/jubilee-indexing

fix: replay + jubilee number increment

Author: Ludo Galabru

Diff for: README.md

@@ -1,16 +1,15 @@
-                      
-       /     /   ▶ Ordhook   
+       /     /   ▶ Ordhook
       / --- /      Ordinal indexing engine based on Chainhook.
      /     /       Build indexes, standards and protocols on top of Ordinals and Inscriptions (BRC20, etc).
-                  
+
 
                                    [![Introduction](https://img.shields.io/badge/%23-%20Introduction%20-orange?labelColor=gray)](#Introduction)
     [![Features](https://img.shields.io/badge/%23-Features-orange?labelColor=gray)](#Features)
     [![Getting started](https://img.shields.io/badge/%23-Quick%20Start-orange?labelColor=gray)](#Quick-start)
     [![Documentation](https://img.shields.io/badge/%23-Documentation-orange?labelColor=gray)](#Documentation)
     [![Contribute](https://img.shields.io/badge/%23-Contribute-orange?labelColor=gray)](#Contribute)
 
-***
+---
 
 # Introduction
 
@@ -22,13 +21,13 @@ The **ordhook** is an indexer designed to help developers build new re-org-resis
 
 The **ordhook** uses [Chainhook SDK](https://github.com/hirosystems/chainhook/tree/develop/components/chainhook-sdk) from the [Chainhook](https://github.com/hirosystems/chainhook/tree/develop) project, which is a re-org-aware transaction indexing engine for Stacks and Bitcoin. The SDK is designed with first-class event-driven principles, so it helps developers extract transactions from blocks efficiently and keeps a consistent view of the chain state.
 
-With **ordhook**, Bitcoin developers can reliably implement feature-rich protocols and business models utilizing  _near-real-time_ Ordinals inscriptions and transfers events.
+With **ordhook**, Bitcoin developers can reliably implement feature-rich protocols and business models utilizing _near-real-time_ Ordinals inscriptions and transfers events.
 
 # Quick Start
 
 ## Installing `ordhook` from source
 
-```console 
+```console
 $ git clone https://github.com/hirosystems/ordhook.git
 $ cd ordhook
 $ cargo ordhook-install
@@ -39,13 +38,14 @@ $ cargo ordhook-install
 ### Explore Ordinal activities in your terminal
 
 Once `ordhook` is installed, Ordinals activities scanning can simply be performed using the following command:
+
 ```console
 $ ordhook scan blocks --interval 767430:767753 --mainnet
 Inscription 6fb976ab49dcec017f1e201e84395983204ae1a7c2abf7ced0a85d692e442799i0 revealed at block #767430 (ordinal_number 1252201400444387, inscription_number 0)
-Inscription 26482871f33f1051f450f2da9af275794c0b5f1c61ebf35e4467fb42c2813403i0 revealed at block #767753 (ordinal_number 727624168684699, inscription_number 1) 
+Inscription 26482871f33f1051f450f2da9af275794c0b5f1c61ebf35e4467fb42c2813403i0 revealed at block #767753 (ordinal_number 727624168684699, inscription_number 1)
 ```
 
-In this command, an interval of blocks to scan (starting at block `767430`, ending at block `767753`) is being provided. `ordhook` will display inscriptions and transfers activities occurring in the range of the specified blocks. 
+In this command, an interval of blocks to scan (starting at block `767430`, ending at block `767753`) is being provided. `ordhook` will display inscriptions and transfers activities occurring in the range of the specified blocks.
 
 The activity for a given inscription can be retrieved using the following command:
 
@@ -56,19 +56,20 @@ Transferred in transaction bc4c30829a9564c0d58e6287195622b53ced54a25711d1b86be7c
 ```
 
 ---
+
 ### Stream Ordinal activities to an indexer
 
 `ordhook` is designed to help developers extract ordinals activities (inscriptions and transfers) from the Bitcoin chain and streaming these activities to their indexer / web application.
 
 In order to get started, a `bitcoind` instance with access to the RPC methods `getblockhash` and `getblock` must be running. The RPC calls latency will directly impact the speed of the scans.
 
-*Note: the configuration of a `bitcoind` instance is out of scope for this guide.*
+_Note: the configuration of a `bitcoind` instance is out of scope for this guide._
 
-Assuming: 
+Assuming:
 
-`1)` a `bitcoind` node correctly configured and 
+`1)` a `bitcoind` node correctly configured and
 
-`2)` a local HTTP server running on port `3000` exposing a `POST /api/events` endpoint, 
+`2)` a local HTTP server running on port `3000` exposing a `POST /api/events` endpoint,
 
 A configuration file `Ordhook.toml` can be generated using the command:
 
@@ -77,15 +78,16 @@ $ ordhook config new --mainnet
 ✔ Generated config file Ordhook.toml
 ```
 
-After adjusting the `Ordhook.toml` settings to make them match the `bitcoind` configuration, the following command can be ran:  
+After adjusting the `Ordhook.toml` settings to make them match the `bitcoind` configuration, the following command can be ran:
 
 ```
 $ ordhook scan blocks --interval 767430:767753 --post-to=http://localhost:3000/api/events --config-path=./Ordhook.toml
 ```
 
-`ordhook` will retrieve the full Ordinals activities (including the inscriptions content) and send all these informations to the `http://localhost:3000/api/events` HTTP POST endpoint. 
+`ordhook` will retrieve the full Ordinals activities (including the inscriptions content) and send all these informations to the `http://localhost:3000/api/events` HTTP POST endpoint.
 
 ---
+
 ### Run `ordhook` as a service for streaming blocks
 
 `ordhook` can be ran as a service for streaming and processing new blocks appended to the Bitcoin blockchain.
@@ -112,6 +114,7 @@ will spin up a HTTP API for managing events destinations.
 A comprehensive OpenAPI specification explaining how to interact with this HTTP REST API can be found [here](https://github.com/hirosystems/chainhook/blob/develop/docs/chainhook-openapi.json).
 
 ---
+
 ### Troubleshooting: Performance and System Requirements
 
 The Ordinals Theory protocol is resource-intensive, demanding significant CPU, memory, and disk capabilities. As we continue to refine and optimize, keep in mind the following system requirements and recommendations to ensure optimal performance:

Diff for: components/ordhook-core/src/core/protocol/inscription_sequencing.rs

@@ -357,7 +357,7 @@ fn get_transactions_to_process(
                 continue;
             }
 
-            if let Some(_) = known_transactions.get(&key) {
+            if let Some(_) = known_transactions.get(&inscription_data.inscription_id) {
                 continue;
             }
 
@@ -417,8 +417,8 @@ impl<'a> SequenceCursor<'a> {
         self.current_block_height = block_height;
 
         let classic = match cursed {
-            true => self.pick_next_neg_number(ctx),
-            false => self.pick_next_pos_number(ctx),
+            true => self.pick_next_neg_classic(ctx),
+            false => self.pick_next_pos_classic(ctx),
         };
         let jubilee_height = match network {
             Network::Bitcoin => 824544,
@@ -435,7 +435,7 @@ impl<'a> SequenceCursor<'a> {
         OrdinalInscriptionNumber { classic, jubilee }
     }
 
-    fn pick_next_pos_number(&mut self, ctx: &Context) -> i64 {
+    fn pick_next_pos_classic(&mut self, ctx: &Context) -> i64 {
         match self.pos_cursor {
             None => {
                 match find_nth_classic_pos_number_at_block_height(
@@ -455,7 +455,7 @@ impl<'a> SequenceCursor<'a> {
     }
 
     fn pick_next_jubilee_number(&mut self, ctx: &Context) -> i64 {
-        match self.pos_cursor {
+        match self.jubilee_cursor {
             None => {
                 match find_nth_jubilee_number_at_block_height(
                     &self.current_block_height,
@@ -473,7 +473,7 @@ impl<'a> SequenceCursor<'a> {
         }
     }
 
-    fn pick_next_neg_number(&mut self, ctx: &Context) -> i64 {
+    fn pick_next_neg_classic(&mut self, ctx: &Context) -> i64 {
         match self.neg_cursor {
             None => {
                 match find_nth_classic_neg_number_at_block_height(
@@ -492,12 +492,12 @@ impl<'a> SequenceCursor<'a> {
         }
     }
 
-    pub fn increment_neg_cursor(&mut self, ctx: &Context) {
-        self.neg_cursor = Some(self.pick_next_neg_number(ctx));
+    pub fn increment_neg_classic(&mut self, ctx: &Context) {
+        self.neg_cursor = Some(self.pick_next_neg_classic(ctx));
     }
 
-    pub fn increment_pos_number(&mut self, ctx: &Context) {
-        self.pos_cursor = Some(self.pick_next_pos_number(ctx))
+    pub fn increment_pos_classic(&mut self, ctx: &Context) {
+        self.pos_cursor = Some(self.pick_next_pos_classic(ctx));
     }
 
     pub fn increment_jubilee_number(&mut self, ctx: &Context) {
@@ -599,9 +599,9 @@ pub fn augment_block_with_ordinals_inscriptions_data(
         inscription_data.inscription_number = inscription_number;
 
         if is_curse {
-            sequence_cursor.increment_neg_cursor(ctx);
+            sequence_cursor.increment_neg_classic(ctx);
         } else {
-            sequence_cursor.increment_pos_number(ctx);
+            sequence_cursor.increment_pos_classic(ctx);
         };
 
         ctx.try_log(|logger| {
@@ -756,10 +756,11 @@ fn augment_transaction_with_ordinals_inscriptions_data(
             );
         });
 
+        sequence_cursor.increment_jubilee_number(ctx);
         if is_cursed {
-            sequence_cursor.increment_neg_cursor(ctx);
+            sequence_cursor.increment_neg_classic(ctx);
         } else {
-            sequence_cursor.increment_pos_number(ctx);
+            sequence_cursor.increment_pos_classic(ctx);
         }
         inscription_subindex += 1;
     }
@@ -773,22 +774,24 @@ fn consolidate_transaction_with_pre_computed_inscription_data(
     tx_index: usize,
     coinbase_txid: &TransactionIdentifier,
     network: &Network,
-    inscriptions_data: &mut BTreeMap<(TransactionIdentifier, usize), TraversalResult>,
+    inscriptions_data: &mut BTreeMap<String, TraversalResult>,
     _ctx: &Context,
 ) {
+    let mut subindex = 0;
     for operation in tx.metadata.ordinal_operations.iter_mut() {
         let inscription = match operation {
             OrdinalOperation::InscriptionRevealed(ref mut inscription) => inscription,
             OrdinalOperation::InscriptionTransferred(_) => continue,
         };
 
-        let Some(traversal) = inscriptions_data.get(&(
-            tx.transaction_identifier.clone(),
-            inscription.inscription_input_index,
-        )) else {
+        let inscription_id = format_inscription_id(&tx.transaction_identifier, subindex);
+        let Some(traversal) = inscriptions_data.get(&inscription_id) else {
+            // Should we remove the operation instead
             continue;
         };
+        subindex += 1;
 
+        inscription.inscription_id = inscription_id.clone();
         inscription.ordinal_offset = traversal.get_ordinal_coinbase_offset();
         inscription.ordinal_block_height = traversal.get_ordinal_coinbase_height();
         inscription.ordinal_number = traversal.ordinal_number;
@@ -857,7 +860,7 @@ pub fn consolidate_block_with_pre_computed_ordinals_data(
         let results =
             find_all_inscriptions_in_block(&block.block_identifier.index, inscriptions_db_tx, ctx);
         // TODO: investigate, sporadically the set returned is empty, and requires a retry.
-        if results.is_empty() && expected_inscriptions_count > 0 {
+        if results.len() != expected_inscriptions_count {
             ctx.try_log(|logger| {
                 warn!(
                     logger,

Diff for: components/ordhook-core/src/db/mod.rs

@@ -1096,7 +1096,7 @@ pub fn find_all_inscriptions_in_block(
     block_height: &u64,
     inscriptions_db_tx: &Connection,
     ctx: &Context,
-) -> BTreeMap<(TransactionIdentifier, usize), TraversalResult> {
+) -> BTreeMap<String, TraversalResult> {
     let transfers_data = find_all_transfers_in_block(block_height, inscriptions_db_tx, ctx);
 
     let args: &[&dyn ToSql] = &[&block_height.to_sql().unwrap()];
@@ -1159,10 +1159,7 @@ pub fn find_all_inscriptions_in_block(
                     transaction_identifier_inscription: transaction_identifier_inscription.clone(),
                     transfer_data: transfer_data.clone(),
                 };
-                results.insert(
-                    (transaction_identifier_inscription, inscription_input_index),
-                    traversal,
-                );
+                results.insert(inscription_id, traversal);
             }
             Ok(None) => break,
             Err(e) => {