every time JDS receives a DeclareMiningJob, it needs to communicate with Bitcoin Core in order to validate whether this Custom Job would lead to a valid block

later on, in case JDC finds a solution, it sends a PushSolution to JDS, which then tries to propagate it to Bitcoin Core

the first implementation of this happened via #24, where we introduced a BlockTemplate.waitNext()-based MempoolMirror approach

this was rather suboptimal, because it restricts the range of optimistic validation (as in: no need for ProvideMissingTransactions round-trip) not only to the txs that are included in Bitcoin Core's mempool, but also on what Bitcoin Core deems as an optimal template

as a consequence, we end up with a relatively high probability of ProvideMissingTransactions round-trips on every DeclareMiningJob, which degrades performance of JD protocol as a whole

#268 builds on bitcoin/bitcoin#34020 (scheduled to land on v32), and allows some improvements in this front

the probability of ProvideMissingTransactions round-trips is substantially lower, because now we only need the WtxId set of DeclareMiningJob to exist in Bitcoin Core's mempool to avoid it

nevertheless, this approach still has some suboptimal tradeoffs, namely:

• A. every call to getTransactionsByWitnessID (triggered on every DeclareMiningJob validation) makes the entire txdata (except txs missing from Core's mempool) travel across the IPC boundary
• B. every DeclareMiningJob validation needs to construct a Block with txdata
• C. every PushSolution needs to construct a Block with txdata
• D. there's no mechanism to feed the contents of ProvideMissingTransactions.Success to Bitcoin Core's mempool (which would avoid future triggers on these specific txs)

bitcoin/bitcoin#35671 aims to mitigate A+B+C, via introduction of TxCollection... this is better because we have substantially less data traveling across the IPC boundary, while Custom Job validation and Solution submission are also slightly more efficient

point D is still a ToDo on Bitcoin Core side, but assuming it is merely an enhancement of how TxCollection.addMissingTxs works under the hood, there's no need for changes on SRI side

maybe one could argue that point D could be mitigated on the short term with an extra tx cache on JDS side, but personally I don't think we need to spend engineering efforts on this, given that:

• this is a good-to-have but not essential optimization
• there's a viable (and hopefully desirable) path to implement this on Bitcoin Core

so TLDR:

this is meant as a follow-up to #268, aiming at whenever bitcoin/bitcoin#35671 lands on Bitcoin Core

finally here's a side-by-side description of getTransactionsByWitnessID vs TxCollection approaches, originally written by @Sjors ' clanker on #598 (comment)

it should help guide the implementation whenever this gets tackled

Legend:

• plain text — identical in both designs
• italics — getTransactionsByWTxID design (bitcoin_core_sv2::BitcoinCoreSv2JDP should leverage getTransactionsByWTxID #268)
• bold — TxCollection design (bitcoin_core_sv2::BitcoinCoreSv2JDP should leverage TxCollection #598 / PoC: replace the mempool mirror with Bitcoin Core's TxCollection #599)
• 🔀 — steps that only happen when Bitcoin Core is missing transactions (the ProvideMissingTransactions trigger)

Combined flow

1. DeclareMiningJob arrives at JDS
2. JDS sends a JdRequest::DeclareMiningJob to BitcoinCoreSv2JDP with populated wtxid_list: Vec<Wtxid> (from DeclareMiningJob)
3. BitcoinCoreSv2JDP fetches the declared transactions from Bitcoin Core: it calls getTransactionsByWTxID() / it instantiates a TxCollection by calling collectTxs()
4. BitcoinCoreSv2JDP checks which transactions are missing: empty elements in the returned array / unknownTxPos()
5. 🔀 Only if some transactions are missing:
   1. BitcoinCoreSv2JDP sends JdResponse::MissingTransactions to JDS, meanwhile keeping the non-empty elements in a cache / holding a reference to TxCollection
   2. JDS sends ProvideMissingTransactions to JDC
   3. JDC sends ProvideMissingTransactions.Success to JDS
   4. JDS sends a second JdRequest::DeclareMiningJob to BitcoinCoreSv2JDP with populated missing_txs: Vec<Transaction> (contents from ProvideMissingTransactions.Success)
   5. BitcoinCoreSv2JDP calls addMissingTxs() to complete the collection
6. BitcoinCoreSv2JDP validates the job: it builds a Block (it now has the full txdata) and calls checkBlock() / it calls makeTemplate() and holds a reference to the resulting BlockTemplate
7. BitcoinCoreSv2JDP sends JdResponse::Success to JDS (includes txdata / does not need to include txdata)
8. JDC sends PushSolution to JDS
9. JDS sends to BitcoinCoreSv2JDP: a full Block it built from the solution / just the solution
10. BitcoinCoreSv2JDP submits to Bitcoin Core via submitBlock() / BlockTemplate.submitSolution()