Skip to content

rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats'#29418

Open
vasild wants to merge 3 commits into
bitcoin:masterfrom
vasild:getnetmsgstats
Open

rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats'#29418
vasild wants to merge 3 commits into
bitcoin:masterfrom
vasild:getnetmsgstats

Conversation

@vasild

@vasild vasild commented Feb 11, 2024

Copy link
Copy Markdown
Contributor

Introduce a new RPC, getnetmsgstats to retrieve traffic bytes and count of messages possibly broken down by:

  • direction (sent or received)
  • network (ipv4, tor, etc)
  • connection type (outbound-full-relay, block-relay-only, etc)
  • message type (verack, ping, etc)

Implements: #26337 Add per message stats to getnettotals rpc


Examples:

bitcoin-cli getnetmsgstats '["direction", "network", "connection_type", "message_type"]'
{
  "bytes": 5392756,
  "count": 32683
}
bitcoin-cli getnetmsgstats '["direction", "network", "connection_type"]'
{
  "sendcmpct": {
    "bytes": 975,
    "count": 31
  },
  "sendaddrv2": {
    "bytes": 816,
    "count": 28
  },
  "headers": {
    "bytes": 1786,
    "count": 28
  },
  "feefilter": {
    "bytes": 816,
    "count": 27
  },
  "version": {
    "bytes": 4843,
    "count": 37
  },
  "ping": {
    "bytes": 1104,
    "count": 36
  },
  "getaddr": {
    "bytes": 384,
    "count": 13
  },
  "block": {
    "bytes": 5128467,
    "count": 20365
  },
  "getheaders": {
    "bytes": 29436,
    "count": 28
  },
  "pong": {
    "bytes": 1040,
    "count": 34
  },
  "getdata": {
    "bytes": 1218925,
    "count": 20008
  },
  "sendheaders": {
    "bytes": 447,
    "count": 16
  },
  "wtxidrelay": {
    "bytes": 816,
    "count": 28
  },
  "addrv2": {
    "bytes": 301161,
    "count": 25
  },
  "verack": {
    "bytes": 816,
    "count": 28
  }
}
bitcoin-cli getnetmsgstats '["direction", "connection_type", "message_type"]'
{
  "i2p": {
    "bytes": 1311861,
    "count": 8226
  },
  "onion": {
    "bytes": 3932760,
    "count": 24272
  },
  "ipv6": {
    "bytes": 631265,
    "count": 3362
  },
  "ipv4": {
    "bytes": 1300106,
    "count": 7773
  }
}
bitcoin-cli getnetmsgstats
{
  "recv": {
    "i2p": {
      "manual": {
        "sendcmpct": {
          "bytes": 90,
          "count": 3
        },
        "sendaddrv2": {
          "bytes": 99,
          "count": 3
        },
        "headers": {
          "bytes": 309,
          "count": 3
        },
        "feefilter": {
          "bytes": 87,
          "count": 3
        },
        "version": {
          "bytes": 408,
          "count": 3
        },
        "ping": {
          "bytes": 87,
          "count": 3
        },
        "block": {
          "bytes": 432950,
          "count": 1794
        },
        "getheaders": {
          "bytes": 3150,
          "count": 3
        },
        "pong": {
          "bytes": 87,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 99,
          "count": 3
        },
        "addrv2": {
          "bytes": 68508,
          "count": 3
        },
        "verack": {
          "bytes": 99,
          "count": 3
        }
      }
    },
    "onion": {
      "block-relay-only": {
        "sendcmpct": {
          "bytes": 66,
          "count": 2
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 106,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 126,
          "count": 1
        },
        "ping": {
          "bytes": 64,
          "count": 2
        },
        "block": {
          "bytes": 478479,
          "count": 1896
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 64,
          "count": 2
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      },
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 195,
          "count": 6
        },
        "sendaddrv2": {
          "bytes": 129,
          "count": 5
        },
        "headers": {
          "bytes": 527,
          "count": 5
        },
        "feefilter": {
          "bytes": 157,
          "count": 5
        },
        "version": {
          "bytes": 639,
          "count": 5
        },
        "ping": {
          "bytes": 189,
          "count": 6
        },
        "block": {
          "bytes": 1465688,
          "count": 5957
        },
        "getheaders": {
          "bytes": 5262,
          "count": 5
        },
        "pong": {
          "bytes": 189,
          "count": 6
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 129,
          "count": 5
        },
        "addrv2": {
          "bytes": 120693,
          "count": 9
        },
        "verack": {
          "bytes": 129,
          "count": 5
        }
      }
    },
    "ipv6": {
      "manual": {
        "sendcmpct": {
          "bytes": 120,
          "count": 4
        },
        "sendaddrv2": {
          "bytes": 132,
          "count": 4
        },
        "headers": {
          "bytes": 412,
          "count": 4
        },
        "feefilter": {
          "bytes": 116,
          "count": 4
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 116,
          "count": 4
        },
        "block": {
          "bytes": 138074,
          "count": 583
        },
        "getheaders": {
          "bytes": 4200,
          "count": 4
        },
        "pong": {
          "bytes": 116,
          "count": 4
        },
        "wtxidrelay": {
          "bytes": 132,
          "count": 4
        },
        "addrv2": {
          "bytes": 93436,
          "count": 4
        },
        "verack": {
          "bytes": 132,
          "count": 4
        }
      },
      "outbound-full-relay": {
        "version": {
          "bytes": 126,
          "count": 1
        }
      }
    },
    "ipv4": {
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 66,
          "count": 2
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 106,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 252,
          "count": 2
        },
        "ping": {
          "bytes": 32,
          "count": 1
        },
        "block": {
          "bytes": 313988,
          "count": 1255
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 32,
          "count": 1
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "addrv2": {
          "bytes": 40,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      }
    }
  },
  "sent": {
    "i2p": {
      "manual": {
        "sendcmpct": {
          "bytes": 90,
          "count": 3
        },
        "sendaddrv2": {
          "bytes": 99,
          "count": 3
        },
        "headers": {
          "bytes": 66,
          "count": 3
        },
        "feefilter": {
          "bytes": 87,
          "count": 3
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 87,
          "count": 3
        },
        "getaddr": {
          "bytes": 99,
          "count": 3
        },
        "getheaders": {
          "bytes": 3150,
          "count": 3
        },
        "pong": {
          "bytes": 87,
          "count": 3
        },
        "getdata": {
          "bytes": 105382,
          "count": 1789
        },
        "sendheaders": {
          "bytes": 99,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 99,
          "count": 3
        },
        "verack": {
          "bytes": 99,
          "count": 3
        }
      }
    },
    "onion": {
      "block-relay-only": {
        "sendcmpct": {
          "bytes": 33,
          "count": 1
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 25,
          "count": 1
        },
        "version": {
          "bytes": 127,
          "count": 1
        },
        "ping": {
          "bytes": 64,
          "count": 2
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 64,
          "count": 2
        },
        "getdata": {
          "bytes": 115932,
          "count": 1884
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      },
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 162,
          "count": 5
        },
        "sendaddrv2": {
          "bytes": 129,
          "count": 5
        },
        "headers": {
          "bytes": 122,
          "count": 5
        },
        "feefilter": {
          "bytes": 157,
          "count": 5
        },
        "version": {
          "bytes": 771,
          "count": 6
        },
        "ping": {
          "bytes": 221,
          "count": 7
        },
        "getaddr": {
          "bytes": 129,
          "count": 5
        },
        "getheaders": {
          "bytes": 5262,
          "count": 5
        },
        "pong": {
          "bytes": 189,
          "count": 6
        },
        "getdata": {
          "bytes": 363478,
          "count": 5932
        },
        "sendheaders": {
          "bytes": 129,
          "count": 5
        },
        "wtxidrelay": {
          "bytes": 129,
          "count": 5
        },
        "addrv2": {
          "bytes": 231,
          "count": 4
        },
        "verack": {
          "bytes": 129,
          "count": 5
        }
      }
    },
    "ipv6": {
      "manual": {
        "sendcmpct": {
          "bytes": 120,
          "count": 4
        },
        "sendaddrv2": {
          "bytes": 132,
          "count": 4
        },
        "headers": {
          "bytes": 88,
          "count": 4
        },
        "feefilter": {
          "bytes": 116,
          "count": 4
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 116,
          "count": 4
        },
        "getaddr": {
          "bytes": 132,
          "count": 4
        },
        "getheaders": {
          "bytes": 4200,
          "count": 4
        },
        "pong": {
          "bytes": 116,
          "count": 4
        },
        "getdata": {
          "bytes": 35492,
          "count": 584
        },
        "sendheaders": {
          "bytes": 99,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 132,
          "count": 4
        },
        "verack": {
          "bytes": 132,
          "count": 4
        }
      },
      "outbound-full-relay": {
        "version": {
          "bytes": 254,
          "count": 2
        }
      }
    },
    "ipv4": {
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 33,
          "count": 1
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 25,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 508,
          "count": 4
        },
        "ping": {
          "bytes": 32,
          "count": 1
        },
        "getaddr": {
          "bytes": 24,
          "count": 1
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 32,
          "count": 1
        },
        "getdata": {
          "bytes": 76531,
          "count": 1231
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      }
    }
  }
}

Previous incarnations of this:
#27534 rpc: add 'getnetmsgstats', new rpc to view network message statistics (Thank you, @satsie!)
#28926 rpc: add 'getnetmsgstats' RPC (Thank you, @willcl-ark!)

@DrahtBot

DrahtBot commented Feb 11, 2024

Copy link
Copy Markdown
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/29418.

Reviews

See the guideline for information on the review process.

Type Reviewers
Concept ACK epiccurious, danielabrozzoni, stickies-v
Stale ACK waketraindev

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #35720 (ipc: Update libmultiprocess subtree and drop fuzz test workaround by maflcko)
  • #34162 (net: Avoid undershooting in GetAddressesUnsafe by fjahr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@DrahtBot

Copy link
Copy Markdown
Contributor

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

Debug: https://github.com/bitcoin/bitcoin/runs/21448984768

@epiccurious

Copy link
Copy Markdown
Contributor

Concept ACK 91c90d759503aa3fa15b330bba2148698cdc271d.

@DrahtBot

DrahtBot commented Apr 4, 2024

Copy link
Copy Markdown
Contributor

Needs rebase if still relevant

@achow101 achow101 requested a review from mzumsande April 9, 2024 15:52
@0xB10C

0xB10C commented Apr 9, 2024

Copy link
Copy Markdown
Contributor

just fyi as this came up: You could count up message stats with the inbound and outbound message tracepoints. However, this does only gives you stats from the points on where you start hooking into the tracepoints, might not be as user friendly (RPC is easier to use than the tracepoints), and the tracepoints currently only work on Linux.

While I personally don't need it, I see why some node operators need traffic statistics. E.g. How much of my (limited monthly) traffic is used up by inbound connections?

@DrahtBot DrahtBot marked this pull request as draft April 23, 2024 06:28
@DrahtBot

Copy link
Copy Markdown
Contributor

Converted to draft due to failing CI and inactivity

@vasild vasild force-pushed the getnetmsgstats branch from 91c90d7 to 6bfda84 Compare May 9, 2024 13:40
@vasild

vasild commented May 9, 2024

Copy link
Copy Markdown
Contributor Author

91c90d7595...6bfda84b92: rebase, pick changes from #29421 and adjust the newly added test.

@DrahtBot DrahtBot removed the CI failed label May 9, 2024
achow101 added a commit that referenced this pull request May 21, 2024
…e constant

b3efb48 protocol: make message types constexpr (Vasil Dimov)
2fa9de0 net: make the list of known message types a compile time constant (Vasil Dimov)

Pull request description:

  Turn the `std::vector` to `std::array` because it is cheaper and allows us to have the number of the messages as a compile time constant: `ALL_NET_MESSAGE_TYPES.size()` which can be used in future code to build other `std::array`s with that size.

  ---

  This change is part of #29418 but it makes sense on its own and would be good to have it, regardless of the fate of #29418. Also, if this is merged, that would reduce the size of #29418, thus the current standalone PR.

ACKs for top commit:
  achow101:
    ACK b3efb48
  jonatack:
    ACK b3efb48
  maflcko:
    utACK b3efb48 🎊
  willcl-ark:
    ACK b3efb48

Tree-SHA512: 6d3860c138c64514ebab13d97ea67893e2d346dfac30a48c3d9bc769a1970407375ea4170afdb522411ced306a14a9af4eede99e964d1fb1ea3efff5d5eb57af
@vasild

vasild commented May 23, 2024

Copy link
Copy Markdown
Contributor Author

6bfda84b92...b5189f543c: rebase due to conflicts (part of this PR was merged via #29421)

@vasild vasild marked this pull request as ready for review May 23, 2024 09:51

@danielabrozzoni danielabrozzoni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, reviewed up to 5ad1f5d, I still haven't finished the review of the last commit

I wonder if in the functional test, instead of hard-coding the message stats values, we could check how many messages/bytes are sent using P2PInterface... I'm not sure how this would look like or if it's possible at all, I'll try it out and report back :)

Comment thread src/net.cpp
Comment thread test/functional/rpc_net.py Outdated
@sedited sedited requested review from danielabrozzoni and removed request for danielabrozzoni March 9, 2026 13:03
@sedited

sedited commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Ping @vasild for rebase

@sedited

sedited commented May 14, 2026

Copy link
Copy Markdown
Contributor

@vasild are you still working on this?

@vasild

vasild commented May 18, 2026

Copy link
Copy Markdown
Contributor Author

7d4e82c0bbe763acac9ba8770a40c162b79054a7...9fc5965c554244ae2db7cec2cf696eb7fd3a83d5: rebase and address suggestions

are you still working on this?

Yes. This and a few others got pushed way back on my TODO list, slowly processing it, thanks for the patience.

@stickies-v stickies-v left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. Seems like useful info to expose.

Approach-wise, I like that it adds very little logic in the RPC layer. I quite strongly opppose the aggregation option, just a simple bitcoin-cli getnetmsgstats provides a user all the info they need. Visualization/aggregation can easily be done by the user, e.g. using jq or any other tool / aggregation they like, so adding that to our interface unnecessarily locks us into backwards compatibility promises and code/interface complexity (even if it's not a crazy amount).

I'm not sure I'll review this in-depth anytime soon, but will try to keep engaging on high-level.

Comment thread src/rpc/net.cpp Outdated
Before this change only per-peer stats were gathered. They
are lost when the peer disconnects.

So, collect the traffic stats globally in `CConnman`, broken down by
* direction (sent or received, (2))
* network of the peer (IPv4, IPv6, Tor, I2P, CJDNS (5))
* connection type (inbound, full outbound, feeler, etc, (6))
* message type (verack, ping, etc, (36))
satsie and others added 2 commits June 12, 2026 16:22
Introduce a new RPC `getnetmsgstats` that renders the global traffic
stats from `CConnman`.

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
@vasild

vasild commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

6293e251350dd2b6da6cd6018606e8f780b606f9...b2d092efcf94e721dab4a36ce3aeefa7f9d9af5f: rebase and address suggestions

I quite strongly oppose the aggregation option, just a simple bitcoin-cli getnetmsgstats provides a user all the info they need. Visualization/aggregation can easily be done by the user

Alright, that's it. I was thinking the same for some time now - "Should we do aggregation on the server side, given that it is more like a visualization/presentation thing?". Thanks for pushing here. I have now dropped the aggregation, which means removing commit d5a1ad9 rpc: make it possible to aggregate the result in getnetmsgstats and omitting the parts of the test that tested the aggregation and tweaking the parts of the test that used the aggregation.

Thanks!

Kino1994 pushed a commit to Kino1994/bitcoin-full-history that referenced this pull request Jun 28, 2026
…s a compile time constant

68c9212 protocol: make message types constexpr (Vasil Dimov)
2eec318 net: make the list of known message types a compile time constant (Vasil Dimov)

Pull request description:

  Turn the `std::vector` to `std::array` because it is cheaper and allows us to have the number of the messages as a compile time constant: `ALL_NET_MESSAGE_TYPES.size()` which can be used in future code to build other `std::array`s with that size.

  ---

  This change is part of bitcoin/bitcoin#29418 but it makes sense on its own and would be good to have it, regardless of the fate of bitcoin/bitcoin#29418. Also, if this is merged, that would reduce the size of bitcoin/bitcoin#29418, thus the current standalone PR.

ACKs for top commit:
  achow101:
    ACK 68c9212
  jonatack:
    ACK 68c9212
  maflcko:
    utACK 68c9212 🎊
  willcl-ark:
    ACK 68c9212

Tree-SHA512: 6d3860c138c64514ebab13d97ea67893e2d346dfac30a48c3d9bc769a1970407375ea4170afdb522411ced306a14a9af4eede99e964d1fb1ea3efff5d5eb57af
BigcoinBGC pushed a commit to BigcoinBGC/bigcoin that referenced this pull request Jun 30, 2026
…s a compile time constant

2dc6997 protocol: make message types constexpr (Vasil Dimov)
df89ced net: make the list of known message types a compile time constant (Vasil Dimov)

Pull request description:

  Turn the `std::vector` to `std::array` because it is cheaper and allows us to have the number of the messages as a compile time constant: `ALL_NET_MESSAGE_TYPES.size()` which can be used in future code to build other `std::array`s with that size.

  ---

  This change is part of bitcoin/bitcoin#29418 but it makes sense on its own and would be good to have it, regardless of the fate of bitcoin/bitcoin#29418. Also, if this is merged, that would reduce the size of bitcoin/bitcoin#29418, thus the current standalone PR.

ACKs for top commit:
  achow101:
    ACK 2dc6997
  jonatack:
    ACK 2dc6997
  maflcko:
    utACK 2dc6997 🎊
  willcl-ark:
    ACK 2dc6997

Tree-SHA512: 6d3860c138c64514ebab13d97ea67893e2d346dfac30a48c3d9bc769a1970407375ea4170afdb522411ced306a14a9af4eede99e964d1fb1ea3efff5d5eb57af
@DrahtBot

Copy link
Copy Markdown
Contributor

🐙 This pull request conflicts with the target branch and needs rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.