Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions ci/test/03_test_script.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,10 @@ echo "=== BEGIN env ==="
env
echo "=== END env ==="

(
# Don't apply patches in the tidy job, because it relies on the `git diff`
# command to detect IWYU errors. It is safe to skip this patch in the tidy job
# because it doesn't run a UB detector.
if [ "$RUN_TIDY" != "true" ]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we fix this in https://github.com/bitcoin-core/leveldb-subtree/pulls instead?
I have opened bitcoin-core/leveldb-subtree#56 (I'm also fine with doing it in a different PR, just wanted to give the possibility to solve it in a cleaner way)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No objection, but I am not sure we should diverge the subtree for a test-only patch. Also, the patch is pre-existing, so it seems a bit unrelated to this pull.

Ideally, Google would fix the UB in some way or another, and then we can pull whatever their fix was, but that shouldn't be a blocker here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main leveledb repo doesn't even compile for the past 8 months, not sure why we would wait for them. But as I mentioned, I don't mind doing it after the PR, just seems cleaner if we didn't need to change this section again.

# compact->outputs[i].file_size is uninitialized memory, so reading it is UB.
# The statistic bytes_written is only used for logging, which is disabled in
# CI, so as a temporary minimal fix to work around UB and CI failures, leave
Expand All @@ -62,7 +65,7 @@ echo "=== END env ==="
mutex_.Lock();
stats_[compact->compaction->level() + 1].Add(stats);
EOF
)
fi

if [ "$RUN_FUZZ_TESTS" = "true" ]; then
export DIR_FUZZ_IN=${DIR_QA_ASSETS}/fuzz_corpora/
Expand Down Expand Up @@ -210,14 +213,30 @@ if [ "${RUN_TIDY}" = "true" ]; then
false
fi

# TODO: Consider enforcing IWYU across the entire codebase.
Comment thread
fanquake marked this conversation as resolved.
FILES_WITH_ENFORCED_IWYU="/src/(crypto|index)/.*\\.cpp"
jq --arg patterns "$FILES_WITH_ENFORCED_IWYU" 'map(select(.file | test($patterns)))' "${BASE_BUILD_DIR}/compile_commands.json" > "${BASE_BUILD_DIR}/compile_commands_iwyu_errors.json"
jq --arg patterns "$FILES_WITH_ENFORCED_IWYU" 'map(select(.file | test($patterns) | not))' "${BASE_BUILD_DIR}/compile_commands.json" > "${BASE_BUILD_DIR}/compile_commands_iwyu_warnings.json"

cd "${BASE_ROOT_DIR}"
python3 "/include-what-you-use/iwyu_tool.py" \
-p "${BASE_BUILD_DIR}" "${MAKEJOBS}" \
-- -Xiwyu --cxx17ns -Xiwyu --mapping_file="${BASE_ROOT_DIR}/contrib/devtools/iwyu/bitcoin.core.imp" \
-Xiwyu --max_line_length=160 \
2>&1 | tee /tmp/iwyu_ci.out
cd "${BASE_ROOT_DIR}/src"
Comment thread
fanquake marked this conversation as resolved.
python3 "/include-what-you-use/fix_includes.py" --nosafe_headers < /tmp/iwyu_ci.out

run_iwyu() {
mv "${BASE_BUILD_DIR}/$1" "${BASE_BUILD_DIR}/compile_commands.json"
python3 "/include-what-you-use/iwyu_tool.py" \
-p "${BASE_BUILD_DIR}" "${MAKEJOBS}" \
-- -Xiwyu --cxx17ns -Xiwyu --mapping_file="${BASE_ROOT_DIR}/contrib/devtools/iwyu/bitcoin.core.imp" \
-Xiwyu --max_line_length=160 \
2>&1 | tee /tmp/iwyu_ci.out
python3 "/include-what-you-use/fix_includes.py" --nosafe_headers < /tmp/iwyu_ci.out
}

run_iwyu "compile_commands_iwyu_errors.json"
if ! ( git --no-pager diff --exit-code ); then
echo "^^^ ⚠️ Failure generated from IWYU"
false
fi

run_iwyu "compile_commands_iwyu_warnings.json"
git --no-pager diff
fi

Expand Down
15 changes: 14 additions & 1 deletion contrib/devtools/iwyu/bitcoin.core.imp
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
# Nothing for now.
[
# Compiler intrinsics.
# See: https://github.com/include-what-you-use/include-what-you-use/issues/1764.
{ "include": [ "<emmintrin.h>", "private", "<immintrin.h>", "public" ] },
{ "include": [ "<smmintrin.h>", "private", "<immintrin.h>", "public" ] },
{ "include": [ "<tmmintrin.h>", "private", "<immintrin.h>", "public" ] },
Comment on lines +2 to +6

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been addressed upstream in include-what-you-use/include-what-you-use@6a113bc.


# libc symbols.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been upstreamed / why is it needed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been upstreamed...

Not yet.

why is it needed?

Otherwise, IWYU will suggest platform-specific headers instead of the standard one provided by the C library.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems fine to upstream this, but this shouldn't be a blocker, i'd say

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet.

Is there anything blocking that? It'd be good if we'd atleast report issues upstream, before adding workarounds (with little-to-no documentation).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{ "symbol": ["AT_HWCAP", "private", "<sys/auxv.h>", "public"] },
{ "symbol": ["AT_HWCAP2", "private", "<sys/auxv.h>", "public"] },

# Fixed in https://github.com/include-what-you-use/include-what-you-use/pull/1706.
{ "symbol": ["SEEK_CUR", "private", "<cstdio>", "public"] },
{ "symbol": ["SEEK_END", "private", "<cstdio>", "public"] },
{ "symbol": ["SEEK_SET", "private", "<cstdio>", "public"] },
]
3 changes: 1 addition & 2 deletions src/crypto/chacha20.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,10 @@
#include <crypto/common.h>
#include <crypto/chacha20.h>
#include <support/cleanse.h>
#include <span.h>

#include <algorithm>
#include <bit>
#include <cstring>
#include <cassert>

#define QUARTERROUND(a,b,c,d) \
a += b; d = std::rotl(d ^ a, 16); \
Expand Down
5 changes: 2 additions & 3 deletions src/crypto/chacha20.h
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,11 @@
#ifndef BITCOIN_CRYPTO_CHACHA20_H
#define BITCOIN_CRYPTO_CHACHA20_H

#include <span.h>

#include <array>
#include <cstddef>
#include <cstdint>
#include <cstdlib>
#include <iterator>
#include <span>
#include <utility>

// classes for ChaCha20 256-bit stream cipher developed by Daniel J. Bernstein
Expand Down
2 changes: 1 addition & 1 deletion src/crypto/chacha20poly1305.h
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@

#include <cstddef>
#include <cstdint>
#include <span>

#include <crypto/chacha20.h>
#include <crypto/poly1305.h>
#include <span.h>

/** The AEAD_CHACHA20_POLY1305 authenticated encryption algorithm from RFC8439 section 2.8. */
class AEADChaCha20Poly1305
Expand Down
2 changes: 2 additions & 0 deletions src/crypto/hex_base.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@
#include <crypto/hex_base.h>

#include <array>
#include <cassert>
#include <cstring>
#include <string>
#include <tuple>

namespace {

Expand Down
1 change: 1 addition & 0 deletions src/crypto/hex_base.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

#include <cstddef>
#include <cstdint>
#include <span>
#include <string>

/**
Expand Down
3 changes: 2 additions & 1 deletion src/crypto/hkdf_sha256_32.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@

#include <crypto/hkdf_sha256_32.h>

#include <crypto/hmac_sha256.h>

#include <cassert>
#include <cstring>

CHKDF_HMAC_SHA256_L32::CHKDF_HMAC_SHA256_L32(const unsigned char* ikm, size_t ikmlen, const std::string& salt)
{
Expand Down
6 changes: 2 additions & 4 deletions src/crypto/hkdf_sha256_32.h
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,8 @@
#ifndef BITCOIN_CRYPTO_HKDF_SHA256_32_H
#define BITCOIN_CRYPTO_HKDF_SHA256_32_H

#include <crypto/hmac_sha256.h>

#include <cstdint>
#include <cstdlib>
#include <cstddef>
#include <string>

/** A rfc5869 HKDF implementation with HMAC_SHA256 and fixed key output length of 32 bytes (L=32) */
class CHKDF_HMAC_SHA256_L32
Expand Down
2 changes: 2 additions & 0 deletions src/crypto/hmac_sha256.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

#include <crypto/hmac_sha256.h>

#include <crypto/sha256.h>

#include <cstring>

CHMAC_SHA256::CHMAC_SHA256(const unsigned char* key, size_t keylen)
Expand Down
3 changes: 1 addition & 2 deletions src/crypto/hmac_sha256.h
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@

#include <crypto/sha256.h>

#include <cstdint>
#include <cstdlib>
#include <cstddef>

/** A hasher class for HMAC-SHA-256. */
class CHMAC_SHA256
Expand Down
2 changes: 2 additions & 0 deletions src/crypto/hmac_sha512.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

#include <crypto/hmac_sha512.h>

#include <crypto/sha512.h>

#include <cstring>

CHMAC_SHA512::CHMAC_SHA512(const unsigned char* key, size_t keylen)
Expand Down
3 changes: 1 addition & 2 deletions src/crypto/hmac_sha512.h
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@

#include <crypto/sha512.h>

#include <cstdint>
#include <cstdlib>
#include <cstddef>

/** A hasher class for HMAC-SHA-512. */
class CHMAC_SHA512
Expand Down
5 changes: 3 additions & 2 deletions src/crypto/muhash.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@
#include <crypto/chacha20.h>
#include <crypto/common.h>
#include <hash.h>
#include <span.h>
#include <uint256.h>
#include <util/check.h>

#include <bit>
#include <cassert>
#include <cstdio>
#include <cstring>
#include <limits>

namespace {
Expand Down
5 changes: 4 additions & 1 deletion src/crypto/muhash.h
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,12 @@
#define BITCOIN_CRYPTO_MUHASH_H

#include <serialize.h>
#include <uint256.h>

#include <cstddef>
#include <cstdint>
#include <span>

class uint256;

class Num3072
{
Expand Down
2 changes: 0 additions & 2 deletions src/crypto/poly1305.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@
#include <crypto/common.h>
#include <crypto/poly1305.h>

#include <cstring>

namespace poly1305_donna {

// Based on the public domain implementation by Andrew Moon
Expand Down
3 changes: 2 additions & 1 deletion src/crypto/poly1305.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
#include <span.h>

#include <cassert>
#include <cstddef>
#include <cstdint>
#include <cstdlib>
#include <span>

#define POLY1305_BLOCK_SIZE 16

Expand Down
2 changes: 1 addition & 1 deletion src/crypto/sha256.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#include <cstring>

#if !defined(DISABLE_OPTIMIZED_SHA256)
#include <compat/cpuid.h>
#include <compat/cpuid.h> // IWYU pragma: keep

#if defined(__linux__) && defined(ENABLE_ARM_SHANI)
#include <sys/auxv.h>
Expand Down
1 change: 0 additions & 1 deletion src/crypto/sha256_arm_shani.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
#include <array>
#include <cstdint>
#include <cstddef>
#include <arm_acle.h>
#include <arm_neon.h>

namespace {
Expand Down
4 changes: 2 additions & 2 deletions src/crypto/sha256_sse4.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
// This is a translation to GCC extended asm syntax from YASM code by Intel
// (available at the bottom of this file).

#if defined(__x86_64__) || defined(__amd64__)

#include <cstdint>
#include <cstdlib>

#if defined(__x86_64__) || defined(__amd64__)

namespace sha256_sse4
{
void Transform(uint32_t* s, const unsigned char* chunk, size_t blocks)
Expand Down
1 change: 1 addition & 0 deletions src/crypto/sha256_x86_shani.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

#if defined(ENABLE_SSE41) && defined(ENABLE_X86_SHANI)

#include <cstddef>
#include <cstdint>
#include <immintrin.h>

Expand Down
4 changes: 2 additions & 2 deletions src/crypto/sha3.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
#include <crypto/common.h>

#include <algorithm>
#include <array>
#include <bit>
#include <cstdint>
#include <cassert>
#include <iterator>
#include <span>

void KeccakF(uint64_t (&st)[25])
Expand Down
3 changes: 1 addition & 2 deletions src/crypto/sha3.h
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,9 @@
#ifndef BITCOIN_CRYPTO_SHA3_H
#define BITCOIN_CRYPTO_SHA3_H

#include <span.h>

#include <cstdint>
#include <cstdlib>
#include <span>

//! The Keccak-f[1600] transform.
void KeccakF(uint64_t (&st)[25]);
Expand Down
4 changes: 4 additions & 0 deletions src/crypto/siphash.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,11 @@

#include <crypto/siphash.h>

#include <uint256.h>

#include <bit>
#include <cassert>
#include <span>

#define SIPROUND do { \
v0 += v1; v1 = std::rotl(v1, 13); v1 ^= v0; \
Expand Down
4 changes: 2 additions & 2 deletions src/crypto/siphash.h
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
#define BITCOIN_CRYPTO_SIPHASH_H

#include <cstdint>
#include <span>

#include <span.h>
#include <uint256.h>
class uint256;

/** SipHash-2-4 */
class CSipHasher
Expand Down
20 changes: 17 additions & 3 deletions src/index/base.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,45 @@
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.

#include <chainparams.h>
#include <common/args.h>
#include <index/base.h>

#include <chain.h>
#include <common/args.h>
#include <dbwrapper.h>
#include <interfaces/chain.h>
#include <interfaces/types.h>
#include <kernel/chain.h>
#include <logging.h>
#include <node/abort.h>
#include <node/blockstorage.h>
#include <node/context.h>
#include <node/database_args.h>
#include <node/interface_ui.h>
#include <primitives/block.h>
#include <sync.h>
#include <tinyformat.h>
#include <uint256.h>
#include <undo.h>
#include <util/fs.h>
#include <util/string.h>
#include <util/thread.h>
#include <util/threadinterrupt.h>
#include <util/time.h>
#include <util/translation.h>
#include <validation.h>
#include <validationinterface.h>

#include <chrono>
#include <cassert>
#include <compare>
#include <cstdint>
#include <memory>
#include <optional>
#include <span>
#include <stdexcept>
#include <string>
#include <thread>
#include <utility>
#include <vector>

constexpr uint8_t DB_BEST_BLOCK{'B'};

Expand Down
Loading
Loading