Visitar URL original
Comparing main...con-905-streams-backfills-audit · quicknode/agent-plugins · GitHub
Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: quicknode/agent-plugins
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: quicknode/agent-plugins
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: con-905-streams-backfills-audit
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 6 commits
  • 6 files changed
  • 2 contributors

Commits on Sep 4, 2026

  1. fix(build-web3): correct streams-reference against the live API

    Every JavaScript filter example in this reference threw or silently returned
    nothing, and the documented `dataset` values are rejected by the API. Agents read
    this file to generate filter code, so each defect became broken generated code.
    
    Verified with POST /streams/test_filter (read-only, creates nothing) on Ethereum
    blocks 21000000 / 21000280 and Solana slot 300000000.
    
    ## Defects
    
    1. dataset enum — `log`, `transaction`, `receipt` are rejected with HTTP 400; the
       API accepts `logs`, `transactions`, `receipts`. The table listed 5 of 19 valid
       values; now complete, sourced from the OpenAPI enum.
    
    2. stream.data shape — it is an array whose nesting depends on the dataset, and
       was treated as a single object throughout. All 12 runnable examples failed:
    
         Block / Receipt Streams       Cannot read properties of undefined ('length')
         Transaction / Utilities       Cannot convert undefined to a BigInt
         Logs / NFT Transfer           Cannot read properties of undefined ('0')
         DEX / Monitor Contract        Cannot read properties of undefined ('toLowerCase')
         Function Signature            processedData is not defined
         Whale, Solana x2              silent null — no error, no data
    
       For `logs` the middle dimension is TRANSACTIONS, not blocks: the real shape is
       data[block][tx][log]. On block 21000000 a block's entry holds 181 groups, one
       per transaction, each group's logs sharing one transactionHash.
    
    3. Batch safety — the outer dimension is the batch. `stream.data.length` equals
       `dataset_batch_size` (default 1), so indexing `stream.data[0]` silently
       processes only the first block. Every example now iterates. This is not
       hypothetical: 9 of 20 streams on our own account run batch sizes of 5 or 50.
    
    4. auth header — Management API examples used `Authorization: Bearer`; the API's
       only securityScheme is `x-api-key`. The webhook destination's own Bearer
       header is unrelated and left alone.
    
    5. Create Stream body — was camelCase `filterFunction` with a nested
       `destination` object. The API takes snake_case `filter_function`
       (base64-encoded), a `destination` string enum, and separate
       `destination_attributes`. Added the required-field list.
    
    6. metadata field names — documented as `streamId`; the API returns `stream_id`,
       and every metadata key is snake_case.
    
    7. DEX swap example — filtered `Swap` events by router address, but the emitter
       is always the pool. Block 21000000 has 22 Swap events and zero from a router,
       so it could never match at any nesting depth. Now filters by event signature.
    
    8. Solana examples — `programs_with_logs` items have no `programId`; it lives at
       `programInvocations[].programId`. Balance deltas come from
       `instruction.accounts[].pre/postBalance`, not `meta.pre/postBalances`.
    
    ## Additions
    
    - A `**Test:**` line on every example naming the network, block, and the result
      it produces, so a reader can confirm their setup before adapting the code.
      Track Whale Transactions is pinned to 21000280 (a 2143 ETH transfer) because
      1000+ ETH transfers are absent from most blocks.
    - Payload Shape section documenting nesting per dataset and the batch dimension.
    - Test a Filter section covering POST /streams/test_filter, including that
      `block` must be a string and that a thrown filter still returns HTTP 201 with
      the error nested in `result.error`.
    
    ## Verified after the change
    
      Block Streams              block               @ 21000000   ok
      Transaction Streams        transactions        @ 21000000   ok
      Logs Streams               logs                @ 21000000   ok
      Receipt Streams            receipts            @ 21000000   ok
      Function Signature         logs                @ 21000000   ok
      Available Utilities        transactions        @ 21000000   ok
      Complex Filter Example     logs                @ 21000000   ok
      Monitor Specific Contract  logs                @ 21000000   ok
      Track Whale Transactions   transactions        @ 21000280   ok
      NFT Transfer Tracking      logs                @ 21000000   ok
      Monitor Program Logs       programs_with_logs  @ 300000000  ok
      Track SOL Transfers        programs_with_logs  @ 300000000  ok
    
    Key-Value Store Integration is the only example not executed; it needs a
    pre-populated KV list.
    
    (cherry picked from commit a6e514b)
    ferhatqn authored and 0xsergen committed Sep 4, 2026
    Configuration menu
    Copy the full SHA
    dc31b77 View commit details
    Browse the repository at this point in the history
  2. [CON-905] Correct Streams Backfills reference against the live product

    Audited streams-backfills-reference.md against qn 0.6.0,
    @quicknode/sdk 3.8.2, and raw REST. 9 confirmed defects.
    `test_filter` is free and creates nothing, so it was the validation
    lane. No Stream was created.
    
    Wrong values:
    - The EVM trace dataset slugs did not exist. `debug_trace` and
      `trace_block` are rejected at every layer. The CLI and SDK take
      `debug-traces` and `trace-blocks`.
    - The destination list gave display names and was incomplete. The five
      wire values are `webhook`, `s3`, `azure`, `postgres`, `kafka`.
      Kafka was missing. "PostgreSQL" and "Azure Storage" are rejected.
    - Compression was listed as a Stream setting. It is a destination
      attribute, and Kafka names it `compressionType`.
    
    Omissions that made the guidance wrong:
    - `qn stream create` builds webhook destinations only. For S3, Azure,
      Postgres, or Kafka you must use `--stream-config-file`, which ignores
      every other flag. An agent asked for an S3 backfill could not
      express it, and no missing flag signalled the problem.
    - `decodeEVMReceipts` was named with no signature. It is
      `decodeEVMReceipts(receipts, abis)`, where `abis` is an array of ABI
      arrays. It adds `decodedLogs` to each receipt. It does not touch
      `logs` and adds no `decoded` key. An agent reading `logs[].decoded`
      gets undefined for every log and concludes nothing matched.
      Measured on block 17811625: 135 receipts in, 135 out, 46 with
      decodedLogs, 110 decoded logs.
    - Bitcoin Cash and XRPL were named without their network keys. They are
      `bch-mainnet` and `xrp-mainnet`. Note `xrplevm-mainnet` is a
      different chain.
    - The API Credits Calculator had no link. It is embedded in the Streams
      billing docs; there is no standalone page.
    - Continuing a backfill into live delivery needs an end range of -1.
    - Elastic Batch and batch size had no field names. They are
      `elasticBatchEnabled` and `datasetBatchSize`, and both are required
      in CreateStreamParams alongside `startRange` and `endRange`.
    
    Also confirmed: there is no /backfills REST resource, filtering does
    not reduce credits, and all 7 documentation links return 200.
    
    `qn stream create` was never executed. Creating a backfill bills
    credits, so that path stays help-and-parse verified.
    0xsergen committed Sep 4, 2026
    Configuration menu
    Copy the full SHA
    b0b77e3 View commit details
    Browse the repository at this point in the history
  3. [CON-905] Fix the two live defects left in the KV filter example

    The cherry-picked commit executed 12 of its 13 examples. Its own
    message names the Key-Value Store example as the one it could not run,
    because that needs a pre-populated KV list. CON-905 seeded the list and
    ran it. Both remaining defects are silent failures.
    
    1. `new Set(await qnLib.qnContainsListItems('watchlist', senders))`
       then `.has(tx.from)`. The helper returns booleans aligned to the
       input by position, not the matched values, so the Set holds `true`
       and `false` and `.has(address)` is always false. The branch is dead
       and the filter returns null on every block.
       Fixed with `senders.filter((_, i) => flags[i])`.
    
    2. `qnAddSet('watched_txs', tx.hash, JSON.stringify({...}))`.
       `qnAddSet.length` is 2. The third argument is dropped without an
       error, so the JSON payload is discarded and `tx.hash` is stored as
       the value. Every hit also overwrote the same single key.
       Fixed to `qnAddSet(`watched:${tx.hash}`, JSON.stringify({...}))`,
       one key per transaction.
    
    Kept from the cherry-picked commit: `stream.data.flatMap(block => block)`.
    Indexing `stream.data[0]` processes only the first block of a batch, and
    batch sizes above 1 are in real use. An earlier CON-905 draft
    reintroduced that index; this version does not.
    
    Added a table for the three qnLib signatures that fail silently, and a
    note that every write helper returns "OK" even on a no-op.
    0xsergen committed Sep 4, 2026
    Configuration menu
    Copy the full SHA
    7036189 View commit details
    Browse the repository at this point in the history
  4. [CON-905] Add the missing Test: line and state qnLib returns as facts

    The cherry-picked commit put a Test: line on 12 of 13 examples. The
    Key-Value Store example was the exception, because it needs a
    pre-populated list. Seeded one and ran it, so it now carries the 13th.
    
    Test: ethereum-mainnet block 21000000 gives 181 transactions and 169
    unique senders. With two of those senders on the watchlist the filter
    returns 2 watchlist_hit objects. The same run against the previous
    code returned 0.
    
    Replaced the trap-shaped table with a plain signature table, per the
    house rule that a reference states facts rather than warnings. Added
    qnGetList and qnGetSet, both checked live: a missing list key gives
    [], a missing set key gives null.
    
    Dropped a claim that qnContainsListItems caps at 1,500 items per call.
    It does not. A 1,501-item call returned 1,501 booleans. The 1,500 cap
    is write-only, on addItems plus removeItems, and kv-reference.md
    already scopes it that way.
    0xsergen committed Sep 4, 2026
    Configuration menu
    Copy the full SHA
    4974e29 View commit details
    Browse the repository at this point in the history
  5. [CON-905] Add Test: lines for the verified backfills slugs and keys

    CON-901 asks for a Test: line on every example. The dataset slugs and
    the network keys were the two defect clusters in this file, so both
    enumerations now carry one.
    
    Verified live with `qn stream test-filter`, which is free and creates
    nothing:
    - All nine asserted EVM dataset slugs pass on ethereum-mainnet block
      21000000: block, block-with-receipts, transactions, logs, receipts,
      debug-traces, trace-blocks, block-with-receipts-debug-trace,
      block-with-receipts-trace-block. The two previously documented
      spellings, debug_trace and trace_block, are rejected by the CLI's
      own enum.
    - bitcoin-mainnet and bch-mainnet pass on block 800000, and
      xrp-mainnet passes on ledger 80000000. bitcoin-cash-mainnet and
      xrpl-mainnet are rejected with the full list of valid keys.
    
    The `qn stream create` example carries no Test: line. Creating a
    backfill bills credits against a real historical range, so it stays
    help-and-parse verified only. Recorded as not-covered in the report.
    0xsergen committed Sep 4, 2026
    Configuration menu
    Copy the full SHA
    3a9b2f5 View commit details
    Browse the repository at this point in the history
  6. [CON-905] Bump build-web3 to 1.0.3 for the CON-901 audit fixes

    The sync runbook in PRODUCT_CATALOG.md step 7 asks for a patch bump on
    renames, wording fixes, and stale-example cleanup. The CON-901 audit is
    all three, across CON-906, CON-902, and CON-905.
    
    One bump for all three PRs, in this one, because three PRs each editing
    the same manifests would conflict. The CON-906 and CON-902 PR bodies say
    the bump lands here.
    
    Runbook step 7 is itself wrong and is not fixed here. It names
    `plugins/build-web3/.claude-plugin/marketplace.json`, which does not
    exist, and it omits two files that do carry the version:
    `.codex-plugin/plugin.json` and `server.json`. Following the runbook
    literally leaves two of four manifests behind. All four are bumped here.
    The runbook text belongs to CON-917.
    0xsergen committed Sep 4, 2026
    Configuration menu
    Copy the full SHA
    45175e1 View commit details
    Browse the repository at this point in the history
Loading