"Revolutionary Update Streamlines Data Management: New Time Filters Enhance Bulk RPC Efficiency!"
"Software developers unveil game-changing feature to optimize data queries and reduce read load, paving the way for smoother operations and cost-effective resource usage."
Commit Details:
feat: add start and stop time to bulk rpcs with large volumes of data (#2255) We want to be able to restrict the reconciliation job's queries to a time range of e.g. past 1-2 weeks to reduce the amount of read load on hubs. The most expensive part of the read operation is serializing the protobufs to send back to the callee and that's what we want to save on. Notes for reviewers: - There are ways to optimize the time query by (1) using the indexes (2) starting at the common prefix of the start and stop time rather than iterating all the results for the fid. I think it's not worth pursuing these optimizations until we have evidence that we need them. The simple solution implemented here will eliminate the protobuf serialization and significantly less risky than the more optimized solutions. - `getUserDataByFid` and `getAllUserDataMessagesByFid` have the exact same implementation. I added time ranges to the one we classified as a bulk rpc but not the other one. We should consider eliminating one of these rpcs. - I didn't add a time range for the link compaction bulk query because (1) we don't query for link compaction messages in reconciliation (2) it seems like we don't store a lot of link compaction state-- each new link compaction message for an fid replaces the older one. - I'm not sure we need the time filter for user data, but I added it anyway. Happy to remove if it seems like a bad complexity tradeoff. - The rpc protocol change is backwards compatible. I copied the format for `TimestampFidRequest` from `FidRequest` and added new optional parameters. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. <!-- start pr-codex --> --- ## PR-Codex overview This PR adds start/stop time filters for bulk queries and validation methods for time inputs. ### Detailed summary - Added start/stop time filters for bulk queries in various modules - Implemented validation methods for farcaster time inputs > The following files were skipped due to too many changes: `apps/hubble/src/addon/src/store/store.rs`, `apps/hubble/src/rpc/server.ts`, `packages/hub-web/src/generated/request_response.ts`, `packages/hub-nodejs/src/generated/request_response.ts`, `packages/core/src/protobufs/generated/request_response.ts`, `packages/hub-web/src/generated/rpc.ts`, `apps/hubble/src/storage/engine/index.ts`, `packages/hub-nodejs/src/generated/rpc.ts`, `apps/hubble/src/rpc/test/bulkService.test.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex -->