"Revolutionary Sync Upgrade: Hubble's New S3 Snapshot Feature Set to Transform Data Management!"

"Embrace rapid data catch-up with the groundbreaking 'catchup-sync-with-snapshot' option, but beware—resetting the database is part of the power!"

Commit Details:

feat(hubble): catchup sync with s3 snapshot (#1833) ## Motivation - If there is a large gap in message count for a hub compared to the latest snapshot, it is faster to sync from s3 snapshot which is publicly available online - **NOTE**: Catchup sync using snapshot WILL RESET THE DATABASE ## Change Summary - Add `--catchup-sync-with-snapshot` and `--catchup-sync-snapshot-message-limit <number>` - By default, `--catchup-sync-with-snapshot` is disabled, and will be used if message difference is greater than `3_000_000` - Add logic for using s3 snapshot for "catch up" sync. Snapshot metadata is used to determine whether snapshot should be used. If metadata JSON is missing attribute for message count, an error is returned - Fix small bug where HTTP API was reporting incorrect sync status - **NOTE**: Catchup sync using snapshot **WILL RESET THE DATABASE** ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [x] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context Confirmed upload and download working: ``` {"level":30,"time":1710988900433,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":1073751179,"totalSize":24114175697,"msg":"Downloading snapshot..."} {"level":30,"time":1710988921461,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":2147499571,"totalSize":24114175697,"msg":"Downloading snapshot..."} {"level":30,"time":1710988941068,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":3221242283,"totalSize":24114175697,"msg":"Downloading snapshot..."} ``` This is using test bucket with transfer acceleration enabled. **NOTE:** If the s3-bucket-name isn't valid DNS endpoint, the snapshot sync fails. This was issue before, but was not problem because we hardcoded download.farcaster.xyz: ``` {"level":30,"time":1710988763240,"pid":18380,"hostname":"mbp.local","component":"Hub","db_location":".rocks/rocks.hub._default","file_count":1,"overwrite":true,"msg":"DB is empty or overwrite is true, fetching snapshot from S3"} {"level":50,"time":1710988763292,"pid":18380,"hostname":"mbp.local","component":"Hub","error":{"message":"getaddrinfo ENOTFOUND wasif-test3","name":"Error","stack":"Error: getaddrinfo ENOTFOUND wasif-test3 ``` For now, it shouldn't be concern since default is download.farcaster.xyz which is public and available to everyone --- ## PR-Codex overview The focus of this PR is to enhance catchup sync functionality and snapshot handling in the Hubble application. ### Detailed summary - Improved catchup sync with snapshot feature - Added option to enable catchup sync using S3 snapshot - Added message limit trigger for catchup sync - Updated default configuration settings for catchup sync - Refactored code for snapshot handling and sync process > The following files were skipped due to too many changes: `apps/hubble/src/hubble.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> <!-- start pr-codex --> --- ## PR-Codex overview This PR introduces support for catch up sync using S3 snapshot in the `Hubble` app. ### Detailed summary - Added support for catch up sync with S3 snapshot - Updated CLI options for catch up sync with snapshot - Implemented catch up sync logic with snapshot - Defined catch up sync snapshot message limit - Modified default configuration for catch up sync snapshot message limit > The following files were skipped due to too many changes: `apps/hubble/src/hubble.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> <!-- start pr-codex --> --- ## PR-Codex overview The focus of this PR is to introduce support for using S3 snapshots for "catch up" sync in the Hubble app. ### Detailed summary - Added support for catch up sync using S3 snapshot - Modified CLI options for enabling catch up sync with snapshot - Implemented logic for catch up sync with snapshot based on message count difference - Updated default configuration for catch up sync with snapshot - Implemented functions for handling snapshot metadata and uploading to S3 > The following files were skipped due to too many changes: `apps/hubble/src/hubble.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}`