"Redis Revolution: Hub Shuttle Takes Flight with Event Streaming Power!"

"Game-changing enhancements empower scalability and introduce reconciliation workers, heralding a new era for ecosystem efficiency!"

Commit Details:

feat(shuttle): Read and write events from redis streams and add reconciliation workers (#1903) ## Motivation Send hub events to redis streams to support scalability and add backfill workers. ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR enhances the `hub-shuttle` package by adding support for Redis streams, reconciliation workers, and scalability improvements. ### Detailed summary - Imported `StatsD` from `@figma/hot-shots` - Added `statsd` export - Updated `RedisClient` class - Added `inBatchesOf` function to `utils.ts` - Updated `package.json` scripts and dependencies - Modified environment variables in `env.ts` - Updated `hubEventProcessor.ts` - Updated `README.md` with backfill process details - Updated `worker.ts` and `eventStream` in `shuttle` folder - Updated `hub.ts` with new functions - Updated `shuttle.integration.test.ts` with new imports and methods > The following files were skipped due to too many changes: `packages/hub-shuttle/src/shuttle.integration.test.ts`, `packages/hub-shuttle/src/shuttle/hubSubscriber.ts`, `packages/hub-shuttle/src/example-app/app.ts`, `packages/hub-shuttle/src/shuttle/eventStream.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex -->