Breakthrough Code Enhancements to Revolutionize On-Chain Event Handling!

New feature promises to eradicate synchronization errors, ensuring seamless data flow and unmatched reliability in Hubble's operations!

Commit Details:

feat: request missing on chain events on error (#2298) ## Why is this change needed? Our hubs regularly fail to get into sync because some of them are missing onchain events. This feature will have us request missing onchain events if there are errors related to missing on chain events when messages are submitted. Deployed to hoyt for testing: [Logs](https://app.datadoghq.com/logs?query=%22Attempting%20to%20retryEventsForFid%22%20OR%20%22Finished%20retryEventsForFid%22%20OR%20%22cacheOnChainEvent%22%20&agg_m=count&agg_m_source=base&agg_q=%40fid&agg_q_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=30&top_o=top&viz=stream&x_missing=true&from_ts=1726433975166&to_ts=1726433977504&live=false) ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. <!-- start pr-codex --> --- ## PR-Codex overview The focus of this PR is to enhance error handling and retry mechanisms for on-chain events in the Hubble application. ### Detailed summary - Added feature to request missing on-chain events on related submit message errors - Improved error code handling for validation failures and unknown signers - Implemented retry mechanism for on-chain events by fid - Enhanced error messages and error handling logic > The following files were skipped due to too many changes: `apps/hubble/src/storage/engine/index.ts`, `apps/hubble/src/eth/l2EventsProvider.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex -->