SDK + S3 Optimization: Moving file uploads to background workers
Building an app that is used around the globe and in locations with slow internet speeds resulted in some logic being built out to protect ourselves from data loss in SDK. Although we have done our best to make sure each submission saves without issue, it’s always good to have backups in case something goes awry. That is why we implemented code to make sure each submission to SDK is saved to S3 before anything happens on our end. This means that if something were to happen after a user submits their form, we would be able to recover that user’s data.
However, down the road we discovered that this logic had introduced a bottleneck, one that needed to be addressed. Although this bottleneck did not affect every submission, we were aware of the issues it could present should we have a heavy flow of submissions into SDK at the same time. Using New Relic we pinpointed the bottleneck to the logic related to S3 uploads.
My approach to solve this was to move the S3 upload off the web process so that when a user submitted their form, the server would respond as fast as possible. To accomplish this I decided to utilize the [‘delayed_job’ gem](https://github.com/collectiveidea/delayed_job). This allows us to push the XML submission saving to S3 into a worker so it can be handled by Redis in the background.
Before the change, the code was taking the submitted file and pushing it into our generic `FileUploader` class that we built. The logic involved utilized the AWS gem to manually place a file where it needs to go.
After the change, the code performs a similar action, however we have now added the delay call, which pushed the method into a background worker that is separate from our web process. This allows the code to move pass the S3 upload before it finishes, which in turn reduces the response time to the user.
After we updated the code, the average response time for the whole submission logic went from 17.2 (ms) to 11.3 (ms), meaning we were able to shave off about 40% of the average response time. Also, since implementing this logic we have yet to have a single form submission request timeout, removing the fear of losing client data.
However, down the road we discovered that this logic had introduced a bottleneck, one that needed to be addressed. Although this bottleneck did not affect every submission, we were aware of the issues it could present should we have a heavy flow of submissions into SDK at the same time. Using New Relic we pinpointed the bottleneck to the logic related to S3 uploads.
My approach to solve this was to move the S3 upload off the web process so that when a user submitted their form, the server would respond as fast as possible. To accomplish this I decided to utilize the [‘delayed_job’ gem](https://github.com/collectiveidea/delayed_job). This allows us to push the XML submission saving to S3 into a worker so it can be handled by Redis in the background.
Before the change, the code was taking the submitted file and pushing it into our generic `FileUploader` class that we built. The logic involved utilized the AWS gem to manually place a file where it needs to go.
After the change, the code performs a similar action, however we have now added the delay call, which pushed the method into a background worker that is separate from our web process. This allows the code to move pass the S3 upload before it finishes, which in turn reduces the response time to the user.
After we updated the code, the average response time for the whole submission logic went from 17.2 (ms) to 11.3 (ms), meaning we were able to shave off about 40% of the average response time. Also, since implementing this logic we have yet to have a single form submission request timeout, removing the fear of losing client data.
