Canada in a Day

I was the sole developer and sysadmin on this project.

It was a bilingual marketing site for the film project and, during the scheduled footage submission period, a submission platform.

The front-end was really polished. It was fully bilingual, including tags and transactional emails. It had nice Lottie/Bodymovin animations, about 80 celebrity “ambassador” pages, and super clear forms.

It needed to have an intuitive and flawless user experience so the bar was as low as possible for participants to upload their footage. It needed to handle a huge influx of video uploads, which were expected to be mostly uploaded within a single 24-hour period. Some of these uploads were expected to be very large files, so the system had to be fault-tolerant.

On top of that, the uploaded videos then needed to be processed. They were scanned for malware with scheduled rescans, probed for their metadata, and transcoded to a format suitable for web. There was then an admin panel with various user permission flags which allowed staff to stream, manage, categorize, and moderate the user-submitted video tags, as well as to view stats and daily charts and download CSV dumps.

To handle the uploads efficiently, the server never touched the data. The server signed upload requests, and then the media was uploaded (in multiple chunks, in parallel, with retrying if necessary) directly to AWS S3. Users could alternatively give a link to a Youtube or Vimeo URL, and if we received one of those a background job used youtube-dl to resolve and download the media and pipe it to S3.

To protect further against failed uploads, I had the application estimate the upload time based on file size and estimated upload speed, and if the file had not been fully received in twice that estimated time, the application emailed the user to prompt them to try again. The user was similarly emailed if the file was found to be corrupt, or not a video file.

Background jobs were managed with Django-Q and AWS SQS.

Deployment and scaling was via custom orchestration scripts and AWS APIs. During deployments, I had the system scale up to at least two application servers, then one at a time take them out of the load balancer, perform all necessary updates, perform sanity checks, then add them back to the pool. We could also scale up and down, though given the short lifetime of the project there was no need for automatic scaling: I was on call for the relatively short peak usage timespan and monitoring it fairly constantly, able to scale up manually if necessary.

The project was a success. The marketing website helped get the public excited, and on Canada Day, July 1, the submission gate opened and over 6,000 video uploads were accepted and processed. Thousands more came in over the following weeks while submissions remained open.

Once the submission period was over, there was a “thank you” state which removed the submission forms and added new messaging and statistics.

Thanks to the way I planned and built this, operational costs were minimal, and performance remained flawless even at the peak moments.