One great thing about computers is that they can be programmed to do things that are repetitive and boring. I try to look around in my life to see what things I can get a computer to do for me, so that I don’t have to do it myself.
Today’s case is scheduling weekly YouTube live streams for my church. Every week someone’s got to schedule the live streams for the upcoming weekend. They look like this:
So you might be thinking, this seems pretty trivial, like it’s just a few clicks to schedule this in YouTube. It can’t take more than 10 minutes. And that’s true, but it’s still straightforward and repetitive. Having to figure out which Sunday is coming up, copying and pasting and ensuring the correct dates and times are replaced in the live stream text, and making sure the scheduled dates and times are correct can become tedious work. And it’s the same procedure week after week: the type of processing that computers love to do.
Sure there are other ways to optimize this process, like batching it to create maybe two or more weeks at a time. However, from a viewer’s perspective it can also be confusing when there are a bunch of upcoming live streams that need to be scrolled through. For the purposes of this project, the optimal frequency is to have one set of upcoming live streams visible at any time, and that means having the computer schedule the next set of live streams once every week.
So let’s get the computer to do this. I’m going to split up this blog post into two parts:
I haven’t had time to work on many side projects lately. I think I generally am most effective at programming when I have a long stretch of uninterrupted time (on the order of a few hours or more at a time), and I have found the same at work. These long stretches of time are difficult to find these days.
However this was one opportunity that came up this week. Many of my side projects are born from automating repetitive manual tasks, and this was one of those.
I’ve been running my “T-Comm” Bus Locator website for over seven years now. As a quick recap, on the backend, it’s a procedural PHP/MySQL website, kind of a typical thing one would see of PHP development circa 2000s.
The high level overview is that every minute or two, the system polls TransLink’s real time information APIs for the locations of the all the buses in the system. The system then calculates some information based on TransLink’s GTFS data, which has the schedule information for the entire system. Then it saves the data into a MySQL database, and some other outputs for use on the web interface.
So naturally as more buses operate during the daytime (up to around a thousand buses), the CPU load increases during the day, then drops off at night (down to several on the Night Bus routes). It could take about two minutes to calculate all the necessary information to update all the buses during the daytime
Last night, I set off to find a quick way to optimize this, without having to rewrite the entire system (which arguably it sort of needs, but that’s for another day). From a cursory at the system resources using tools such as htop, it was pretty obvious that the bottleneck was on the MySQL database and its reading and writing to the relatively slow hard disks.
I can’t believe it’s September already. The weather is starting to become cool and wet, days are becoming shorter, marking the end of what has been an incredible summer (and year to date). It’s been a while since I’ve written here, so with the changing season I thought I’d share a bit of an update of 2016 so far.
Some of these warrant their own blog posts, but until I have time to write the full thing here is a summary. Continue Reading