Development, Testing, and Reliability

Hi,  this post is from Bruce-Robert Fenn Pocock. I’m the “Benevolent Dictator” of the Violet Volts project, and the author of the game system from Tootsville IV and the new Tootsville V. Today I’m going to talk a little bit about (a) Why is it taking so long to get back online? and (b) Software testing and some great partnerships that are helping us with that.

Now, the first question on your mind, if you’re reading this, is “what’s taking so long?”

I’m going to start off by pointing out that building an entire world — particularly a shared, persistent, multi-player online world — is a pretty big job to begin with. Doing it in people’s spare time and with no budget (well, my husband and I are footing the bill, anyway, and we love y’all but if we go broke doing this, it all goes away …) is hard. It’s going to take some time.

We’re doing it the hardest way possible in order to build in some reliability that most software just does not need to have. For one thing, you may have seen mentioned, we’re not going to be running this game “only” on a big stack of expensive “cloud” servers like Res Interactive (and most on-line services) have done in the past. As programmers like to point out, “there is no ‘cloud’ — it’s just someone else’s computer.” Well, the Gossipnet system we’re developing for the game is based on the idea of running “your share” of the game on your computer. You can imagine that in a single-player game, your computer does all the work — say, in Pac-Man, your one computer handles not only your actions as Pac-Man, but also moves the Ghost Monsters around, as well. In Tootsville V, your own actions will be handled by your own computer.

Making it so that this system works, though, means that your computer has to “talk” to everyone around you. The biggest problem with that is that your computer is probably not a failure-hardened Internet server with a great network connection and power back-ups. It’s more likely to be a laptop computer on a WiFi connection where you might, at any moment, close the lid and put it to sleep, or microwave some pizza and disrupt your wireless connection, or try to close an annoying message from your buddy and accidentally quit Firefox and drop out of the game altogether.

So that means we have to “harden” the game against any one (or more!) computer dropping out suddenly, and make sure it doesn’t cause a disruption to other players. It’s a kinda new way of building things, and it’s usually done with server machines in datacenters, not a game being played in web browsers, so we’re inventing new technology as we go along. (I’m going to speak up and take most of the credit for this, myself, but I do want to particularly give a “shout out” to Hans Sense who’s helped me think through some of the particularly complicated things.)

What we’re doing, also, is trying to build things in a way that lasts. The goal here is to create a new Tootanga that can keep on going with very little costs. That means we don’t have to get a lot of donations to keep on running, and even if something happens and my family can’t keep paying for it, it won’t be too expensive for a few people’s donations to keep things going forever. (The expenses are all listed on the Wiki-Wiki, by the way. We are dedicated to being totally transparent about the operation of the game.)

Testing Partnerships

As a side effect, we need to ensure that all our software is as reliable as possible. The last thing we need is a software glitch causing problems when it’s running on hundreds or (we hope!) thousands of web browsers around the world. (We are also making plans for how to “patch” bugs, as they are discovered, in all those running systems, while you’re still playing the game, without interrupting you. But, that’s another story.)

Software tests — automated tests that the computer can use to verify that the programs are working as-designed — are the key to that effort. Running these tests is partly the responsibility of the developers. While I make changes to the software, I periodically run some (or all) of these tests to make sure that everything is working as expected.

In order to ensure that these tests run before any changes make it to our players, we also have taken advantage of opportunities offered by both Travis CI, and Circle CI. The CI in their names stands for “Continuous Integration.” They “continuously” run our software tests whenever we “integrate” new changes into our code. The status of those tests is then displayed at http://tootsville.org/development/ as a “pass/fail” status. Since the “integrated” code is open, you can even “click through” and review the results. Getting the main component, the “violet-volts” program that actually creates the game software itself, totally working in the test environments has been a challenge; as I write this, it’s still failing. That’s a good thing, though: By knowing that these tests are failing, we know not to push that program out to our players.

Running this software across myriad systems means there’s a lot of — let’s call it “technological diversity.” Different devices, different operating systems, different screen sizes, different web browsers, all can influence how the game program works on your own computer. Not everyone has the luxury of running Tootsville on a high-definition display on a Fedora™ Linux® computer with the latest stable release of Firefox™, and that’s OK. (If they did, though, it would sure make my job easier!)

That’s where our newest partnership comes into play. Over the next few weeks, I’ll be rolling out integration with BrowserStack, who have agreed (in exchange for some advertising-like mentions in our programming areas) to give us free, limited access to their library of over 1,000 web browser, device, and operating system combinations.

I’ve decided that the “advertising” they want is OK with the “no ads” policy because it will not appear to players, only people who are actively looking “under the hood.” Their links will appear on our GitHub page, the Developers page, the “about this program” screen, and places like this blog. The average player will probably never know, or care, about this partnership. What I hope they will notice is that we are able to detect possible problems with certain devices or browsers before we push out software to our players, which means less bugs reaching the player.

Are We There Yet?

It’s still a long ways to go before we can rest. The next Milestone will be our new Hillside Demo, and I don’t want anybody’s expectations to be too high. It’s not going to look like much. Your Toots won’t be able to do lots of things, yet. The goal of this milestone will be to make sure that the major pieces of the game are working: that players can log in, talk, move around; that the 3D scene draws properly on your display; that the software doesn’t crash or misbehave horribly when someone drops out of the game. These are really exciting things for a programmer, and horribly under-whelming things for a player. I can’t even promise that the Toots characters will be animated or have their patterns and colors on, yet.

Once we reach that point, though, things will start looking a little more interesting to you, as a player. You’ll be able to see as new features, new items, and so forth are introduced. Changes will build up and we’ll start checking off the “Issues” on our GitHub project pages one by one.

I hope that the Hillside Demo will still happen in 2016, but as you know, the holidays are coming and that always causes unforeseeable delays in “hobby” projects like this. So — no promises. It’s still possible, though. Hang in there!

About the Gossip Mice

In the previous versions of Tootsville, the Gossip Mouse named Little Reminder might appear sometimes to give players news and information from the staff. In the new game, things are a little different.

The Gossip Mice now are actually a user interface to the mesh networking technology that makes it possible for Tootsville V to operate.

When you join the game, your web browser connects up to other play in the game world directly, using a technology called a mesh network.

In most games, every player’s computer talks to a cluster of game servers “in the cloud,” at a data center somewhere. As more players join, those computers have to do more and more work.

With a mesh network, each player’s computer is doing a little bit of the work that keeps the game running. That spreads out the work, so we don’t need to charge a subscription fee to run the big cluster of servers.

There are still servers, and it still does cost something to maintain them, so we are also working to organize some fund raising activities for the Violet Volts project to offset those costs. Right now, they’re being paid out of pocket by the Fenn Pocock family.

When the servers are back online, soon, you’ll get messages from the Gossip Mice to let you know if there are problems with that mesh network.

As always there’s more information in the Wiki-Wiki. Check out the Mesh Networking article.