Moving to the Web
This is the first of four articles I am in the middle of writing for them about the problems we encountered moving our game to the web. Though it was written for a technical audience, you don't have to have a technical background to understand it.
I should perhaps explain for newer readers that my company writes and runs massively multiplayer games. Until last autumn we had always run our games on proprietary consumer networks, relying on the host network to handle billing, bulletin boards, chat rooms and other ancillary services. Last summer AOL decided that as part of that week's long term strategy it no longer wanted to support games on its system in the old way. We were therefore offered a deal whereby they paid compensation for the fact that our contract still had several years to run, and we moved off within 3 months. We decided to use the compensation money to move onto the web, and that was the start of quite a saga. The more experienced among you will have already spotted that three months is a ludicrously short time to write all the infrastructure we would need to support our product - particularly since only one of our programmers had any experience of working with commercial databases, and his experience was mainframe hierarchical databases. Some decisions were relatively easy. Our main product, the game Federation, was already running under HP-UX10.10 on an HP server. We bought three shiny new HP-9000 series servers, one for the production version of the game, one to run the web and admin services, and one to use for testing and development. Physically obtaining these machines burned up a precious 4 weeks of our allotted time; a harbinger of things to come. The next question was, of course, where to put the machines! We don't have an office since everyone works from home - we are the ultimate telecommuting firm. We contracted the New York branch of Telehouse to host our machines - they handled all the power and air-conditioning requirements and have operators there 24 hours a day, and we access the machines remotely. This has the added advantage that we can link to our Internet access provider whose routers are in the same building, via Ethernet instead of dedicated lines. This was probably the best decision we made during the whole move. The first mistake we made was to buy HP-UX10.20 with the new machines. It was then we discovered that the 10.10 binaries for the GNU tools we were using wouldn't run under 10.20. Not a huge problem, but it meant nothing could be done until the sources were located on the net and recompiled under 10.20. Several days of our precious time slipped away while this was fixed. We were now faced with four major programming tasks and a number of minor ones, plus a major administrative headache. We needed to set up a database for our customer accounts, we needed to move our product onto the new machines, we needed to set up a web site so that customers could access their accounts, and we needed to write billing software so that we could collect money from our customers. It was clear from a very early stage that we needed to buy in commercial products for the database and the web server. Obviously, we would have to port our own product ourselves, and, as it turned out, write the accounting software. There was nothing on the market that even remotely met our accounting needs. With the latter went the major administrative headache of finding some way to allow our users to pay online by credit card. The first requirement was to get Federation up and running on the web before our time on AOL expired, so that we wouldn't lose our customer base. This meant that there had to be three things running - the game, the customer database, and the web site (otherwise customers couldn't set up accounts for themselves). None of us had used a relational database before, which made choosing one more difficult than it would have otherwise been. However, we knew that there were three relatively well regarded 'industrial strength' databases, produced by Company A, Company B and Company C. My chief programmer, Nick, and I put our heads together and considered what we knew about the firms involved. We took Company A out of the equation right at the start. I felt that they had their own agenda involving Network Computers and Company A servers, which didn't fit what we wanted - just the database and then to be left to get on with it. Nick had had dealings with their sales force in his previous job and found them pushy and abrasive. That left Company B and Company C. Both, we knew, had a reasonable reputation in the market, and it really came down to a toss up between them. We eventually chose one of them and I set off to try to purchase their software. I should just mention at this stage that all this was taking place in the US, the UK companies may be a little different - though I suspect not. I'm sure the more knowledgeable of you are going to laugh, but I, in my naivete, went down to the office/business centre the chosen company had just set up near New York's Wall Street, and tried to buy the database we wanted with my credit card. I was met with complete incomprehension by the staff there on two counts - one that I wanted it now, two that I wanted to pay for it. Their first suggestion was that they take my name and address so that one of their reps could call on me 'in the next three to four weeks'! It took me several days to get the company to understand that I wanted it now, not in a few months time. Eventually, they condescended to allow me to pay them some money and arranged for the product to be shipped to us from California - they didn't actually have any in their 'business centre'. The product took about three days to arrive - a single CD, no supporting manuals or anything. All of the so-called manuals were on the CD in the execrable .pdf format. (Adobe's invention of Acrobat and the .pdf format is to my mind responsible for a major deterioration in the standard of documentation of software products.) We also discovered that although we had ordered the Unix version of the product, all its administration tools only ran under Windows NT (the database itself ran under UNIX, as we specified). And you can't step up the database without the tools... Fortunately, Nick happened to have NT on his laptop, so he hooked up the laptop to the network, fired off the install program for the tools, and went off to get some cigarettes. When he came back he was greeted with the well known blue screen of death on the laptop. But, as he soon realized, it wasn't just the tools installation program that was screwed - it had taken out half of the registry with it, and now the computer wouldn't even boot. Time to re-install Windows NT - not a trivial task, and a time consuming one as well. When this had been done Nick went through the .pdf file to find out what he had done wrong. Eventually he found a note saying that the installation program was broken and giving instructions for installing the tools by hand. Can you believe it - shipping a product with a known broken installation program which could potentially (and in our case actually) take out the operating system. (This isn't the time to go into the merits or otherwise of vendors who ship operating systems and treat security as an application type add on...) I have to say I was stunned, but we needed to press on, and after installing the tools by hand we were able to get the database running. It was at this stage we hit our next snag - we discovered that there was no software development kit (SDK) to allow us to interface to the database from our C/C++ code. I rang the New York office, and after a long delay finally got through to someone who knew what I was talking about. 'Oh', he said brightly, 'you need to buy the C/C++ SDK'. I'm afraid I was a little sharp in my questioning of why no one had thought to mention this when we bought the database, so that we could have bought the SDK at the same time. Another three days passed while the SDK was shipped out to us from California. While this was going on Nick and I had a discussion on documentation. The .pdf files - scans of the original printed documentation - were proving a real productivity bar, and they were far to large too run out on his printer. We decided to get them printed out and bound by Kinko's - a sort of 24 hour Kall-Quik with self service facilities - and our administrator took them down to the nearest branch to have them printed out. This seemed like a good solution - if a trifle expensive - until she called to pick up the printout and was told by the manager that when they tried to print the documents they got a big copyright notice and a statement expressly forbidding the printing of the files! Of course they hadn't continued with the printing. Our administrator called direct to California this time, and made some rather sharp comments to the person at the other end. It turned out that they had the original manuals (the books) there - for sale needless to say - so, to cut a long story short we ordered the manual set, and my credit card took yet another hit. California agree to overnight the manuals to Nick. A week later the manuals still hadn't turned up - time for another long distance call. Yes, they had the order, we were told - but we hadn't rung to 'confirm' that we wanted the order! Apparently telling them we wanted it, paying for overnight delivery and making a credit card payment weren't enough indication that we were serious about using their products... After a few more sharp words they agreed to ship the books that day - and sure enough they turned up a couple of days later, having been sent on two day delivery instead of overnight. We had by now nearly used up our allotted three month, and we clearly weren't going to be ready in time to transfer our customers across from AOL. We had to go back cap in hand to AOL and ask them for a one month extension. AOL were generous and granted the extension without any hassling, though we were warned that they wouldn't be able to give any further extensions, because that would conflict with their own plans for the old games channel. (Actually, I know AOL gets a lot of stick for crassness, but I must say that their games channel management staff were very supportive during this transition period.) Meanwhile, things were proceeding apace now we had all the bits. Our Apache web server was running and Federation had been ported to the new system (perhaps if Francis twists my arm and people are interested, I'll tell you something about the problems involved in that in the next issue). The database was now up and running, and we were linking it to CGI forms on the web server when we ran into a brick wall. We kept getting reports of duplicate keys when we tried to set up new accounts. Nick went through the SQL code and the documentation with a fine tooth comb in order to find out what was wrong. Nothing showed up. Eventually, in desperation, because our extension time was running out, he turned to the Deja News archive of the newsgroup for our database. Surprise, surprise, a series of reports from well before the period when we bought the product, detailing the bug in the database code that we had hit. This was serious - if we didn't get a fix for this problem we would go bust. We were out of time, but I was damned if we were going to shell out any more cash to fix a product that had been broken when we bought it. And what is more the manufacturers knew it was broken. I have to confess that at this stage we resorted to a subterfuge... I briefed our administrator, Barbara, on the situation and she rang up the support desk in California. 'Did we have a support contract?' inquired the customer service rep on the other end. 'No,' said Barbara. The rep explained to her in some detail all about support contracts. Barbara heard him out politely and then explained that we didn't want support, we wanted a broken program they had sold us fixing. She then went on to tell him about how she was a mother of three kids and she had bought a washing machine from a New York department store. The washing machine didn't work. She rang the store and they sent out an engineer to fix it the next day - no support contract, 'No charge lady, we are sorry you had the problem, don't hesitate to call us if anything else goes wrong.' There was a silence on the other end while the rep decided what to do next... Then he started explaining about support contracts again, this time in simpler terms. When he finished she explained about her broken washing machine again. The rep tried to explain again in even simpler terms, and Barbara repeated her story about the washing machine. At this stage the rep knew he was out of his depth and passed the call up to his supervisor. Barbara explained about her washing machine to the supervisor and asked for the patch to fix our database. The supervisor tried the patronizing line - 'You wouldn't understand the technical details,' he told her. 'Oh but I know all about products that don't work when you buy them,' said Barbara and switched into New York harpy mode, wondering what would happen if she went to talk to the NYC Consumer affairs department, the State Attorney, her Congresswoman, and Senator... It didn't take all that long for the supervisor to decide he was outclassed and pass the call further up the chain of command. Two and a half hours later we finally reached someone with authority to give us the patch without charging! We didn't pay. Actually, they never stood a chance, Barbara used to work as a customer service rep for General Electric - she knew every move they would make before they made it. Did we make it on time? Yes - just. But we nearly went to the wall because of the badly flawed database software we ended up with. What is really terrifying is the fact that this was a reputable industrial strength database package, running on a stable long standing operating system. If I had to do it again, I would take the time and spend the money buying one of each of the contenders and trying them out. It seemed like we didn't have time or spare cash to do that when we started, but we would have made up the time spent later on. Continuing problems with the database meant that we weren't able to start charging our customers for a further three months. We nearly went bust, not just because of the database - our inexperience cost us a lot as well - but the database's inadequacies must take a large share of the blame. The reason I haven't revealed any names is because I don't believe that the problem is unique to our vendor. We ran into other problems whenever we were involved with what is laughingly called 'Enterprise' computing - for instance on the hardware side the four weeks it took to get our 'D' series servers. These vendors are going to get trounced by the likes of Microsoft and other shrink wrapped software vendors. Even while this saga was unfolding news was coming in that all three of the vendors we originally considered were in financial trouble with flat sales and no one buying their upgrades or new products. These vendors desperately need to find a new market - and it is one that is already there. That market is companies like mine who, while not large in themselves, have IT commitments that are in the Enterprise range. But these customers have very different imperatives to those of the large companies. For a start, they do not have a six months to a year purchasing cycle. They expect to be able to buy the product immediately they have decided what they want - which only takes a week or so. They don't want extensive 'consulting' services from the vendors, or an endless stream of sales reps. They are used to making their own way, and their own mistakes. but they do expect the product to work, or at the very least bugs to be properly document with workarounds. These companies don't have enormous amounts of money - but there are a lot of them, and the number is growing as web services grow. For these people Enterprise software such as relational databases and reporting tools are a commodity, and one which should come with all the bits bundled in and shouldn't cost the earth. This is a whole new culture for the vendors, but one they will have to grasp - and soon. Change or die, Enterprise vendors. If you don't Microsoft will eat your breakfast, your dinner and your tea, and eventually it will swallow you too.
|
If you have any questions or comments about the articles on my web site, click here to send me email. |