Items Duplication

About a week after the crash, a player asked me if he can buy some items in the game, for real money. After careful consideration (which lasted maybe 2 seconds) I agreed. It wasn't very unreasonable to sell items in a free game, and the donations weren't going that well. So he decided to buy the best items in the game, for about 50 USD (total).

Next day, when I got home from work, I had a few private messages on the forum. One of them was from a contributor (programmer) who wasn't, at that time, part of the team, but he was reliable and trustworthy. He told me that there are rumors that there is a modified client out there that can duplicate some items. This is the nightmare of any MMORPG maker, but at that time I was very confident that our game has no such exploit that would allow items duplication. After all, I was extremely careful when I wrote all the server functions dealing with user input, and even more careful with the functions that dealt with items. So I dismissed it as an unsubstantiated rumor.
Then, later on that evening, the person that previously bought those items contacted me and told me the same thing: there is a modified client out there that can modify non-stackable items (non-stackable items, in our game, are those items that occupy individual positions in the inventory. For example, gold coins take only one slot, regardless of their number. Big items, such as armor, swords, etc. take a slot for each individual piece).

I didn't exactly believe him either, but he offered to e-mail me that modified client, so I can test for myself. I was reluctant to run a 3rd party binary program, fearing that it might be some virus, trojan, keylogger, etc. As an MMORPG owner, many people would like to know my passwords. I asked the other developers if they have any means of running that program in some sand box, such as VMware. One of them did have VMware and windows installed, so he tested it. To our surprise and horror, it actually worked! I was very disappointed and confused, and I started to look in the server code. Fortunately, we were able to inspect the packets sent to the server by that program, and it appeared that the problem occurred when an item was dropped into a bag, then the client tried to pick up a 0 quantity of that item. The item was given to the player, but at the same time it remained in the bag, so this action could be repeated over and over again. Having all this information, it was easy to isolate the problem and pinpoint the exact code that was responsible for that. It was an easy fix, and I immediately updated the main server. Give the fact that the exploit was relatively new, I didn't want to actually restore all the users from a previous backup, to get rid of those duplicated items. After all, how many users could they be anyway?

I was tired that night, so right after updating the server I went to bed. However, just a few minutes before going to bed, I decided to take another look in the game to see if there are any problems. That was a good thing, because someone reported a bag with a negative quantity of bread! I checked it for myself, seen it was there, and realized where the problem was (while fixing the previous bug, I inadvertently changed something from unsigned int to signed int). But I was too tired, and didn't consider it a really big problem. After all, you can't take more than 0 apples from a bag that lacks apples, right?
So I finally went to bed, then went to work, without checking the game. When I came home, there was general anarchy in the game: everyone was extremely rich, people having 1 billion gold coins, all the good weapons, etc. And it didn't even require them to use a modified client; the official client was pretty good at removing positive quantities from negative quantities in the bag, and the server was happy to overlook the fact that this wouldn't be possible in real life.

Now, finding and fixing the problem was trivial. Finding out who abused that bug and how much they abused it was a whole different issue. In fact, it was pretty much impossible to determine who, what, and how much. I was pretty angry, and I told the players: "Guys, you know what, I am not playing this game, you are playing it, so if you abused the bugs than great, I hope you will enjoy playing with so many gold coins!". Obviously, no one liked that, because, while it is fun and cool to exploit bugs, it's not fun when everyone else is exploiting the same bug(s). Eventually, another team member convinced me to do what had to be done, and restore everything from the backups (which were less than 48 hours ago). I asked them if they want that, and the overwhelming majority said: "Yes, restore from the backups". I didn't restore the backups myself, I let someone else do it, so I won't be their scapegoat again. They were a little unhappy that they lost some of their 'work', especially that it was so close after the big crash, but most of them accepted it and moved on.

The Forum Problem

Just a few weeks after the server hard disk crash, we had another problem, this time with our forums. We were using phpBB, and we didn't keep track of it's security problems, which proved to be a big mistake. One nice July day, while I was at work, a script kiddie used an exploit to gain moderator access, and deleted most of our forums. Fortunately, some of the forums were moderated by admins, so he couldn't delete those forums. However, the general chat, programming, art, and some other forums didn't require mod access, and all or most of their posts were erased. There were some admins online when it happened, and they managed to back up some data, but they didn't know that the best solution to this problem is to put the boards in the administration mode, so nothing can be altered.

When that happened, no one knew what was wrong. We suspected that maybe the database had a problem, but the script kiddie took care to let us know it wasn't an accident and that it was deliberate. The damage wasn't really too extensive, there was nothing really important there, and so it didn't really affect us, or the players. Sure, we would have rather not had that happen, but since it did happen, we tried to make the best out of it. After learning that our phpBB version had some critical security bugs, I was quite displeased with it, and decided to use some other forum software. Of course, we could have just updated it and used a newer version that had that bug fixed, but it is my belief that once a software has such bugs, it will always have them. This belief was confirmed at least once, when I got rooted with two RedHat versions (out of the box configuration).

Naturally, I was looking for some other forums software. I didn't mind paying a fee, as long as the forum was good and secure. Initially, I wanted to get vBulletin, but they required an e-mail address from your ISP (yahoo and hotmail were not good enough) even though the payment was done via PayPal. For me, that seemed an unreasonable requirement, so I had to look for an alternative. Someone recommended Invision Power Board, and after visiting their site I decided to buy it. But it turned out that I didn't even have to buy it, they had an unlimited free version (however, last time I checked their website for updates, they didn't have a free version, so eventually, we will have to pay for it in order to get the latest versions. $185US for a lifetime of free updates seems very reasonable), which behaved pretty much like the paid version except for the fact that there was no tech support, and it lacked some extra features we didn't need anyway. The conversion of the old (phpBB) database to the IPB format was relatively painless using a freely available program. There were some minor issues such as some categories not showing up before their main categories were created manually, and some unimportant fields not imported, but overall I was very pleased with how well it worked. Even now, 6 months later, it works fine and had no more security problems with the forums. Needless to say, now we also back up the forums database every few weeks, just in case.

User Input

We have a suggestions forum where people can post their ideas about how the game can be improved. There are a lot of unrealistic ideas, ranging from things that are technically impossible, to things that would transform our game into some other game, typically Runescape. There are, however, some decent ideas that are relatively easy to implement, and would make the game much more interesting. One of such suggestion came not from the average player, but from the head of map makers, Moonshadow. He told me one day that making items (such as potions, magic essences, etc.) in industrial quantities is very boring, we should have some random events while manufacturing, for example a rare chance of producing a higher quality (more potent) version of the item that is to be produced.

As I had some free time, I decided to give it a try. The implementation didn't take too long, maybe 10 hours of work. Only some items had a 'super item' associated with them, and that 'super item' looked, initially, exactly like the normal item, so we didn't need a client update. The next client update, however, had more item images so we were able to give those super items a new look. In the beginning we started with some more potent potions (for example an extra healing potion that restored 50 HP, rather than 20). Then we added some enriched magic essences, which were used in the manufacturing of some special items, such as full plate iron armor.

The players reacted very well to this addition and no one was upset (not surprising). That was the exception from the rule, though; usually whenever we change how something works, there are at least a few players complaining, threatening to leave the game, etc. In fact, whenever we are making something that was previously simple more complex, a lot of players take it almost as a personal insult. I always believed that making things more complex will add more to the game. I can't understand what is the fun in endlessly repeating the same actions over and over, especially simple actions such as putting some ingredients together and clicking "mix", then repeating the same sequence over again.

Game Crowding

One issue we faced as our player-base increased was that there were some major points of interest where at least 40% of the players congregated and spent most of their time there. The biggest point of interest were the storages since it facilitated "industrial manufacturing". What the players did was harvest a lot, bring the resources to the storage, then possibly trade resources with others (what better place for trade than near a storage?), and after they had all the ingredients necessary for whatever they were set to make, started to produce products in industrial quantities. Some were meant for selling to NPCs and make money, while other were meant for improving some skill, such as potions making, and finally, other products were used as a helper for a different skill (an example being the mana potions, used to increase the summoning skill by being able to summon a lot of monsters and animals over and over again).

That was all good and nice, except for the people with dial-up connections or slower computers that had the misfortune to be around the storages to just deposit/withdraw some items. Since so many players were in a small area, the dial-up users were very lagged and could barely move around. Besides, it made macroing trivial; a lot of players were sitting there with simple mouse recording programs and used them to withdraw raw materials from the storage, manufacture items, deposit the items to the storage, and then repeat the whole process again and again. This problem was partially solved by placing some interdiction areas near the storage NPCs, so that people will have at least move away a few tiles if they wanted to manufacture or summon. Many players did not like this change, but on the other hand they did realize that we had to do something about it, so they stopped complaining after a while.

Yet another problem was that some "wild maps" (i.e. no cities or merchant NPCs in there) were almost empty, while the city maps were always crowded. The only thing we could do about it was to place some relatively low-level monsters in there, such as a map full of snakes. That way, players that needed snake skins for summoning or for some god quests knew where to go. This measure did populate those maps a little more, but it didn't bring the change I was expecting. Another change was to add a few NPCs in there that were either part of some quests, or they were offering some unique services that no other NPC in the game offered (such as repairing damaged weapons). The most extreme measure to make those maps attractive was to place some storages that didn't have any manufacturing interdiction around them. This did indeed work, and it worked a little better than I had expected.

Selling Items

As I said earlier, just after the crash a player e-mailed me asking if we are willing to sell some in-game items for money. This was supposed to be our little secret but other players found out. While a few complained a little, we got more and more requests to sell in-game items for money. Eventually, everyone knew that we do that, and even though we didn't have any official price list, they already knew the prices. Months later, we finally decided to make that 'official' and put up a "shop page" with the price for the most common items. With the money we made we were able to 'pay' some of our volunteers (i.e. give them small monetary gifts), and a big part of the income will be spent soon for some more advertising and for renting a dedicated server once there will not be enough room for players on this server (I estimate this will happen in the summer).

About a month after selling the first item, a player approached us and asked if he can buy levels for money. I didn't know what to say, so I said "yes". It was supposed to be a really well guarded secret, but apparently the other players found out somehow. Then they complained a lot on the forums, and the vast majority disagreed with the practice, so we had to promise them that we will never again sell levels, experience, pick points, etc. in the game. I can't really blame them for being so outraged; I guess it is a little unfair for us to do that and many people view this practice as plain cheating.

What made me quit

Something I and many other players didn't like was the fact that the harvesting skill was too boring and it was very susceptible to macroing. All it took was to click on a resource, then click again, and again, and again. Then as you got overcrowded with the resource you were harvesting, drop it into a bag at your feet, harvest again, drop, harvest, drop, and so on. Not exactly the most exciting or dangerous thing one could do.

So I wanted to change this by adding some more excitement, while also preventing the players from macroing. What I did was to add various random events, triggered by the type of resource being harvested. I only put two types: vegetal and mineral. The random events were both good and bad. For example, good events included finding rare items, money, or getting more harvesting experience. Bad events included having your pickax break, loosing HP due to accidents, or, the most extreme, being teleported in various, often nasty places.

The random events had a low probability of happening, depending on the resource. For example, harvesting diamonds had a chance of about 1/150, while harvesting flowers had a chance of maybe 1/1000.
I was thinking that the chance is pretty reasonable, and was hoping the players will enjoy this change, as it was designed to add some spice into a boring activity while reducing macroing as well. The reaction from the community was mixed. About 20% of the players liked it, 30% didn't care, while the other 50% were more or less very mad. This wasn't new, or even unexpected. However, the intensity of the complaints was overwhelming. Someone went so far as removing one of our download mirrors. Fortunately that didn't affect us at all, we already had 2 high bandwidth, unlimited traffic mirrors. But on top of that, people were raging on our forums, threatening to quit, saying how much the game sucks now, and so on and so forth. Even players that were previously on our side whenever we made an unpopular change started to complain to me, saying they will quit playing if we don't change it back.

This is where I snapped, so to speak. At that moment I took the toll of all the previous complaints, hard work, and general dissatisfaction with the player base. The game wasn't going in the direction I wanted it to go. I was hoping to make a complex game, rather than satisfy the ego of power-players. Some of the previous quests I implemented were not very popular due to their not entirely medieval fantasy style. Nothing was going as I wanted it to go.

The decision to leave wasn't an easy one. Nevertheless, I knew that there are other developers that can carry on what I started, and I did say that I might come back in the future. It wasn't like abandoning the project, it was more like letting others take care of it, while I focus on what I wanted to do in the first place. I still reserved the right to come back and take care of the things if it didn't go well. And it didn't go really well, after I left, so I had to come back about a month ago, and take care of some stuff.

But more about what happened after I left, the new project I started, and other info about sharing code between the projects will be in the next article. Until then, enjoy!