A Major Change in Game Design

As I mentioned in my previous article, in February 2004 we decided to fix the game a little bit, by doing a partial reset (all the items gone, some of the experience scaled down, and the attributes removed as well). This was caused mainly by poor design, because we were too busy working on the technical stuff, such as programming and art, and we neglected the gameplay. It wasn't the smartest thing, but given the fact that we had such limited resources, there was no other way.

So, an announcement was made in our forums and in the game. Basically, we had to tell our user base something they already knew: the game is not going well, and the only way to fix it is a partial reset. They took the news surprisingly well, but of course, some players didn't like the idea and left the game. Others decided to take a leave until the restart, since they figured what's the point in playing since a lot of their work will be left. Before the announcement we had about 250 players online in the afternoon, but after the announcement the population started to decrease day by day. A few weeks after, we barely had 150 players online.
Meanwhile, we were working hard at balancing the items prices, and also adding some new content to the game.

The first idea we came up with was to introduce the concept of "knowledge", which means that in addition to skills and materials, you will also need the knowledge to do various things, such as harvesting, crafting, summoning, and even fighting. The only way to acquire knowledge was by reading books. Some of the books were sold by merchants, and some others could be obtained only from monsters, as a drop. Each book 'self destroyed' after being read, so you couldn't read a book then sell it to someone else, since that would have defied the purpose of removing some items from the game. Another idea was to create the concept of "overall experience", which means that whenever you gain some experience in any field, that experience is added to the overall skill. Whenever you gain a new overall level, you get a "pickpoint" which can be used to increase an attribute, a nexus, or chose a perk. The perks idea was shamelessly 'borrowed' from Fallout. Of course, we used different names and effects. In addition to that, we also introduced the concept of negative perks. Basically, a negative perk gives (instead of taking) pickpoints. Some examples of negative perks include: "hellspawn", which gives you 10 pick points, but every time you enter a new map, or enter/exit a building, you have a 1/5 chance to end up in hell (the place where you go after you die). You don't lose anything, but it can be very annoying nevertheless. Another negative perk is "antisocial", which basically prevents the NPC merchants from buying or selling stuff to you. There are about 10 negative perks, and close to 25 positive perks. Some positive perks give you the ability to fight better, some give you the ability to harvest faster, etc.

Some of our players are very creative when it comes to using the perks. For example, a guild called "Bombers" has only members that have at least two positive perks: "Selfdestruct" and "Mirror Skin". Selfdestruct causes 5 points of damage, regardless of defence, to any opponent the player is fighting at the moment he/she dies. The "Mirror skin" perk gives you a small chance of reflecting incoming blows, causing damage to the enemy that attacked you. The strategy of that guild is to go in PK maps, in large numbers, and attack strong players. As they die, they inflict damage, and a horde of newbies with those two perks can kill even the most skilled fighter.

Anyways, 5 weeks after the announcement was made, we had the new version ready to be tested. So we set up a new server on a different port, distributed the new client, and asked our players to come and help us test the new system before releasing the official version. This test was a very good idea, because our players discovered a lot of bugs, many of them critical. We've been very lucky that they did report all the bugs they found, rather than keeping them secret and abusing them once the official version is released. Initially, the testing server was supposed to last for one week, but due to the overwhelming number of bugs and other problems, we had to test it for 2 weeks. We did tell the players that there is no fixed release date, and we will release the new version when it's ready, so we were able to take our time testing for problems. The most annoying bug was in the new combat system, which had to be totally rewritten, as the old combat system didn't allow multiple opponents.

There were actually two bugs:

1. Every once in a while a player would take damage even though s/he wasn't fighting with anyone.
2. Every once in a while a player was "locked in combat", so s/he couldn't move or do anything else, even while not fighting.

Those two bugs required a lot of debugging using different approaches, such as putting a lot of tests throughout the code and dumping data in files or on screen. Eventually, a few days after they were reported I fixed them, but it was a very frustrating experience.

In late April, the new version was up. Most of the feedback was positive, as the players had a lot of new things to do, in addition to restoring their lost wealth. As expected, the number of online players started to increase again, but we didn't reach the numbers just before the announcement. Even after a few weeks, we had only about 200 players on, which was quite frustrating and made me wonder whether or not this reset was a good idea.

The Scripting Engine

Meanwhile, after we finished tuning up the new system, I started to focus on a long dream of mine: Having a scripting system embedded in the server. The whole scripting thing was new to me, I had no experience when it came to embedding a scripting language. If in a single player game the speed of the scripting engine does have to be fast, in an MMORPG the speed is critical. So, obviously, I was looking only for the scripting languages that compiled their script in bytecode so it can be executed relatively fast. Initially, I wanted to write our own scripting engine including the virtual machine (VM, which interprets the bytecode). However, after about 2 weeks of research, I realized that making a scripting language would take a lot of time, and most likely it won't be as good as a mature scripting engine. After all, we are a team of game makers, and we had to focus on the game. The next few weeks I had to evaluate a lot of scripting languages. A lot of people recommended Lua, but after looking at it's syntax and other characteristics such as speed in addition to how easy it is to embed it, I decided against it. I also considered PHP, but there was no good documentation on how to embed it in an engine. Besides, it wasn't as fast as I wanted it to be. There were a few scripting languages that I liked, such as some interpreted C library (that had a VM as well), but the license was a problem: it was GPL, and we couldn't live with the GPL license. Java and JavaScript were also considered, but I didn't really like Java, and there were also some reports about security problems and speed issues.

After evaluating all the options, I remembered that a few years ago an ex-colleague of mine told me about 'Small', a C-like, typeless language. After reading it's manual, embedding guide and so on, I have decided to use it. Embedding it into the engine was extremely easy, which took only a few days to link it with my engine, and another few days (maybe a week) to write all the C wrappers for the functions that I needed the Small script to call. The scripting language was meant to serve 3 objectives:

1. Make it easier to write quests.

2. Have the quests out of the game engine, so if we sell a license of the game, the people that buy the license will be able to make their own quests, without having to wade through thousands of lines of code.

3. Making it more secure for the quests programming. If there is a bug in the C code, that can easily crash the entire server. If there is a problem with the Small script, the consequences are far less drastic.

The Newbie Tutorial

One of the first things that had  to be done was a new newbie tutorial. The old one was pretty much a failure; the new one was sublime but was completely missing. Newbies were asking for such a tutorial, and many were leaving the game because they didn't know how to play the game. So now that the scripting subsystem was working, I had decided to finally do that very important thing and write a tutorial. We decided to keep it simple, but at the same time have enough info to give the user an idea about the game and how to play it. The tutorial consisted of 3 parts:

• The first part was a "get familiar with the maps". Basically, the player had to go through the first 2 maps, find the local taverns, and touch the counter of each tavern in the map.

• The next step was harvesting 5 different flowers from the starting map. This was intended as an introduction to the harvesting skill, and some minor exploration of the starting map (which is pretty small).

• The 3rd and last part was the fighting. Basically, the player had to gather 20 rabbit furs. The rabbits are very common animals, so it is easy to find them. Being rabbits, it's also easy and relatively safe to kill them. Of course, I didn't bother to check whether the players actually fought those rabbits or just got their furs from someone. After all, it was meant as a tutorial, so cheating on it is not really the smartest thing to do if you are a newbie wanting to know how to play the game.

Upon completion, a player gets the worst armor and weapon available in the game. This was added so the players will stop begging for items, and to give them a reason to complete the tutorial. Plus, if a player has some items he or she will be a little less likely to leave the game. In addition to giving you those 3 small quests, the tutorial NPC was also designed to respond to various questions about how to play the game, how to navigate between maps, and various other useful tips and advises.

Getting a Job

After the scripting engine was implemented and tested, the next step was adding some new quests. Initially, my idea was to add a lot of quests, which would tell the background story and entertain the players. I was thinking of adding a new quest every 2 weeks or so, which seemed a reasonable time. However, during that time I was working at the game more or less full-time. In June I got a full-time job (as an embroidery operator, no less!). So I had less and less time to work at Eternal Lands. Usually, when you come home from your 8 hours job, you are not really in the mood for programming. Besides, there are other things to do as well, like eating, sleeping, and so on. The time allocated for the game was maybe 2-3 hours/day, which was not enough. However, working at an MMORPG is not only programming. You also have to do some advertising, public relations, dealing with the players, read/moderator the forums and the game, and many other things you don't even think of before you actually have a working game full of players. So, unfortunately, I didn't really have much time for the quests; there were many other server related things I had to work at, such as improving various routines, seeking and fixing bugs, and write code for other functions that were to be implemented. I wanted to assign another developer with writing quests, but we were such a small team that all the developers were busy over their heads with other things, such as client work. Besides, all of them had other things to do as well, such as jobs, school, social life. But I always planed to eventually go back to writing quests, and have a very deep game that can also be considered a RPG, with a story line, characters (NPCs) connected to each other, and well, something like the good old RPGs used to be. In fact, even now I hope that, eventually, I will find a good quest writer and programmer that will complete this vision of mine.

The Proxies Problem

A previous user, who was banned multiple times for various offences most of them including spamming in the game and being rude in general, figured a way to use anonymous proxies in order to get access in the game. I don't know what kind of proxies he was using, but I assume they were Winsocks. This was a real problem because I wasn't online all the time, and not many people in our game have the privilege necessary to ban people. So there were sometimes playing when no one that could ban was online, and that player was disrupting the game. We did think of various ways to block proxies, but none of them was really effective. The most obvious one was, at character creation time, try to log to the IP of the player on some ports, and see if you can connect or not. If you can, most likely it's a proxy, so place that IP on the banned list. However, the problem was that our server is not threaded, so by doing such checks everyone in the game would have been slowed down considerably whenever someone created a new character. Not to mention wasting a lot of bandwidth in the process.

Another idea, which we eventually implemented, was more elegant: The http://opm.blitzed.org/ way! From their web page, here is how it works:

"Anyone can query our DNSBL through normal DNS means. Just reverse the octets and do a name lookup. For example, to check if 127.0.0.2 is present in opm.blitzed.org, do a DNS lookup on 2.0.0.127.opm.blitzed.org. Each entry in the DNSBL has an A record and a TXT record associated with it, the TXT record contains a URL to the proxy information page specific to that IP address telling the user a little information about how to sort out the proxy."

Checking each user via this methods is still kind of slow, and we don't want to swamp that server either. It is, after all, a free service, and we don't give anything in return. So we wanted to make it as less resource intensive as possible. Therefore, we just gave our newbie helpers (which are a group of relatively trusted players) access to a server command which allows them the check if a player is using a proxy or not. If that command determines that a player is indeed using a proxy, than that player is banned. So it is a very simple and effective solution, and it worked quite nicely.

The Great Crash

One morning before going to work, a few weeks after starting my full time job, I took a quick look at our forums, and noticed that the server was down (a lot of users asked what was going on). I wasn't really alarmed by this problem, because sometimes (although very rarely) there are all kind of problems with various Internet routes and routers. However, when I came home I noticed that our server admin posted in that thread that our server suffered a critical harddisk failure, and all the data was lost. The good news was that he replaced that machine with a better one (Dual P3, 1.1GHZ). The bad news was that we had no backups whatsoever. So all the player files, the guild files, and pretty much all the dynamic data, plus the logs and ban list was gone. I know, it was really stupid of us not to have backups. In fact, I did have some CORRUPTED tar.gz file that contained all the users, about 2 weeks old. However, for some reason it got damaged while downloading it, and I didn't bother to check for it's integrity, assuming that it's intact and that nothing wrong will happen with the server.

So I had to tell our users about this bad news. I was expecting 90% of them to leave and never come back. Needless to say, I wasn't really thrilled. Surprisingly though, they didn't complain as much as I expected. Reinstalling the game on the new server was relatively easy (there were a few problems with the headers, some missing libraries, and the scripting engine), and it took about 4 hours. So the server was up again that evening, sans the aforementioned things that were lost. The amazing thing was that immediately after we restarted the server we had about 200 players online, which was kind of the record after the previous character reset. In the following days, we started to have more and more players online, around 250-270 on average, but exceeding 300 in the weekends. I can never explain why exactly this happened. My best guess is that some of the players that abandoned the game before finally found a new reason to start playing, since everyone had the chance to become the best.

Of course, many (maybe 20%) of our veteran players did quit the game, but it could have been much worse than that. Some players assumed that the crash was deliberate, and I did that to complete the reset we had in April. However, that was really not the case, it was a total accident. But the truth is, in retrospective, that crash was a good thing, because we were finally able to get rid of the mistakes of the past, that were not completely erased by the reset.

Another problem associated with the crash was that our banned IPs list was gone. Some old troublemakers took advantage of it, and for a few days they were able to come back in the game and disrupt it again, by generally doing the same things that caused them to be banned. While banning someone requires only a few key strokes, it does take some time because you want to be sure you are banning someone who is actually causing trouble. So you have to go and investigate the problem yourself.

Someone also took advantage of the fact that all the characters were gone so he took a lot of names that didn't belong to him. That took quite a while to fix, by banning the offending IP range, and restoring those characters to their rightful owners. Since none of the previous records were available, we had to use the forums to see if their names/e-mails/IPs are matching. Sometimes, if that user had no previous forum account, we just went by their friend's word.

Obviously, the players were afraid that a crash like this might happen again, and many said that if it does happen and they lose their characters, then they will give up forever. I can't really blame them, no one really likes it when something like this happens. So we gave them our word that from now on we will have weekly backups, on remote computers. And as you will see in the 4th part of this article, the backups were really useful, they saved our skin at least once!

Well, this concludes part III of the series. In the next part, I will mention some related problems that happened, how we solved them, how we started to make some money from the game, and a summary of our biggest mistakes.