Web-Based Real-Time Application Analysis

Hi all,

To analyze a real-time application you typically have to write a custom solution. Think about a developer console, or a transparent head-up display (HUD). But that can be quite a bit of work even it you try to keep things simple. The poor man's option is often to use a log file to output diagnostic information, and a configuration file for input parameters. But that's limited to just text and clearly not interactive...

So what I'd like to present here is a solution that is as powerful as a console & HUD, but almost as simple as a log & configuration file. The key is to use existing web technology. By letting a browser take care of the interface rendering you get a lot of possibilities, just by writing a bit of HTML code. To make it interactive, you need an embedded server that will serve the web page and responds to input.

So were do we find a simple embeddable server? Actually, it's really easy to write our own!

The first thing we need to do is to open a network socket and make it listen for TCP/IP connection requests. In Windows, we can use Winsock for this (similar API's exist for other platforms). Note that you'll have to link in Ws2_32.lib before you can make use of Winsock. The code for opening a socket looks like this:

// #include <winsock2.h>
// #include <ws2tcpip.h>

SOCKET listenSocket;

WSADATA winsocksData;
WSAStartup(MAKEWORD(2, 2), &winsocksData);   // Initialize Winsocks 2.2

addrinfo hints = {0};
hints.ai_family = AF_INET;         // Internet Protocol (IPv4)
hints.ai_socktype = SOCK_STREAM;   // Sequenced, reliable, byte stream
hints.ai_protocol = IPPROTO_TCP;   // Transmission Control Protocol
hints.ai_flags = AI_PASSIVE;       // Listen for connection requests

addrinfo *info;
getaddrinfo("localhost", "80", &hints, &info);   //   localhost, port 80 (HTTP)

listenSocket = socket(info->ai_family, info->ai_socktype, info->ai_protocol);   // Create socket
bind(listenSocket, info->ai_addr, (int)info->ai_addrlen);   // Bind it to the address
listen(listenSocket, 1);   // Start listening for one connection

This may seem a little complicated at first, but this is just some one-time initialization code. Note that sockets are extremely powerful and you can use practically the same code for lots of other communication protocols between various applications and devices. So now that we have a socket that listens for incoming TCP/IP connections on localhost, we have to actually wait till a connection request arrives. This will happen when you open a browser and type the URL http://localhost, possibly followed by a path to a resource. Waiting for an incoming connection is done by the accept function:

SOCKET clientSocket = accept(listenSocket, 0, 0);

Note that this will freeze execution. Or more precisely, it suspends the current thread until a connection is accepted. So in an actual real-time application you will want to put this and the remaining embedded server code in its own thread. Note also that the accept function returns a new socket. That is to allow multiple clients to connect to a single server, each with their own independent data stream. Here we're accepting only one client though.

Once accept returns, you can start exchanging data with the browser! You only need to know two more functions; recv and send. As the names already imply, recv will wait for a request from the client, and send can be used to reply. All communication happens using simple byte strings, and the HTTP protocol uses messages that are fairly easy to read. So let's receive our first request message:

char request[65536];
int bytesReceived = recv(clientSocket, request, 65535, 0);

This function again freezes until the client actually sends something. But when you type for instance http://localhost/panel into your browser and hit enter, the accept function will first establish a communication session and right after that the recv function will return the following text in the request buffer:

GET /panel [...]

The [...] is a bunch of additional information the browser decided to send us, like the browser name and language, and what formats it accepts.

So let's reply with "Hello World!":

send(clientSocket, "HTTP/1.1 200 OK\r\n"
                   "Content-Type: text/html; charset=UTF-8\r\n"
                   "Content-Length: 25\r\n"
                   "\r\n"
                   "<html>Hello World!</html>", 104, 0);

You should now magically see the words Hello World! in your browser!

The first line of the response message tells the browser that the request was understood and this message contains the reply. The second line tells it the reply consists of html code, encoded using UTF-8 (a popular ASCII extension that can also represent Unicode). This is followed by the length of the body. Next is another carriage return and newline to separate the headers from the body of the message. And finally we have the HTML code (not standards compliant, but most browsers will take it anyway). The send function also takes the client socket and the total message length as a parameter.

From this point forward the possibilities are almost endless. HTTP and HTML are fairly simple so I'll leave it up to you to decide what you want to do and explore tutorials and documentation. To make things interactive you probably want to create a form with a submit buttom. This is what SwiftShader 2.0 does. Just run the demo and point your browser to http://localhost:8080/swiftconfig (and have a peek at the HTML code if you're not sure how it's done). With a form the browser will send the server a "POST" request, containing the enabled checkboxes and selected drop down box elements. Beware that some browsers will send that in a second message. After parsing just resend the entire page with appropriately selected controls.

Note that the essential code is really just a few dozen lines, hardly any longer than for reading and writing configuration and log files. You now have all the power of HTML at your fingertips to create any interface you like. You can even change your application's behavior remotely (try with your smartphone)!

You can also get back diagnostic information from your application. However, when you use forms you'll have to click a button to get updated information. That can be annoying, especially when you're waiting for a specific event to happen. Also, forms require redrawing the entire web page. So isn't there a way to automatically get regular updates, without reloading the entire page?

Absolutely, but for this we'll need JavaScript. So make sure you're fully comfortable with using HTTP and HTML first. The key to dynamically update your web page is the use of the XMLHttpRequest JavaScript API. It can be used to make an HTTP request from within JavaScript, and the response can be plain text or XML. The latter case gave birth to a suite of techniques known as Ajax, but here I'll keep things simple by using the plain text version to create a framerate counter. Here's the essential JavaScript code for making a request:

var xhr;
function request()
{
    xhr = new XMLHttpRequest();
    xhr.open('POST', 'http://localhost/panel/fps', true);
    xhr.onreadystatechange = update;
    xhr.send();
}

Notice that we're making a "POST" request. This avoids any browser caching. Next comes the URL for our framerate service. The last parameter to the open function specifies that we want to make an asynchronous request. This means the browser won't halt waiting for the server's response. Instead, it will call the onreadystatechange function when it receives something. Here we set that function to our "update" function, which I'll explain below. Up to this point no actual request has been made yet, we've just set up our XMLHttpRequest object. The send function sends it off to the server. Here's the update function, responsible for refreshing part of the web page with the data we receive:

function update()
{
    if(xhr.readyState == 4 && xhr.status == 200)
    {
        document.getElementById('fps').innerHTML = xhr.responseText;
        setTimeout('request()', 1000);
    }				
}

The XMLHttpRequest object will call the onreadystatechange function (which we've associated our update function with) multiple times before the complete response has been received. That's why we first have to check the readyState and status properties to make sure that we can use the response. The plain text sent by the server will be available in the responseText field.

So we want to assign that string to a certain HTML element where the framerate should go. This can be achieved by navigating the Document Object Model. The DOM is a tree-like structure representing all the elements of an HTML (or XML) document. In this case we locate the element that has the "fps" id and we overwrite the HTML contained within that element with the server response containing the framerate as a string.

Finally, we also make the script call the request function again after 1000 milliseconds. This way you'll see the framerate get updated automatically every second. Note that this is a 'pull' approach. If the sever has some updated data sooner it can't send that to the browser before it asks for it. However, an alternative solution is to immediately make the next request, and have the sever send it when ready. Anyway, on a local network you can have hundreds if not thousands of updates per second without stressing anything.

So finally lets put it all together:

#include <winsock2.h>
#include <ws2tcpip.h>
#include <math.h>
#include <string>

class Server
{
public:
    Server();

    ~Server();

private:
    enum Status
    {
        OK = 200,
        NotFound = 404
    };

    void loop();
    std::string page();
    std::string fps();
    void send(SOCKET client, Status code, std::string body = "");

    SOCKET listenSocket;
};

Server::Server()
{
    WSADATA winsocksData;
    WSAStartup(MAKEWORD(2, 2), &winsocksData);

    addrinfo hints = {0};
    hints.ai_family = AF_INET;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_protocol = IPPROTO_TCP;
    hints.ai_flags = AI_PASSIVE;

    addrinfo *info;
    getaddrinfo("localhost", "80", &hints, &info);

    listenSocket = socket(info->ai_family, info->ai_socktype, info->ai_protocol);
    bind(listenSocket, info->ai_addr, (int)info->ai_addrlen);
    listen(listenSocket, 1);

    loop();
}

Server::~Server()
{
    closesocket(listenSocket);
    WSACleanup();
}

bool match(char *url, char *string)
{
    return strncmp(url, string, strlen(string)) == 0;
}

void Server::loop()
{
    SOCKET clientSocket = accept(listenSocket, 0, 0);
    char request[65536];
    int bytesReceived;

    do
    {
        bytesReceived = recv(clientSocket, request, 65535, 0);

        if(bytesReceived > 0)
        {
            request[bytesReceived] = 0;

            if(match(request, "GET /panel "))
            {
                send(clientSocket, OK, page());
            }
            else if(match(request, "POST /panel/fps "))
            {
                send(clientSocket, OK, fps());
            }
            else
            {
                send(clientSocket, NotFound);
            }
        }
    }
    while(bytesReceived > 0);

    closesocket(clientSocket);
}

std::string Server::page()
{
    return
    "<!DOCTYPE HTML PUBLIC '-//W3C//DTD HTML 4.01//EN' 'http://www.w3.org/TR/html4/strict.dtd'>     \n"
    "<html>                                                                                         \n"
    "    <head>                                                                                     \n"
    "        <title>Panel</title>                                                                   \n"
    "    </head>                                                                                    \n"
    "    <body>                                                                                     \n"
    "        <p>FPS: <span id='fps'>-</span></p>                                                    \n"
    "        <script type='text/javascript'>                                                        \n"
    "            var xhr;                                                                           \n"
    "            request();                                                                         \n"
    "            function request()                                                                 \n"
    "            {                                                                                  \n"
    "                xhr = new XMLHttpRequest();                                                    \n"
    "                xhr.open('POST', 'http://localhost/panel/fps', true);                          \n"
    "                xhr.onreadystatechange = update;                                               \n"
    "                xhr.send();                                                                    \n"
    "            }                                                                                  \n"
    "            function update()                                                                  \n"
    "            {                                                                                  \n"
    "                if(xhr.readyState == 4 && xhr.status == 200)                                   \n"
    "                {                                                                              \n"
    "                    document.getElementById('fps').innerHTML = xhr.responseText;               \n"
    "                    setTimeout('request()', 1000);                                             \n"
    "                }                                                                              \n"
    "            }                                                                                  \n"
    "        </script>                                                                              \n"
    "    </body>                                                                                    \n"
    "</html>                                                                                        \n";
}

std::string Server::fps()
{
    char buffer[256];
    sprintf(buffer, "%f", 45.0 + 15.0 * sin(timeGetTime() / 1000.0));
    return buffer;
}

void Server::send(SOCKET client, Status code, std::string body)
{
    std::string status;
    char header[256];

    switch(code)
    {
    case OK:       status += "HTTP/1.1 200 OK\r\n";        break;
    case NotFound: status += "HTTP/1.1 404 Not Found\r\n"; break;
    }

    sprintf(header, "Content-Type: text/html; charset=UTF-8\r\n"
                    "Content-Length: %d\r\n"
                    "Host: localhost\r\n"
                    "\r\n", body.size());

    std::string message = status + header + body;
    ::send(client, message.c_str(), (int)message.length(), 0);
}

int main()
{
    Server server;

    return 0;
}

You'll have to link with Winmm.lib for the timeGetTime function. This application produces a sine shaped "framerate". Just run it and open http://localhost/panel. In this case the HTML code does pass conformance validation. Also, if the server gets an invalid request it will respond with status code 404. This may occur when some browsers make a request for the vaficon.

That said, this code may still not work under all circumstances. The XMLHttpRequest API has only recently become available on all popular browsers, so you may want to update your browser to the latest version or look for backward compatible implementations. I also didn't add any kind of error checking, so if you need this to be more robust check the documentation of the used APIs. And of course you may want to extend this with convenient methods to extend the HTML code with certain controls and asynchronously updated diagnostic information. You can even update images to create things like graphs...

Either way I hope I've convinced you that with relatively little code you can get access to a wide range of possibilities to diagnose and control you application with a neat interface. Please let me know if you use this anywhere, or if you know any other web based technology that is useful during development!

Cheers,

Nicolas "Nick" Capens

A big thanks goes out to the people in the Framerate in a browser thread for helping me get started with asynchronous updates!

Very inspiring, thanks!

_{Offtopic: I never bothered to do more Javascript than an occasional assignment for my study, but I'm very surprised that it allows (or forces?) things like "xhr.readyState == 4". Now that is ugly!}

Oh my God, so application analysis is what it was all about :> Although I really like the nice tutorial on implementing quick-and-dirty dynamic web servers I really don't think it's worth it given that creating another form with .NET (for example) takes two lines of code. As always though, you manage to demonstrate a very large number of concept within a few lines and you do it very clearly. And I'm serious on it, it's not just blabla nice job blabla.

Originally Posted by roel

I'm very surprised that it allows (or forces?) things like "xhr.readyState == 4". Now that is ugly![/sub]

Yeah, the specification defines symbolic names for these constants, but no browser actually implements them. I don't think it's really that ugly though. Whether it's a number or a symbolic name you still have to look at the documentation to write the correct code. Also, you could define the symbolic names yourself, but what exactly would you win by doing so?

You may want to add a clarifying comment though...

Originally Posted by Mihail121

Oh my God, so application analysis is what it was all about :>

Actually it can be used for more than that. For instance you could create a test suite that can be accessed from all over the world (checking status and starting new test runs). And I'm sure there are many other creative uses.

But yeah, personally I'm using it to analyse and control real-time applications.

I really don't think it's worth it given that creating another form with .NET (for example) takes two lines of code.

Sure, in many cases that's a quick and easy solution. But it has some limitations. First of all you need a visual framework like that, which isn't always an option. Also, it wouldn't work remotely (or at least not with two lines of code). And when you're creating just a component instead of a complete application you often lose control over keyboard and mouse input.

Originally Posted by Nick

Actually it can be used for more than that. For instance you could create a test suite that can be accessed from all over the world (checking status and starting new test runs). And I'm sure there are many other creative uses.

But yeah, personally I'm using it to analyse and control real-time applications.

Yes, it certainly adds another perspective to browsing and abstracts the idea of "what" is being browsed. In that sense, it's an innovative and therefore interesting work. Besides, it's implementable with a few strokes.

Sure, in many cases that's a quick and easy solution. But it has some limitations. First of all you need a visual framework like that, which isn't always an option. Also, it wouldn't work remotely (or at least not with two lines of code). And when you're creating just a component instead of a complete application you often lose control over keyboard and mouse input.

I was about to point at X11 before reading the "not with two lines". After reading, I agree, although technologies are evolving fast and it's certainly possible (using a decent framework) to achieve the same although it will not reach that many people as your solution. I'm personally interested in trying it out with Lynx.

Originally Posted by Nick

Yeah, the specification defines symbolic names for these constants, but no browser actually implements them. I don't think it's really that ugly though. Whether it's a number or a symbolic name you still have to look at the documentation to write the correct code. Also, you could define the symbolic names yourself, but what exactly would you win by doing so?

You may want to add a clarifying comment though...

Ok: I skimmed through your code, and thought: "what on earth could these weird numbers mean?". Then I read the documentation. If the line would have been something like if (xhr.readyState == XMLHttpRequest.DONE && xhr.status == HTTP.OK) I'd probably not even noticed that line.

I see this as a tremendous debugging utiliy, you run your engine on the main pc and with an internet connection you can debug on another pc connected trhough the net , both local or global, brilliant...