State of the Snail - Debugging Hell

It has been a while! SnailLife work has been moving at a snail’s pace. Why? Because debugging the snail brain has turned into a highly demotivating, hellish endeavour. The snails make decisions and perform actions based on so much crap, that the simple log files I was using all long are just not cutting it anymore. The Laravel log for the original app has turned into 3 Laravel logs because I now have 3 independent apps (the front-end, BrainRunner, and BrainHub to coordinate all the brain runners). That then turned into individual log files per snail plus individual log files per brain task. And still making sense of why snails are choosing to behave in certain ways is nigh impossible - the brain and infrastructure issues I have been facing have been seemingly neverending.

Logging issues aside I have added some debug settings to help me narrow down issues. The BrainRunner app now has a snail debug config file which allows me to disable the following actions that would otherwise take place on each brain check:

More settings will be added, such as disabling sensory/short term/long term memories, for example. The “idle action check” is where the bulk of the “magic” happens, that is the biggest black hole.

Adding options to disable certain functionality has helped, but the whole thing has been demotivating to say the least. I got into this to simulate snails, not work on debugging tools. And yes, of course I realize when dealing with a convoluted system like this I should have known what to expect. I did sort of expect this…I just chose to ignore it until the last minute to work on more fun things ;) After putting in a great deal of effort to remain organized and do things right in my projects at work, I’ve allowed good behaviour to fly out the window at home.

Anyway, I have now moved logging out of local log files and into Loggly. It’s not done yet, but the main logs like Laravel logs from the BrainRunner and BrainHub, snail logs, and brain task logs are all in Loggly now (in addition to the local machine).

To send the default Laravel logs to Loggly for each app I added the configureMonologUsing method in bootstrap/app.php:

$app->configureMonologUsing(function ($monolog) use ($app)
    $today = date("Y-m-d");
    $logFileName = "logs/laravel-$today.log";
    $monolog->pushHandler(new MonoStreamHandler(storage_path($logFileName), MonoLogger::INFO));
    $monolog->pushHandler(new MonoStreamHandler(storage_path($logFileName), MonoLogger::WARNING));
    $monolog->pushHandler(new MonoStreamHandler(storage_path($logFileName), MonoLogger::ERROR));
    $monolog->pushHandler(new MonoStreamHandler(storage_path($logFileName), MonoLogger::CRITICAL));

    if (CoreUtility::InternetOn())
        $envlabel = strtolower(env('APP_ENV'));
        $maintag = "brainrunner_$envlabel";
        $envtag = "env_$envlabel";
        $logglyString = CoreUtility::buildLogglyPushHandler(array($maintag, $envtag));
        $monolog->pushHandler(new LogglyHandler($logglyString, MonoLogger::INFO));
        $monolog->pushHandler(new LogglyHandler($logglyString, MonoLogger::WARNING));
        $monolog->pushHandler(new LogglyHandler($logglyString, MonoLogger::ERROR));
        $monolog->pushHandler(new LogglyHandler($logglyString, MonoLogger::CRITICAL));
    return $app;

Aside from having all of my logs in a central place, Loggly lets me query the logs easily to get more relevant information. I use tags to distinguish between individual brain tasks, snails, apps, environments, etc.

But that is not enough. I feel to debug the brain in its current state having a visual representation of the brain would help a great deal. What originally made me think about this was my earlier experimentation with TensorFlow and TensorBoard. TensorBoard provides a visual of your TensorFlow graph and lets you interact with said visual to see how tensors flow between operations. The brain as I have it can also be represented as a set of nodes (each node representing a neuron), and I should be able to visualize the flow of inputs between neurons in a similar way. What if I could have a graph for every brain task that runs on every BrainRunner and see exactly what path each input took and why?

I think I will look into this next. As you can see I have not been doing much work on the actual snails here. But I think the return on investment in improved debugging tools will be worth it if I have any hope of getting any further with simulating a snail brain.

On another note it looks like ECAL 2017 ticket prices are out and I’d better start saving:

© - 2021 · Liza Shulyayeva ·