Long Stache

I've been working with Logstash a lot at work lately, and I noticed something about the plugins. There's both an HTTP input plugin and an HTTP output plugin. That gave me a genius idea: Chain a bunch of Logstash instances together, and see what happens!

So let's start with the Logstash pipeline configuration. It's pretty simple actually. I don't care about processing any data at any point in the pipeline of pipelines, so it's only 18 lines long.

input {
  http { } 
}

output {
  stdout { codec => rubydebug }

  http {
    http_method=>"get"
    url=>"${OBSERVER}/ls/${LSNAME}"
  }

  if("${NEXT_CONTAINER}" != "") {
    http {
        http_method=>"get"
        url=>"${NEXT_CONTAINER}"
    }
  }
}

As you can see the HTTP input plugin has no options defined, the defaults work just fine. The stdout plugin is just for debugging, in case I need to diagnose any issues. The first HTTP output plugin sends data (just a ping basically) to an observer app that I'll get into later. The second HTTP output plugin sends data to the next Logstash instance that's defined in an environment variable. Obviously given the if statement, if there is no next Logstash instance, then it doesn't attempt to send any data. This is important because Logstash is designed to guarantee at least one delivery to the output destination. And since there is no place to deliver data after the last Logstash instance, the last Logstash instance will continually try to send the data over and over again because it isn't receiving a successful response. That would cause the last instance to get bogged down.

Next I built a Logstash Docker image with this configuration file inside. I realize now writing this that I could have just used the default image and mounted a pipeline directory with the config file inside instead, but oh well doesn't matter.

I then threw together a quick Python script that would create as many Logstash containers as I specified, and then shut them all down when I hit CTRL + C. If you're curious you can look at it here, it's nothing special.

Next I had to decide how I wanted to receive all the messages from the Logstash instances. I decided the easiest solution would be to have a NodeJS app running on the same Docker network that the Logstash containers are running on, and then to have that NodeJS app communicate through a websocket to a web app in the browser. And for the web app I just went with Svelte because I've been using it recently.

Here are the key parts of the NodeJS observer app. To start I just serve the Svelte app at the root, and serve the requested assets as well.

app.get('/', (req, res) => {
  res.sendFile(new URL('./frontend/dist/index.html', import.meta.url).pathname);
});

app.get('/assets/:assetName', (req, res) => {
  res.sendFile(new URL('./frontend/dist/assets/' + req.params.assetName, import.meta.url).pathname);
})

And then when the app receives any requests at the /ls/[Logstash App Name] route, it will forward the number of the Logstash instance that is in the Logstash app name to the web app through a websocket.

app.get('/ls/:lsName', (req, res) => {
  const lsNum = Number(req.params.lsName.substring(2));

  console.log('ping from: ' + lsNum);

  io.emit('ping', lsNum);
  res.json('success');
})

For the Svelte app I made a row of 10 black boxes. When the web app receives a message through the websocket, the nth box in the row lights up to the corresponding nth Logstash instance. The code is nothing special, just some quickly put together HTML, CSS, Svelte, and Javascript. You can look at it here if you're curious.

Here's a basic MSPaint diagram if you're confused about the setup.

So with all that written and set up, It's time to test it out!

So I've got a working setup. Next I added a little bit more styling to make it look cooler. I also added a button that sends a request to the NodeJS app, that then sends a message to the Logstash pipeline of pipelines.

10 Logstash instances is kinda lame though… How about 64 instances?

So I bumped the number of Logstash containers to make in my Python script and attempted to start them all. Each Logstash instance uses about a gigabyte of ram, and I have 64 gigabytes. So I should have just enough to run 64 containers. The first time around it only launched 56 containers, then the Docker daemon stopped responding and the script exited. I closed the containers manually, then I added a 10 second delay in my Python script between starting containers. Logstash does a lot of processing in the beginning, but then idles down after everything is initialized. So trying to launch them all one after the other was probably a little too much for my Ryzen 5 2600X.

The second time around I got up to 39 containers. But the everything froze. I tried to switch TTYs but all I got was a blinking cursor. I waited 15 minutes before forcefully shutting down my computer.

Third time's the charm. With the computer freshly rebooted, I should hopefully have most of my resources available for getting this monster pipeline of pipelines running.

First I started up the NodeJS and web app. Then I ran the Python script, and the Logstash containers started to run, one every 10 seconds. With everything set up, I clicked the button for the moment of truth.

Slowly, choppily, but surely, the message made it all the way through! I showed the docker stats output as well at the beginning so you don't think I'm pulling a trick on you with just CSS. When the first message makes it's way through it's pretty slow. But subsequent messages go faster. I guess just because the Logstash instances have woken up and are ready to receive more messages after the first.

Here's the the visualization of the pipeline of pipelines, along with the system monitor.

As you can see, as one message makes it's way through, the CPU usage goes up a little. But then once I start spamming that button, CPU usages shoots up to 100% and the video frames start to suffer. But they do make it through, Logstash guarantees it.

Sometimes when I talk about computers, I get asked "Why do you need that much RAM?" or "Why do you want a CPU with that many cores?" This is why. When I want to make a Logstash Centipede, I don't want my computer holding me back!

This was a pretty fun little project, and a creative way to light up 64 cubes while using up all my RAM and CPU. My only regret is that I didn't have a more powerful computer :-)