100 Days of Code - Days 3-4

For days 3 and 4 I continued working on my work project where I experienced both elation and extreme frustration within the same very short window.

In it’s current state the project is a bit of a mess, and there is some proprietary stuff that I can’t share at the moment. However, after I got the bare bones of the project set up, my next goal was to establish an SSH connection between the web application and a target system. This proved much easier than anticipated! Using the paramiko SSH library made it very easy to get up and running. For the project, I created a separate module that could be imported and included in the application factory. I added some custom parameters to the app.config, which I now load from a Python file. This will allow the application to be customizable since it will, in theory, be used for multiple jurisdictions and systems.

With the configuration in place, it’s a simple matter to get a connection established with paramiko:

import paramiko

from flask import current_app

def get_ssh():
    conn = paramiko.SSHClient()

    conn.load_system_host_keys()
    conn.set_missing_host_key_policy(paramiko.AutoAddPolicy)
    conn.connect(
            current_app.config['SSH_HOSTNAME'],
            username=current_app.config['SSH_USER'],
            password=current_app.config['SSH_PASSWORD']
    )

    return conn

That’s all I needed to establish a connection to the server! I return the connection from the function and store this within the application configuration. This way everything in the app can use a single SSH connection to the target server. One of the big issues with the previous version of this application is that it is CONSTANTLY SSHing into the server to pull data, which renders the /var/logs/auth.log file completely useless, since it’s completely filled with these connections. Establishing a single connection will eliminate this issue. I’m not certain if I’ll run into contention issues with this single connection, but it should be easy enough to establish connections as needed to handle the various aspects that will be monitored all at once.

I include this SSH connection into the application factory:

from . import ssh
with app.app_context():
    app.config.update(SSH = ssh.get_ssh())

The issue came up early on that I wasn’t closing my SSH connection properly when the program would exit, so it would push the connection into a TIME_WAIT status. I wasn’t sure what the best way to handle this would be, since I was just using CTRL+C to end the application when I was done testing. A quick Google search revealed the atexit library! Think of this as a finally statement for your program. I import it into the __init__.py file, and then register the task to call when the application ends. Since I stored the connection in the app.config, I add this to atexit and it appears that my problem has been solved.

atexit.register(app.config['SSH'].close)

With the connection mostly handled and stable, I moved on to the task of executing commands on the server and getting the output into the browser. Again, this proved easy at first. I was able to put together a new route and view that handled this without issue.

@bp.route('/cmd')
@bp.route('/cmd/<cmd>')
def cmd(cmd=None):
    if cmd:
        client = current_app.config['SSH']

        try:
            _, stdout, stderr = client.exec_command(cmd)

            stdout = stdout.readlines()
            stderr = stderr.readlines()
        except TimeoutError as e:
            flash(e)

        if stderr:
            flash(stderr)
    else:
        stdout = ['No command given.']

    return render_template('cmd.html', title=cmd, stdout=stdout)

{% extends 'base.html' %}

{% block title %}{{ title }}{% endblock %}

{% block content %}
    {% if stdout|length %}
        {% for line in stdout %}
            {{ line }}<br />
        {% endfor %}
    {% endif %}
{% endblock %}

Very simple, ugly, but straight forward. Essentially the route takes in a command, with zero validation, grabs the SSH connection from the configuration, and executes it. It grabs stdin, stdout, and strerr from the SSH stream. If we get an error, we use the flash() method to display that which is picked up in the base.html template. Otherwise it passes a list of lines from stdout to the view where it parses every line and adds a break. This works great, and I was super happy that it was so easy. Obviously I’ll need to add some validation methodologies here. We wouldn’t want anyone being able to execute any command, but this will be the foundation for future additions. And thus ended day 3.

Day 4 was the day of frustration. The primary function of this application will be to provide users an interface with which they can monitor log files in real time. These are simple text files that are written to by the system and alert the user of abnormalities. I had my SSH connection, and I was able to execute commands… How hard would it be to stream files in real time to a web page? As it turns out, much harder than I thought.

In Linux, it’s easy to just run tail -f on a file and have it stream out new contents to the console. However, executing this through paramiko never releases the stream back to its caller. Essentially, you never get data back out to stdout to be captured and sent to the web app. I wasn’t sure what to do so I looked up some possible solutions, one of which I really liked. It involves a generator, something I’ve never used or heard of in Python before, but now I really like the idea behind. Unfortunately through my trial and error I was never able to get it working completely. I was able to stream the file up to a certain point but then never got any updates (I had to get JavaScript involved to attempt to run dynamic updates from the streaming data). I was able to continuously stream the data, but each iteration and call to the generator duplicated the file contents into the XMLHttpRequest I was trying to use. SO I could stream new data, but it kept adding lines to the page, continuously growing out of control. I was able to get this working in a fashion by constantly calling a function to open and stream all of the contents of the file, but this seemed inefficient. It was the best I could do before walking away in frustration however.

What I believe needs to take place is this:

1. Have the generator start at beginning of the file and stream through the whole file, returning that to the web app.
2. Update what line/byte the generator is at and have it pick up where it left off in subsequent calls, only returning new content from the file to the web app.
3. Have the web app append each line it gets from the generator into the contents of the page.

This seems simple enough, I’m just not certain how to pull it off yet. I’ve been kicking around a few ideas over the weekend, and when I get back to work on Monday I believe I’ll be able to get this figured out. Until then I’ll be giving my brain a little rest with some easier problems and some board games over the weekend.

Tomorrow I’ll write about what I was able to accomplish tonight doing some updates to the blog. Until next time…