Whenever I share data between programs I often write the data out to disk. But what would the relationship between the two programs look like if I want to stream the data instead of write it out to disk? Using the subprocess module in Python we can pass data between two python programs. The subprocess module enables you to execute command line arguments from within a Python script.

For example, the Python code below executes the pwd command and stores the results as a variable.

import subprocess

In order to accomplish this, imagine we have a sensor that generates a timestamp. This represented by the code below. The sensor output is printed to stdout.

# sensor.py

import time
import datetime

# print('sensor timestamp: {0}.'.format(ts), flush=True)
while True:
    ts = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
    result = 'sensor timestamp: {0}.'.format(ts)
    print(result, flush=True)

Within print I have set flush=True since I need the buffer to be flushed (i.e. emptied to the terminal) every time the print statement is executed. The buffer is a temporary location for your data when it is moved between processes. Flushing the buffer when writing to standard out forces it to write the results to the terminal (even when it normally wouldn’t do so). On its own, Python’s print statement automatically flushes the buffer everytime it’s called. However, in this situation we have to force this behavior since standard output is buffered when connected via a pipe. If we didn’t flush the buffer then we wouldn’t see the results of the print statement until the process terminated (in this case never, since we’re using a while loop).

In order to collect this data suppose we need to listen the output of the sensor. This is accomplished with the code below.

# listener.py

import subprocess

process = subprocess.Popen(['python', 'sensor.py'], stdout=subprocess.PIPE)
# stdout, stderr = process.communicate() # not useful for this example cause communicate will wait for the process to finish

while process.poll() is None:
    result = process.stdout.readline().decode()

The listener.py code is doing several things by way of the subprocess module. subprocess.Popen allows you to spawn a process. The argument stdout allows you to capture the standard output of the process. We set stdout = subprocess.PIPE since we want a pipe to be open to standard output. I check to see if the spawed process is still running with process.poll. We then read the entire line of output with process.stdout.readline() and print the result.

Executing the following command in terminal,

python listener.py

gives us the following output,

sensor timestamp: 20200705-005927.
sensor timestamp: 20200705-005929.
sensor timestamp: 20200705-005931.