There are a variety of ways python applications can talk to each other. APIs might be the most common way, but APIs have a fair bit of overhead - there has to be something there listening all the time. One other way is asychronous communication via a message broker.

I'd not been familiar with message brokers before - Amazon Web Services Simple Queue Service or SQS was definitely my first foray into them. There are many SQS alternatives out there, some of which are open source, some hosted. In research for this blog post, I came across Celery which is python based, and looks pretty interesting. I defintely want to check it out at some point.

Anyway, the scope of this article is on using SQS, and particularly the boto3 SDK to work with SQS. The basic architecture of the system that I implemented that used SQS was one 'hub' daemon, and multiple 'spoke' daemons. Each daemon polled SQS for messages at configurable intervals. There a number of message queues, one specifically for the hub, and one for each spoke.

As I mentioned in the previous articles in this series, I created a set of modules that abstracted the SQS process. You can get the queue as an object, then do actions on that object as in these examples.

    session = boto3.session.Session(profile_name=profile, region_name=region)
    sqs = session.resource('sqs')
    try:
        queue = sqs.get_queue_by_name(QueueName=queue_name)
    except:
        print("Queue doesn't exist!")

    # get messages
    messages = queue.receive_messages()
    all_messages = []
    for message in messages:
        all_messages.append(message.body)

    #send messages
    queue.send_message(
            MessageBody=message
        )

    # create queue
    new_queue = sqs.create_queue(QueueName=queue_name)

    # purge queue
    queue.purge()

Because I did this in many places in my application, I ended up creating modules that:

* check to see if a queue exists
* creates a queue
* polls a queue for messages (and returns them)
* sends a message to a queue
* purges a queue

It's easy to parse the messages to find the keywords needed to direct the apps to make specific actions based on the messages (in my case, things like sending log files to an S3 store, updating a google spreadsheet, looking for specific files in an S3 store, and a few others.)

If you have apps that need to talk to each other, SQS or any message broker/queue system is a really powerful way to make that happen, especially if synchronous communication is not required. You can configure the retention time and policies of SQS pretty easily - and it will likely fit almost any use case.