Logging in python

It doesn't matter how long you have been coding, even today while writing any script we need some manual validation of output and we tend to use print statement a lot.

print: At the initial phase when you have just started writing the code, you are in the flow to set the logic and the flow of the code, and during this phase, you need to verify different things and so developers use print to verify things like the value of the variable, error/exceptions, and print statement to verify whether the code has been entered if/else statement.

It is very handy and quick and that's why it is built for.

Print makes the development interactive and fast. Interactive development is a very fast way to iterate on your idea, and logic and that's why many developers like jupyter notebooks.

why logging is important

But as the application matures it is put to production in the server, and while code is running in the server we don't know what is happening inside it, but we need that visibility to track input and errors. And for this we have logging

how to quickly setup logs in a python application

why log the logs in JSON format

Json makes it easy to directly push logs to different log monitoring tools and services.

Logging in a file in the server

# pip install python-json-logger
from pythonjsonlogger import jsonlogger
def get_logger(logger_name):
    file_name = "/tmp/logs/{}.log".format(logger_name)
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)

    formatter = jsonlogger.JsonFormatter(fmt='%(asctime)s :: %(lineno)s :: %(levelname)-8s :: %(name)s ::  %(message)s')

    logHandler = logging.FileHandler(filename=file_name)
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    return logger

logger_name = "login"
login_logger = get_logger(logger_name)

why and when to use cloudwatch

Setting up cloudwatch

Setting up the cloud watch client on the server

sudo aws configure
sudo yum install amazon-cloudwatch-agent

verify the installation using the below commands

cat ~/.aws/credentials
cat ~/.aws/config

To check the cloud watch status use the below commands or here

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a start

Setting up the python environment


#pip install python-json-logger
#pip install boto3
#pip install watchtower

import boto3
from pythonjsonlogger import jsonlogger
import watchtower

credentials = boto3.Session().get_credentials()
access_key = credentials.access_key
secret_key = credentials.secret_key
region = ""

cloudwatch_client = boto3.client("logs", 
region_name=region,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key
)



def get_logger(log_group, logger_name):    
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)

    formatter = jsonlogger.JsonFormatter(fmt='%(asctime)s :: %(lineno)s :: %(levelname)-8s :: %(name)s ::  %(message)s')

    handler = watchtower.CloudWatchLogHandler(log_group_name=log_group, log_stream_name=logger_name, boto3_client=cloudwatch_client)

    handler.setFormatter(formatter)
    logger.addHandler(handler)
    return logger


log_group = "app_name"
login_logger = get_logger(log_group, "login")

Reference:

Vivek Blog