Dynamic Logging in FastAPI with Python

How can you change logging levels for a FastAPI application without restarting? This solution uses a log filter that is dynamically configurable via the Prefab UI. This approach allows you to change logging levels for both uvicorn and FastAPI without the need to restart the server, providing greater flexibility and control over your application's logging behavior.

Setting Up the Environment

First, let's set up our project dependencies. We'll be using the prefab-cloud-python library to manage our dynamic logging. Add the following to your pyproject.toml file:

[tool.poetry.dependencies]
prefab-cloud-python = "^0.10.10"

Next, import the necessary modules from the prefab-cloud-python library:

import prefab_cloud_python
from prefab_cloud_python import Options, LoggerFilter

Configuring the Logger

Now, let's configure our logger in the main.py file of our FastAPI application:

#main.py

# this will read PREFAB_API_KEY from env
prefab_cloud_python.set_options(prefab_cloud_python.Options()) 

# normal logging setup
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG) # set to DEBUG so that LoggerFilter will see all log records
ch = logging.StreamHandler(sys.stdout)
ch.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
root_logger.addHandler(ch)
# key step - add the Prefab LoggerFilter to the StreamHandler
ch.addFilter(LoggerFilter())

# get an instance of the logger
logger = logging.getLogger("my-application")

In this configuration:

1. We set up Prefab Cloud options.
2. We configure the root logger to use DEBUG level, ensuring all log records are captured.
3. We create a StreamHandler to output logs to stdout.
4. We add a formatter to structure our log messages.
5. We add the Prefab LoggerFilter to the StreamHandler, which will enable dynamic log level control.
6. Finally, we create an instance of the logger for our application.

Get a Free Api Key

Create an account at Prefab and get a free API key.

Run your application with the PREFAB_API_KEY environment variable set to your API key.

Adding Logging to Your FastAPI Routes

With our logger configured, we can now add logging to our FastAPI routes. Here's an example of how to use different log levels in a route:

@app.get("/")
async def root():
    logger.debug("debug something")
    logger.info("info something")
    logger.warning("warning something")
    logger.error("error something")
    logger.critical("critical something")
    return {"message": "Hello World from FastAPI"}

This route demonstrates logging at various levels: debug, info, warning, error, and critical. With our dynamic logging setup, we can control which of these messages are actually output based on the current log level configuration.

If we run this and goto localhost:8000 we see the following in the logs:

2024-09-25 10:26:07,128 - my-application - WARNING - warning something
2024-09-25 10:26:07,129 - my-application - ERROR - error something
2024-09-25 10:26:07,129 - my-application - CRITICAL - critical something

Change the log levels dynamically

Now that we have our dynamic logging setup in place, we can change the log levels on the fly using the Prefab Cloud UI. Here's how you can do it:

1. Log in to your Prefab dashboard.
2. Navigate to the "Logging" section.
3. Set the value to the desired log level (e.g., "DEBUG", "INFO", "WARNING", "ERROR", or "CRITICAL").

The new log level will be applied to your application almost immediately, without requiring a restart. For example, if you set the log level to "DEBUG", all log messages with a severity of DEBUG and above will be displayed, like this:

2024-09-25 10:27:54,920 - my-application - DEBUG - debug something
2024-09-25 10:27:54,920 - my-application - INFO - info something
2024-09-25 10:27:54,921 - my-application - WARNING - warning something
2024-09-25 10:27:54,921 - my-application - ERROR - error something
2024-09-25 10:27:54,921 - my-application - CRITICAL - critical something

Here's a quick look of how you might adjust the log level. In this exampel we've set the root logger to WARN and the my-application logger to DEBUG for a particular user, else WARN, and we've set the uvicorn logger to INFO.

To help you estimate how many log lines will be output at each log level, you can hover over the logger to see the log volume.

Configuring uvicorn Logging

To ensure that uvicorn (the ASGI server we're using to run our FastAPI application) also uses our custom logger, we need to modify how we run the application and set log_config=None.

if __name__ == "__main__":
    import uvicorn
    uvicorn.run("__main__:app", host="0.0.0.0", port=8000, reload=True, log_config=None)

By setting log_config=None, we're telling uvicorn to use the root logger we've configured, which includes our dynamic logging setup. If you're running uvicorn from the command line you can run uvicorn --reload --log-config empty_log_config.json with the following empty_log_config.json

{
    "version": 1,
    "disable_existing_loggers": False
}

Conclusion

With this setup, you now have a FastAPI application with dynamic logging capabilities. You can change log levels on the fly using Prefab, allowing you to adjust the verbosity of your logs without restarting your application. This can be incredibly useful for debugging issues in production or temporarily increasing log detail for specific components of your system.