Redirect standard out to Python's logging module with contextlib

Posted on Wed 22 May 2019 • Tagged with python, snippets • 5 min read

Python's logging functionality is very nice once you get the hang of it, but many people either disagree or don't bother to use it. Learn how to redirect other people's pesky print statements into your nice logging setup.


Continue reading...

Cleaner Spark UDF definitions with a little decorator

Posted on Thu 16 November 2017 • Tagged with spark, python, data, snippets • 3 min read

One of the handy features that makes (Py)Spark more flexible than database tools like Hive even for just transforming tabular data is the ease of creating User Defined Functions (UDFs). However, one thing that still remains a little annoying is that you have to separately define a function and declare it as a UDF. With four lines of code you can clean those definitions right up.


Continue reading...