Streaming input and output in Flask
I always found Flask painful to use when it comes to streaming binary data, such as when I need to send a binary file or receive one. The official documentation is very sparse, explaining how to receive files submitted through a form, how to send a file or how to send data in chunks.
All those approaches may be very convenient for web applications with ordinary needs, that is to receive small files sent through forms, to send files located on disk back to the client and to use HTTP's chunked transfer encoding.
But what if I'm not writing a basic web application, but a REST API which needs to send and receive large blobs, and by large, I mean at least a few dozen of gigabytes? Here, the basic usage of Flask becomes very non-intuitive, and none of the basic approaches can be used any longer. There are no forms, and there are no files per se. Too bad.
Search resulted in nothing particularly useful. Stack Overflow kindly suggested to tweak the client to send data in chunks—thanks, but telling in the documentation of the REST API that data should be sent somehow differently from the common approach is lame.
Finally, I found the solution, which is extremely simple, but not documented anywhere. Since Flask is based on Werkzeug, there is always a possibility to move down one abstraction and use Werkzeug's capabilities, which is convenient, since Werkzeug's documentation mentions the
request.stream property. Here's an example where the input stream is copied to a file with no memory footprint:
@flask.route("", methods=["POST"]) def demo(): with open("/tmp/flask-stream-demo", "bw") as f: chunk_size = 4096 while True: chunk = flask.request.stream.read(chunk_size) if len(chunk) == 0: return f.write(chunk)
This makes it possible to upload (and in the same way, download) files independently of their actual size, would it be a few kilobytes or terabytes, without any impact on the server's memory, as soon as the “other side”, that is a file, an SQL Server's FILESTREAM, whatever, is also a stream.