Streaming input and output in Flask

Arseni Mourzenko
Founder and lead developer
177
articles
May 1, 2015
Tags: short 50 flask 2 python 5 performance 13

I al­ways found Flask painful to use when it comes to stream­ing bi­na­ry data, such as when I need to send a bi­na­ry file or re­ceive one. The of­fi­cial doc­u­men­ta­tion is very sparse, ex­plain­ing how to re­ceive files sub­mit­ted through a form, how to send a file or how to send data in chunks.

All those ap­proach­es may be very con­ve­nient for web ap­pli­ca­tions with or­di­nary needs, that is to re­ceive small files sent through forms, to send files lo­cat­ed on disk back to the client and to use HTTP's chun­ked trans­fer en­cod­ing.

But what if I'm not writ­ing a ba­sic web ap­pli­ca­tion, but a REST API which needs to send and re­ceive large blobs, and by large, I mean at least a few dozen of gi­ga­bytes? Here, the ba­sic us­age of Flask be­comes very non-in­tu­itive, and none of the ba­sic ap­proach­es can be used any longer. There are no forms, and there are no files per se. Too bad.

Search re­sult­ed in noth­ing par­tic­u­lar­ly use­ful. Stack Over­flow kind­ly sug­gest­ed to tweak the client to send data in chunks—thanks, but telling in the doc­u­men­ta­tion of the REST API that data should be sent some­how dif­fer­ent­ly from the com­mon ap­proach is lame.

Fi­nal­ly, I found the so­lu­tion, which is ex­treme­ly sim­ple, but not doc­u­ment­ed any­where. Since Flask is based on Werkzeug, there is al­ways a pos­si­bil­i­ty to move down one ab­strac­tion and use Werkzeug's ca­pa­bil­i­ties, which is con­ve­nient, since Werkzeug's doc­u­men­ta­tion men­tions the request.stream prop­er­ty. Here's an ex­am­ple where the in­put stream is copied to a file with no mem­o­ry foot­print:

@flask.route("", methods=["POST"])
def demo():
    with open("/tmp/flask-stream-demo", "bw") as f:
        chunk_size = 4096
        while True:
            chunk = flask.request.stream.read(chunk_size)
            if len(chunk) == 0:
                return

            f.write(chunk)

This makes it pos­si­ble to up­load (and in the same way, down­load) files in­de­pen­dent­ly of their ac­tu­al size, would it be a few kilo­bytes or ter­abytes, with­out any im­pact on the serv­er's mem­o­ry, as soon as the “oth­er side”, that is a file, an SQL Serv­er's FILESTREAM, what­ev­er, is also a stream.