Machine learning model serving with streamlit and FastAPI - PyConES 2020

hello i am davide fiocco and i will be talking about serving machine learning models with web applications developed using two cool python libraries: streamlit and fast api if you want to write me afterwards this is my twitter handle it sounds a bit funny in Spanish i know i didn’t know back then when i chose it anyway i think it’s easy to remember so maybe that’s useful um about me i started using python now more than 10 years ago i was using it to analyze data coming from computer simulations or physical systems my background is indeed Physics and currently i’m a senior data scientist at Frontiers i will spend some time talking about Frontiers Frontiers is an open access scientific publisher so we have this we have this website www.frontesin.org and um scientists can send their work in the form of scientific articles to our website and afterwards the articles are reviewed by other experts in the same field and if they pass the peer review they are eventually published online currently we have 92 journals online that are open for submissions and Frontiers has published so far 171 000 articles i was looking into this yesterday and if you look for python while programming just to distinguish it from snakes we have indeed scientific articles that are dealing with the usage of python in in science so applications in biology or libraries that are used in a scientific domain so if you’re interested check it out - something else about us we are over 500 employees in different offices in the world does one engineering office in Madrid and the company currently processes over 60 000 scientific articles per year this is pretty labor intensive and therefore we have a technology stack that also uses python uh we use python mainly in our big data pipelines and to power our machine learning components in the platform so yeah we’ll be talking about machine learning and as you know already it’s a very powerful technology at work we use natural language processing to so machine learning to process text data for instance to classify scientific articles based on their content in this slide you find an example which is sentiment analysis here we want to classify sentences based on their mood “Granada is really beautiful” is a happy sentence so we’ll have an algorithm which is able to recognize this actually if you visit this repository by hugging face there’s even algorithms that are able to read sentences and assign to them an emoji which is correct to describe their tone but machine learning is also applied to image and videos in those cases we talk about computer vision an example of a computer vision application is image segmentation and that’s what we will play with a bit in the rest of the talk so in this case we have an input image and we want to assign to different regions of it different classes depending on the content so here we have all this area which is the background so we assign it to color black here we have a vehicle so we colored the motorcycle in different way and we colored the motorcyclist in a different way still what’s nice nowadays is that it has become easier to at least apply machine learning models if we program python because there are frameworks like pytorch or tensorflow or scikit-learn that allow to run machine learning models easily so for instance if we wanted to solve the image segmentation task that we just have seen using pytorch we could do the following we could load a pre-trained model here we trust the model provider to have trained correctly the model so it’s both fair and accurate and then we can essentially use this model a bit like a python function on some input image and get in output some model prediction which is essentially this colored mask here um this is very nice however if we want to make the models useful uh for other users or make the models usable for other computer application we need some other elements that are not provided by for instance pytorch or tensorflow we need often to construct a front end so some kind of interface so that people can for instance upload an image and then invoke the model by pressing a button and getting results like in this example here or well not only front end we also need a back end so that another um application can call the back end and by sending an image and then getting a result this typically would be done for instance with a curl call like this that references the backend endpoint so having a front end and a back end is really useful to make machine learning working in a way for users and applications the problem is that you may need to master several technologies in order to build both and wouldn’t it be nice to build both the front end and the back end using pure python and hopefully without writing a lot of code so this can be done so i will propose a simple solution I will show it here that uses python and docker so python is uses as used here to build the front end by employing the streamlit library and instead the back end can be built using the fastapi library docker and in particular docker compose is used to make the front-end and back-end work together and also can facilitate the deployment let’s talk a bit about the libraries that constitute the building blocks for the solution so one as i said is streamlit and this is their website as they say here they want to make it easy to build beautiful web applications used in machine learning and data science and i agree that this library can be very helpful to build simple uis without writing a lot of code you don’t need to be an expert of javascript you don’t need to know html to to build these user interfaces the libraries the the streamlit is pretty intuitive to use you essentially just write one one line of code and this can be translated by the library into a widget in your in your user interface the project has become pretty popular it has 10 1 000 stars on github and if you want to compare it with other frameworks like dash or or shiny which is popular among our users you can check this webpage on plotly.com also let’s talk about fast api - they want to be high performance is to learn faster code and ready for production and indeed i found fast api to be capable of building apis with documentation without writing a lot of code fastapi is especially easy to use if you know already flask so anyway i would agree that it’s quite easy to learn also because it has an extensive documentation and lead developer sebastian ramirez has recorded a lot of videos about how to use the library the project has become pretty popular it has 20 000 stars on github and you can compare with other popular frameworks like flask or django by visiting this webpage okay so let’s look more in detail inside our our solution this is on on github very happy to to respond to issues if there are any and yeah the code may have bugs so this is also open for pull requests the project is described on this tenders blog post um the architecture of the project is represented here we have two containers one takes responsibility for running the front end and that’s powered by the streamlit library this serves a user interface to users with descriptions and buttons and so on the front end also is capable of firing a request to the backend using the requests library and the back end runs using fast api and the core machine learning task as we just said is done by pytorch code so okay let’s have a little walk through really quickly of the of the of the code - okay so the code as it is on github is here so you have a readme file that explains a bit how to run the web application and also how to deploy on heroku and how to debug it the architecture is formalizing code in this docker compose dot yaml file as i said we have two services one for the backend which is in the same docker network as the front end the streamlit container um we also expose ports so that both the front the back end and the front end are reachable from the outside so we can peek into the python code of the backend so as we said there is pytorch code to run the machine learning model on a given image and give an output result the colored map so i won’t go too much into detail because this is really taken from the pytorch hub guide we essentially take the pre-trained model from the pytorch hub and then we have a function which takes this model and runs it over a binary image there’s also some pre-processing involved this is where we invoke the the model we call the model to to get the output predictions that are also given in output uh in the end by by the function so this pytorch code is used inside the back end as we said so here we have fast api um we indeed create a fast api object with a title for this is used in the app documentation and also the same the same is true for the description and then the main part is this where we declare an endpoint that the front end or another application can call by passing an input an image this piece of code runs the model here and then returns to the to the client the machine learning model result - let’s have a look at the front end so the the front-end code is here and it uses streamlit and a request library requests are used to send the input given by the user to the fast api backend in this processing function here and instead the construction of the ui layout is very simply done by streamlight here so we with this instruction we create a title for our web application we will see this in a second we write to the web page some description and also we create widgets so that users can upload an image which will be stored in this variable this input image will be then kind of passed to the backend by invoking the the process function that we have seen above here and yeah the call to the back end is triggered by the pressing of this button here finally the output result is shown to the user with this final streamlit call so that’s pretty much it if we want to run the application we would just need to use docker compose so let’s do a docker-compose up - this activates the fastapi backend and the streamlit frontend containers and yeah so let’s have a look we can visit the front end at this address here and we can check that everything works indeed you can see the title here the description of the application and different widgets that we have created using streamlit okay yeah this is made with streamlining teeth we have to wait a bit for the back end to process the image but in the end we get the result the backend documentation is now visible at this address so yeah everything seems to be working in case we want to deploy the application on the web we can use a tool like heroku as far as i know we need an add-on DocHero to to run the application it takes about 10 minutes to deploy using the instruction that you find on the on the github repository and at the end of the process you have a web application which is public on the web and is accessible at this address for the front end and here we can visit the the back end so that’s pretty much it for the code walkthrough i hope that was enjoyable uh thanks very much for attending the talk and i’m happy to take questions .