Functions-as-a-Service on Fly.io with FastAPI

Fly.io recently came with a very intriguing service called Machines. Machines are designed to quickly(in around 400ms) start a docker image when a request arrives and turn off when the process exits. This is amazing because it allowed me to run getimages.social without worrying about the costs. I only pay for storage when a machine isn't running, which is around $0.15/mo for a 1GB Docker image. Here's a snippet of log messages related to one start/stop sequence.

2022-07-06T07:29:07.991 proxy[39080523a19875] fra [info] Machine started in 454.077286ms
2022-07-06T07:29:08.794 app[39080523a19875] fra [info] INFO: Started server process [509]
2022-07-06T07:29:08.794 app[39080523a19875] fra [info] INFO: Waiting for application startup.
2022-07-06T07:29:08.794 app[39080523a19875] fra [info] INFO: Application startup complete.
2022-07-06T07:29:08.804 app[39080523a19875] fra [info] INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2022-07-06T07:29:09.172 proxy[39080523a19875] fra [info] Machine became reachable in 1.180665787s
2022-07-06T07:29:09.186 app[39080523a19875] fra [info] INFO: 27.50.59.67:0 - "GET / HTTP/1.1" 200 OK
2022-07-06T07:29:38.796 app[39080523a19875] fra [info] before condition self.last_request=datetime.datetime(2022, 7, 6, 7, 29, 9, 186501)
2022-07-06T07:29:38.797 app[39080523a19875] fra [info] self.last_request < datetime.now() - timedelta(seconds=25)=True
2022-07-06T07:29:38.871 app[39080523a19875] fra [info] INFO: Shutting down
2022-07-06T07:29:38.972 app[39080523a19875] fra [info] INFO: Waiting for application shutdown.
2022-07-06T07:29:38.972 app[39080523a19875] fra [info] INFO: Application shutdown complete.
2022-07-06T07:29:38.972 app[39080523a19875] fra [info] INFO: Finished server process [509]

From my experience machine starts in 300-500ms. 1GB docker image then becomes reachable in around 1400ms. More lightweight image(200MB) starts in 500-800ms.

I'm using async web framework FastAPI so I created a simple coroutine that will tell the process to terminate when there were no requests in the last 30 seconds. So I added middleware that updates the last request time and coroutine regularly checks when was the last request made. I'm using uvicorn as a web server and the best way to self-terminate I found is to send SIGTERM.

class ActivityWatcher:
    last_request: datetime

    def update_last_request(self):
        self.last_request = datetime.now()

    async def start(self):
        self.last_request = datetime.now()
        while True:
            await asyncio.sleep(30)
            print(f"before condition {self.last_request=}")
            print(f"{self.last_request < datetime.now() - timedelta(seconds=25)=}")
            if self.last_request < datetime.now() - timedelta(seconds=25):
                import os
                import signal

                os.kill(os.getpid(), signal.SIGTERM)


activity_watcher = ActivityWatcher()


@app.on_event("startup")
async def startup_event():
    asyncio.create_task(activity_watcher.start())


@app.middleware("http")
async def update_last_request_middleware(request: Request, call_next):
    response = await call_next(request)
    activity_watcher.update_last_request()
    return response

Setting up a Machine is more involved than classic Fly Virtual Machines as it's currently available only via API and requires you to build and push the docker image. But it's not that big of a problem as entire process is well documented.

Want to try it for yourself? Go ahead and try my demo app.