smallsec
about | contact

Building a DigitalOcean OpenAI API Proxy

Update: Fabric, the OpenAI proxy discussed below is now public! Definitely worth a peek :)

I recently took Daniel Miessler's Augmented course and thought I'd take a stab at implementing the OpenAI API proxy he discussed. And since I like playing with the plumbing rather than operating it on a long-term basis I wanted to do something other than just host it on a VPS somewhere.

I'd initially planned on building my proxy using a serverless architecture, but the entry-level tiers for the providers I looked at all timeouts in the 10-30s range, which often isn't enough time for OpenAI to work its magic.

I eventually settled on DigitalOcean's App platform. This is sort of halfway between a VPS and a serverless function. You write code for Flask the same way you would normally, but then you upload it to the platform and don't have to worry about the base OS. Score!

The rest of this post details the process required to build our own authenticated API that hosts Daniel's useful ExtWis tool and a CLI to go along with it.

Note: This is a proof-of-concept project that is optimized for quick and simple deployment. There are likely a few unhandled bugs and a bucketful of improvements to be made. Hopefully it does a good enough job getting you acquainted with the various pieces that you can take what I've built and run with it!

The full code can be found here.

Pre-Requisites

There are a few things you need to set up before getting into it.

Project Layout

This is the project layout we're working towards. I'll explain this in subsequent sections:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

├── .do
│   └── app.yaml
├── client
│   ├── extwis
│   │   ├── extwis
│   │   │   ├── __init__.py
│   │   │   └── main.py
│   │   ├── requirements.txt
│   │   └── setup.py
│   └── README.md
├── README.md
└── server
    ├── app.py
    ├── gunicorn_config.py
    ├── Procfile
    ├── README.md
    ├── requirements.txt
    ├── api
    │   ├── extwis
    │   │   ├── extwis.py
    │   │   ├── __init__.py
    │   │   └── system.md
    │   └── __init__.py
    └── util
        └── auth.py

GitHub Configuration

DigitalOcean Apps build when you push to GitHub (or GitLab, weirdos) by default, so our first step is going to be configuring one of those.

  1. Log in to GitHub and create the repository.
  2. Make sure you can clone it locally.
  3. Log into DigitalOcean and then give it read/write access to that repository so it can do it's buildy magic.
  4. Do a little dance.

Server

We're going to start with the server configuration. We'll cover both creating the Flask API server and deploying it to the DigitalOcean App platform.

Flask Configuration

We're using Flask because it's quick and easy to set up and is more than capable of hosting our APIs.

The Flask configuration lives in app.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import logging

from flask import Flask, request

from api.extwis import extwis
from util.auth import token_required

app = Flask(__name__)
logging.basicConfig(level=logging.INFO)

@app.route("/extwis", methods=["POST"])
@token_required
def route_extwis():
    logging.info("running extwis")
    resp = extwis.send(request.data.decode("utf-8"))
    return resp.encode("utf-8")

This looks pretty minimal because it is. The code defines a single API route, /extwis, accessible via the POST method. The @token_required line is called a decorator, and this one ensures that any incoming requests have a valid token provided in the Authorization header. The logic is implemented in util/auth.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

from functools import wraps
from flask import request, make_response, jsonify
import os

def token_required(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
				# Get value of API_SECRET environment variable
        api_secret = os.environ.get("API_SECRET")
        authorization_header = request.headers.get("Authorization")
        if not authorization_header:
						# Authorization header is missing
            return make_response(jsonify({"message": "Unauthorized"}), 401)
        if authorization_header != api_secret:
						# Authorization header is present, but the value is incorrect
            return make_response(jsonify({"message": "Forbidden"}), 403)
        return func(*args, **kwargs)
    return wrapper

Next we have the code that loads our prompt from disk and sends it to OpenAI. This is copied right out of OpenAI's documentation, more or less verbatim.

Let's take a look at what it does:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

from openai import OpenAI

def send(user_prompt: str):
		# Load prompt
    with open("api/extwis/system.md") as file:
        system_prompt = file.read()
    client = OpenAI()
    completion = client.chat.completions.create(
        model="gpt-4-1106-preview",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
    )
    return f"{completion.choices[0].message.content}"

The from api.extwis import extwis line imports the functions stored api/extwis/extwis.py, which we then call with extwis.send(). The system prompt is loaded from system.md (grab a copy here), and the user's input is sent as the user prompt. The output is returned to the calling function, route_extwis(), and then back to the user.

And that's basically it for Flask! You should be able to run the server locally doing the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ cd server/
$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip3 install flask openai gunicorn
$ pip3 freeze > requirements.txt
$ export API_SECRET=$(openssl rand -base64 32)
$ FLASK_APP=app.py flask run
* Serving Flask app 'app.py'
 * Debug mode: off
INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
INFO:werkzeug:Press CTRL+C to quit

You can call the API using httpie:

1
2
3
4

$ echo "is this wise" | http http://localhost:5000/extwis Authorization:$API_SECRET
SUMMARY:
The user presented a brief, ambiguous query related to the concept of wisdom, potentially seeking validation or insight into the wisdom of a specific action or thought.
...omitted for brevity...

Let's launch this sucker into the cloud.

DigitalOcean App Deploy

For a platform that was released in 2020, there is a dearth of documentation available about this product. Luckily, there isn't a whole lot to deploying app so I was able to cobble together a working configuration.

I'm going to focus on using the doctl CLI tool to deploy things since that's where I tend to spend most of my time, but you can definitely do this using the UI if you prefer lots of unnecessary clicking.

To start, we're going to define the App Spec. This is a YAML file stored in .do/app.yml that handles all the platform-specific configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
features:
- buildpack-stack=ubuntu-22
ingress:
  rules:
  - component:
      name: APPNAME
    match:
      path:
        prefix: /
name: APPNAME
region: tor
services:
- environment_slug: python
  github:
    branch: main
    deploy_on_push: true
    repo: REPONAME
  http_port: 8080
  instance_count: 1
  instance_size_slug: basic-xxs
  name: APPNAME
  source_dir: /server

Here's some notes about the values

With the above values configured, it's time to push this all up to GitHub:

1
2
3
4

$ cd your-repo-name
$ git add .
$ git commit -m 'initial commit'
$ git push

Finally, we create the App:

1

doctl apps create --spec .do/app.yaml

Browse to the DigitalOcean Apps panel and see what ye hath wrought! If all goes well, your app should go from Building to Deploying!

Our next step is configuring the environment variables needed by the application for authentication:

  1. Click on your app name.
  2. Click Settings.
  3. Beside App-Level Environment Variables, click Edit.
  4. Add the following:
    • OPENAI_API_KEY - Your OpenAI API key.
    • API_SECRET - The API secret you created.
  5. Click Save.

You can get the URL from the UI or by running:

1

doctl apps list

You can test whether your API is alive the same way you did earlier with your local Flask server:

1
2
3
4

$ echo "is this wise" | http https://appname-random.ondigitalocean.app/extwis Authorization:$API_SECRET
SUMMARY:
The user presented a brief, ambiguous query related to the concept of wisdom, potentially seeking validation or insight into the wisdom of a specific action or thought.
...omitted for brevity...

Huzzah! Let's make it a bit easier to use this API from our desktop.

Client

Compared to the server, the client is dead simple.

1
2
3
4
5
6
7
8

├── client
│   ├── extwis
│   │   ├── extwis
│   │   │   ├── __init__.py
│   │   │   └── main.py
│   │   ├── requirements.txt
│   │   └── setup.py
│   └── README.md

All of the client logic is stored in client/extwis/main.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

#!/usr/bin/env python3
import os

import click
import requests

def send_data(text, api_key):
		# The API URL
    url = "https://appname-random.ondigitalocean.app/extwis"
		# Add our authorization header
    headers = {"Authorization": api_key}
    resp = requests.post(url, data=text, headers=headers)
    return resp

@click.command()
def main():
    """A simple CLI tool for interacting with Jarvis"""

    # Read API key from OS environment variable
    api_key = os.environ.get("API_SECRET")
    if not api_key:
        click.echo("API_SECRET environment variable is not set.")
        return
    input_text = click.get_text_stream("stdin").read()
    if not input_text:
        click.echo("No input received!")
        return
    resp = send_data(input_text, api_key)
    print(resp.text)

if __name__ == "__main__":
    main()

This takes input on stdin, attaches an authorization header, and sends it to our API. Ezpz.

Requirements live in client/extwis/requirements.txt:

requests
click

And to make life easier for us on the CLI, we're also going to define client/extwis/setup.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

from setuptools import find_packages, setup

with open("requirements.txt") as f:
    requirements = f.read().splitlines()

setup(
    name="extwis",
    version="0.1.0",
    author="yourname",
    author_email="you@there.com",
    description="Extract wisdom from articles",
    url="https://github.com/orgname/reponame",
    packages=find_packages(exclude=("tests", "docs")),
    install_requires=requirements,
    classifiers=[
        "Programming Language :: Python :: 3.10",
        "Operating System :: OS Independent",
    ],
    python_requires=">=3.10",
    entry_points={"console_scripts": ["extwis=extwis.main:main"]},
)

This is mostly straightforward, but one area that took me a minute to understand was the entry_points block. This is basically saying “map the extwis command to the main function of extwis/main.py”.

With these pieces in place, we can finally install our CLI tool using pipx:

1
2
3
4
5
6

$ cd client/extwis
$ pipx install .
$ echo "is this wise" | extwis
SUMMARY:
The user presented a brief, ambiguous query related to the concept of wisdom, potentially seeking validation or insight into the wisdom of a specific action or thought.
...omitted for brevity...

Conclusion

The above steps should give you a halfway decent understanding of what we're doing. Extending the APIs just involves creating and incorporating additional configuration under client/ and server/. Happy hacking!

<<
>>