Google Gemini Receipt Parser for YNAB

Tired of manually logging expenses in YNAB? Discover how I automated the importing and categorising of transactions in YNAB from receipt images using Google Gemini. You'll also see how easy it is to run the Docker container for your own budget.

My wife and I use YNAB (You Need A Budget) to manage our envelope-based budget. We've found enveloped budgeting to be incredibly effective.

Now, YNAB is excellent, but our biggest challenge in maintaining an accurate budget is entering transactions in the app as they occur. Ideally, we should quickly open the app and enter that transaction every time we spend money. But let's be honest, who remembers to always do that? When shopping in a rush or eating out at a nice place, the last thing I think of is logging what I just spent.

This inspired me to find a better solution. With the latest AI, I knew I could leverage something like Google Gemini to analyze my receipts and automatically add the expense to YNAB. So, I built a Docker-based service that does precisely that (and even saves the receipts to storage for me).

Key Features

I quickly whipped up a proof-of-concept app in Bun and TypeScript that exposes an HTTP endpoint where I can upload a receipt. I've refined it since then and built a small Docker container that's incredibly easy to run and has some cool features like:

Effortless Receipt Parsing

Upload any receipt, and Google Gemini will parse it into a structured JSON object, saving you time and effort of doing all that manually. I've even sent screenshots of payment notifications from my bank, and it can figure out what I spent on.

Automated YNAB Categorization

The service retrieves a list of your YNAB categories so that Google Gemini can pick the most appropriate category for the transaction. There's no need to manually assign a category after the transaction is imported, saving you time when reconciling your transactions.

Flexible API Endpoint for Integration

The endpoint is really easy to use and can be called from anywhere. Just send a POST request to /upload with the image file and name of the account you used to pay. It's also secured using an API Key and Secret.

Convenient Receipt Storage

You can optionally store all the receipt files once they've been processed. Currently, it supports storing these files in a local directory or in an S3-compatible object storage.

How It Works

The service integrates with Google Gemini and the YNAB API to parse and categorise incoming receipts. The diagram below provides a high-level overview of the process:

1. Send Request
   First, you send a request to the Docker service with an image of your receipt and the account you used to pay.
2. Retrieve YNAB Categories
   To get the most up-to-date categories in your budget, the service queries the YNAB API for your budget categories (you can even filter which category groups should be used).
3. Parse Receipt with Google Gemini
   The receipt and list of your budget categories are sent to Google Gemini to parse and categorise. When parsing, Google Gemini parses each line item on the receipt and categorises it.
4. Insert Parsed Transaction into YNAB
   Once parsed, the transaction is uploaded into YNAB with the merchant and category. If multiple categories apply, the transaction is split appropriately. For example, if you buy groceries and household cleaning items in the same shopping trip, the transaction will be split across those categories.
5. Save Receipt to File Storage
   Finally, you can optionally configure a file storage. The uploaded receipt is then saved there. Great for keeping receipts for warranty purposes or to refer back to later while reconciling your budget.

Running the Docker Container

The entire service is packaged as a Docker container, making it straightforward to run on your local machine or deploy to a remote server.

Prerequisites

Before you can run the container, you must sort out some prerequisites. Most importantly, you'll need Docker installed to run the container.

You'll also need API credentials for the YNAB API and Google Gemini API. The /upload endpoint is secured using Basic authentication, so you'll also need to generate an API Key and Secret that you'll use to secure the upload endpoint. These can be any random strings, just make sure they're long and unguessable.

If you're using an S3-compatible object storage to store your receipts, you'll also need those connection details. Some options for S3-compatible storage are AWS S3, Cloudflare R2 and Backblaze B2.

Running the Container

Once you've sorted out the prerequisites above, open a new terminal and execute the following Docker command, making sure to replace the placeholder values with your API credentials:

docker run \
    -e APP_API_KEY=your_api_key \
    -e APP_API_SECRET=your_api_secret \
    -e GEMINI_API_KEY=your_gemini_api_key \
    -e GEMINI_MODEL=gemini-2.0-flash-exp
    -e YNAB_API_KEY=your_ynab_api_key \
    -e YNAB_BUDGET_ID=your_ynab_budget_id \
    -p 3000:3000 \
    ivankahl/ynab-slip-uploader

The container has a health endpoint you can use to check if it's running. Call it using the following cURL command:

curl http://localhost:3000/healthz

If everything is running fine, you should receive an OK response back.

Uploading a Receipt

Now for the exciting part: uploading a receipt. Take a picture of a recent receipt and send it to the /upload endpoint, along with the name of the account in YNAB that you used to pay that receipt:

curl --location 'http://localhost:3000' \
     --user '<API_KEY>:<API_SECRET>' \
     --form 'account=<YNAB_ACCOUNT_NAME>'
     --form 'file=@<PATH_TO_RECEIPT_FILE>' 

Here's an example of the cURL request with populated values:

curl --location 'http://localhost:3000' \
     --user 'myapikey:thisisverysecret' \
     --form 'account="Checking Account"'
     --form '[email protected]'

If everything goes well, the endpoint will return a 200 response with a JSON object containing the parsed transaction details. If you log into YNAB, you should also find the new transaction logged, ready for you to approve:

That's it! So easy!

Bonus: Apple Shortcut to Upload Receipts

My wife and I both use iPhones, so I created an Apple Shortcut that, when run:

1. Opens the camera so you can take a picture of the receipts.
2. Asks you which account you used to pay
3. Sends the transaction to the Docker service (you'll need to host this on a remote server).
4. Notifies you when the transaction has been imported.

You can download and configure the shortcut using the link below:

Conclusion

Budgeting is important, but it shouldn't feel like a burden. Otherwise, you just won't do it. This Docker service, along with the Apple Shortcut, has been hugely beneficial for my wife and I. Hopefully, you find it useful as well.

If you have any feature requests or find any bugs, please submit an issue on the GitHub repository. You're also welcome to ask for help in the comments below or on the repository.

Excited to hear how this helps you.