From First User to Ban: Building (and Losing) a WhatsApp AI Food Tracker
Everything I learned building a food-tracking assistant on WhatsApp — from 0 to 1.
I’ve been tinkering with side-projects for a while now, but CalorIA - the WhatsApp-based food assistant that estimates calories from photos, voice notes, and text - was my newest and most absorbing project. I kicked it off in April 2025.
In this (longest-so-far!) post, I’ll walk through the exact steps (and landmines) I hit while bringing the idea to life, so you can replicate or remix the process for your own bot idea.
I've split this into four sections:
Business reasoning
Tech Stack & Project Structure
Bot Usage & Getting It Out There (Marketing)
What’s Next - for the project and for me
TL;DR: Built a WhatsApp-based food tracker from scratch using AI. Got it running, got 100+ users, made some money. Then WhatsApp banned my number. Here’s everything I learned building it - from tech stack to distribution pitfalls.
Business reasoning
While brainstorming part-time projects, one theme kept popping up: “WhatsApp + AI” mash-ups. Here’s why that combination felt interesting.
Why WhatsApp?
Distribution is built-in. In Brazil, WhatsApp has >93 % penetration; meeting users where they already are beats convincing them to download yet another app.
Rich media support. Photos, audio, documents - perfect for these AI integrations as you just need to handle the different file types.
It’s the cool kid on the block. From “bank-in-WhatsApp” startups like Magie/Jota to indie experiments, WhatsApp + AI is the hot combo right now.
Context-savvy AI. LLMs thrive in chat; text and voice give you the richest context for personalized answers.
A friend of mine (shout-out to Pedro R.) kept repeating: “Leverage WhatsApp’s distribution - it’s a no-brainer.” He was right. The only hurdle was choosing what to build first.
I also dragged my feet a bit - everyone kept warning me about the pains of building inside WhatsApp’s ecosystem.
Reality check: Building inside WhatsApp’s walled garden isn’t hard, but it does come with quirks and limits you’ll feel along the way.
Why a diet tracking app?
I wanted to lose weight so I thought of making something that would be useful for me. After I started building, I actually found it's quite a common/successful idea. You have a US-based company called Cal AI doing close to $30m revenue (but they use their own app) and here in Brazil, the biggest WhatsApp based project seems to be dieta.ai - who seem to be doing quite well.
So the idea of leveraging AI to analyse food is not novel nor unique - but it's quite useful.
The Photo Caveat
Computer vision can’t see salad dressing or hidden butter, so image-based calorie counts are ballpark figures at best. Whenever possible, I encourage users to send a quick voice note instead:
“Ate two slices of bread with 1 slice of cheese”
That extra context tightens the estimate dramatically.
The Nutritionist’s Point of View
Several nutritionists I spoke with warned against “calorie tunnel vision.” Weight management is more than numbers, and obsessing over them can be counter-productive. Of course, if you're being pragmatic and you have more of a mathematical approach to it, that's fine.
WhatsApp Setup
Choosing and Configuring the WhatsApp APIs
My first decision was official vs. unofficial APIs. I went with Meta’s official Business API for compliance, even though plenty of private wrappers promise quicker results.
The trade-off? A surprisingly bureaucratic setup. I had to resurrect a dormant Facebook account, verify my business, add a payment method, create an “app,” enable webhooks - the whole nine yards. It ate nearly four hours of a Saturday, and on the fun-meter I’d call it a 4/10. Still, once you clear that hurdle, you’re on solid ground for anything you build next.
The webhook setup was the real speed-bump. I’d never wired one up before, so it took multiple false starts before Meta finally acknowledged my callback URL. After experimenting with third-party tunneling tools, I scrapped the shortcuts and spun up a clean endpoint on my own domain. More than a few ChatGPT searches later, the test ping came back green. Painful…
Once that was done, I had gotten a test WhatsApp number and went to town.
Buying a Dedicated Number
With the webhook verified, I needed a proper WhatsApp Business number. I grabbed a prepaid line from Claro - shockingly painless:
Go to their website, show ID, pay.
Activate the SIM (or, in my case, load it as an eSIM).
Register the number in the WhatsApp Business Manager and run the SMS/voice verification.
No hidden dragons here; the process was far easier than I’d feared.
Ok so after this setup, technically you have everything that you need to build your own bot.
Technology
I’m not a traditional engineer, so much of this architecture was stitched together via ChatGPT, trial-and-error, and stubborn curiosity. Still, it worked - and I’m sharing it in case it saves someone else time.
This section is a summary of my conversations with ChatGPT - surprisingly good at extracting the relevant stuff. I just copied the entire chat into Gemini Pro 2.5 (because of the long context window) and asked for a recap.
1. The Core Idea & Tech Stack
Backend: Python with the Flask framework.
Because I knew Python already and it was easier for me to manage request/response. Plus Flask is one of the easiest frameworks to set up.
AI: OpenAI's GPT-4o-mini for all analysis and Whisper for audio transcription.
Database: MongoDB for flexible data storage (user profiles, food entries, conversations).
Deployment: An AWS EC2 instance, with Gunicorn as the application server, Nginx as a reverse proxy, and Certbot for SSL.
2. Application Architecture: A Modular Approach
To keep the project organized and scalable, I used the following modular structure:
app/
routes/: Handles incoming WhatsApp webhooks (webhook.py).
services/: Contains the core business logic.
media_analyzer.py & text_analyzer.py: Interface with the OpenAI API to process different user inputs.
food_processor.py: Parses the AI's response to extract structured data (calories, protein, etc.).
command_handler.py: Manages all slash commands like /hoje or /meta.
reminder_service.py: Logic for sending meal reminders.
database/: Manages all MongoDB interactions, with separate files for each collection (e.g., user_profiles.py, food_entries.py).
utils/: Helper functions, primarily for interacting with the WhatsApp API (whatsapp.py) and handling timezones.
config/: Stores settings, API keys, and system prompts.
3. The AI Brain: How It Understands Food
The more complex part of CalorAI lies in the AI analysis:
The System Prompt: I created a detailed system prompt that instructs the AI to act as "CalorIA." Most importantly, it enforces a strict JSON-like output format (Alimento: ..., Calorias: ...) which allows my food_processor.py service to reliably parse the response and save it to the database.
Multi-Modal Analysis:
Text: User descriptions are sent directly to GPT for analysis.
Images: Images are downloaded, converted to base64, and sent to GPT's vision model.
Audio: Audio messages are transcribed using OpenAI's Whisper, and the resulting text is then analyzed.
Documents: PDFs (like diet plans) have their text extracted using the PyMuPDF library, which is then summarized by the AI.
Contextual Conversations: The app saves conversation history in MongoDB and includes the last few messages in new API calls. This allows users to ask follow-up questions or make corrections naturally.
4. Key Features and Their Implementation
Food Logging & Summaries: Users can send food descriptions or use commands like /hoje, /semana, and /mes. The user_profiles.py and user_stats.py modules handle calculating daily totals and generating these summary reports by querying the food_entries collection in MongoDB.
Confirmation Flow: To prevent incorrect entries, the bot doesn't automatically save a food item. It first sends an interactive message with "✅ Sim" and "❌ Não" buttons. The food data is temporarily stored in a pending_food_entries collection until the user confirms.
Meal Reminders & The Scheduler:
The reminder_service.py checks for active users who haven't logged food yet or opted out of reminders, then sends a relevant meal prompt.
Usage Analytics: The analytics.py module queries the database to generate detailed insights, tracking metrics like total users, active users (daily/weekly/monthly), popular foods, and top-used commands. This is crucial for understanding user engagement.
5. Deployment to Production
Getting CalorAI online involved a standard setup on an AWS EC2 instance:
Server Setup: Installed Python, Nginx, and Git.
Code Deployment: Cloned the project from GitHub.
Application Server: Used Gunicorn to run the Flask app with multiple workers for concurrency.
Reverse Proxy: Configured Nginx to sit in front of Gunicorn, handle SSL, and forward traffic.
Automation: Created a systemd service (calorias.service) to manage Gunicorn, ensuring the app runs on boot and restarts automatically if it crashes.
6. Evolving the Product: Adding a Trial and Paywall
To turn this into a potential business, I implemented a trial and payment system.
Trial System:
Created a new user_subscriptions collection in MongoDB.
When a user sends their first-ever message, a new document is created with status: "trial" and an expiry date 7 days in the future.
The Paywall:
In the main webhook handler (webhook.py), I added a middleware check.
If a user's status is "expired," the bot intercepts their message, sends them payment instructions, and alerts me via a WhatsApp message.
Manual Activation:
I added a special, admin-only command: /activate <wa_id>.
This command updates the user's status in the database to "active", unlocking the bot's features for them and sending a confirmation message.
This entire process was actually an interesting challenge for me, given I hadn't done a end-to-end full-cycle product build: from a core idea and architecture to AI integration, feature development and deployment.
One small piece on Payments:
One thing I deliberately skipped early on was integrating a proper payment system.
Sure, I could’ve spent a few days adding Stripe or credit card support, but it felt like overkill. Instead, I went with something simple: manual PIX transfers.
Was it manual? Yes. Was it annoying? A little. But honestly, I figured that if this ever became a real problem - dozens of people pinging me asking where to pay - it’d be what I call a “happy pain.”
Better to feel the friction of success than waste time optimizing for scale too early.
Bot Usage & Getting It Out There (Marketing)
Once the bot was running smoothly, I started sharing it with a few close friends. I even got 10 of them to pay me R$19,90 for a month — which was already enough to make the project profitable.
As people used it, I began noticing patterns I hadn't anticipated. One user asked, “how do I send photos?” - which caught me off guard. I had assumed it was obvious: just send them like any image. Turns out, UX is mostly about over-clarifying the obvious.
After getting some early validation, I felt confident enough to try broader outreach. Here's the playbook I followed:
Spun up a quick landing page at usecaloria.com
Created an Instagram account
Launched some paid ads through Meta
Because WhatsApp, Instagram, and Facebook are all under Meta, setting up the ads was surprisingly smooth. I even had a video perform well — which was a cool milestone.
I generated most of the visuals and video content using ChatGPT and Canvas, with some prompt tuning to make the AI-generated photos look more “natural.”
User Behavior & What I Learned
Over the next few weeks, the number of people trying it out slowly grew — from a handful of friends to over 100 registered users.
That sounds like a modest number, but it gave me enough of a dataset to start noticing patterns.
Some takeaways:
Getting people to try it was easy. A 7-day free trial and the familiarity of WhatsApp meant there was no real friction. You just say “Hi” to a number — and you’re in.
A few users became what I’d call power users — sending dozens of messages right after onboarding. But most would send one or two messages and then ghost the bot.
What I learned:
The hardest part isn’t getting users to try it - it’s getting them to come back. Even though the whole idea is “you already have WhatsApp open,” people forget. Constantly.
The only way I see a service like this sticking is if it nails the timing of its reminders - pinging people when it actually makes sense to log a meal.
And here’s the catch: WhatsApp’s API charges for messages if the user hasn’t interacted in the last 24 hours. So unless someone’s already using the bot regularly, reminder messages cost money.
I even spoke to a friend who used something similar with a different provider - and his biggest complaint? “I just keep forgetting to use it.”
Turns out, building the assistant is the easy part.
Building the habit? That’s the real product.
What’s Next - for the project and for me
So… what made me pause everything and write this post?
My WhatsApp number got banned. Just like that.
On Sunday night, I was celebrating - +50 people trying it out in just 2 days, ads working, real momentum. By Monday morning, the bot had stopped responding. A few tests later, I logged into Facebook and saw the dreaded message.
That’s when I learned the hard way: building inside Meta’s ecosystem means you’re fully at the mercy of their moderation systems. And WhatsApp bans - even for business numbers - happen more often than you'd expect.
I still don’t know why it happened. I tried appealing, sent emails, no luck. Maybe it was related to the ad video I had running. Maybe not. A few weeks earlier, even Magie - a known WhatsApp partner - had their number banned for a good number of hours.
So yeah, fickle is the word.
Technically, I could just buy a new WhatsApp Business number and spin everything up again, it'd be a matter of just switching the number on the website and then reaching out to all of the users about this. It would probably take me a couple of hours. But the ban gave me a moment to pause and reflect: Do I actually want to keep pushing this forward?
After all, the original goal was simple: lose weight and learn how to build something on WhatsApp. I did both. (To be fair, I wasn’t only using CalorIA - I was following a diet plan and using the bot to help track things more easily.)
Despite the abrupt pause, CalorIA was one of the most rewarding projects I’ve built. I’m still reflecting on what’s next - but in the meantime, I did what I always do after a project stalls: I bought a new domain this Sunday and spun up a fresh landing page.
usefarol.com. Let's see where it goes.
