How I score Reddit job posts for scams with an LLM

If you've ever looked for freelance work on Reddit, you know how it goes. You open r/forhire or r/slavelabour, sort by New, and start scrolling. The first post is someone advertising their logo design services. The next one asks whether Upwork is dead. Then there's another freelancer's portfolio, and after that a "job" that wants you to message them on Telegram about an exciting opportunity in crypto.

The real jobs are in there, buried, and some of the ones you find aren't real either.

So I built Only Hiring. It reads every new post from 18 job subreddits and lists only the ones where someone is paying for work, on one page you can scan in a couple of minutes. Each job gets a short summary and a scam score from 0 to 1, and the pay is pulled out into numbers so you can sort by it.

This is how the LLM side of it works, with the prompt rules that mattered most.

How many of the posts are jobs

Here's what the 712 posts the pipeline has read so far turned out to be:

Those 712 are what's left after skipping every post tagged [For Hire] or [Offer], which never reach the model at all. Even after that, fewer than half are jobs.

Some subreddits are much worse than others. On r/softwareengineerjobs, 14 of the 147 posts I read were hiring. That's under 10%, and most of the rest were career questions. r/RemoteJobs and r/developerjobs weren't much better. On subreddits that enforce their tags, like r/forhire and r/DesignJobs, nearly every post that's left is a real job.

The pipeline

It's a small Python pipeline that runs on my machine:

scrape new posts ──> skip [For Hire] tags ──> LLM grades each post ──> clean the output
                                                                            │
website <── Postgres <── publish jobs only <── drop reposts and deleted posts

The website is Next.js on Vercel, and it only ever reads from the database.

Step 1: Don't send everything to the model

The cheapest filter is one that doesn't need an LLM. Most job subreddits make people tag their posts, so anything tagged [For Hire], [Offer] or Hire Me is skipped before it's even fetched. On r/forhire, a regex gets rid of most of the subreddit.

Everything else goes to the model, because tags don't catch the rest. People tag their own services as [Hiring] to get more views, and some "hiring" posts turn out to be course sales or questions.

Step 2: One prompt, one JSON object

Each post goes to an LLM (I use DeepSeek) with a single prompt. The model decides what kind of post it is and, if it's a job, pulls out everything useful:

{
  "is_job_post": true,
  "post_type": "hiring_job_offer",
  "job_title": "Video Editor",
  "summary": "...",
  "role_category": "video_animation",
  "seniority": null,
  "employment_type": "freelance_project",
  "pay_type": "per_unit",
  "pay_min": 25,
  "pay_max": 40,
  "pay_currency": "USD",
  "pay_raw": "$25-40 per video",
  "work_mode": "remote",
  "location": null,
  "skills": ["Premiere Pro", "After Effects"],
  "apply_method": "reddit_dm",
  "scam_score": 0.1,
  "red_flags": [],
  "reasoning": "A client offering paid editing work per video."
}

The rule I care most about for extraction is this one, copied from the prompt:

If a detail is not stated, set it to null. Missing information is expected and fine; a wrong value is much worse than null.

LLMs love to fill gaps. If a post doesn't say whether the work is remote, a model will often guess "remote" because most Reddit jobs are. That guess then shows up on the site as a fact, and someone filters by it. So only three fields may be inferred: the job title, the summary and the role category. Every other field has to come from the post itself. Every key is required too, so the model has to write null explicitly instead of leaving a field out.

Pay gets the most rules, because it's the field people care about most and it's the messiest. The model first copies the shortest exact bit of the post that states the pay ("$20-25/hr"), then splits it into a type, a min, a max and a currency. Keeping the raw excerpt means I can always check the parsed numbers against what the post said. "Competitive salary" or a bare [PAID] tag keeps its excerpt but gets no numbers.

Step 3: Scoring scams

I spent more time on this part than any other. The prompt has a scoring guide with three bands:

0.0-0.2: identifiable employer or concrete project, specific deliverables, realistic pay.
0.3-0.5: vague work and no details, commission-only with no clear product,
         pay that seems off for the work.
0.6-1.0: classic scam or exploitation signs, such as: upfront fees or paid "training";
         crypto or forex trading and "investment" schemes; contact only through
         Telegram/WhatsApp; requests for ID, bank or personal details; renting or
         "borrowing" someone's LinkedIn/Upwork/bank account; reshipping or payment
         processing; MLM; unpaid "test" tasks that are real work; very high pay for
         trivial work.

The list that made the biggest difference is the opposite one, the things that are not red flags:

Normal on Reddit and NOT warning signs on their own: applying by Reddit DM or comment,
an individual client who gives no company name, pay not stated, payment in crypto,
and job requirements such as location, age, language or equipment.

Without it, a model that has absorbed generic "how to spot a job scam" advice flags half of Reddit. Real clients on r/slavelabour are anonymous individuals who want a DM and don't name a company, because they don't have one. Advice written for LinkedIn treats that as suspicious, and on Reddit it's how most hiring works. So the score has to measure how unusual a post is for Reddit specifically.

The model also has to list each warning sign it found in red_flags. The site shows them next to the warning, so you can see why a post was flagged and decide for yourself. "Scam score: 0.7" on its own tells you very little, while "Telegram-only contact" and "unpaid sample before the go-ahead" tell you what to look out for.

On the site, a score of 0.4 or more gets a visible warning, and 0.6 or more hides the post unless you choose to see likely scams.

Step 4: Never trust the JSON

Even with a strict format, model output is untrusted input. Every response goes through a sanitiser before it's stored:

• Unknown keys are dropped, and enum values that aren't on the list become null.
• Strings like "N/A", "null", "unknown" and "not specified" become real nulls.
• Numbers are parsed ("1,500" becomes 1500), and negative pay is thrown away.
• If pay_min is bigger than pay_max, they're swapped.
• If the post isn't a job, every job field is cleared, whatever the model said.
• Currency codes have to be three letters.

None of it is clever, but without it, bad values leak through to the filters. It's the same rule I follow in client work: check an AI step's output in code before anything else uses it.

Step 5: Reposts and deleted posts

Hiring posts get crossposted to three subreddits at once, or reposted every few days to bump them. A post counts as a duplicate when the same author posts near-identical text (90% similar or more, using Python's difflib). The copy reuses the first one's grade instead of going through the model again, and only the newest copy is published.

Posts also disappear. Authors delete them once the job's filled, and Reddit's spam filters remove others. Every run asks Reddit which of the last 14 days' jobs are gone and takes those off the site. On one day, 24 of the 221 jobs from the previous two weeks had already gone.

What the scores found

I expected the scam scores to catch a lot of scams. They mostly didn't, because there aren't many. Of the 311 hiring posts, 40 got a warning (13%), and only 6 scored high enough to be hidden.

What I found much more often was exploitation: real jobs on bad terms. These are some of the red flags the model wrote down:

• "unpaid sample clip requested before go-ahead"
• "payment only after all 30 clips are approved"
• "full usage rights granted for a $30 bulk set"
• "very low pay for live screen-share work"
• "unpaid substantive rewriting assessment"
• "WhatsApp-only contact"

Most of these jobs are real. They pay very little, or want a lot of free work first, which is why I think the red flags matter more than the score: "payment only after all 30 clips are approved" tells you exactly what you'd be agreeing to. The guide to Reddit job scams breaks it down by subreddit.

An LLM writing tic I didn't expect

The first version of the prompt asked for a summary of "what the work is and who it suits". The model took the second half literally. At one point 65% of the summaries had the word "suit" in them, and 44% had a sentence starting "It suits...". A page of jobs read like a page of horoscopes.

The fix was to ban stock phrases in the prompt ("It suits...", "ideal for", "great opportunity", "Seeking a...") and ask for the most useful specific detail instead, like the deliverables or the timeline. Re-grading every listed job took the summaries mentioning "suit" from 171 to 1.

If you're generating text for lots of similar items, read twenty of them in a row. A tic you'd never spot in one output is obvious in a list.

What you get on the site

All that extracted data is what makes the board quick to scan. Every job from the 18 subreddits is in one list, newest first. You can filter by role (19 categories, from video editing to data and AI), by experience level, by remote, hybrid or on-site, and by how it pays: hourly, fixed price, per piece, monthly and so on. Sorting by highest pay only works because pay is parsed into numbers. One tick hides commission-only jobs, and search covers titles, skills, locations and subreddits.

Scam warnings show their reasons, and likely scams are hidden by default. You can mark jobs you've applied to, then hide them or see only those. That works without an account, and signing in keeps the list across devices. There's also an RSS feed of new jobs, and a page for each subreddit with what it pays and how many of its posts are real jobs.

You still apply on Reddit. Every job links to the original post, and the site only keeps a summary and the post's opening lines.

What it gets wrong

The model makes mistakes. Sometimes it misses a red flag or misreads a pay figure, and now and then it flags something harmless. So the site says so and links every job to its source, and the footer tells you to read the original post before you reply to anyone. Treat a scam score as a hint and make the call yourself.

Some fields are often empty because the posts don't say. Only about one job in five states a seniority level, for example, so the experience filter has a "Not stated" option. I'd rather show that than guess.

Try it

It's free, with no sign-up needed: onlyhiring.hassandev.me. The website's code is on GitHub under the MIT licence.

If you know a subreddit with real job posts that I'm not reading, or you find a post it got wrong, tell me through the Feedback link on the site.