Spaces:
Sleeping
Fix thin proposals, the hire-rate verdict, the sector list and the thin-dossier warning
Browse filesFour problems students reported with the proposal platform.
Proposals were thin and quoted the "selected offer". The dossier reader
joined each PDF page or Word file into one block, so the keyword scanner
filed whole pages (cut at 600 characters) as single facts, coaching
headings and market statistics included.
- PDF and Word files are now read with their headings and paragraph
breaks; running headers, footers and page numbers are dropped, and a
file named loosely is placed by its cover-page title.
- Each section of the team templates gets a role: facts are kept; offer,
target client and value proposition only steer the angle; weaknesses
and skills marked "Develop" become things never to claim; roadmaps,
SWOT opportunities and threats, market sizing, income targets and
"how this helps the client" are skipped. Snapshot rows become separate
claims, and promised outcomes are never kept as results.
- The Upwork Profile Optimisation document now outranks the Dossier, and
the Offer Bank is ignored once either is uploaded.
- The proposal prompt uses the offer only for the angle, never names it,
and rewrites coaching text in the first person; internal labels and
pasted capitalised headings are stripped from the output.
- On Groq the request is trimmed to fit the free per-minute allowance,
the critique re-ask is skipped, and GPT-OSS runs at low reasoning
effort. A size retry is always smaller than the first try.
- The no-AI outline is labelled "AI writing was unavailable" instead of
"Your draft is ready", and quotes strength titles, not coaching text.
Every job read "hire rate below 25%". A hire rate the screenshot did not
show was scored as a low one.
- A missing hire rate is now a caution ("check it on Upwork"); only a
visible low rate from a client with 5+ jobs posted is a NO GO, and a
visible rate above zero outranks a stray "Hires: 0".
- The screenshot reader is told that "Hires" under "Activity on this
job" is not the client's total, and the screen asks for the "About the
client" box. The verdict screen lets the user type hire rate, jobs
posted and payment status when they are missing.
- The proposal screen's "not strongly recommended" warning follows the
verdict the user sees, not the hidden fit-score tier.
The interview's industry list had 10 choices. It now has the 30 sectors
named across the discovery-call scripts (Legal and Academia first), the
dropdown accepts typed entries, and the text box is labelled. The skill
categories gain Legal, Accounting, Academic writing, Finance, HR and
Translation.
Good dossiers were called thin because the strength score read file
names only. The folder is now re-scored from the evidence actually read,
and the gap-fill message names what is missing. A 36-page team dossier
moves from 30 to 85 out of 100.
Pricing detection no longer matches "rate" inside words such as
"strategist" or "rated".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- app/models/schemas.py +8 -4
- app/prompts/proposal_prompt.py +12 -1
- app/services/beginner_evaluator.py +49 -17
- app/services/dossier_reader.py +265 -16
- app/services/evidence_index.py +295 -6
- app/services/folder_validator.py +89 -1
- app/services/interview.py +26 -6
- app/services/llm_client.py +7 -0
- app/services/proposal_generator.py +91 -20
- app/services/screenshot_parser.py +6 -1
- app/ui/analysis_screen.py +80 -0
- app/ui/dossier_screen.py +37 -5
- app/ui/interview_screen.py +14 -3
- app/ui/proposal_screen.py +41 -12
- app/ui/screenshot_screen.py +4 -2
|
@@ -42,10 +42,14 @@ SourceType = Literal[
|
|
| 42 |
|
| 43 |
SOURCE_PRIORITY: dict[str, int] = {
|
| 44 |
"structured_profile_json": 1,
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
"linkedin_profile": 6,
|
| 50 |
"pricing_or_service_package": 7,
|
| 51 |
"portfolio_or_case_study": 8,
|
|
|
|
| 42 |
|
| 43 |
SOURCE_PRIORITY: dict[str, int] = {
|
| 44 |
"structured_profile_json": 1,
|
| 45 |
+
# The Upwork Profile Optimisation document is already written for
|
| 46 |
+
# clients, in the first person, with the student's real numbers, so it
|
| 47 |
+
# outranks the Dossier, which is written to the student. The Offer Bank
|
| 48 |
+
# lists offers the student did not choose, so it ranks below both.
|
| 49 |
+
"upwork_profile": 2,
|
| 50 |
+
"skillarbitrage_dossier_roadmap": 3,
|
| 51 |
+
"linkedin_optimization": 4,
|
| 52 |
+
"offer_blueprint": 5,
|
| 53 |
"linkedin_profile": 6,
|
| 54 |
"pricing_or_service_package": 7,
|
| 55 |
"portfolio_or_case_study": 8,
|
|
@@ -94,10 +94,21 @@ anywhere in the proposal.
|
|
| 94 |
Evidence marked "Practice project (self-initiated, not client work)" is real
|
| 95 |
work the person did on their own. Never present it as a client engagement, a
|
| 96 |
paid project or a client result.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
|
| 98 |
CHECK A2 — PROFILE ALIGNMENT (use the <profile> for positioning + voice)
|
| 99 |
The proposal must read as if written BY the person in <profile>:
|
| 100 |
-
-
|
|
|
|
|
|
|
|
|
|
| 101 |
- Lead with their core strength / unique background from "strengths"
|
| 102 |
(e.g. a sales/communications background, a specific industry, a metric).
|
| 103 |
- Match the writing voice and tone implied by their profile.
|
|
|
|
| 94 |
Evidence marked "Practice project (self-initiated, not client work)" is real
|
| 95 |
work the person did on their own. Never present it as a client engagement, a
|
| 96 |
paid project or a client result.
|
| 97 |
+
Much of the evidence comes from coaching documents written TO the freelancer
|
| 98 |
+
("You review every product page…", "Your strength is…"). Rewrite it in the
|
| 99 |
+
freelancer's own first-person voice; never copy it word for word. Outcomes an
|
| 100 |
+
offer promises ("increase conversion by 40-80%", "at least 30% within 90
|
| 101 |
+
days") are goals, not past results: never present them as something the
|
| 102 |
+
freelancer has achieved. Never mention market-size statistics, income
|
| 103 |
+
targets, the freelancer's training programme, or what a document calls
|
| 104 |
+
itself.
|
| 105 |
|
| 106 |
CHECK A2 — PROFILE ALIGNMENT (use the <profile> for positioning + voice)
|
| 107 |
The proposal must read as if written BY the person in <profile>:
|
| 108 |
+
- Use "selected_offer" and "title_or_positioning" only to choose the angle.
|
| 109 |
+
Never quote them and never name the offer: internal labels such as
|
| 110 |
+
"Offer 2", "Part A", "Niche 1A", "selected offer" or "offer statement"
|
| 111 |
+
must not appear anywhere in the proposal.
|
| 112 |
- Lead with their core strength / unique background from "strengths"
|
| 113 |
(e.g. a sales/communications background, a specific industry, a metric).
|
| 114 |
- Match the writing voice and tone implied by their profile.
|
|
@@ -15,7 +15,8 @@ offline (and in tests).
|
|
| 15 |
The 9 signals (each votes GO / CAUTION / NO GO):
|
| 16 |
|
| 17 |
1. Proposals — <20 GO · 20-50 CAUTION · 50+ NO GO
|
| 18 |
-
2. Hire rate — 50%+ GO · 25-50% CAUTION · <25%
|
|
|
|
| 19 |
3. Payment — verified GO · not verified NO GO · no data CAUTION
|
| 20 |
4. Client rating — 4.8+ GO · 4.5-4.8 CAUTION · <4.5 NO GO · no rating CAUTION
|
| 21 |
5. Job desc. — clear scope GO · everything else CAUTION (never NO GO)
|
|
@@ -65,8 +66,8 @@ REASON_PROPOSALS_50_PLUS = (
|
|
| 65 |
"for a beginner profile."
|
| 66 |
)
|
| 67 |
REASON_HIRE_RATE_LOW = (
|
| 68 |
-
"The client
|
| 69 |
-
"hire — a high risk of wasted connects."
|
| 70 |
)
|
| 71 |
REASON_PAYMENT_NOT_VERIFIED = (
|
| 72 |
"Payment is not verified, so there is a higher risk of not getting paid."
|
|
@@ -92,6 +93,20 @@ WARN_PROPOSALS_20_49 = (
|
|
| 92 |
"Competition is moderate (20-50 proposals), so the proposal must be very strong."
|
| 93 |
)
|
| 94 |
WARN_HIRE_RATE_MID = "The client's hire rate is 25-50%, so a hire is not guaranteed."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
WARN_PAYMENT_UNKNOWN = "Payment verification was not visible, so confirm it before applying."
|
| 96 |
WARN_RATING_MID = (
|
| 97 |
"The client's rating is 4.5-4.8 — read the past reviews carefully before applying."
|
|
@@ -403,6 +418,10 @@ def _client_activity_bucket(jobs: Optional[int], hires: Optional[int], hire_rate
|
|
| 403 |
"""
|
| 404 |
if hires is None and jobs is None and hire_rate is None:
|
| 405 |
return "not_visible"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 406 |
# Explicit "posted jobs, hired no one" signal.
|
| 407 |
if hires == 0 and ((jobs or 0) >= 1 or (hire_rate is not None and hire_rate == 0)):
|
| 408 |
return "dead"
|
|
@@ -493,16 +512,22 @@ def evaluate(
|
|
| 493 |
jobs_n = (_ints(jobs_value)[0] if _ints(jobs_value) else None)
|
| 494 |
hires_n = (_ints(hires_value)[0] if _ints(hires_value) else None)
|
| 495 |
|
| 496 |
-
# If the hire-rate % isn't shown but the client
|
| 497 |
-
#
|
| 498 |
-
#
|
| 499 |
-
|
| 500 |
-
|
|
|
|
| 501 |
if _is_missing(hire_value):
|
| 502 |
-
hire_value = "
|
| 503 |
hire_b = _hire_rate_bucket(hire)
|
| 504 |
activity_b = _client_activity_bucket(jobs_n, hires_n, hire)
|
| 505 |
new_client = _is_new_client(jobs_n, hires_n)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 506 |
|
| 507 |
# Signal 5 — job description quality (LLM if supplied, else heuristic).
|
| 508 |
if isinstance(job_desc_quality, dict) and job_desc_quality.get("status") in (GO, CAUTION):
|
|
@@ -530,9 +555,10 @@ def evaluate(
|
|
| 530 |
else:
|
| 531 |
add("proposals", "Proposals", NOT_VISIBLE, CAUTION, "Proposal count was not visible.")
|
| 532 |
|
| 533 |
-
# 2 — Hire rate. A low
|
| 534 |
-
#
|
| 535 |
-
# often win work from new clients
|
|
|
|
| 536 |
if hire_b == "high":
|
| 537 |
add("hire_rate", "Hire Rate", hire_value, GO)
|
| 538 |
elif hire_b == "mid":
|
|
@@ -540,10 +566,12 @@ def evaluate(
|
|
| 540 |
elif new_client:
|
| 541 |
add("hire_rate", "Hire Rate", (hire_value or NOT_VISIBLE) + " — new client", GO,
|
| 542 |
"")
|
| 543 |
-
elif hire_b == "low":
|
| 544 |
add("hire_rate", "Hire Rate", hire_value, NO_GO, REASON_HIRE_RATE_LOW)
|
|
|
|
|
|
|
| 545 |
else:
|
| 546 |
-
add("hire_rate", "Hire Rate", NOT_VISIBLE,
|
| 547 |
|
| 548 |
# 3 — Payment verified
|
| 549 |
if payment == "verified":
|
|
@@ -618,8 +646,10 @@ def evaluate(
|
|
| 618 |
(data if data != NOT_VISIBLE else "New client") + " — new client", GO, "")
|
| 619 |
elif activity_b == "mixed":
|
| 620 |
add("client_activity", "Client Activity", data, CAUTION, WARN_ACTIVITY_MIXED)
|
| 621 |
-
elif activity_b == "dead":
|
| 622 |
add("client_activity", "Client Activity", data, NO_GO, REASON_ACTIVITY_DEAD)
|
|
|
|
|
|
|
| 623 |
else:
|
| 624 |
add("client_activity", "Client Activity", NOT_VISIBLE, CAUTION, WARN_ACTIVITY_UNKNOWN)
|
| 625 |
|
|
@@ -696,8 +726,10 @@ def evaluate(
|
|
| 696 |
|
| 697 |
score_signals = {
|
| 698 |
"payment_not_verified": payment == "not_verified",
|
| 699 |
-
#
|
| 700 |
-
|
|
|
|
|
|
|
| 701 |
"hire_rate_mid": hire_b == "mid",
|
| 702 |
"hire_rate_high": hire_b == "high",
|
| 703 |
"rating_below_4_5": rating_b == "low",
|
|
|
|
| 15 |
The 9 signals (each votes GO / CAUTION / NO GO):
|
| 16 |
|
| 17 |
1. Proposals — <20 GO · 20-50 CAUTION · 50+ NO GO
|
| 18 |
+
2. Hire rate — 50%+ GO · 25-50% CAUTION · <25% NO GO (only for a client
|
| 19 |
+
with a track record) · not visible CAUTION · new client GO
|
| 20 |
3. Payment — verified GO · not verified NO GO · no data CAUTION
|
| 21 |
4. Client rating — 4.8+ GO · 4.5-4.8 CAUTION · <4.5 NO GO · no rating CAUTION
|
| 22 |
5. Job desc. — clear scope GO · everything else CAUTION (never NO GO)
|
|
|
|
| 66 |
"for a beginner profile."
|
| 67 |
)
|
| 68 |
REASON_HIRE_RATE_LOW = (
|
| 69 |
+
"The client has posted several jobs but hires for fewer than 1 in 4, so "
|
| 70 |
+
"they rarely hire — a high risk of wasted connects."
|
| 71 |
)
|
| 72 |
REASON_PAYMENT_NOT_VERIFIED = (
|
| 73 |
"Payment is not verified, so there is a higher risk of not getting paid."
|
|
|
|
| 93 |
"Competition is moderate (20-50 proposals), so the proposal must be very strong."
|
| 94 |
)
|
| 95 |
WARN_HIRE_RATE_MID = "The client's hire rate is 25-50%, so a hire is not guaranteed."
|
| 96 |
+
# A hire rate that could not be read is unknown, not low: it is checked by the
|
| 97 |
+
# user, never held against the client.
|
| 98 |
+
WARN_HIRE_RATE_UNKNOWN = (
|
| 99 |
+
"The client's hire rate was not visible in the screenshot. Check it in the "
|
| 100 |
+
"'About the client' box on Upwork before applying."
|
| 101 |
+
)
|
| 102 |
+
WARN_HIRE_RATE_LOW_UNPROVEN = (
|
| 103 |
+
"The hire rate looks low, but the client's job history was not visible. "
|
| 104 |
+
"Check how many jobs they have posted before applying."
|
| 105 |
+
)
|
| 106 |
+
WARN_ACTIVITY_UNPROVEN = (
|
| 107 |
+
"The client shows no hires yet, but their job history was not visible. "
|
| 108 |
+
"Check how many jobs they have posted before applying."
|
| 109 |
+
)
|
| 110 |
WARN_PAYMENT_UNKNOWN = "Payment verification was not visible, so confirm it before applying."
|
| 111 |
WARN_RATING_MID = (
|
| 112 |
"The client's rating is 4.5-4.8 — read the past reviews carefully before applying."
|
|
|
|
| 418 |
"""
|
| 419 |
if hires is None and jobs is None and hire_rate is None:
|
| 420 |
return "not_visible"
|
| 421 |
+
# A visible hire rate above zero outranks a hires count of 0: that 0 is
|
| 422 |
+
# most likely the "Hires: 0" Upwork shows for this one open post.
|
| 423 |
+
if hire_rate is not None and hire_rate > 0:
|
| 424 |
+
return "consistent" if hire_rate >= 50 else "mixed"
|
| 425 |
# Explicit "posted jobs, hired no one" signal.
|
| 426 |
if hires == 0 and ((jobs or 0) >= 1 or (hire_rate is not None and hire_rate == 0)):
|
| 427 |
return "dead"
|
|
|
|
| 512 |
jobs_n = (_ints(jobs_value)[0] if _ints(jobs_value) else None)
|
| 513 |
hires_n = (_ints(hires_value)[0] if _ints(hires_value) else None)
|
| 514 |
|
| 515 |
+
# If the hire-rate % isn't shown but both the client's jobs posted and
|
| 516 |
+
# total hires are, the rate can be worked out from them. Both counts are
|
| 517 |
+
# needed: a hires count on its own may be the "Hires: 0" that Upwork shows
|
| 518 |
+
# for this one open post, which says nothing about the client.
|
| 519 |
+
if hire is None and hires_n is not None and jobs_n:
|
| 520 |
+
hire = min(100.0, 100.0 * hires_n / jobs_n)
|
| 521 |
if _is_missing(hire_value):
|
| 522 |
+
hire_value = f"{round(hire)}% ({hires_n} hires from {jobs_n} jobs)"
|
| 523 |
hire_b = _hire_rate_bucket(hire)
|
| 524 |
activity_b = _client_activity_bucket(jobs_n, hires_n, hire)
|
| 525 |
new_client = _is_new_client(jobs_n, hires_n)
|
| 526 |
+
# A low hire rate only means something once the client has a real record:
|
| 527 |
+
# enough jobs posted (or, failing that, enough hires) to judge them by.
|
| 528 |
+
# Without that evidence, low or missing numbers are a reason to check,
|
| 529 |
+
# never a reason to skip the job.
|
| 530 |
+
has_history = not new_client and (jobs_n is not None or hires_n is not None)
|
| 531 |
|
| 532 |
# Signal 5 — job description quality (LLM if supplied, else heuristic).
|
| 533 |
if isinstance(job_desc_quality, dict) and job_desc_quality.get("status") in (GO, CAUTION):
|
|
|
|
| 555 |
else:
|
| 556 |
add("proposals", "Proposals", NOT_VISIBLE, CAUTION, "Proposal count was not visible.")
|
| 557 |
|
| 558 |
+
# 2 — Hire rate. A low hire rate is a NO GO only for a client with a real
|
| 559 |
+
# track record. For a NEW client (few jobs posted) it is expected and fine —
|
| 560 |
+
# beginners often win work from new clients. A rate that was not visible,
|
| 561 |
+
# or a low rate with no visible history, is a CAUTION: check it on Upwork.
|
| 562 |
if hire_b == "high":
|
| 563 |
add("hire_rate", "Hire Rate", hire_value, GO)
|
| 564 |
elif hire_b == "mid":
|
|
|
|
| 566 |
elif new_client:
|
| 567 |
add("hire_rate", "Hire Rate", (hire_value or NOT_VISIBLE) + " — new client", GO,
|
| 568 |
"")
|
| 569 |
+
elif hire_b == "low" and has_history:
|
| 570 |
add("hire_rate", "Hire Rate", hire_value, NO_GO, REASON_HIRE_RATE_LOW)
|
| 571 |
+
elif hire_b == "low":
|
| 572 |
+
add("hire_rate", "Hire Rate", hire_value, CAUTION, WARN_HIRE_RATE_LOW_UNPROVEN)
|
| 573 |
else:
|
| 574 |
+
add("hire_rate", "Hire Rate", NOT_VISIBLE, CAUTION, WARN_HIRE_RATE_UNKNOWN)
|
| 575 |
|
| 576 |
# 3 — Payment verified
|
| 577 |
if payment == "verified":
|
|
|
|
| 646 |
(data if data != NOT_VISIBLE else "New client") + " — new client", GO, "")
|
| 647 |
elif activity_b == "mixed":
|
| 648 |
add("client_activity", "Client Activity", data, CAUTION, WARN_ACTIVITY_MIXED)
|
| 649 |
+
elif activity_b == "dead" and has_history:
|
| 650 |
add("client_activity", "Client Activity", data, NO_GO, REASON_ACTIVITY_DEAD)
|
| 651 |
+
elif activity_b == "dead":
|
| 652 |
+
add("client_activity", "Client Activity", data, CAUTION, WARN_ACTIVITY_UNPROVEN)
|
| 653 |
else:
|
| 654 |
add("client_activity", "Client Activity", NOT_VISIBLE, CAUTION, WARN_ACTIVITY_UNKNOWN)
|
| 655 |
|
|
|
|
| 726 |
|
| 727 |
score_signals = {
|
| 728 |
"payment_not_verified": payment == "not_verified",
|
| 729 |
+
# Only a visible low rate from a client with a track record counts
|
| 730 |
+
# against them; a blank rate, or a new client's low one, does not.
|
| 731 |
+
"hire_rate_below_25": hire_b == "low" and has_history,
|
| 732 |
+
"hire_rate_unknown": hire_b == "not_visible" and not new_client,
|
| 733 |
"hire_rate_mid": hire_b == "mid",
|
| 734 |
"hire_rate_high": hire_b == "high",
|
| 735 |
"rating_below_4_5": rating_b == "low",
|
|
@@ -6,8 +6,10 @@ raw extracted text is never logged.
|
|
| 6 |
|
| 7 |
Supported file types and their handlers:
|
| 8 |
|
| 9 |
-
* ``.pdf`` -> ``pdfplumber`` (one chunk per page
|
| 10 |
-
|
|
|
|
|
|
|
| 11 |
* ``.txt`` / ``.md`` -> read as utf-8 text.
|
| 12 |
* ``.json`` -> parsed via ``json``. The structured payload is preserved.
|
| 13 |
* ``.csv`` -> read as text plus a small ``json_data`` preview.
|
|
@@ -22,6 +24,8 @@ from __future__ import annotations
|
|
| 22 |
import csv
|
| 23 |
import hashlib
|
| 24 |
import json
|
|
|
|
|
|
|
| 25 |
from dataclasses import dataclass, field
|
| 26 |
from pathlib import Path
|
| 27 |
from typing import Iterable, Optional
|
|
@@ -129,6 +133,179 @@ def _read_csv_preview(path: Path, max_rows: int = 50) -> tuple[str, list[list[st
|
|
| 129 |
return "\n".join(text_lines), rows, None
|
| 130 |
|
| 131 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
def _pdf_chunks(
|
| 133 |
path: Path,
|
| 134 |
base_meta: dict,
|
|
@@ -142,27 +319,41 @@ def _pdf_chunks(
|
|
| 142 |
try:
|
| 143 |
with pdfplumber.open(str(path)) as pdf:
|
| 144 |
page_iter = pdf.pages[:_PDF_MAX_PAGES]
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
for index, page in enumerate(page_iter, start=1):
|
| 146 |
try:
|
| 147 |
-
|
| 148 |
except Exception as exc: # noqa: BLE001 - per-page tolerance
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
records.append(
|
| 150 |
ChunkRecord(
|
| 151 |
chunk_id=_chunk_id(base_meta["file_path"], f"p{index}"),
|
| 152 |
page_number=index,
|
| 153 |
extracted_text="",
|
| 154 |
extraction_status="failed",
|
| 155 |
-
extraction_warning=
|
| 156 |
**base_meta,
|
| 157 |
)
|
| 158 |
)
|
| 159 |
continue
|
|
|
|
|
|
|
| 160 |
records.append(
|
| 161 |
ChunkRecord(
|
| 162 |
chunk_id=_chunk_id(base_meta["file_path"], f"p{index}"),
|
| 163 |
page_number=index,
|
| 164 |
-
|
| 165 |
-
|
|
|
|
| 166 |
**base_meta,
|
| 167 |
)
|
| 168 |
)
|
|
@@ -171,6 +362,51 @@ def _pdf_chunks(
|
|
| 171 |
return records, None
|
| 172 |
|
| 173 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 174 |
def _docx_chunk(
|
| 175 |
path: Path,
|
| 176 |
base_meta: dict,
|
|
@@ -185,15 +421,28 @@ def _docx_chunk(
|
|
| 185 |
except Exception as exc: # noqa: BLE001
|
| 186 |
return [], f"docx open failed: {exc.__class__.__name__}"
|
| 187 |
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 195 |
|
| 196 |
-
full_text = "\n".join(
|
| 197 |
status: ExtractionStatus = "ok" if full_text else "empty"
|
| 198 |
return (
|
| 199 |
[
|
|
@@ -324,13 +573,13 @@ def _read_one_file(path: Path, rel_path: Path, ext: str) -> list[ChunkRecord]:
|
|
| 324 |
if pdf_chunks:
|
| 325 |
if warning:
|
| 326 |
pdf_chunks[-1].extraction_warning = _sanitize_warning(warning)
|
| 327 |
-
return pdf_chunks
|
| 328 |
return [_failed_chunk(path, base_meta, warning or "no pages")]
|
| 329 |
|
| 330 |
if ext == ".docx":
|
| 331 |
docx_chunks, warning = _docx_chunk(path, base_meta)
|
| 332 |
if docx_chunks:
|
| 333 |
-
return docx_chunks
|
| 334 |
return [_failed_chunk(path, base_meta, warning or "no content")]
|
| 335 |
|
| 336 |
if ext in {".txt", ".md"}:
|
|
|
|
| 6 |
|
| 7 |
Supported file types and their handlers:
|
| 8 |
|
| 9 |
+
* ``.pdf`` -> ``pdfplumber`` (one chunk per page; headings and paragraph
|
| 10 |
+
breaks rebuilt from the layout).
|
| 11 |
+
* ``.docx`` -> ``python-docx`` (paragraphs + tables in document order, single
|
| 12 |
+
chunk per file; headings marked).
|
| 13 |
* ``.txt`` / ``.md`` -> read as utf-8 text.
|
| 14 |
* ``.json`` -> parsed via ``json``. The structured payload is preserved.
|
| 15 |
* ``.csv`` -> read as text plus a small ``json_data`` preview.
|
|
|
|
| 24 |
import csv
|
| 25 |
import hashlib
|
| 26 |
import json
|
| 27 |
+
import re
|
| 28 |
+
import statistics
|
| 29 |
from dataclasses import dataclass, field
|
| 30 |
from pathlib import Path
|
| 31 |
from typing import Iterable, Optional
|
|
|
|
| 133 |
return "\n".join(text_lines), rows, None
|
| 134 |
|
| 135 |
|
| 136 |
+
# ---------------------------------------------------------------------------
|
| 137 |
+
# Document layout: headings and paragraphs
|
| 138 |
+
# ---------------------------------------------------------------------------
|
| 139 |
+
#
|
| 140 |
+
# The evidence scanner works paragraph by paragraph and reads a Markdown-style
|
| 141 |
+
# "## Heading" paragraph as the start of a section. PDF and Word files carry
|
| 142 |
+
# neither marker on their own — pdfplumber returns a page as single-spaced
|
| 143 |
+
# lines and Word paragraphs used to be joined with single line breaks — so a
|
| 144 |
+
# whole page or file reached the scanner as one paragraph. The helpers below
|
| 145 |
+
# rebuild both: headings come out as "## " (top level) or "### " (sub level)
|
| 146 |
+
# paragraphs, and body text is split where the layout shows a paragraph break.
|
| 147 |
+
|
| 148 |
+
# A line set this many points larger than the body text is a heading, and
|
| 149 |
+
# this many more is a top-level heading.
|
| 150 |
+
_HEADING_SIZE_STEP = 1.5
|
| 151 |
+
_TOP_HEADING_SIZE_STEP = 4.0
|
| 152 |
+
_HEADING_MAX_CHARS = 160
|
| 153 |
+
# A gap between lines this many times the usual line gap starts a paragraph.
|
| 154 |
+
_PARAGRAPH_GAP_FACTOR = 1.8
|
| 155 |
+
_LIST_ITEM = re.compile(r"^\s*(?:\d{1,2}[.)]|[•▪●◦]|\\?#)\s")
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def _is_caps_heading(text: str) -> bool:
|
| 159 |
+
"""True for a short ALL-CAPS line such as "HOW THIS HELPS THE CLIENT".
|
| 160 |
+
|
| 161 |
+
Five letters is the floor, so a lone acronym ("HSBC") in body text is not
|
| 162 |
+
mistaken for a heading.
|
| 163 |
+
"""
|
| 164 |
+
stripped = text.strip()
|
| 165 |
+
letters = [c for c in stripped if c.isalpha()]
|
| 166 |
+
return (
|
| 167 |
+
len(letters) >= 5
|
| 168 |
+
and len(stripped) <= 80
|
| 169 |
+
and all(c.isupper() for c in letters)
|
| 170 |
+
and not stripped.endswith((".", ","))
|
| 171 |
+
)
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
def _heading_marker(level: int, text: str) -> str:
|
| 175 |
+
return ("## " if level == 1 else "### ") + " ".join(text.split())[:_HEADING_MAX_CHARS]
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def _pdf_line_size(line: dict) -> float:
|
| 179 |
+
sizes = [float(c.get("size") or 0.0) for c in line.get("chars") or []]
|
| 180 |
+
return max(sizes) if sizes else 0.0
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
def _pdf_line_bold(line: dict) -> bool:
|
| 184 |
+
chars = [c for c in line.get("chars") or [] if str(c.get("text") or "").strip()]
|
| 185 |
+
return bool(chars) and all("bold" in str(c.get("fontname") or "").lower() for c in chars)
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def _pdf_body_size(pages_lines: list[list[dict]]) -> float:
|
| 189 |
+
"""The most common text size in the document, weighted by characters."""
|
| 190 |
+
weights: dict[float, int] = {}
|
| 191 |
+
for lines in pages_lines:
|
| 192 |
+
for line in lines:
|
| 193 |
+
size = round(_pdf_line_size(line), 1)
|
| 194 |
+
weights[size] = weights.get(size, 0) + len(str(line.get("text") or ""))
|
| 195 |
+
return max(weights, key=weights.get) if weights else 0.0
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def _pdf_page_text(
|
| 199 |
+
lines: list[dict], body_size: float, open_headings: list[str]
|
| 200 |
+
) -> str:
|
| 201 |
+
"""Rebuild one page's text with heading markers and paragraph breaks.
|
| 202 |
+
|
| 203 |
+
``open_headings`` holds the top-level and sub-level heading in force when
|
| 204 |
+
the page starts; it is repeated at the top of the page (so a section that
|
| 205 |
+
runs across a page break keeps its name) and updated in place.
|
| 206 |
+
"""
|
| 207 |
+
gaps = [
|
| 208 |
+
float(cur["top"]) - float(prev["bottom"])
|
| 209 |
+
for prev, cur in zip(lines, lines[1:])
|
| 210 |
+
if float(cur["top"]) >= float(prev["bottom"])
|
| 211 |
+
and abs(_pdf_line_size(cur) - body_size) < _HEADING_SIZE_STEP
|
| 212 |
+
]
|
| 213 |
+
usual_gap = statistics.median(gaps) if gaps else 0.0
|
| 214 |
+
paragraph_gap = max(usual_gap * _PARAGRAPH_GAP_FACTOR, usual_gap + body_size * 0.5, 3.0)
|
| 215 |
+
|
| 216 |
+
blocks: list[str] = [h for h in open_headings if h]
|
| 217 |
+
paragraph: list[str] = []
|
| 218 |
+
heading: Optional[tuple[int, list[str]]] = None
|
| 219 |
+
prev: Optional[dict] = None
|
| 220 |
+
|
| 221 |
+
def close_paragraph() -> None:
|
| 222 |
+
if paragraph:
|
| 223 |
+
blocks.append("\n".join(paragraph))
|
| 224 |
+
paragraph.clear()
|
| 225 |
+
|
| 226 |
+
def close_heading() -> None:
|
| 227 |
+
nonlocal heading
|
| 228 |
+
if heading:
|
| 229 |
+
level, parts = heading
|
| 230 |
+
marker = _heading_marker(level, " ".join(parts))
|
| 231 |
+
blocks.append(marker)
|
| 232 |
+
if level == 1:
|
| 233 |
+
open_headings[:] = [marker, ""]
|
| 234 |
+
else:
|
| 235 |
+
open_headings[1] = marker
|
| 236 |
+
heading = None
|
| 237 |
+
|
| 238 |
+
for line in lines:
|
| 239 |
+
text = str(line.get("text") or "").strip()
|
| 240 |
+
if not text:
|
| 241 |
+
continue
|
| 242 |
+
size = _pdf_line_size(line)
|
| 243 |
+
level = 0
|
| 244 |
+
if size >= body_size + _TOP_HEADING_SIZE_STEP:
|
| 245 |
+
level = 1
|
| 246 |
+
elif size >= body_size + _HEADING_SIZE_STEP or _is_caps_heading(text):
|
| 247 |
+
level = 2
|
| 248 |
+
gap = float(line["top"]) - float(prev["bottom"]) if prev else 0.0
|
| 249 |
+
if level:
|
| 250 |
+
close_paragraph()
|
| 251 |
+
# A heading that wraps onto a second line continues the same
|
| 252 |
+
# heading rather than starting another one.
|
| 253 |
+
if heading and heading[0] == level and gap < paragraph_gap:
|
| 254 |
+
heading[1].append(text)
|
| 255 |
+
else:
|
| 256 |
+
close_heading()
|
| 257 |
+
heading = (level, [text])
|
| 258 |
+
else:
|
| 259 |
+
close_heading()
|
| 260 |
+
# A bold lead-in line ("14 Years of Sales-Driven…") titles the
|
| 261 |
+
# paragraph under it, so the gap just after it is not a break.
|
| 262 |
+
after_lead = (
|
| 263 |
+
len(paragraph) == 1 and prev is not None
|
| 264 |
+
and _pdf_line_bold(prev) and len(paragraph[0]) < 100
|
| 265 |
+
)
|
| 266 |
+
if paragraph and (
|
| 267 |
+
(gap > paragraph_gap and not after_lead) or _LIST_ITEM.match(text)
|
| 268 |
+
):
|
| 269 |
+
close_paragraph()
|
| 270 |
+
paragraph.append(text)
|
| 271 |
+
prev = line
|
| 272 |
+
close_heading()
|
| 273 |
+
close_paragraph()
|
| 274 |
+
return "\n\n".join(blocks)
|
| 275 |
+
|
| 276 |
+
|
| 277 |
+
def _drop_running_lines(
|
| 278 |
+
pages_lines: list[Optional[list[dict]]],
|
| 279 |
+
) -> list[Optional[list[dict]]]:
|
| 280 |
+
"""Remove page numbers and the header/footer lines repeated on most pages.
|
| 281 |
+
|
| 282 |
+
A footer such as "SKILL ARBITRAGE — UPWORK OPTIMISATION | 7" is set in
|
| 283 |
+
capitals, so left in place it would read as a heading at the foot of
|
| 284 |
+
every page and change the section the next page belongs to.
|
| 285 |
+
"""
|
| 286 |
+
def key(line: dict) -> str:
|
| 287 |
+
return re.sub(r"\d+", "#", " ".join(str(line.get("text") or "").lower().split()))
|
| 288 |
+
|
| 289 |
+
readable = [lines for lines in pages_lines if lines]
|
| 290 |
+
counts: dict[str, int] = {}
|
| 291 |
+
for lines in readable:
|
| 292 |
+
for k in {key(line) for line in lines}:
|
| 293 |
+
counts[k] = counts.get(k, 0) + 1
|
| 294 |
+
threshold = max(3, int(len(readable) * 0.3))
|
| 295 |
+
running = {k for k, n in counts.items() if n >= threshold}
|
| 296 |
+
|
| 297 |
+
cleaned: list[Optional[list[dict]]] = []
|
| 298 |
+
for lines in pages_lines:
|
| 299 |
+
if lines is None:
|
| 300 |
+
cleaned.append(None)
|
| 301 |
+
continue
|
| 302 |
+
cleaned.append([
|
| 303 |
+
line for line in lines
|
| 304 |
+
if key(line) not in running and key(line).strip("# ")
|
| 305 |
+
])
|
| 306 |
+
return cleaned
|
| 307 |
+
|
| 308 |
+
|
| 309 |
def _pdf_chunks(
|
| 310 |
path: Path,
|
| 311 |
base_meta: dict,
|
|
|
|
| 319 |
try:
|
| 320 |
with pdfplumber.open(str(path)) as pdf:
|
| 321 |
page_iter = pdf.pages[:_PDF_MAX_PAGES]
|
| 322 |
+
# Read every page's lines first: the body text size, which tells
|
| 323 |
+
# a heading from body text, is a property of the whole document.
|
| 324 |
+
pages_lines: list[Optional[list[dict]]] = []
|
| 325 |
+
failures: dict[int, str] = {}
|
| 326 |
for index, page in enumerate(page_iter, start=1):
|
| 327 |
try:
|
| 328 |
+
pages_lines.append(page.extract_text_lines(return_chars=True) or [])
|
| 329 |
except Exception as exc: # noqa: BLE001 - per-page tolerance
|
| 330 |
+
pages_lines.append(None)
|
| 331 |
+
failures[index] = f"page extract failed: {exc.__class__.__name__}"
|
| 332 |
+
pages_lines = _drop_running_lines(pages_lines)
|
| 333 |
+
body_size = _pdf_body_size([lines for lines in pages_lines if lines])
|
| 334 |
+
open_headings = ["", ""]
|
| 335 |
+
for index, lines in enumerate(pages_lines, start=1):
|
| 336 |
+
if lines is None:
|
| 337 |
records.append(
|
| 338 |
ChunkRecord(
|
| 339 |
chunk_id=_chunk_id(base_meta["file_path"], f"p{index}"),
|
| 340 |
page_number=index,
|
| 341 |
extracted_text="",
|
| 342 |
extraction_status="failed",
|
| 343 |
+
extraction_warning=failures.get(index),
|
| 344 |
**base_meta,
|
| 345 |
)
|
| 346 |
)
|
| 347 |
continue
|
| 348 |
+
section = (open_headings[1] or open_headings[0]).lstrip("# ") or None
|
| 349 |
+
text = _pdf_page_text(lines, body_size, open_headings).strip()
|
| 350 |
records.append(
|
| 351 |
ChunkRecord(
|
| 352 |
chunk_id=_chunk_id(base_meta["file_path"], f"p{index}"),
|
| 353 |
page_number=index,
|
| 354 |
+
section_name=section,
|
| 355 |
+
extracted_text=text,
|
| 356 |
+
extraction_status="ok" if text else "empty",
|
| 357 |
**base_meta,
|
| 358 |
)
|
| 359 |
)
|
|
|
|
| 362 |
return records, None
|
| 363 |
|
| 364 |
|
| 365 |
+
# A team document names itself on its cover page even when the file name
|
| 366 |
+
# does not ("Final.pdf"). A PDF or Word file its name could not place is
|
| 367 |
+
# placed by these title phrases instead; checked in order, first match wins.
|
| 368 |
+
_TITLE_RULES: tuple[tuple[SourceType, tuple[str, ...]], ...] = (
|
| 369 |
+
("upwork_profile", ("upwork profile optimi", "upwork optimi")),
|
| 370 |
+
("offer_blueprint", ("offer bank", "offer blueprint")),
|
| 371 |
+
("linkedin_optimization", ("linkedin optimi",)),
|
| 372 |
+
("skillarbitrage_dossier_roadmap", ("dossier", "roadmap")),
|
| 373 |
+
("discovery_call_transcript", ("transcript",)),
|
| 374 |
+
)
|
| 375 |
+
_UNPLACED_SOURCES = frozenset({"unknown_supported_file", "generic_profile_document"})
|
| 376 |
+
_TITLE_WINDOW_CHARS = 600
|
| 377 |
+
|
| 378 |
+
|
| 379 |
+
def _place_by_title(chunks: list[ChunkRecord]) -> list[ChunkRecord]:
|
| 380 |
+
"""Re-type an unplaced file's chunks from the title on its first page."""
|
| 381 |
+
if not chunks or chunks[0].source_type not in _UNPLACED_SOURCES:
|
| 382 |
+
return chunks
|
| 383 |
+
head = " ".join(chunks[0].extracted_text[:_TITLE_WINDOW_CHARS].lower().replace("#", " ").split())
|
| 384 |
+
# Covers often letter-space their titles ("S K I L L A R B I T R A G E").
|
| 385 |
+
squashed = head.replace(" ", "")
|
| 386 |
+
for source_type, phrases in _TITLE_RULES:
|
| 387 |
+
if any(p in head or p.replace(" ", "") in squashed for p in phrases):
|
| 388 |
+
for chunk in chunks:
|
| 389 |
+
chunk.source_type = source_type
|
| 390 |
+
chunk.source_priority = SOURCE_PRIORITY[source_type]
|
| 391 |
+
break
|
| 392 |
+
return chunks
|
| 393 |
+
|
| 394 |
+
|
| 395 |
+
def _docx_heading_level(paragraph) -> int:
|
| 396 |
+
"""0 for body text, 1 for a top-level heading, 2 for a sub-heading."""
|
| 397 |
+
text = paragraph.text.strip()
|
| 398 |
+
style = str(getattr(getattr(paragraph, "style", None), "name", "") or "").lower()
|
| 399 |
+
if style in {"title", "heading 1"}:
|
| 400 |
+
return 1
|
| 401 |
+
if style.startswith("heading"):
|
| 402 |
+
return 2
|
| 403 |
+
runs = [r for r in paragraph.runs if r.text.strip()]
|
| 404 |
+
all_bold = bool(runs) and all(r.bold for r in runs)
|
| 405 |
+
if _is_caps_heading(text) or (all_bold and len(text) <= 80 and not text.endswith(".")):
|
| 406 |
+
return 2
|
| 407 |
+
return 0
|
| 408 |
+
|
| 409 |
+
|
| 410 |
def _docx_chunk(
|
| 411 |
path: Path,
|
| 412 |
base_meta: dict,
|
|
|
|
| 421 |
except Exception as exc: # noqa: BLE001
|
| 422 |
return [], f"docx open failed: {exc.__class__.__name__}"
|
| 423 |
|
| 424 |
+
# Paragraphs and tables are read in document order, so a table (such as a
|
| 425 |
+
# key/value snapshot) stays inside the section its heading opens. Each
|
| 426 |
+
# Word paragraph is its own paragraph here, and headings are marked.
|
| 427 |
+
blocks: list[str] = []
|
| 428 |
+
for item in document.iter_inner_content():
|
| 429 |
+
if hasattr(item, "rows"):
|
| 430 |
+
for row in item.rows:
|
| 431 |
+
cells: list[str] = []
|
| 432 |
+
for cell in row.cells:
|
| 433 |
+
value = cell.text.strip()
|
| 434 |
+
if value and value not in cells:
|
| 435 |
+
cells.append(value)
|
| 436 |
+
if cells:
|
| 437 |
+
blocks.append(" | ".join(cells))
|
| 438 |
+
continue
|
| 439 |
+
text = item.text.strip()
|
| 440 |
+
if not text:
|
| 441 |
+
continue
|
| 442 |
+
level = _docx_heading_level(item)
|
| 443 |
+
blocks.append(_heading_marker(level, text) if level else text)
|
| 444 |
|
| 445 |
+
full_text = "\n\n".join(blocks).strip()
|
| 446 |
status: ExtractionStatus = "ok" if full_text else "empty"
|
| 447 |
return (
|
| 448 |
[
|
|
|
|
| 573 |
if pdf_chunks:
|
| 574 |
if warning:
|
| 575 |
pdf_chunks[-1].extraction_warning = _sanitize_warning(warning)
|
| 576 |
+
return _place_by_title(pdf_chunks)
|
| 577 |
return [_failed_chunk(path, base_meta, warning or "no pages")]
|
| 578 |
|
| 579 |
if ext == ".docx":
|
| 580 |
docx_chunks, warning = _docx_chunk(path, base_meta)
|
| 581 |
if docx_chunks:
|
| 582 |
+
return _place_by_title(docx_chunks)
|
| 583 |
return [_failed_chunk(path, base_meta, warning or "no content")]
|
| 584 |
|
| 585 |
if ext in {".txt", ".md"}:
|
|
@@ -105,10 +105,6 @@ _SERVICE_LINE_HINTS = ("services:", "service:", "offerings:", "i offer", "we off
|
|
| 105 |
_DELIVERABLE_HINTS = ("deliverables:", "deliverable:", "what you get", "you receive")
|
| 106 |
_TARGET_CLIENT_HINTS = ("target client", "ideal client", "client avatar", "icp")
|
| 107 |
_OFFER_HINTS = ("selected offer", "primary offer", "headline offer", "offer:", "blueprint:")
|
| 108 |
-
_PRICING_HINTS = (
|
| 109 |
-
"rate", "rates", "pricing", "package", "retainer", "hourly", "fixed-fee",
|
| 110 |
-
"deposit", "fee:", "rate card",
|
| 111 |
-
)
|
| 112 |
_CERTIFICATION_HINTS = ("certified", "certification", "course completion", "credential")
|
| 113 |
_EDUCATION_HINTS = ("bachelor", "master", "phd", "b.sc", "m.sc", "diploma", "degree")
|
| 114 |
_LANGUAGE_HINTS = ("languages:", "language:", "fluent in", "native speaker")
|
|
@@ -130,6 +126,186 @@ def _split_csv_like(value: str) -> list[str]:
|
|
| 130 |
return [s.strip() for s in re.split(r"[,;\n•·]+", value) if s.strip()]
|
| 131 |
|
| 132 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
def _make_proof(
|
| 134 |
*,
|
| 135 |
chunk: ChunkRecord,
|
|
@@ -173,8 +349,26 @@ def _scan_text_chunk(chunk: ChunkRecord) -> list[ProofPoint]:
|
|
| 173 |
|
| 174 |
paragraphs = _split_paragraphs(text)
|
| 175 |
current_section: Optional[str] = chunk.section_name
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
|
| 177 |
for idx, paragraph in enumerate(paragraphs):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
lower = paragraph.lower()
|
| 179 |
location = f"{base_location}#para{idx + 1}"
|
| 180 |
if current_section:
|
|
@@ -184,6 +378,51 @@ def _scan_text_chunk(chunk: ChunkRecord) -> list[ProofPoint]:
|
|
| 184 |
if _SECTION_HEADER.match(first_line):
|
| 185 |
current_section = first_line.strip().strip("#").strip()
|
| 186 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 187 |
def line_after(hint: str) -> Optional[str]:
|
| 188 |
pos = lower.find(hint)
|
| 189 |
if pos == -1:
|
|
@@ -296,7 +535,7 @@ def _scan_text_chunk(chunk: ChunkRecord) -> list[ProofPoint]:
|
|
| 296 |
)
|
| 297 |
)
|
| 298 |
|
| 299 |
-
if
|
| 300 |
proofs.append(
|
| 301 |
_make_proof(
|
| 302 |
chunk=chunk,
|
|
@@ -420,7 +659,45 @@ def _scan_text_chunk(chunk: ChunkRecord) -> list[ProofPoint]:
|
|
| 420 |
)
|
| 421 |
)
|
| 422 |
|
| 423 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 424 |
# We still want a placeholder proof from unknown content so that
|
| 425 |
# the file shows up in the canonical profile's source summary.
|
| 426 |
proofs.append(
|
|
@@ -572,10 +849,22 @@ def build(chunks: Iterable[ChunkRecord]) -> list[ProofPoint]:
|
|
| 572 |
``other_relevant_evidence`` placeholder is emitted so the file is
|
| 573 |
not silently dropped from the canonical profile's source summary.
|
| 574 |
"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 575 |
index: list[ProofPoint] = []
|
| 576 |
for chunk in chunks:
|
| 577 |
if chunk.extraction_status in {"failed", "metadata_only", "empty"}:
|
| 578 |
continue
|
|
|
|
|
|
|
| 579 |
if chunk.file_type == ".json" and chunk.json_data is not None:
|
| 580 |
index.extend(_scan_json_chunk(chunk))
|
| 581 |
else:
|
|
|
|
| 105 |
_DELIVERABLE_HINTS = ("deliverables:", "deliverable:", "what you get", "you receive")
|
| 106 |
_TARGET_CLIENT_HINTS = ("target client", "ideal client", "client avatar", "icp")
|
| 107 |
_OFFER_HINTS = ("selected offer", "primary offer", "headline offer", "offer:", "blueprint:")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
_CERTIFICATION_HINTS = ("certified", "certification", "course completion", "credential")
|
| 109 |
_EDUCATION_HINTS = ("bachelor", "master", "phd", "b.sc", "m.sc", "diploma", "degree")
|
| 110 |
_LANGUAGE_HINTS = ("languages:", "language:", "fluent in", "native speaker")
|
|
|
|
| 126 |
return [s.strip() for s in re.split(r"[,;\n•·]+", value) if s.strip()]
|
| 127 |
|
| 128 |
|
| 129 |
+
# ---------------------------------------------------------------------------
|
| 130 |
+
# Team document sections
|
| 131 |
+
# ---------------------------------------------------------------------------
|
| 132 |
+
#
|
| 133 |
+
# The program team writes every Dossier, Offer Bank and Upwork Profile
|
| 134 |
+
# Optimisation document from one template, and each heading says what its
|
| 135 |
+
# section is for. Only some sections hold facts a client may read; the rest
|
| 136 |
+
# coach the student and must never reach a proposal. Each heading is matched
|
| 137 |
+
# by phrase, in this order, first match wins:
|
| 138 |
+
#
|
| 139 |
+
# * skip — coaching: how-to notes, roadmaps, SWOT opportunities and
|
| 140 |
+
# threats, outreach plans, video scripts, income targets.
|
| 141 |
+
# * avoid — weaknesses and gaps: kept only as things never to claim.
|
| 142 |
+
# * deliverables — what the offer delivers: one capability per paragraph,
|
| 143 |
+
# never a result (its numbers are promises, not outcomes).
|
| 144 |
+
# * services — service tiers and rates: services, pricing, and any skill
|
| 145 |
+
# the document says is still being developed.
|
| 146 |
+
# * facts — snapshot, strengths, bio, employment, education, skills.
|
| 147 |
+
# * angle — offer statement, target client, value proposition: they
|
| 148 |
+
# steer the pitch, so only positioning-type claims are kept.
|
| 149 |
+
#
|
| 150 |
+
# A sub-heading the table does not know inherits its parent's role (so
|
| 151 |
+
# "SECTION 1 — HOOK" under "Video Script" is skipped too); an unknown
|
| 152 |
+
# top-level heading, or a document with no known heading at all, is scanned
|
| 153 |
+
# exactly as before.
|
| 154 |
+
_SECTION_ROLES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
| 155 |
+
("skip", (
|
| 156 |
+
"skill arbitrage", "how to use", "contents", "roadmap", "phase ", "week ", "by now you must",
|
| 157 |
+
"tasks for this week", "before moving to", "completion checklist",
|
| 158 |
+
"acquisition", "opportunit", "threat", "how this helps", "what life looks like",
|
| 159 |
+
"smartest move", "profile photo", "video script", "recording guideline",
|
| 160 |
+
"upload your video", "how to upload", "linked accounts", "availability setting",
|
| 161 |
+
"categories", "income", "client kpis", "upgrade trigger", "swot analysis",
|
| 162 |
+
"target list", "content pack",
|
| 163 |
+
)),
|
| 164 |
+
("avoid", ("weakness", "additional gaps", "biggest challenge", "limitation", "out of scope")),
|
| 165 |
+
("deliverables", ("what you will deliver", "deliverables", "what you get")),
|
| 166 |
+
("services", ("service stack", "service fit", "tier ", "rate recommendation", "pricing")),
|
| 167 |
+
("facts", (
|
| 168 |
+
"snapshot", "strength", "bio", "overview", "headline", "employment",
|
| 169 |
+
"work history", "experience", "education", "language", "skill",
|
| 170 |
+
"certific", "portfolio", "testimonial", "about me", "background",
|
| 171 |
+
)),
|
| 172 |
+
("angle", (
|
| 173 |
+
"selected offer", "offer statement", "your offer", "offer ", "niche",
|
| 174 |
+
"target client", "who you serve", "ideal client", "value proposition",
|
| 175 |
+
"the pain", "positioning",
|
| 176 |
+
)),
|
| 177 |
+
)
|
| 178 |
+
|
| 179 |
+
# Claim types an "angle" section may contribute. Its metrics are promises
|
| 180 |
+
# ("increase conversion by 40-80%"), never results, so they are not kept;
|
| 181 |
+
# nor is pricing, since the money it mentions is usually the client's
|
| 182 |
+
# revenue ("brands doing Rs 10-50 lakh/month"), not the student's rate.
|
| 183 |
+
_ANGLE_CLAIM_TYPES = frozenset({
|
| 184 |
+
"positioning", "selected_offer", "target_client", "service",
|
| 185 |
+
"deliverable", "industry", "guarantee", "proposal_preference",
|
| 186 |
+
})
|
| 187 |
+
|
| 188 |
+
# Paragraphs that coach the student wherever they appear: sales copy about
|
| 189 |
+
# the client's benefit, market sizing, and week-by-week task lists.
|
| 190 |
+
_COACHING_PARAGRAPH = re.compile(
|
| 191 |
+
r"^\s*how this helps|market (?:was )?valued|market size|market basis|\bcagr\b|"
|
| 192 |
+
r"\bbillion\b|income target|by now you must have|tasks for this week|"
|
| 193 |
+
# A table of contents: three or more "N Title P" entries in a row.
|
| 194 |
+
r"(?:\d{1,2}\s+[A-Z][^\d\n]{2,60}?\s+\d{1,3}\s+){3,}",
|
| 195 |
+
re.IGNORECASE,
|
| 196 |
+
)
|
| 197 |
+
|
| 198 |
+
# An outcome the offer promises to deliver ("conversion improvement of at
|
| 199 |
+
# least 30% within 30 days"). The student has not achieved it yet, so it is
|
| 200 |
+
# never kept as a service or a result.
|
| 201 |
+
_PROMISED_OUTCOME = re.compile(
|
| 202 |
+
r"\bat least \d|\bimprovement (?:of|from)\b|\breduction of\b|\bgrowth of\b|"
|
| 203 |
+
r"^\s*upgrade trigger",
|
| 204 |
+
re.IGNORECASE,
|
| 205 |
+
)
|
| 206 |
+
|
| 207 |
+
# Pricing language. Whole words only: as substrings, "rate" matched
|
| 208 |
+
# "strategist", "rated" and "conversion rate", and filed them as pricing.
|
| 209 |
+
_PRICING_PATTERN = re.compile(
|
| 210 |
+
r"\bpricing\b|\bretainers?\b|\bhourly\b|\bper hour\b|/\s?hr\b|\brate card\b|"
|
| 211 |
+
r"\bfixed[- ](?:fee|price)\b|\bdeposit\b|\bfee:|\bpackages?\b|"
|
| 212 |
+
r"(?:\brs\.?|\binr|\busd|\$|₹)\s?\d",
|
| 213 |
+
re.IGNORECASE,
|
| 214 |
+
)
|
| 215 |
+
|
| 216 |
+
# A skill the document says is not ready to sell yet.
|
| 217 |
+
_DEVELOPING_SKILL = re.compile(
|
| 218 |
+
r"\bdevelop \(|requires (?:training|learning)|★★☆|★☆☆|outside (?:your|my) comfort",
|
| 219 |
+
re.IGNORECASE,
|
| 220 |
+
)
|
| 221 |
+
|
| 222 |
+
_HEADING_LINE = re.compile(r"^\s*(#{1,6})\s+(.+?)\s*$")
|
| 223 |
+
|
| 224 |
+
# Documents written around the one offer the student chose.
|
| 225 |
+
_CHOSEN_OFFER_SOURCES = frozenset({"skillarbitrage_dossier_roadmap", "upwork_profile"})
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
def _section_role(heading: str, level: int, current: str) -> str:
|
| 229 |
+
"""The role a heading opens; see ``_SECTION_ROLES``."""
|
| 230 |
+
text = " ".join(heading.lower().split())
|
| 231 |
+
for role, phrases in _SECTION_ROLES:
|
| 232 |
+
if any(phrase in text for phrase in phrases):
|
| 233 |
+
return role
|
| 234 |
+
if level >= 3 and current != "unknown":
|
| 235 |
+
return current
|
| 236 |
+
return "unknown"
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
# Labels on the dossier's "Student Snapshot" table, each with the claim it
|
| 240 |
+
# becomes. ``None`` marks private planning notes that are dropped.
|
| 241 |
+
_SNAPSHOT_LABELS: tuple[tuple[str, Optional[str]], ...] = (
|
| 242 |
+
("professional qualifications", "certification"),
|
| 243 |
+
("tools you already know", "tool"),
|
| 244 |
+
("target client type", "target_client"),
|
| 245 |
+
("english proficiency", "language"),
|
| 246 |
+
("biggest challenge", None),
|
| 247 |
+
("income target", None),
|
| 248 |
+
("selected offer", "selected_offer"),
|
| 249 |
+
("subject domain", "industry"),
|
| 250 |
+
("target client", "target_client"),
|
| 251 |
+
("languages", "language"),
|
| 252 |
+
("education", "education"),
|
| 253 |
+
("location", "location"),
|
| 254 |
+
("name", "identity"),
|
| 255 |
+
)
|
| 256 |
+
_SNAPSHOT_SPLIT = re.compile(
|
| 257 |
+
r"(?im)^\s*(" + "|".join(re.escape(label) for label, _ in _SNAPSHOT_LABELS)
|
| 258 |
+
+ r")\b\s*[:|]?\s*"
|
| 259 |
+
)
|
| 260 |
+
# "Offer 2 — Academic Writing…" → "Academic Writing…": the offer's number is
|
| 261 |
+
# an internal label from the Offer Bank, never something a client should see.
|
| 262 |
+
_OFFER_LABEL = re.compile(r"^\s*(?:offer|niche)\s*\d+\s*[a-z]?\s*[—–:\-]\s*", re.IGNORECASE)
|
| 263 |
+
|
| 264 |
+
|
| 265 |
+
def _strip_offer_label(text: str) -> str:
|
| 266 |
+
return _OFFER_LABEL.sub("", text).strip()
|
| 267 |
+
|
| 268 |
+
|
| 269 |
+
def _snapshot_proofs(chunk: ChunkRecord, paragraph: str, location: str) -> list[ProofPoint]:
|
| 270 |
+
"""Turn "Label value" rows from a Student Snapshot into separate claims."""
|
| 271 |
+
proofs: list[ProofPoint] = []
|
| 272 |
+
parts = _SNAPSHOT_SPLIT.split(paragraph)
|
| 273 |
+
claim_for = dict(_SNAPSHOT_LABELS)
|
| 274 |
+
# re.split with one group gives [before, label, value, label, value, ...].
|
| 275 |
+
for label, value in zip(parts[1::2], parts[2::2]):
|
| 276 |
+
claim_type = claim_for.get(label.lower())
|
| 277 |
+
value = " ".join(value.split())
|
| 278 |
+
if not claim_type or not value or value.lower().startswith(("no ", "none")):
|
| 279 |
+
continue
|
| 280 |
+
if claim_type == "selected_offer":
|
| 281 |
+
value = _strip_offer_label(value)
|
| 282 |
+
kwargs: dict[str, Any] = {}
|
| 283 |
+
if claim_type == "tool":
|
| 284 |
+
kwargs["tools"] = [re.sub(r"\(.*?\)", "", t).strip() for t in _split_csv_like(value)]
|
| 285 |
+
basic = [
|
| 286 |
+
re.sub(r"\(.*?\)", "", t).strip()
|
| 287 |
+
for t in _split_csv_like(value)
|
| 288 |
+
if re.search(r"\(\s*basic", t, re.IGNORECASE)
|
| 289 |
+
]
|
| 290 |
+
if basic:
|
| 291 |
+
proofs.append(_make_proof(
|
| 292 |
+
chunk=chunk, claim_type="weakness_or_constraint",
|
| 293 |
+
claim_text="Basic level only, do not oversell: " + ", ".join(basic),
|
| 294 |
+
location=location,
|
| 295 |
+
))
|
| 296 |
+
elif claim_type == "industry":
|
| 297 |
+
kwargs["industries"] = [
|
| 298 |
+
re.sub(r"\(.*?\)", "", t).strip() for t in _split_csv_like(value)
|
| 299 |
+
]
|
| 300 |
+
proofs.append(_make_proof(
|
| 301 |
+
chunk=chunk, claim_type=claim_type, # type: ignore[arg-type]
|
| 302 |
+
claim_text=value, location=location,
|
| 303 |
+
normalized_value=value if claim_type in {"identity", "selected_offer", "location"} else None,
|
| 304 |
+
**kwargs,
|
| 305 |
+
))
|
| 306 |
+
return proofs
|
| 307 |
+
|
| 308 |
+
|
| 309 |
def _make_proof(
|
| 310 |
*,
|
| 311 |
chunk: ChunkRecord,
|
|
|
|
| 349 |
|
| 350 |
paragraphs = _split_paragraphs(text)
|
| 351 |
current_section: Optional[str] = chunk.section_name
|
| 352 |
+
# The role of the section being read (see _SECTION_ROLES), and whether
|
| 353 |
+
# any heading in this chunk was a known template heading.
|
| 354 |
+
role = "unknown"
|
| 355 |
+
saw_template_section = False
|
| 356 |
+
snapshot_label = ""
|
| 357 |
|
| 358 |
for idx, paragraph in enumerate(paragraphs):
|
| 359 |
+
# Marked headings ("## Your Strengths") open a section and are never
|
| 360 |
+
# claims themselves. A paragraph whose first line is the heading and
|
| 361 |
+
# whose body follows (a Markdown file) keeps the body.
|
| 362 |
+
lines = paragraph.splitlines()
|
| 363 |
+
while lines and _HEADING_LINE.match(lines[0]):
|
| 364 |
+
hashes, heading = _HEADING_LINE.match(lines[0]).groups() # type: ignore[union-attr]
|
| 365 |
+
current_section = heading.strip()
|
| 366 |
+
role = _section_role(heading, len(hashes), role)
|
| 367 |
+
saw_template_section = saw_template_section or role != "unknown"
|
| 368 |
+
lines = lines[1:]
|
| 369 |
+
if not lines:
|
| 370 |
+
continue
|
| 371 |
+
paragraph = "\n".join(lines)
|
| 372 |
lower = paragraph.lower()
|
| 373 |
location = f"{base_location}#para{idx + 1}"
|
| 374 |
if current_section:
|
|
|
|
| 378 |
if _SECTION_HEADER.match(first_line):
|
| 379 |
current_section = first_line.strip().strip("#").strip()
|
| 380 |
|
| 381 |
+
if role == "skip" or _COACHING_PARAGRAPH.search(paragraph):
|
| 382 |
+
continue
|
| 383 |
+
# A page number or a stray fragment is not evidence.
|
| 384 |
+
if role != "unknown" and not re.search(r"[A-Za-z]{3}", paragraph):
|
| 385 |
+
continue
|
| 386 |
+
if role == "avoid":
|
| 387 |
+
proofs.append(_make_proof(
|
| 388 |
+
chunk=chunk, claim_type="weakness_or_constraint",
|
| 389 |
+
claim_text=paragraph[:300], location=location,
|
| 390 |
+
))
|
| 391 |
+
continue
|
| 392 |
+
if role == "deliverables":
|
| 393 |
+
proofs.append(_make_proof(
|
| 394 |
+
chunk=chunk, claim_type="deliverable",
|
| 395 |
+
claim_text=paragraph[:300], location=location,
|
| 396 |
+
))
|
| 397 |
+
continue
|
| 398 |
+
if role == "services":
|
| 399 |
+
if _DEVELOPING_SKILL.search(paragraph):
|
| 400 |
+
claim_type: ClaimType = "weakness_or_constraint"
|
| 401 |
+
elif _PROMISED_OUTCOME.search(paragraph):
|
| 402 |
+
continue
|
| 403 |
+
elif _PRICING_PATTERN.search(paragraph):
|
| 404 |
+
claim_type = "pricing"
|
| 405 |
+
else:
|
| 406 |
+
claim_type = "service"
|
| 407 |
+
proofs.append(_make_proof(
|
| 408 |
+
chunk=chunk, claim_type=claim_type,
|
| 409 |
+
claim_text=paragraph[:300], location=location,
|
| 410 |
+
))
|
| 411 |
+
continue
|
| 412 |
+
if role == "facts" and "snapshot" in (current_section or "").lower():
|
| 413 |
+
# A label laid out on its own line ("Tools You Already Know")
|
| 414 |
+
# names the value in the paragraph that follows it.
|
| 415 |
+
if snapshot_label:
|
| 416 |
+
paragraph = snapshot_label + " " + paragraph
|
| 417 |
+
snapshot_label = ""
|
| 418 |
+
if _SNAPSHOT_SPLIT.fullmatch(paragraph.strip() + " "):
|
| 419 |
+
snapshot_label = paragraph.strip()
|
| 420 |
+
continue
|
| 421 |
+
proofs.extend(_snapshot_proofs(chunk, paragraph, location))
|
| 422 |
+
continue
|
| 423 |
+
snapshot_label = ""
|
| 424 |
+
start = len(proofs)
|
| 425 |
+
|
| 426 |
def line_after(hint: str) -> Optional[str]:
|
| 427 |
pos = lower.find(hint)
|
| 428 |
if pos == -1:
|
|
|
|
| 535 |
)
|
| 536 |
)
|
| 537 |
|
| 538 |
+
if _PRICING_PATTERN.search(paragraph):
|
| 539 |
proofs.append(
|
| 540 |
_make_proof(
|
| 541 |
chunk=chunk,
|
|
|
|
| 659 |
)
|
| 660 |
)
|
| 661 |
|
| 662 |
+
if role == "angle":
|
| 663 |
+
proofs[start:] = [p for p in proofs[start:] if p.claim_type in _ANGLE_CLAIM_TYPES]
|
| 664 |
+
for p in proofs[start:]:
|
| 665 |
+
if p.claim_type == "selected_offer":
|
| 666 |
+
p.claim_text = _strip_offer_label(p.claim_text)
|
| 667 |
+
elif (
|
| 668 |
+
role == "facts"
|
| 669 |
+
and "strength" in (current_section or "").lower()
|
| 670 |
+
and not any(p.claim_type == "achievement" for p in proofs[start:])
|
| 671 |
+
):
|
| 672 |
+
# Each paragraph under "Strengths" is one strength; it is filed as
|
| 673 |
+
# one even when a keyword also filed it as, say, education.
|
| 674 |
+
proofs.append(_make_proof(
|
| 675 |
+
chunk=chunk, claim_type="achievement",
|
| 676 |
+
claim_text=paragraph, location=location,
|
| 677 |
+
))
|
| 678 |
+
elif role == "facts" and len(proofs) == start:
|
| 679 |
+
# A fact section's paragraph is evidence even when no keyword
|
| 680 |
+
# fires: a strength reads as one, a role or bio entry as history.
|
| 681 |
+
section = (current_section or "").lower()
|
| 682 |
+
if "skill" in section:
|
| 683 |
+
# A skills list: "1 Academic Writing" or "Academic Writing".
|
| 684 |
+
skill = re.sub(r"^\s*\d+[.)]?\s*", "", " ".join(paragraph.split()))
|
| 685 |
+
if skill and len(skill) <= 60:
|
| 686 |
+
proofs.append(_make_proof(
|
| 687 |
+
chunk=chunk, claim_type="skill", claim_text=skill,
|
| 688 |
+
location=location, skills=[skill],
|
| 689 |
+
))
|
| 690 |
+
else:
|
| 691 |
+
proofs.append(_make_proof(
|
| 692 |
+
chunk=chunk,
|
| 693 |
+
claim_type="achievement" if "strength" in section else "experience",
|
| 694 |
+
claim_text=paragraph,
|
| 695 |
+
location=location,
|
| 696 |
+
))
|
| 697 |
+
|
| 698 |
+
# A page made only of coaching sections yields nothing on purpose; the
|
| 699 |
+
# placeholder is for files the scanner could not place at all.
|
| 700 |
+
if not proofs and not saw_template_section:
|
| 701 |
# We still want a placeholder proof from unknown content so that
|
| 702 |
# the file shows up in the canonical profile's source summary.
|
| 703 |
proofs.append(
|
|
|
|
| 849 |
``other_relevant_evidence`` placeholder is emitted so the file is
|
| 850 |
not silently dropped from the canonical profile's source summary.
|
| 851 |
"""
|
| 852 |
+
chunks = list(chunks)
|
| 853 |
+
# The Offer Bank lists every offer drafted for the student, including
|
| 854 |
+
# the ones they did not choose. Once the Dossier (written around the
|
| 855 |
+
# chosen offer) or the Upwork Profile Optimisation document is present,
|
| 856 |
+
# the Offer Bank would only mix unchosen offers into the proposals.
|
| 857 |
+
has_chosen_offer_source = any(
|
| 858 |
+
c.source_type in _CHOSEN_OFFER_SOURCES
|
| 859 |
+
and c.extraction_status not in {"failed", "metadata_only", "empty"}
|
| 860 |
+
for c in chunks
|
| 861 |
+
)
|
| 862 |
index: list[ProofPoint] = []
|
| 863 |
for chunk in chunks:
|
| 864 |
if chunk.extraction_status in {"failed", "metadata_only", "empty"}:
|
| 865 |
continue
|
| 866 |
+
if has_chosen_offer_source and chunk.source_type == "offer_blueprint":
|
| 867 |
+
continue
|
| 868 |
if chunk.file_type == ".json" and chunk.json_data is not None:
|
| 869 |
index.extend(_scan_json_chunk(chunk))
|
| 870 |
else:
|
|
@@ -68,7 +68,7 @@ _BUCKETS: tuple[tuple[str, str, int], ...] = (
|
|
| 68 |
_FILENAME_RULES: tuple[tuple[SourceType, tuple[str, ...]], ...] = (
|
| 69 |
("linkedin_optimization", ("linkedin optimization", "linkedin optimisation", "linkedin rewrite")),
|
| 70 |
("skillarbitrage_dossier_roadmap", ("skillarbitrage", "dossier", "roadmap")),
|
| 71 |
-
("offer_blueprint", ("offer blueprint", "offer-blueprint", "offer_blueprint", "blueprint", "selected offer")),
|
| 72 |
("upwork_profile", ("upwork",)),
|
| 73 |
("linkedin_profile", ("linkedin",)),
|
| 74 |
("discovery_call_transcript", ("transcript", "discovery call", "discovery-call", "discovery_call", "call notes")),
|
|
@@ -414,6 +414,94 @@ def score_rubric_labels() -> dict[str, str]:
|
|
| 414 |
return {key: label for key, label, _w in _BUCKETS}
|
| 415 |
|
| 416 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 417 |
# ---------------------------------------------------------------------------
|
| 418 |
# Main entry point
|
| 419 |
# ---------------------------------------------------------------------------
|
|
|
|
| 68 |
_FILENAME_RULES: tuple[tuple[SourceType, tuple[str, ...]], ...] = (
|
| 69 |
("linkedin_optimization", ("linkedin optimization", "linkedin optimisation", "linkedin rewrite")),
|
| 70 |
("skillarbitrage_dossier_roadmap", ("skillarbitrage", "dossier", "roadmap")),
|
| 71 |
+
("offer_blueprint", ("offer blueprint", "offer-blueprint", "offer_blueprint", "blueprint", "selected offer", "offer bank")),
|
| 72 |
("upwork_profile", ("upwork",)),
|
| 73 |
("linkedin_profile", ("linkedin",)),
|
| 74 |
("discovery_call_transcript", ("transcript", "discovery call", "discovery-call", "discovery_call", "call notes")),
|
|
|
|
| 414 |
return {key: label for key, label, _w in _BUCKETS}
|
| 415 |
|
| 416 |
|
| 417 |
+
# ---------------------------------------------------------------------------
|
| 418 |
+
# Strength from what was read
|
| 419 |
+
# ---------------------------------------------------------------------------
|
| 420 |
+
#
|
| 421 |
+
# ``validate`` scores a folder before anything in it is read: it sees file
|
| 422 |
+
# names, and the contents of plain-text files only, so a 36-page dossier PDF
|
| 423 |
+
# named "Dossier.pdf" scored 30/100 and was called thin. Once the files have
|
| 424 |
+
# been read into proof points, ``score_evidence`` scores what they actually
|
| 425 |
+
# hold, on the same seven buckets and weights.
|
| 426 |
+
|
| 427 |
+
# The kinds of claim that fill each bucket.
|
| 428 |
+
_EVIDENCE_BUCKETS: dict[str, frozenset[str]] = {
|
| 429 |
+
"positioning_and_offer": frozenset({"positioning", "selected_offer", "target_client"}),
|
| 430 |
+
"work_history_background": frozenset({"experience", "work_history", "education", "achievement"}),
|
| 431 |
+
"skills_tools_services": frozenset({"skill", "tool", "service", "deliverable"}),
|
| 432 |
+
"proof_credibility": frozenset({"metric", "testimonial", "project", "portfolio", "certification"}),
|
| 433 |
+
"proposal_preferences_pricing": frozenset({"pricing", "proposal_preference", "guarantee"}),
|
| 434 |
+
}
|
| 435 |
+
# The profile facts a proposal can draw on; how many are covered fills the
|
| 436 |
+
# "structured profile" bucket.
|
| 437 |
+
_PROFILE_FACT_GROUPS: tuple[frozenset[str], ...] = (
|
| 438 |
+
frozenset({"identity"}),
|
| 439 |
+
frozenset({"positioning", "selected_offer"}),
|
| 440 |
+
frozenset({"target_client"}),
|
| 441 |
+
frozenset({"service", "deliverable"}),
|
| 442 |
+
frozenset({"skill", "tool"}),
|
| 443 |
+
frozenset({"experience", "work_history", "achievement"}),
|
| 444 |
+
frozenset({"education", "certification"}),
|
| 445 |
+
frozenset({"language"}),
|
| 446 |
+
frozenset({"pricing"}),
|
| 447 |
+
)
|
| 448 |
+
_PROFILE_FACTS_FOR_FULL_MARKS = 7
|
| 449 |
+
_DISTINCT_CLAIMS_FOR_DIVERSITY = 25
|
| 450 |
+
|
| 451 |
+
# What each empty bucket means, in words a student understands.
|
| 452 |
+
MISSING_IN_PLAIN_WORDS: dict[str, str] = {
|
| 453 |
+
"structured_profile_data": "basic profile details (education, languages, tools)",
|
| 454 |
+
"positioning_and_offer": "who you help and what you offer",
|
| 455 |
+
"work_history_background": "your work history or background",
|
| 456 |
+
"skills_tools_services": "your skills, tools and services",
|
| 457 |
+
"proof_credibility": "past results, samples or testimonials",
|
| 458 |
+
"proposal_preferences_pricing": "your rates or pricing",
|
| 459 |
+
"completeness_diversity": "enough detail overall",
|
| 460 |
+
}
|
| 461 |
+
|
| 462 |
+
|
| 463 |
+
def score_evidence(result: FolderValidationResult, proofs: list) -> FolderValidationResult:
|
| 464 |
+
"""Re-score ``result`` from the proof points read out of its files.
|
| 465 |
+
|
| 466 |
+
The higher of the two scores stands, so a folder of well-named plain-text
|
| 467 |
+
files is never marked down. ``missing_categories`` is rebuilt to name
|
| 468 |
+
only what the files really lack.
|
| 469 |
+
"""
|
| 470 |
+
claim_types = {str(getattr(p, "claim_type", "") or "") for p in proofs or []}
|
| 471 |
+
breakdown: dict[str, int] = {}
|
| 472 |
+
facts = sum(1 for group in _PROFILE_FACT_GROUPS if group & claim_types)
|
| 473 |
+
breakdown["structured_profile_data"] = round(
|
| 474 |
+
20 * min(facts, _PROFILE_FACTS_FOR_FULL_MARKS) / _PROFILE_FACTS_FOR_FULL_MARKS
|
| 475 |
+
)
|
| 476 |
+
weights = {key: weight for key, _label, weight in _BUCKETS}
|
| 477 |
+
for key, kinds in _EVIDENCE_BUCKETS.items():
|
| 478 |
+
breakdown[key] = weights[key] if kinds & claim_types else 0
|
| 479 |
+
distinct = {str(getattr(p, "claim_text", "") or "") for p in proofs or []}
|
| 480 |
+
sources = {str(getattr(p, "source_type", "") or "") for p in proofs or []}
|
| 481 |
+
breakdown["completeness_diversity"] = (
|
| 482 |
+
10 if len(distinct) >= _DISTINCT_CLAIMS_FOR_DIVERSITY or len(sources) >= 3 else 0
|
| 483 |
+
)
|
| 484 |
+
total = min(sum(breakdown.values()), 100)
|
| 485 |
+
if total >= result.strength_score:
|
| 486 |
+
result.strength_score = total
|
| 487 |
+
result.score_breakdown = breakdown
|
| 488 |
+
result.missing_categories = _missing_categories(breakdown)
|
| 489 |
+
# The pre-read warning about a low score no longer applies once the
|
| 490 |
+
# content has been scored above it.
|
| 491 |
+
if not result.below_threshold:
|
| 492 |
+
result.warnings = [w for w in result.warnings if "strength score" not in w]
|
| 493 |
+
return result
|
| 494 |
+
|
| 495 |
+
|
| 496 |
+
def missing_in_plain_words(result: FolderValidationResult) -> list[str]:
|
| 497 |
+
"""The empty buckets of ``result``, phrased for a student."""
|
| 498 |
+
return [
|
| 499 |
+
MISSING_IN_PLAIN_WORDS[key]
|
| 500 |
+
for key, awarded in (result.score_breakdown or {}).items()
|
| 501 |
+
if not awarded and key in MISSING_IN_PLAIN_WORDS
|
| 502 |
+
]
|
| 503 |
+
|
| 504 |
+
|
| 505 |
# ---------------------------------------------------------------------------
|
| 506 |
# Main entry point
|
| 507 |
# ---------------------------------------------------------------------------
|
|
@@ -264,11 +264,28 @@ QUESTIONS: list[Question] = [
|
|
| 264 |
"and why that one?",
|
| 265 |
"strengths", "proposal", placeholder="The work, and why it stands out"),
|
| 266 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
_q("q13", _S_DOMAIN, "Which industries or domains have you actually worked in?",
|
| 268 |
"industries", "score", "multi",
|
| 269 |
-
("
|
| 270 |
-
"
|
| 271 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 272 |
placeholder="Pick all that apply, or type your own"),
|
| 273 |
_q("q14", _S_DOMAIN,
|
| 274 |
"Which of those do you know best? Tell me something about how that industry "
|
|
@@ -291,9 +308,12 @@ QUESTIONS: list[Question] = [
|
|
| 291 |
|
| 292 |
_q("q18", _S_SKILLS, "What is your main skill category?",
|
| 293 |
"skills", "score", "chips",
|
| 294 |
-
("Writing & content", "
|
| 295 |
-
"
|
| 296 |
-
"
|
|
|
|
|
|
|
|
|
|
| 297 |
placeholder="Or type your main category"),
|
| 298 |
_q("q19", _S_SKILLS, "Name the five skills a client would genuinely hire you for.",
|
| 299 |
"skills", "score", placeholder="Five skills, comma separated",
|
|
|
|
| 264 |
"and why that one?",
|
| 265 |
"strengths", "proposal", placeholder="The work, and why it stands out"),
|
| 266 |
|
| 267 |
+
# The sectors named across the programs' discovery-call scripts, with the
|
| 268 |
+
# programs' own domains first. Labels avoid commas and " - " (the answer
|
| 269 |
+
# splitter cuts on both) and very short words such as "HR" or "CA", which
|
| 270 |
+
# industry matching would find inside unrelated words in a job post.
|
| 271 |
_q("q13", _S_DOMAIN, "Which industries or domains have you actually worked in?",
|
| 272 |
"industries", "score", "multi",
|
| 273 |
+
("Legal / Law firms / LPO", "Academia / Universities / Research",
|
| 274 |
+
"Accounting / CPA firms / Bookkeeping", "Finance / Banking / Fintech",
|
| 275 |
+
"Human resources / Recruitment / Staffing",
|
| 276 |
+
"Healthcare / Clinics / Wellness", "Pharma / Biotech / Life sciences",
|
| 277 |
+
"Education / EdTech / Coaching", "E-commerce / D2C / Retail",
|
| 278 |
+
"SaaS / Tech / IT services", "Startups", "Marketing / Advertising agencies",
|
| 279 |
+
"Media / Publishing / Journalism",
|
| 280 |
+
"Real estate / Construction / Infrastructure",
|
| 281 |
+
"Manufacturing / Supply chain / Logistics",
|
| 282 |
+
"Hospitality / Travel / Events", "Non-profit / NGOs / Development",
|
| 283 |
+
"Government / Public sector / PSUs", "Think tanks / Policy",
|
| 284 |
+
"Consulting / Professional services", "GCCs / Large enterprises / MNCs",
|
| 285 |
+
"Energy / Renewables / Utilities", "Agriculture / Agri-business",
|
| 286 |
+
"Fashion / Beauty / Lifestyle", "Food & beverage / Consumer products",
|
| 287 |
+
"Coaches / Creators / Course businesses", "ESG / Sustainability",
|
| 288 |
+
"Insurance", "Telecom", "Automotive"),
|
| 289 |
placeholder="Pick all that apply, or type your own"),
|
| 290 |
_q("q14", _S_DOMAIN,
|
| 291 |
"Which of those do you know best? Tell me something about how that industry "
|
|
|
|
| 308 |
|
| 309 |
_q("q18", _S_SKILLS, "What is your main skill category?",
|
| 310 |
"skills", "score", "chips",
|
| 311 |
+
("Writing & content", "Academic & research writing", "Legal services",
|
| 312 |
+
"Accounting & bookkeeping", "Finance & business consulting",
|
| 313 |
+
"Human resources & recruiting", "Translation", "Design & creative",
|
| 314 |
+
"Web / software development", "Marketing & SEO",
|
| 315 |
+
"Admin & virtual assistance", "Data & analytics", "Video & audio",
|
| 316 |
+
"Customer support"),
|
| 317 |
placeholder="Or type your main category"),
|
| 318 |
_q("q19", _S_SKILLS, "Name the five skills a client would genuinely hire you for.",
|
| 319 |
"skills", "score", placeholder="Five skills, comma separated",
|
|
@@ -822,6 +822,13 @@ def _groq_text(
|
|
| 822 |
kwargs: dict[str, Any] = {"model": model, "messages": messages, "temperature": 0}
|
| 823 |
if expected_json:
|
| 824 |
kwargs["response_format"] = {"type": "json_object"}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 825 |
|
| 826 |
try:
|
| 827 |
from groq import Groq # type: ignore
|
|
|
|
| 822 |
kwargs: dict[str, Any] = {"model": model, "messages": messages, "temperature": 0}
|
| 823 |
if expected_json:
|
| 824 |
kwargs["response_format"] = {"type": "json_object"}
|
| 825 |
+
# GPT-OSS models think before they answer, and on Groq's free plan that
|
| 826 |
+
# thinking counts against the same per-minute token allowance as the
|
| 827 |
+
# answer. Low effort leaves the room for the answer itself.
|
| 828 |
+
# Sent as extra_body so an older `groq` SDK, whose create() predates the
|
| 829 |
+
# parameter, passes it through instead of rejecting it.
|
| 830 |
+
if "gpt-oss" in model.lower():
|
| 831 |
+
kwargs["extra_body"] = {"reasoning_effort": "low"}
|
| 832 |
|
| 833 |
try:
|
| 834 |
from groq import Groq # type: ignore
|
|
@@ -170,11 +170,29 @@ def _detect_complexity(confirmed_job: dict) -> str:
|
|
| 170 |
return "standard"
|
| 171 |
|
| 172 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
def _strip_forbidden(text: str) -> str:
|
| 174 |
# Strip any stray HTML/markup tags the model may emit (e.g. a trailing
|
| 175 |
# </div>). Proposals are plain prose, so this is safe and keeps the copied
|
| 176 |
# / downloaded text clean — not just the on-screen render.
|
| 177 |
text = re.sub(r"</?\s*[a-zA-Z][a-zA-Z0-9]*[^>]*>", "", text)
|
|
|
|
|
|
|
| 178 |
for phrase in FORBIDDEN_PHRASES:
|
| 179 |
pattern = re.compile(re.escape(phrase), re.IGNORECASE)
|
| 180 |
text = pattern.sub("", text)
|
|
@@ -238,12 +256,42 @@ def _opening(confirmed_job: dict, claims: list[dict]) -> str:
|
|
| 238 |
return "Here is how I'd approach the brief you've shared."
|
| 239 |
|
| 240 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 241 |
def _credibility(evidence: list, matched_skills: list[str], claims: list[dict]) -> str:
|
| 242 |
parts: list[str] = []
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
experience_proofs = _proofs_by_type(evidence, {"experience", "work_history"})
|
| 244 |
-
if experience_proofs:
|
| 245 |
proof = experience_proofs[0]
|
| 246 |
-
text =
|
| 247 |
if text:
|
| 248 |
parts.append(text)
|
| 249 |
claims.append(_claim(text, proof, kind="experience"))
|
|
@@ -263,10 +311,10 @@ def _credibility(evidence: list, matched_skills: list[str], claims: list[dict])
|
|
| 263 |
if cited_skills:
|
| 264 |
parts.append("Hands-on with " + ", ".join(cited_skills[:4]) + ".")
|
| 265 |
|
| 266 |
-
|
| 267 |
-
if
|
| 268 |
-
proof =
|
| 269 |
-
text =
|
| 270 |
if text:
|
| 271 |
parts.append(text)
|
| 272 |
claims.append(_claim(text, proof, kind="achievement"))
|
|
@@ -296,7 +344,9 @@ def _approach(evidence: list, claims: list[dict]) -> str:
|
|
| 296 |
|
| 297 |
if services:
|
| 298 |
for proof in services[:2]:
|
| 299 |
-
|
|
|
|
|
|
|
| 300 |
if text:
|
| 301 |
bullets.append(text)
|
| 302 |
claims.append(_claim(text, proof, kind="service"))
|
|
@@ -317,17 +367,12 @@ def _approach(evidence: list, claims: list[dict]) -> str:
|
|
| 317 |
|
| 318 |
|
| 319 |
def _differentiator(evidence: list, claims: list[dict]) -> str:
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
proof = positioning[0]
|
| 323 |
-
text = str(_proof_attr(proof, "claim_text", "")).strip()
|
| 324 |
-
if text:
|
| 325 |
-
claims.append(_claim(text, proof, kind="positioning"))
|
| 326 |
-
return f"Why me, briefly: {text}"
|
| 327 |
portfolio = _proofs_by_type(evidence, {"portfolio", "testimonial", "project"})
|
| 328 |
if portfolio:
|
| 329 |
proof = portfolio[0]
|
| 330 |
-
text =
|
| 331 |
if text:
|
| 332 |
claims.append(_claim(text, proof, kind="portfolio"))
|
| 333 |
return f"For context on past work: {text}"
|
|
@@ -887,6 +932,10 @@ def _build_profile_block(canonical_profile: Any) -> str:
|
|
| 887 |
"weaknesses_to_account_for", # things NOT to claim (tools marked basic)
|
| 888 |
):
|
| 889 |
value = _val(name)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 890 |
if value:
|
| 891 |
summary[name] = value if not isinstance(value, str) else value[:300]
|
| 892 |
|
|
@@ -914,6 +963,16 @@ def _build_profile_block(canonical_profile: Any) -> str:
|
|
| 914 |
return json.dumps(summary, ensure_ascii=False, indent=2) if summary else "{}"
|
| 915 |
|
| 916 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 917 |
_CRITIQUE_CORRECTION = """
|
| 918 |
|
| 919 |
CORRECTION — your previous response did not return three completed critique
|
|
@@ -948,6 +1007,7 @@ def _try_llm_generate(
|
|
| 948 |
claim_text_cap: int,
|
| 949 |
max_output_tokens: int,
|
| 950 |
canonical_profile: Any = None,
|
|
|
|
| 951 |
) -> tuple[Optional[str], list[dict[str, Any]], list[dict[str, str]], Any]:
|
| 952 |
"""Call the LLM via the central client.
|
| 953 |
|
|
@@ -980,7 +1040,8 @@ def _try_llm_generate(
|
|
| 980 |
# succeeded and simply skipped the rounds — a failed or oversized call
|
| 981 |
# falls through untouched so the existing size-error retry still owns it.
|
| 982 |
if (
|
| 983 |
-
|
|
|
|
| 984 |
and isinstance(result.response_json, dict)
|
| 985 |
and not _has_three_critique_rounds(result.response_json)
|
| 986 |
):
|
|
@@ -1067,13 +1128,19 @@ def generate(
|
|
| 1067 |
if match_data:
|
| 1068 |
matched_skills = list(match_data.get("skill_match", {}).get("matched", []) or [])
|
| 1069 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1070 |
primary_context = build_proposal_context(
|
| 1071 |
confirmed_job,
|
| 1072 |
evidence_list,
|
| 1073 |
match_result=match_data,
|
| 1074 |
recommendation_result=recommendation,
|
| 1075 |
-
max_evidence_points=
|
| 1076 |
-
max_context_chars=
|
| 1077 |
)
|
| 1078 |
|
| 1079 |
used_api = False
|
|
@@ -1101,6 +1168,7 @@ def generate(
|
|
| 1101 |
claim_text_cap=compact_meta["claim_text_cap"],
|
| 1102 |
max_output_tokens=settings.proposal_max_output_tokens,
|
| 1103 |
canonical_profile=canonical_profile,
|
|
|
|
| 1104 |
)
|
| 1105 |
if llm_result is not None:
|
| 1106 |
llm_status = llm_result.status
|
|
@@ -1114,13 +1182,15 @@ def generate(
|
|
| 1114 |
# shrink and retry ONCE with the hard fallback settings.
|
| 1115 |
if proposal is None and _is_size_error(llm_result):
|
| 1116 |
retry_used = True
|
|
|
|
|
|
|
| 1117 |
retry_context = build_proposal_context(
|
| 1118 |
confirmed_job,
|
| 1119 |
evidence_list,
|
| 1120 |
match_result=match_data,
|
| 1121 |
recommendation_result=recommendation,
|
| 1122 |
-
max_evidence_points=8,
|
| 1123 |
-
max_context_chars=8000,
|
| 1124 |
)
|
| 1125 |
chosen_context = retry_context
|
| 1126 |
retry_meta = retry_context["__meta__"]
|
|
@@ -1133,6 +1203,7 @@ def generate(
|
|
| 1133 |
claim_text_cap=retry_meta["claim_text_cap"],
|
| 1134 |
max_output_tokens=1200,
|
| 1135 |
canonical_profile=canonical_profile,
|
|
|
|
| 1136 |
)
|
| 1137 |
if llm_result is not None:
|
| 1138 |
llm_status = llm_result.status
|
|
|
|
| 170 |
return "standard"
|
| 171 |
|
| 172 |
|
| 173 |
+
# Headings and labels from the team's coaching documents. They mean nothing
|
| 174 |
+
# to a client, so any that slip into a draft (from the model or the local
|
| 175 |
+
# outline) are removed: "YOUR SELECTED OFFER", "PART A — YOUR OFFER
|
| 176 |
+
# STATEMENT", "HOW THIS HELPS THE CLIENT", "Offer 2 — ".
|
| 177 |
+
_INTERNAL_LABELS = re.compile(
|
| 178 |
+
r"\byour selected offer\b[\s,:—–-]*|\bpart [ab]\s*[—–:-]\s*|"
|
| 179 |
+
r"\byour offer statement\b[\s,:—–-]*|\bhow this helps the (?:client|brand)\b[\s:—–-]*|"
|
| 180 |
+
r"\b(?:offer|niche)\s*\d+\s*[a-z]?\s*[—–:]\s*",
|
| 181 |
+
re.IGNORECASE,
|
| 182 |
+
)
|
| 183 |
+
# Four or more capitalised words in a row ("ACADEMIC WRITING SUPPORT FOR
|
| 184 |
+
# MANAGEMENT") are a pasted document heading; acronyms such as "MBA" or
|
| 185 |
+
# "APA" stand alone and are left as they are.
|
| 186 |
+
_CAPS_HEADING_RUN = re.compile(r"\b(?:[A-Z][A-Z&/'’-]+\s+){3,}[A-Z][A-Z&/'’-]+\b\s*")
|
| 187 |
+
|
| 188 |
+
|
| 189 |
def _strip_forbidden(text: str) -> str:
|
| 190 |
# Strip any stray HTML/markup tags the model may emit (e.g. a trailing
|
| 191 |
# </div>). Proposals are plain prose, so this is safe and keeps the copied
|
| 192 |
# / downloaded text clean — not just the on-screen render.
|
| 193 |
text = re.sub(r"</?\s*[a-zA-Z][a-zA-Z0-9]*[^>]*>", "", text)
|
| 194 |
+
text = _INTERNAL_LABELS.sub("", text)
|
| 195 |
+
text = _CAPS_HEADING_RUN.sub("", text)
|
| 196 |
for phrase in FORBIDDEN_PHRASES:
|
| 197 |
pattern = re.compile(re.escape(phrase), re.IGNORECASE)
|
| 198 |
text = pattern.sub("", text)
|
|
|
|
| 256 |
return "Here is how I'd approach the brief you've shared."
|
| 257 |
|
| 258 |
|
| 259 |
+
_SECOND_PERSON = re.compile(r"\s(?:You|Your)\b")
|
| 260 |
+
|
| 261 |
+
|
| 262 |
+
def _lead_phrase(text: str) -> str:
|
| 263 |
+
"""The short title of a coaching-document paragraph.
|
| 264 |
+
|
| 265 |
+
Dossier strengths read "Natural Communication Confidence You described
|
| 266 |
+
communication as your forte…": the title is the part a client may see;
|
| 267 |
+
the rest is written to the student. Text without such a title is cut to
|
| 268 |
+
its first sentence.
|
| 269 |
+
"""
|
| 270 |
+
text = " ".join(str(text or "").split())
|
| 271 |
+
match = _SECOND_PERSON.search(text)
|
| 272 |
+
if match and 8 <= match.start() <= 120:
|
| 273 |
+
return text[: match.start()].strip(" .,:;—-")
|
| 274 |
+
sentence = re.split(r"(?<=[.!?])\s", text, maxsplit=1)[0]
|
| 275 |
+
return sentence[:160].strip()
|
| 276 |
+
|
| 277 |
+
|
| 278 |
def _credibility(evidence: list, matched_skills: list[str], claims: list[dict]) -> str:
|
| 279 |
parts: list[str] = []
|
| 280 |
+
# Strength titles first: they are the dossier's own summary of what the
|
| 281 |
+
# freelancer brings, and they read cleanly to a client.
|
| 282 |
+
strengths = [
|
| 283 |
+
_lead_phrase(_proof_attr(p, "claim_text", ""))
|
| 284 |
+
for p in _proofs_by_type(evidence, {"achievement"})
|
| 285 |
+
]
|
| 286 |
+
# A paragraph that starts in lower case is the tail of one cut by a page
|
| 287 |
+
# break, not a strength of its own.
|
| 288 |
+
strengths = [s for s in strengths if s and s[0].isupper()][:3]
|
| 289 |
+
if strengths:
|
| 290 |
+
parts.append("What I bring: " + "; ".join(strengths) + ".")
|
| 291 |
experience_proofs = _proofs_by_type(evidence, {"experience", "work_history"})
|
| 292 |
+
if experience_proofs and not strengths:
|
| 293 |
proof = experience_proofs[0]
|
| 294 |
+
text = _lead_phrase(_proof_attr(proof, "claim_text", ""))
|
| 295 |
if text:
|
| 296 |
parts.append(text)
|
| 297 |
claims.append(_claim(text, proof, kind="experience"))
|
|
|
|
| 311 |
if cited_skills:
|
| 312 |
parts.append("Hands-on with " + ", ".join(cited_skills[:4]) + ".")
|
| 313 |
|
| 314 |
+
metrics = _proofs_by_type(evidence, {"metric"})
|
| 315 |
+
if metrics:
|
| 316 |
+
proof = metrics[0]
|
| 317 |
+
text = _lead_phrase(_proof_attr(proof, "claim_text", ""))
|
| 318 |
if text:
|
| 319 |
parts.append(text)
|
| 320 |
claims.append(_claim(text, proof, kind="achievement"))
|
|
|
|
| 344 |
|
| 345 |
if services:
|
| 346 |
for proof in services[:2]:
|
| 347 |
+
# A dossier deliverable reads "1. Content Audit You review every
|
| 348 |
+
# page…": its title is the step, the rest is written to the student.
|
| 349 |
+
text = re.sub(r"^\d+\\?[.)]\s*", "", _lead_phrase(_proof_attr(proof, "claim_text", "")))
|
| 350 |
if text:
|
| 351 |
bullets.append(text)
|
| 352 |
claims.append(_claim(text, proof, kind="service"))
|
|
|
|
| 367 |
|
| 368 |
|
| 369 |
def _differentiator(evidence: list, claims: list[dict]) -> str:
|
| 370 |
+
# The offer statement is not quoted here: it is the angle, not proof, and
|
| 371 |
+
# its numbers are promises ("increase conversion by 40-80%").
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 372 |
portfolio = _proofs_by_type(evidence, {"portfolio", "testimonial", "project"})
|
| 373 |
if portfolio:
|
| 374 |
proof = portfolio[0]
|
| 375 |
+
text = _lead_phrase(_proof_attr(proof, "claim_text", ""))
|
| 376 |
if text:
|
| 377 |
claims.append(_claim(text, proof, kind="portfolio"))
|
| 378 |
return f"For context on past work: {text}"
|
|
|
|
| 932 |
"weaknesses_to_account_for", # things NOT to claim (tools marked basic)
|
| 933 |
):
|
| 934 |
value = _val(name)
|
| 935 |
+
if isinstance(value, list):
|
| 936 |
+
# A long list (every weakness a call transcript mentions, every
|
| 937 |
+
# deliverable in a dossier) would crowd out the evidence.
|
| 938 |
+
value = [str(v)[:200] for v in value if v][:_PROFILE_LIST_CAP]
|
| 939 |
if value:
|
| 940 |
summary[name] = value if not isinstance(value, str) else value[:300]
|
| 941 |
|
|
|
|
| 963 |
return json.dumps(summary, ensure_ascii=False, indent=2) if summary else "{}"
|
| 964 |
|
| 965 |
|
| 966 |
+
_PROFILE_LIST_CAP = 8
|
| 967 |
+
|
| 968 |
+
# Groq's free plan allows roughly 8,000 tokens a minute for its default model,
|
| 969 |
+
# and the full proposal request is about that size before any answer. On Groq
|
| 970 |
+
# the evidence is trimmed to fit, and the extra critique re-ask (a second
|
| 971 |
+
# full-size call in the same minute) is skipped.
|
| 972 |
+
_GROQ_MAX_EVIDENCE_POINTS = 10
|
| 973 |
+
_GROQ_MAX_CONTEXT_CHARS = 6000
|
| 974 |
+
|
| 975 |
+
|
| 976 |
_CRITIQUE_CORRECTION = """
|
| 977 |
|
| 978 |
CORRECTION — your previous response did not return three completed critique
|
|
|
|
| 1007 |
claim_text_cap: int,
|
| 1008 |
max_output_tokens: int,
|
| 1009 |
canonical_profile: Any = None,
|
| 1010 |
+
enforce_critique: bool = True,
|
| 1011 |
) -> tuple[Optional[str], list[dict[str, Any]], list[dict[str, str]], Any]:
|
| 1012 |
"""Call the LLM via the central client.
|
| 1013 |
|
|
|
|
| 1040 |
# succeeded and simply skipped the rounds — a failed or oversized call
|
| 1041 |
# falls through untouched so the existing size-error retry still owns it.
|
| 1042 |
if (
|
| 1043 |
+
enforce_critique
|
| 1044 |
+
and result.success
|
| 1045 |
and isinstance(result.response_json, dict)
|
| 1046 |
and not _has_three_critique_rounds(result.response_json)
|
| 1047 |
):
|
|
|
|
| 1128 |
if match_data:
|
| 1129 |
matched_skills = list(match_data.get("skill_match", {}).get("matched", []) or [])
|
| 1130 |
|
| 1131 |
+
on_groq = getattr(settings, "active_provider", "") == "groq"
|
| 1132 |
+
max_points = settings.max_proposal_evidence_points
|
| 1133 |
+
max_chars = settings.max_proposal_context_chars
|
| 1134 |
+
if on_groq:
|
| 1135 |
+
max_points = min(max_points, _GROQ_MAX_EVIDENCE_POINTS)
|
| 1136 |
+
max_chars = min(max_chars, _GROQ_MAX_CONTEXT_CHARS)
|
| 1137 |
primary_context = build_proposal_context(
|
| 1138 |
confirmed_job,
|
| 1139 |
evidence_list,
|
| 1140 |
match_result=match_data,
|
| 1141 |
recommendation_result=recommendation,
|
| 1142 |
+
max_evidence_points=max_points,
|
| 1143 |
+
max_context_chars=max_chars,
|
| 1144 |
)
|
| 1145 |
|
| 1146 |
used_api = False
|
|
|
|
| 1168 |
claim_text_cap=compact_meta["claim_text_cap"],
|
| 1169 |
max_output_tokens=settings.proposal_max_output_tokens,
|
| 1170 |
canonical_profile=canonical_profile,
|
| 1171 |
+
enforce_critique=not on_groq,
|
| 1172 |
)
|
| 1173 |
if llm_result is not None:
|
| 1174 |
llm_status = llm_result.status
|
|
|
|
| 1182 |
# shrink and retry ONCE with the hard fallback settings.
|
| 1183 |
if proposal is None and _is_size_error(llm_result):
|
| 1184 |
retry_used = True
|
| 1185 |
+
# Always smaller than the first try: 8 points / 8,000 chars on the
|
| 1186 |
+
# default budget, and proportionally less on Groq's smaller one.
|
| 1187 |
retry_context = build_proposal_context(
|
| 1188 |
confirmed_job,
|
| 1189 |
evidence_list,
|
| 1190 |
match_result=match_data,
|
| 1191 |
recommendation_result=recommendation,
|
| 1192 |
+
max_evidence_points=min(8, max(4, max_points - 4)),
|
| 1193 |
+
max_context_chars=min(8000, int(max_chars * 0.66)),
|
| 1194 |
)
|
| 1195 |
chosen_context = retry_context
|
| 1196 |
retry_meta = retry_context["__meta__"]
|
|
|
|
| 1203 |
claim_text_cap=retry_meta["claim_text_cap"],
|
| 1204 |
max_output_tokens=1200,
|
| 1205 |
canonical_profile=canonical_profile,
|
| 1206 |
+
enforce_critique=not on_groq,
|
| 1207 |
)
|
| 1208 |
if llm_result is not None:
|
| 1209 |
llm_status = llm_result.status
|
|
@@ -83,7 +83,12 @@ _EXTRACTION_INSTRUCTIONS = (
|
|
| 83 |
"For \"client_jobs_posted\" and "
|
| 84 |
"\"client_hires\", capture the counts the client profile shows (e.g. "
|
| 85 |
"\"12 jobs posted\", \"8 hires\"); the hire-rate percentage still goes in "
|
| 86 |
-
"\"hire_rate\".
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
"shown. For \"hidden_keyword\", capture any explicit instruction-style "
|
| 88 |
"phrase planted in the description that asks the applicant to include a "
|
| 89 |
"specific word/phrase to prove they read it (else \"Not visible\"). For "
|
|
|
|
| 83 |
"For \"client_jobs_posted\" and "
|
| 84 |
"\"client_hires\", capture the counts the client profile shows (e.g. "
|
| 85 |
"\"12 jobs posted\", \"8 hires\"); the hire-rate percentage still goes in "
|
| 86 |
+
"\"hire_rate\". Take these three ONLY from the 'About the client' panel. "
|
| 87 |
+
"The 'Activity on this job' section also shows a 'Hires' line, but that "
|
| 88 |
+
"counts hires for this one post (usually 0 while it is open) — never put "
|
| 89 |
+
"it in \"client_hires\" and never use it to work out \"hire_rate\". If "
|
| 90 |
+
"the 'About the client' panel is not in the images, mark those fields "
|
| 91 |
+
"\"Not visible\". For \"client_last_active\", capture how recent activity is "
|
| 92 |
"shown. For \"hidden_keyword\", capture any explicit instruction-style "
|
| 93 |
"phrase planted in the description that asks the applicant to include a "
|
| 94 |
"specific word/phrase to prove they read it (else \"Not visible\"). For "
|
|
@@ -197,6 +197,85 @@ def _missing_heads_up_fields(confirmed_job: dict) -> list[str]:
|
|
| 197 |
return [f for f in _HEADS_UP_FIELDS if _field_is_missing(confirmed_job, f)]
|
| 198 |
|
| 199 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
def _visible_job_text(confirmed_job: dict, *, cap: int = 600) -> str:
|
| 201 |
"""Short context string from the job's visible fields for the labeling LLM."""
|
| 202 |
parts = [
|
|
@@ -777,6 +856,7 @@ def render() -> None:
|
|
| 777 |
|
| 778 |
# ---- Full 10-signal table + niche note ----------------------------
|
| 779 |
_render_signal_table(beginner_eval)
|
|
|
|
| 780 |
_render_niche_note(beginner_eval)
|
| 781 |
|
| 782 |
# ---- Strengths & concerns ----------------------------------------
|
|
|
|
| 197 |
return [f for f in _HEADS_UP_FIELDS if _field_is_missing(confirmed_job, f)]
|
| 198 |
|
| 199 |
|
| 200 |
+
# The client details a user can type in when the screenshot missed Upwork's
|
| 201 |
+
# 'About the client' box. These are the fields the hire-rate and client checks
|
| 202 |
+
# hang on, so without them the verdict can only say "check it yourself".
|
| 203 |
+
_TYPED_CLIENT_FIELDS = ("hire_rate", "client_jobs_posted", "payment_verification")
|
| 204 |
+
_PAYMENT_CHOICES = {
|
| 205 |
+
"Not sure": "",
|
| 206 |
+
"Yes, verified": "Payment verified",
|
| 207 |
+
"No, not verified": "Payment not verified",
|
| 208 |
+
}
|
| 209 |
+
|
| 210 |
+
|
| 211 |
+
def _typed_client_value(field: str, raw: str) -> str:
|
| 212 |
+
"""Clean one typed client detail, or "" when it can't be used.
|
| 213 |
+
|
| 214 |
+
The hire rate must be a 0-100 percentage and jobs posted a whole number;
|
| 215 |
+
anything else is dropped rather than guessed at.
|
| 216 |
+
"""
|
| 217 |
+
text = " ".join(str(raw or "").split())[:40]
|
| 218 |
+
numbers = re.findall(r"\d+(?:\.\d+)?", text.replace(",", ""))
|
| 219 |
+
if field == "hire_rate":
|
| 220 |
+
if not numbers or float(numbers[0]) > 100:
|
| 221 |
+
return ""
|
| 222 |
+
return f"{numbers[0]}% hire rate"
|
| 223 |
+
if field == "client_jobs_posted":
|
| 224 |
+
return f"{int(float(numbers[0]))} jobs posted" if numbers else ""
|
| 225 |
+
return text
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
def _render_client_details_form(confirmed_job: dict) -> None:
|
| 229 |
+
"""Let the user fill in the client details the screenshot did not show.
|
| 230 |
+
|
| 231 |
+
Typed values only ever replace a "Not visible" field. Saving them clears
|
| 232 |
+
the cached analysis, so the verdict is worked out again with them.
|
| 233 |
+
"""
|
| 234 |
+
missing = [f for f in _TYPED_CLIENT_FIELDS if _field_is_missing(confirmed_job, f)]
|
| 235 |
+
if not missing:
|
| 236 |
+
return
|
| 237 |
+
with st.container(border=True, key="ups_panel_analysis_client"):
|
| 238 |
+
theme.section_label("Add the client's details")
|
| 239 |
+
st.caption(
|
| 240 |
+
"These weren't in your screenshot. You'll find them in the "
|
| 241 |
+
"'About the client' box on the Upwork job page. Fill in what you "
|
| 242 |
+
"can and the verdict is checked again."
|
| 243 |
+
)
|
| 244 |
+
with st.form("client_details_form", border=False):
|
| 245 |
+
cols = st.columns(len(missing))
|
| 246 |
+
typed: dict[str, str] = {}
|
| 247 |
+
for col, field in zip(cols, missing):
|
| 248 |
+
if field == "hire_rate":
|
| 249 |
+
typed[field] = col.text_input("Hire rate", placeholder="e.g. 75%")
|
| 250 |
+
elif field == "client_jobs_posted":
|
| 251 |
+
typed[field] = col.text_input("Jobs posted", placeholder="e.g. 12")
|
| 252 |
+
else:
|
| 253 |
+
choice = col.selectbox("Payment verified?", list(_PAYMENT_CHOICES))
|
| 254 |
+
typed[field] = _PAYMENT_CHOICES[choice]
|
| 255 |
+
submitted = st.form_submit_button("Check again with these details")
|
| 256 |
+
if not submitted:
|
| 257 |
+
return
|
| 258 |
+
cleaned = {
|
| 259 |
+
field: _typed_client_value(field, value)
|
| 260 |
+
for field, value in typed.items()
|
| 261 |
+
}
|
| 262 |
+
cleaned = {field: value for field, value in cleaned.items() if value}
|
| 263 |
+
if not cleaned:
|
| 264 |
+
st.warning(
|
| 265 |
+
"Nothing to add yet: type the hire rate as a percentage and "
|
| 266 |
+
"jobs posted as a number."
|
| 267 |
+
)
|
| 268 |
+
return
|
| 269 |
+
updated = dict(confirmed_job)
|
| 270 |
+
for field, value in cleaned.items():
|
| 271 |
+
updated[field] = {"value": value, "confidence": "high", "source": "typed by user"}
|
| 272 |
+
st.session_state.confirmed_job_fields = updated
|
| 273 |
+
# Jobs posted is not part of the job fingerprint, so drop the cached
|
| 274 |
+
# result outright to make sure the verdict is worked out again.
|
| 275 |
+
st.session_state.match_data = None
|
| 276 |
+
st.rerun()
|
| 277 |
+
|
| 278 |
+
|
| 279 |
def _visible_job_text(confirmed_job: dict, *, cap: int = 600) -> str:
|
| 280 |
"""Short context string from the job's visible fields for the labeling LLM."""
|
| 281 |
parts = [
|
|
|
|
| 856 |
|
| 857 |
# ---- Full 10-signal table + niche note ----------------------------
|
| 858 |
_render_signal_table(beginner_eval)
|
| 859 |
+
_render_client_details_form(confirmed_job)
|
| 860 |
_render_niche_note(beginner_eval)
|
| 861 |
|
| 862 |
# ---- Strengths & concerns ----------------------------------------
|
|
@@ -37,6 +37,8 @@ from app.services.folder_validator import (
|
|
| 37 |
FolderValidationResult,
|
| 38 |
SUPPORTED_EXTENSIONS,
|
| 39 |
THIN_DOSSIER_THRESHOLD,
|
|
|
|
|
|
|
| 40 |
score_rubric_labels,
|
| 41 |
validate,
|
| 42 |
)
|
|
@@ -315,10 +317,17 @@ def _sync_dossier_uploads() -> list[dict]:
|
|
| 315 |
accept_multiple_files=True,
|
| 316 |
key=uploader_key,
|
| 317 |
help=(
|
| 318 |
-
"
|
| 319 |
-
"
|
|
|
|
|
|
|
| 320 |
),
|
| 321 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 322 |
|
| 323 |
stored: list[dict] = ss.get(_UPLOAD_RECORDS_KEY) or []
|
| 324 |
if files:
|
|
@@ -536,6 +545,7 @@ def _card_read_evidence() -> None:
|
|
| 536 |
st.session_state.evidence_index = proofs
|
| 537 |
st.session_state.canonical_profile = profile
|
| 538 |
st.session_state.evidence_index_meta = meta
|
|
|
|
| 539 |
|
| 540 |
proofs = st.session_state.get("evidence_index")
|
| 541 |
if proofs:
|
|
@@ -563,6 +573,18 @@ def _card_read_evidence() -> None:
|
|
| 563 |
# ---------------------------------------------------------------------------
|
| 564 |
|
| 565 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 566 |
def _continue_chain() -> None:
|
| 567 |
"""Run the full dossier pipeline silently, then advance.
|
| 568 |
|
|
@@ -593,6 +615,7 @@ def _continue_chain() -> None:
|
|
| 593 |
st.session_state.evidence_index = proofs
|
| 594 |
st.session_state.canonical_profile = profile
|
| 595 |
st.session_state.evidence_index_meta = meta
|
|
|
|
| 596 |
if st.session_state.get("evidence_index"):
|
| 597 |
# Persist the path so the app can auto-reload it on the next restart.
|
| 598 |
if path:
|
|
@@ -713,10 +736,12 @@ def _card_interview_gap_fill(result: FolderValidationResult, *, debug: bool) ->
|
|
| 713 |
"""Offer the interview as a gap-filler for a thin dossier."""
|
| 714 |
with st.container(border=True, key="ups_panel_dossier_6"):
|
| 715 |
theme.section_label("Your files gave us less than usual")
|
|
|
|
| 716 |
st.warning(
|
| 717 |
-
"
|
| 718 |
-
|
| 719 |
-
"work
|
|
|
|
| 720 |
)
|
| 721 |
if debug:
|
| 722 |
st.caption(
|
|
@@ -748,6 +773,13 @@ def _card_interview_gap_fill(result: FolderValidationResult, *, debug: bool) ->
|
|
| 748 |
st.rerun()
|
| 749 |
|
| 750 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 751 |
def _should_offer_gap_fill(result: FolderValidationResult | None) -> bool:
|
| 752 |
"""True when real files were read but they scored thin.
|
| 753 |
|
|
|
|
| 37 |
FolderValidationResult,
|
| 38 |
SUPPORTED_EXTENSIONS,
|
| 39 |
THIN_DOSSIER_THRESHOLD,
|
| 40 |
+
missing_in_plain_words,
|
| 41 |
+
score_evidence,
|
| 42 |
score_rubric_labels,
|
| 43 |
validate,
|
| 44 |
)
|
|
|
|
| 317 |
accept_multiple_files=True,
|
| 318 |
key=uploader_key,
|
| 319 |
help=(
|
| 320 |
+
"Your Upwork Profile Optimisation document and your Dossier work "
|
| 321 |
+
"best; add a portfolio, résumé or testimonials if you have them. "
|
| 322 |
+
"Your files are processed in this session only and are never "
|
| 323 |
+
"stored on the server."
|
| 324 |
),
|
| 325 |
)
|
| 326 |
+
st.caption(
|
| 327 |
+
"Upload your **Upwork Profile Optimisation** document and your "
|
| 328 |
+
"**Dossier**. Leave out the Offer Bank — it lists offers you didn't "
|
| 329 |
+
"choose."
|
| 330 |
+
)
|
| 331 |
|
| 332 |
stored: list[dict] = ss.get(_UPLOAD_RECORDS_KEY) or []
|
| 333 |
if files:
|
|
|
|
| 545 |
st.session_state.evidence_index = proofs
|
| 546 |
st.session_state.canonical_profile = profile
|
| 547 |
st.session_state.evidence_index_meta = meta
|
| 548 |
+
_score_read_files(proofs)
|
| 549 |
|
| 550 |
proofs = st.session_state.get("evidence_index")
|
| 551 |
if proofs:
|
|
|
|
| 573 |
# ---------------------------------------------------------------------------
|
| 574 |
|
| 575 |
|
| 576 |
+
def _score_read_files(proofs: list) -> None:
|
| 577 |
+
"""Re-score the folder from what its files turned out to hold.
|
| 578 |
+
|
| 579 |
+
The validation runs before anything is read and sees file names only, so
|
| 580 |
+
a full dossier PDF scored as thin. Scoring the proof points instead is
|
| 581 |
+
what decides whether the "less than usual" offer appears.
|
| 582 |
+
"""
|
| 583 |
+
result = st.session_state.get("dossier_validation")
|
| 584 |
+
if result is not None and proofs:
|
| 585 |
+
st.session_state.dossier_validation = score_evidence(result, proofs)
|
| 586 |
+
|
| 587 |
+
|
| 588 |
def _continue_chain() -> None:
|
| 589 |
"""Run the full dossier pipeline silently, then advance.
|
| 590 |
|
|
|
|
| 615 |
st.session_state.evidence_index = proofs
|
| 616 |
st.session_state.canonical_profile = profile
|
| 617 |
st.session_state.evidence_index_meta = meta
|
| 618 |
+
_score_read_files(proofs)
|
| 619 |
if st.session_state.get("evidence_index"):
|
| 620 |
# Persist the path so the app can auto-reload it on the next restart.
|
| 621 |
if path:
|
|
|
|
| 736 |
"""Offer the interview as a gap-filler for a thin dossier."""
|
| 737 |
with st.container(border=True, key="ups_panel_dossier_6"):
|
| 738 |
theme.section_label("Your files gave us less than usual")
|
| 739 |
+
missing = missing_in_plain_words(result)
|
| 740 |
st.warning(
|
| 741 |
+
"We read your files but couldn't find "
|
| 742 |
+
+ (_join_words(missing) if missing else "much to quote from")
|
| 743 |
+
+ ". A few questions about your work fill the gaps — your files "
|
| 744 |
+
"are kept either way."
|
| 745 |
)
|
| 746 |
if debug:
|
| 747 |
st.caption(
|
|
|
|
| 773 |
st.rerun()
|
| 774 |
|
| 775 |
|
| 776 |
+
def _join_words(items: list[str]) -> str:
|
| 777 |
+
"""Join phrases as a sentence would: "a", "a and b", "a, b and c"."""
|
| 778 |
+
if len(items) <= 1:
|
| 779 |
+
return "".join(items)
|
| 780 |
+
return ", ".join(items[:-1]) + " and " + items[-1]
|
| 781 |
+
|
| 782 |
+
|
| 783 |
def _should_offer_gap_fill(result: FolderValidationResult | None) -> bool:
|
| 784 |
"""True when real files were read but they scored thin.
|
| 785 |
|
|
@@ -1382,14 +1382,22 @@ def _card_question(state: Any, question: Any, index: int, *, debug: bool) -> Non
|
|
| 1382 |
):
|
| 1383 |
_submit(state, option, from_chip=True)
|
| 1384 |
else:
|
|
|
|
|
|
|
| 1385 |
picked = st.multiselect(
|
| 1386 |
"Pick everything that applies",
|
| 1387 |
options,
|
| 1388 |
key=f"iv_multi_{index}_{seq}",
|
|
|
|
|
|
|
| 1389 |
)
|
| 1390 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1391 |
text = st.text_area(
|
| 1392 |
-
"Your answer",
|
| 1393 |
key=f"iv_answer_{index}_{seq}",
|
| 1394 |
value=earlier if (editing and kind == "text") else "",
|
| 1395 |
placeholder=str(
|
|
@@ -1397,7 +1405,7 @@ def _card_question(state: Any, question: Any, index: int, *, debug: bool) -> Non
|
|
| 1397 |
),
|
| 1398 |
height=110,
|
| 1399 |
max_chars=_MAX_ANSWER_CHARS,
|
| 1400 |
-
label_visibility="collapsed",
|
| 1401 |
)
|
| 1402 |
|
| 1403 |
col_send, col_prev, col_skip, col_finish = st.columns([2, 1, 1, 1])
|
|
@@ -1469,7 +1477,10 @@ def _card_question(state: Any, question: Any, index: int, *, debug: bool) -> Non
|
|
| 1469 |
# to read. Sent as a chip it costs no call against the free tier.
|
| 1470 |
# Typed words are not on that list, so as soon as there are any
|
| 1471 |
# the whole answer goes through validation.
|
| 1472 |
-
|
|
|
|
|
|
|
|
|
|
| 1473 |
if not answer:
|
| 1474 |
st.warning("Type an answer first, or use Skip.")
|
| 1475 |
else:
|
|
|
|
| 1382 |
):
|
| 1383 |
_submit(state, option, from_chip=True)
|
| 1384 |
else:
|
| 1385 |
+
# Typing a name that is not in the list adds it as a new
|
| 1386 |
+
# option, so a sector the list misses can be picked here too.
|
| 1387 |
picked = st.multiselect(
|
| 1388 |
"Pick everything that applies",
|
| 1389 |
options,
|
| 1390 |
key=f"iv_multi_{index}_{seq}",
|
| 1391 |
+
accept_new_options=True,
|
| 1392 |
+
placeholder="Pick from the list, or type your own and press Enter",
|
| 1393 |
)
|
| 1394 |
|
| 1395 |
+
# On a pick-several question the box is the "not in the list" half of
|
| 1396 |
+
# the answer, so it carries a visible label saying so; elsewhere it is
|
| 1397 |
+
# the answer itself and the question above is its label.
|
| 1398 |
+
multi_box = bool(options) and kind == "multi"
|
| 1399 |
text = st.text_area(
|
| 1400 |
+
"Not in the list? Type it here" if multi_box else "Your answer",
|
| 1401 |
key=f"iv_answer_{index}_{seq}",
|
| 1402 |
value=earlier if (editing and kind == "text") else "",
|
| 1403 |
placeholder=str(
|
|
|
|
| 1405 |
),
|
| 1406 |
height=110,
|
| 1407 |
max_chars=_MAX_ANSWER_CHARS,
|
| 1408 |
+
label_visibility="visible" if multi_box else "collapsed",
|
| 1409 |
)
|
| 1410 |
|
| 1411 |
col_send, col_prev, col_skip, col_finish = st.columns([2, 1, 1, 1])
|
|
|
|
| 1477 |
# to read. Sent as a chip it costs no call against the free tier.
|
| 1478 |
# Typed words are not on that list, so as soon as there are any
|
| 1479 |
# the whole answer goes through validation.
|
| 1480 |
+
# A name typed into the dropdown is a new option, not one of the
|
| 1481 |
+
# list's, so it goes through validation like typed words do.
|
| 1482 |
+
new_picks = [p for p in picked if p not in options]
|
| 1483 |
+
from_picks = bool(picked) and not typed and not new_picks
|
| 1484 |
if not answer:
|
| 1485 |
st.warning("Type an answer first, or use Skip.")
|
| 1486 |
else:
|
|
@@ -27,6 +27,12 @@ from app.ui import output_screen, theme
|
|
| 27 |
# Background-task id for proposal generation (one at a time).
|
| 28 |
_PROPOSAL_TASK = "proposal_generation"
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
def _copy_button(text: str) -> None:
|
| 32 |
"""A real one-click copy button (works in deployed Streamlit)."""
|
|
@@ -80,16 +86,32 @@ def _copy_button(text: str) -> None:
|
|
| 80 |
def _render_proposal(proposal_result: dict, show_debug: bool, *, settings) -> None:
|
| 81 |
prop_meta = proposal_result.get("__meta__") or {}
|
| 82 |
verify_meta = proposal_result.get("verification_meta") or {}
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
proposal_text = proposal_result.get("proposal", "")
|
| 95 |
|
|
@@ -221,7 +243,14 @@ def render() -> None:
|
|
| 221 |
unsafe_allow_html=True,
|
| 222 |
)
|
| 223 |
|
| 224 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 225 |
|
| 226 |
with st.container(border=True, key="ups_panel_proposal_3"):
|
| 227 |
theme.section_label("Generate")
|
|
@@ -312,7 +341,7 @@ def render() -> None:
|
|
| 312 |
st.session_state.proposal_generating = False
|
| 313 |
st.session_state.generated_proposal = None
|
| 314 |
st.session_state.verified_proposal = None
|
| 315 |
-
st.error(
|
| 316 |
return
|
| 317 |
|
| 318 |
proposal_result = st.session_state.get("verified_proposal")
|
|
|
|
| 27 |
# Background-task id for proposal generation (one at a time).
|
| 28 |
_PROPOSAL_TASK = "proposal_generation"
|
| 29 |
|
| 30 |
+
_PROPOSAL_FAILED_MESSAGE = (
|
| 31 |
+
"The AI couldn't write the proposal this time. Wait a minute, then press "
|
| 32 |
+
"Generate Proposal again. If it keeps failing, check the API key on the "
|
| 33 |
+
"Setup screen."
|
| 34 |
+
)
|
| 35 |
+
|
| 36 |
|
| 37 |
def _copy_button(text: str) -> None:
|
| 38 |
"""A real one-click copy button (works in deployed Streamlit)."""
|
|
|
|
| 86 |
def _render_proposal(proposal_result: dict, show_debug: bool, *, settings) -> None:
|
| 87 |
prop_meta = proposal_result.get("__meta__") or {}
|
| 88 |
verify_meta = proposal_result.get("verification_meta") or {}
|
| 89 |
+
# Without the AI the app can only fill a fixed outline from the files.
|
| 90 |
+
# That outline must never pass for a finished proposal.
|
| 91 |
+
written_by_ai = bool(prop_meta.get("used_api"))
|
| 92 |
+
if written_by_ai:
|
| 93 |
+
output_screen.render_clean_stage_banner(
|
| 94 |
+
output_screen._stage_user_state(prop_meta),
|
| 95 |
+
output_screen._stage_user_state(verify_meta),
|
| 96 |
+
)
|
| 97 |
+
theme.screen_head(
|
| 98 |
+
"proposal",
|
| 99 |
+
"Your draft is ready",
|
| 100 |
+
"A grounded first draft built only from facts in your dossier. "
|
| 101 |
+
"Read it through, then copy it straight into Upwork.",
|
| 102 |
+
)
|
| 103 |
+
else:
|
| 104 |
+
theme.screen_head(
|
| 105 |
+
"proposal",
|
| 106 |
+
"AI writing was unavailable",
|
| 107 |
+
"This is only a rough outline, filled in from your files without "
|
| 108 |
+
"the AI. Don't send it as it is: wait a minute, then press "
|
| 109 |
+
"Generate Proposal again.",
|
| 110 |
+
)
|
| 111 |
+
st.warning(
|
| 112 |
+
"Not a finished proposal. The AI service didn't respond this time, "
|
| 113 |
+
"so the outline below is not tailored to the job."
|
| 114 |
+
)
|
| 115 |
|
| 116 |
proposal_text = proposal_result.get("proposal", "")
|
| 117 |
|
|
|
|
| 243 |
unsafe_allow_html=True,
|
| 244 |
)
|
| 245 |
|
| 246 |
+
# The warning follows the verdict the user was shown on the Analysis
|
| 247 |
+
# screen — the job checklist — not the hidden fit-score tier, which could
|
| 248 |
+
# read "Do Not Proceed" under a "Proceed with caution" verdict.
|
| 249 |
+
beginner_eval = (match_data or {}).get("beginner_evaluation")
|
| 250 |
+
if isinstance(beginner_eval, dict) and beginner_eval.get("result"):
|
| 251 |
+
do_not_proceed = bool(beginner_eval.get("instant_no"))
|
| 252 |
+
else:
|
| 253 |
+
do_not_proceed = recommendation.get("verdict") == "Do Not Proceed"
|
| 254 |
|
| 255 |
with st.container(border=True, key="ups_panel_proposal_3"):
|
| 256 |
theme.section_label("Generate")
|
|
|
|
| 341 |
st.session_state.proposal_generating = False
|
| 342 |
st.session_state.generated_proposal = None
|
| 343 |
st.session_state.verified_proposal = None
|
| 344 |
+
st.error(_PROPOSAL_FAILED_MESSAGE)
|
| 345 |
return
|
| 346 |
|
| 347 |
proposal_result = st.session_state.get("verified_proposal")
|
|
@@ -387,8 +387,10 @@ def render() -> None:
|
|
| 387 |
_clear_screenshots_for_new_job()
|
| 388 |
st.rerun()
|
| 389 |
st.caption(
|
| 390 |
-
"
|
| 391 |
-
"
|
|
|
|
|
|
|
| 392 |
)
|
| 393 |
|
| 394 |
# Groq is text-only: a Groq vision provider can't read screenshots.
|
|
|
|
| 387 |
_clear_screenshots_for_new_job()
|
| 388 |
st.rerun()
|
| 389 |
st.caption(
|
| 390 |
+
"Include the **About the client** box (hire rate, jobs posted, "
|
| 391 |
+
"payment verified) — add a second screenshot if it doesn't fit in "
|
| 392 |
+
"one. Missing details are marked “Not visible”; the tool won't "
|
| 393 |
+
"guess hidden information."
|
| 394 |
)
|
| 395 |
|
| 396 |
# Groq is text-only: a Groq vision provider can't read screenshots.
|