Spaces:
Paused
Paused
Download api/connectors_tt.py from aios-loopable/cfo-os: direct link, hf CLI and curl.
- Browser
- Download file 34.2 kB
-
https://huggingface.co/spaces/aios-loopable/cfo-os/resolve/main/api/connectors_tt.py
- Command line
-
hf download hf://spaces/aios-loopable/cfo-os/api/connectors_tt.py
-
curl -L -o connectors_tt.py https://huggingface.co/spaces/aios-loopable/cfo-os/resolve/main/api/connectors_tt.py
34.2 kB
| """connectors_tt.py β the TIKTOK connector (wave 29 Β· item 7 Β· DEBT D-9 Β· rulings R1 + R2). | |
| Everything in this file knows what a VENDOR's TikTok row looks like. Nothing in it knows what an | |
| automation is. That split is `connectors_ig.py`'s (wave 27 item 23) and it is the reason this file | |
| exists at all rather than another thousand lines inside the engine. | |
| β **EVERY VENDOR FIELD NAME HERE WAS PROBED, NOT GUESSED.** The whole schema β 40 profile / 43 | |
| post / 17 comment fields, each with the vendor's own type, description and `pii` flag β was read | |
| live from `GET /datasets/{id}/metadata` for **$0.00** and written down in | |
| `.claude/wiki/waves/wave29/proto/tiktok-schema.md` (promoted to `tiktok-capture.md` at | |
| close-out). That document is the AUTHORITY: do not re-probe it, and do not invent a key. Where a | |
| name below reads through a candidate list it is because the vendor has two names for one fact | |
| (`biography`/`signature`, `region`/`country`), never because the name is uncertain. | |
| β **NOTHING HERE EVER AUTHENTICATES TO TIKTOK.** No login, no cookie, no account to get banned β | |
| public data through a supplier, exactly the rail `connectors_ig.py` states for Instagram. The | |
| vendor key is a key to a SUPPLIER. | |
| β **THE TRANSPORT IS `connectors_bd.py` β SHARED, VENDOR-NAMED, AND NO LONGER BORROWED FROM THE | |
| OTHER PLATFORM'S CONNECTOR** (WAVE 30 Β· T09, DEBT D-128). `bd_call`, `bd_scrape` and | |
| `bd_filter_start` take the dataset id as a PARAMETER β they are Bright Data's wire, not | |
| Instagram's β and re-implementing them here would be a second copy of the deferral handling, the | |
| truncation guard, the SSRF rail and the snapshot-progress reader, i.e. five places for one bug. | |
| Until wave 30 the code was right and the NAME was wrong: this file imported thirteen symbols from | |
| `connectors_ig`, which read as a dependency on Instagram and was really a dependency on a supplier. | |
| β **This file now imports ZERO names from `connectors_ig`, and a gate check asserts that**, because | |
| the sentence above is the kind that quietly stops being true. | |
| β **WHAT $0 COULD NOT BUY, so nobody reads this file as more measured than it is:** | |
| 1. the real ROW shape β `/metadata` describes a DATASET, and Instagram's rows carry undeclared | |
| envelope keys (`timestamp`, `input`) that no metadata call mentions; | |
| 2. that a declared field POPULATES β Bright Data's Instagram Reels *declares* `views: number` | |
| and delivers an account-grain wrong number (Β§4e). **Declared is not delivered**, and the one | |
| TikTok claim that matters most (`play_count`) is exactly a declaration. | |
| """ | |
| from __future__ import annotations | |
| import automation_engine as engine | |
| # β T09 β FOUR NAMES CAME OFF THIS LIST AND NOTHING BROKE, which is the point of deriving an | |
| # import block from the AST both ways. `bd_call`, `bd_filter_start`, `bd_key` and `bd_ready` were | |
| # imported here under a comment claiming they were *"re-exported for the runners"*; no runner ever | |
| # read them off this module (the engine imports them from the transport itself), so they were four | |
| # lines of dependency nobody was paying for. `[[artifact-with-no-importer]]` in its smallest form. | |
| from connectors_bd import ( | |
| _bd_first_url, | |
| _bd_flag, | |
| _bd_list, | |
| _bd_source_payload, | |
| _first, | |
| _ig_int, | |
| bd_scrape, | |
| ) | |
| # --------------------------------------------------------------------------------------------- | |
| # THE DATASETS | |
| # --------------------------------------------------------------------------------------------- | |
| # β CATALOGUE PRESENCE IS NOT ENTITLEMENT β the same finding Instagram produced. `GET | |
| # /datasets/list` returned 1,735 rows of which 12 are TikTok; the three below answer 200 with a | |
| # full field list, and the two DISCOVERY halves answer **404 for our key**: | |
| # `gd_lj71gn6l68bz7y9hc` (posts by profile) and `gd_lilwhto81z415d9mdl` (posts by keyword). | |
| # β TikTok discovery routes through the PROFILES dataset's corpus filter, exactly as Instagram's | |
| # does. A ticket that reaches for a by-keyword endpoint is reaching for a 404. | |
| TT_DS_PROFILES = "gd_l1villgoiiidt09ci" # TikTok - Profiles. 40 fields, 152,000,000 records | |
| TT_DS_POSTS = "gd_lu702nij2f790tmv9h" # TikTok - Posts. 43 fields | |
| TT_DS_COMMENTS = "gd_lkf2st302ap89utw5k" # TikTok - Comments. 17 fields | |
| #: The vendor's two post-type tokens, verbatim from the dataset's own `ai_description` | |
| #: (*"strictly these two"*, video = 99.5% of rows). Ours are `image`/`video`/`carousel`. | |
| TT_POST_TYPE_VIDEO = "video" | |
| TT_POST_TYPE_CONTENT = "content" | |
| #: What a TikTok profile URL looks like, for the runner that has a handle and needs a URL. Kept | |
| #: beside the dataset ids because it is the same class of vendor fact. | |
| #: β ONE SPELLING. `platform/core/user_tables.profile_url(handle, 'tiktok')` builds the identical | |
| #: string from `_PROFILE_RULES` (contract C2, E's half) β this exists for the connector's own | |
| #: batch calls, and the gate asserts the two agree rather than trusting that they do. | |
| TT_PROFILE_URL = "https://www.tiktok.com/@{handle}" | |
| def tt_profile_url(handle): | |
| """`nurilab` β `https://www.tiktok.com/@nurilab`. `''` for a blank handle, never a bare `@`.""" | |
| h = str(handle or "").strip().lstrip("@") | |
| return TT_PROFILE_URL.format(handle=h) if h else "" | |
| # --------------------------------------------------------------------------------------------- | |
| # THE FIELD MAPS | |
| # --------------------------------------------------------------------------------------------- | |
| # Each function turns ONE vendor row into the cell dict for one of the `ut_tt_*` schemas declared | |
| # in `automation_engine`. The rules they all obey, stated once: | |
| # | |
| # * **BLANK MEANS NOT READ, NEVER "THEY HAVE NONE".** A key the vendor did not send is OMITTED, | |
| # so a later, richer pull fills it instead of being overwritten by this one's silence. `_first` | |
| # returns `None` (never 0) when nothing matches, which is what makes that possible. | |
| # * **A zero from the vendor is a MEASUREMENT and survives.** (Instagram's `_ig_zero_is_blank` | |
| # rule is scoped to a paid rung whose zeros were proven fictional; nothing here has earned it.) | |
| # * **Every unpromoted vendor key stays whole in `source_payload`.** A schema addition on the | |
| # vendor's side is preserved rather than silently discarded while our column model catches up. | |
| # * **Nothing here writes a session token.** `tt_chain_token`, `secu_id` (~85% null), `short_id` | |
| # (100% null in the sample), `ftc` (100% null) and `relation` are deliberately unmapped; they | |
| # ride in `source_payload` where they make no claim. | |
| def _tt_str(node, *names): | |
| """The first non-empty string among `names`, or None when the vendor sent nothing. | |
| β `None`, NOT `""`. The callers below drop `None` keys, which is what keeps a blank honest β | |
| an empty string written into a cell claims "we looked and it is empty". | |
| """ | |
| v = _first(node, *names) | |
| if v is None: | |
| return None | |
| s = str(v).strip() | |
| return s or None | |
| def _tt_pct(node, *names): | |
| """A vendor 0β1 engagement fraction β our stored 0β100 percentage, or None. | |
| β THE Γ100 IS NOT COSMETIC (wave 26, amendment C1-a). Our `pct` renderer appends the sign to | |
| the STORED number, so writing the vendor's raw 0.0656 would print a 6.6% creator as `0.0%` β | |
| measured on the Instagram side, and TikTok sends the same shape on all three of its rates. | |
| """ | |
| v = _first(node, *names) | |
| if v is None: | |
| return None | |
| out = engine._pct100(v) | |
| return out or None | |
| def _tt_day(node, *names): | |
| """A vendor stamp β `YYYY-MM-DD`, or None. Our `date` columns store a day.""" | |
| v = _first(node, *names) | |
| if v is None: | |
| return None | |
| return engine._day(v) or None | |
| def _drop_blanks(row): | |
| """The one place a mapped row loses its `None`s β see the BLANK MEANS NOT READ rule above.""" | |
| return {k: v for k, v in row.items() if v is not None and v != ""} | |
| def normalize_profile(node, handle=""): | |
| """A TikTok Profiles row β the `ut_tt_profile` / `ut_tt_snapshots` cell shape. | |
| β TWO FIELDS READ THROUGH A CANDIDATE PAIR, and both pairs are the vendor's, not a guess: | |
| * `biography` is PRIMARY and `signature` the FALLBACK β the probe measured `signature` | |
| populated on 85% of rows and they carry the same text; | |
| * `region` is PRIMARY and `country` the FALLBACK β `region` is the one with a documented | |
| two-letter-ISO description, `country` has no description at all. | |
| β `videos_count` is mapped to `posts_count` with a caveat recorded rather than hidden: its | |
| `ai_description` ranges 1-89, so it may be a WINDOW rather than a lifetime total. It is the | |
| only count of its kind the dataset offers. | |
| """ | |
| node = node if isinstance(node, dict) else {} | |
| account = _tt_str(node, "account_id") or str(handle or "").strip().lstrip("@") | |
| return _drop_blanks({ | |
| "platform": engine.PLATFORM_TIKTOK, | |
| "handle": account, | |
| "full_name": _tt_str(node, "nickname"), | |
| "tt_id": _tt_str(node, "id"), | |
| "profile_url": _tt_str(node, "url") or (tt_profile_url(account) or None), | |
| "bio": _tt_str(node, "biography", "signature"), | |
| "external_url": _bd_first_url(_first(node, "bio_link")), | |
| "verified": _bd_flag(node, "is_verified"), | |
| "is_private": _bd_flag(node, "is_private"), | |
| # β APPROXIMATE, and the word is the vendor's: `is_commerce_user` has *"many null values"* | |
| # on their own description. It is the closest thing TikTok has to Instagram's | |
| # `is_business_account`, and `_bd_flag` writes nothing at all when the key is absent β so | |
| # the approximation only ever fills a cell the vendor actually answered. | |
| "is_business": _bd_flag(node, "is_commerce_user"), | |
| "followers": _ig_int(_first(node, "followers")), | |
| "following": _ig_int(_first(node, "following")), | |
| "posts_count": _ig_int(_first(node, "videos_count")), | |
| # β `likes` on a PROFILE row is likes RECEIVED across the account's videos (18-110,200, no | |
| # nulls). It is not a post-level number and it is not our `likes` column, which is why it | |
| # is stored under a different name. | |
| "likes_received": _ig_int(_first(node, "likes")), | |
| "avg_engagement": _tt_pct(node, "awg_engagement_rate"), | |
| "like_engagement": _tt_pct(node, "like_engagement_rate"), | |
| "comment_engagement": _tt_pct(node, "comment_engagement_rate"), | |
| "country_code": _tt_str(node, "region", "country"), | |
| "region": _tt_str(node, "region"), | |
| "predicted_lang": _tt_str(node, "predicted_lang"), | |
| # β ACCOUNT AGE, NOT A MEASUREMENT STAMP β `create_time` on a profile is when the ACCOUNT | |
| # was made. TikTok stamps nothing with "when this number was true", exactly like Instagram, | |
| # which is why the append law dates a snapshot by when WE read it. | |
| "account_created_at": _tt_day(node, "create_time"), | |
| "source_payload": _bd_source_payload(node), | |
| }) | |
| def tt_post_type(node): | |
| """A TikTok post row β one of OUR three type options, or None. | |
| β THE FIRST OF THE PROBE DOC'S TWO NAMED BLOCKERS. The vendor's vocabulary is `"video"` / | |
| `"content"`; ours is `image` / `video` / `carousel` and has no `"content"`. Writing the | |
| vendor's token would fail `_clean_field`'s option check on the way in and would put an | |
| untranslated API word in front of a user on the way out. | |
| So: `video` is `video`, and `content` β TikTok's photo-mode post β is `image`, EXCEPT when the | |
| row carries more than one `carousel_images` entry, which is what a carousel IS on either | |
| network. β The multi-image branch is decided from the IMAGES, never from the type token: the | |
| token cannot express it, so inferring `carousel` from the word would be inventing a fact. | |
| β An UNKNOWN token returns None rather than defaulting to `video` (99.5% of rows are video, and | |
| that is exactly what would make the wrong default invisible). | |
| """ | |
| raw = str((node or {}).get("post_type") or "").strip().lower() | |
| if raw == TT_POST_TYPE_VIDEO: | |
| return "video" | |
| if raw != TT_POST_TYPE_CONTENT: | |
| return None | |
| images = (node or {}).get("carousel_images") | |
| return "carousel" if isinstance(images, list) and len(images) > 1 else "image" | |
| def normalize_post(node): | |
| """A TikTok Posts row β the `ut_tt_posts` cell shape. None when it carries no identity. | |
| β THE SECOND NAMED BLOCKER, RESOLVED HERE AND NOWHERE ELSE: `play_count` is ONE number and | |
| Instagram's schema has TWO columns for it (`plays` and `views`). On TikTok they are the same | |
| fact β `play_count` IS the count TikTok displays under a video β so it maps to `views` and | |
| `ut_tt_posts` HAS NO `plays` COLUMN. Copying one vendor number into two of our columns would | |
| manufacture a second measurement that a rollup could average or double-count, which is a worse | |
| outcome than the missing column it would paper over. | |
| β `shortcode` reads `shortcode` then `post_id`: both are 19-digit numerics on this dataset and | |
| the probe measured them as the same shape. That equality is what lets the commentsβposts link | |
| join with NO normaliser, which the Instagram side never had. | |
| β `num_share_count` (a number) is preferred over `share_count` (typed TEXT by the vendor). | |
| β `commerce_info` is a business/commerce LOCATION per its own description β cities and | |
| countries. It is NOT a paid-partnership flag, and nothing in this dataset is: TikTok declares | |
| no equivalent, so `paid_partnership`/`partner` have no column on this family at all. | |
| """ | |
| node = node if isinstance(node, dict) else {} | |
| shortcode = _tt_str(node, "shortcode", "post_id") | |
| if not shortcode: | |
| return None | |
| return _drop_blanks({ | |
| "platform": engine.PLATFORM_TIKTOK, | |
| "shortcode": shortcode, | |
| # ββ `account_id`, NOT `profile_username` β MEASURED on a real Posts row 2026-08-12. | |
| # The vendor's `profile_username` is the DISPLAY NAME (`"Dina"`), while `account_id` is the | |
| # @handle (`"d1na_th"`) β the same field `normalize_profile` already reads for `handle`, so | |
| # one name means one thing across both corpora. Reading the display name silently broke the | |
| # only join this table has: `ut_tt_posts.influencer_key` -> `ut_tt_profile.handle` matched | |
| # NOTHING, so a person could not filter posts by creator and a rollup would count zero. | |
| # β NO FALLBACK TO `profile_username`, deliberately. It is not a degraded handle, it is a | |
| # different fact, and filling a join key with it is worse than leaving it blank β a blank | |
| # is visibly missing, a display name looks like an answer [[one-question-two-normalizers]]. | |
| # The URL is the honest second source: it carries the handle by construction. | |
| "influencer_key": (_tt_str(node, "account_id") | |
| or tt_handle(_tt_str(node, "profile_url", "url") or "") or None), | |
| "posted_at": _tt_day(node, "create_time"), | |
| "type": tt_post_type(node), | |
| "caption": _tt_str(node, "description"), | |
| "url": _tt_str(node, "url"), | |
| "hashtags": _bd_list(node, "hashtags"), | |
| "tagged_location": _tt_str(node, "commerce_info"), | |
| "views": _ig_int(_first(node, "play_count")), | |
| "likes": _ig_int(_first(node, "digg_count")), | |
| "comments": _ig_int(_first(node, "comment_count")), | |
| "shares": _ig_int(_first(node, "num_share_count")), | |
| "saves": _ig_int(_first(node, "collect_count")), | |
| "video_duration": _ig_int(_first(node, "video_duration")), | |
| "source_payload": _bd_source_payload(node), | |
| }) | |
| def normalize_comment(node): | |
| """A TikTok Comments row β the `ut_tt_comments` cell shape. None without a comment id. | |
| β THE NAME COLLISION, RESOLVED: TikTok's `replies` is an ARRAY of reply objects and OUR | |
| `replies` column is an INT count. The count comes from `num_replies`; the array stays whole in | |
| `source_payload`. Reading the array's length instead would be a second, disagreeing answer to | |
| a question the vendor already answers β and it would disagree, because a page of replies is not | |
| all of them. | |
| β `date_created` is typed `date` by this vendor, unlike Instagram's `comment_date` which is | |
| text and needs defensive parsing. It still goes through `_tt_day` β one date path, so a vendor | |
| that changes its mind cannot change ours. | |
| β The comment TEXT and every identifiable commenter field (`commenter_user_name` is flagged | |
| PII) stay in `source_payload` and are promoted to no column, which is the same posture the | |
| Instagram comment schema takes for the same D-24 reason. | |
| """ | |
| node = node if isinstance(node, dict) else {} | |
| comment_key = _tt_str(node, "comment_id") | |
| if not comment_key: | |
| return None | |
| return _drop_blanks({ | |
| "platform": engine.PLATFORM_TIKTOK, | |
| "comment_key": comment_key, | |
| "shortcode": _tt_str(node, "post_id"), | |
| # ββ OWNER RULING 2026-08-12: the comment's CONTENT gets a column. `comment_text` is the | |
| # vendor's own key and `comment_text_only` its stripped variant (the probe recorded both); | |
| # primary first, so a row carrying the rich form is not silently served the plain one. | |
| # β The commenter's identity is deliberately NOT promoted β see `TT_COMMENT_FIELDS`. | |
| "text": _tt_str(node, "comment_text", "comment_text_only"), | |
| "commented_at": _tt_day(node, "date_created"), | |
| "likes": _ig_int(_first(node, "num_likes")), | |
| "replies": _ig_int(_first(node, "num_replies")), | |
| "source_payload": _bd_source_payload(node), | |
| }) | |
| #: β The three maps, addressable by name β so a gate (and the runners in T04-T06) can walk them | |
| #: rather than naming three functions, and so adding a fourth dataset is one entry. | |
| TT_NORMALIZERS = { | |
| "tt_profile": normalize_profile, | |
| "tt_post_metrics": normalize_post, | |
| "tt_comments": normalize_comment, | |
| } | |
| # --------------------------------------------------------------------------------------------- | |
| # THE FETCH β wave 30 Β· W30-T08 (carrying wave-29's dropped T05) | |
| # --------------------------------------------------------------------------------------------- | |
| def tt_handle(url): | |
| """A TikTok profile URL **or** a bare handle β the handle. `''` when it is neither. | |
| Deliberately permissive about the input and strict about the output, because the two callers | |
| hand it different things: an automation stores whatever a person typed in the profile column | |
| (`@nurilab`, `nurilab`, or the full URL), while the discovery runner already holds a clean | |
| `account_id`. One normaliser, so a row found by discovery and a row typed by hand cannot | |
| resolve to two different handles. | |
| """ | |
| s = str(url or "").strip() | |
| if not s: | |
| return "" | |
| if "tiktok.com" in s.lower(): | |
| # Everything after the first `@`, up to the next path segment or query. | |
| tail = s.split("@", 1)[1] if "@" in s else "" | |
| s = tail.split("/")[0].split("?")[0].split("#")[0] | |
| s = s.strip().lstrip("@").strip() | |
| # A handle is the vendor's `account_id` shape: alphanumerics, dots and underscores. | |
| return s if s and all(c.isalnum() or c in "._" for c in s) else "" | |
| def tt_post_urls(node, limit=0): | |
| """β WAVE 30 Β· T10 β the profile row's own post permalinks, newest-first as the vendor sends. | |
| β THIS IS WHY TIKTOK POST CAPTURE COSTS NO EXTRA DISCOVERY. `top_videos` rides the PROFILE row | |
| we have already bought, so the posts read is a scrape of links we hold, never a search for them. | |
| The two TikTok DISCOVERY datasets (posts-by-profile, posts-by-keyword) are **404 for our key**, | |
| so a design that reached for either would not merely be dearer, it would not work. | |
| ββ CORRECTED 2026-08-12 β THIS DOCSTRING USED TO SAY *"the probe MEASURED `top_videos` as an | |
| array of video permalinks, NO empties"*, AND THAT SENTENCE IS WHAT SHIPPED THE BUG. The probe | |
| read `/datasets/{id}/metadata` β a DATASET description β and the phrase quoted was the field's | |
| `ai_description`, not an observation of a row. The real row sends **dicts keyed `video_url`** | |
| (measured below), so the reader built against the quoted sentence found nothing, forever, in | |
| silence. β The transferable half: *"the probe measured X"* and *"the probe read a declaration | |
| of X"* are different claims, and prose cannot be told apart by a reader downstream β which is | |
| why the correction names the method, not just the value. | |
| β `top_posts_data` is deliberately NOT read: the probe calls it *"a thin dup of `top_videos`"*, | |
| and preferring whichever happened to be longer is how one creator's window silently differs | |
| from another's. | |
| β `limit <= 0` means "everything the row carried". The CAP IS THE CALLER'S β `config.maxPosts`, | |
| validated 1..12 β and it is applied here rather than after the scrape so an unwanted post is | |
| never bought. [[a-constant-two-features-share]]: the 12 is the vendor's measured profile window, | |
| not a number this function may invent. | |
| """ | |
| raw = (node or {}).get("top_videos") | |
| out = [] | |
| for item in raw if isinstance(raw, list) else []: | |
| # ββ MEASURED ON A REAL ROW 2026-08-12, AND IT IS NOT WHAT THE SCHEMA SAID. | |
| # `top_videos` is NOT an array of permalink strings. One paid TikTok Profiles scrape of a | |
| # live handle returns **19 DICTS**, keyed | |
| # `video_url Β· video_id Β· playcount Β· diggcount Β· commentcount Β· share_count Β· | |
| # favorites_count Β· create_date Β· cover_image`. | |
| # The docstring above cites the $0 probe as having "measured" permalinks β it had not, and | |
| # could not: `/datasets/{id}/metadata` describes a DATASET, and the probe's own verdict says | |
| # so in terms (*"what $0 cannot buy β¦ the real ROW shape β¦ that a declared field | |
| # POPULATES"*, `wave29/proto/tiktok-schema.md`). This is the SECOND time this vendor's | |
| # declaration has diverged from its delivery on this exact axis; BD's IG Reels `views` was | |
| # the first. [[reachable-is-not-the-same-as-built]] | |
| # β The consequence, live: `TT_DS_POSTS` was never reached, because this returned an EMPTY | |
| # list on every real profile β post capture could not have worked for anybody, and the | |
| # T10 gate stayed green because its canned fixture encoded the DECLARED shape. A fixture | |
| # written from a schema tests the schema. | |
| # β `video_url` is FIRST because it is the key the vendor actually sends; `url` is kept | |
| # because it costs nothing and is what a future corpus revision would most likely use. The | |
| # bare-string branch stays for the same reason β this widens what is ACCEPTED and invents | |
| # nothing: a shape that yields no `httpβ¦` value still degrades to "no posts", exactly as | |
| # before, rather than to a URL built out of a guess. | |
| # β `top_posts_data` is STILL not read (it carries `post_url` and would work): preferring | |
| # whichever array happened to be longer is how one creator's window silently differs from | |
| # another's, and that reasoning is unchanged by this correction. | |
| if isinstance(item, dict): | |
| u = str(item.get("video_url") or item.get("url") or "").strip() | |
| else: | |
| u = str(item or "").strip() | |
| if u.lower().startswith("http") and u not in out: | |
| out.append(u) | |
| return out[:limit] if limit and limit > 0 else out | |
| def pull_posts_tt(post_urls, log=print, deferred=None): | |
| """The TikTok Posts dataset for a list of permalinks β `(rows, note)`, already normalised. | |
| β ONE CALL FOR THE WHOLE WINDOW. `bd_scrape` has always taken a list, and the Instagram side | |
| measured what happens when a caller forgets: 25 records, one billed snapshot each, a walk still | |
| running at 67 minutes. Nothing here loops per URL. | |
| """ | |
| urls = [str(u) for u in (post_urls or []) if str(u or "").strip()] | |
| if not urls: | |
| return [], "" | |
| rows, note = bd_scrape(TT_DS_POSTS, urls, deferred=deferred) | |
| if note: | |
| log(f"[aios-tt] posts: {note}") | |
| return [], note | |
| out = [r for r in (normalize_post(n) for n in rows) if r] | |
| return out, "" | |
| def pull_comments_tt(post_urls, log=print, deferred=None): | |
| """The TikTok Comments dataset for a list of POST permalinks β `(rows, note)`, normalised. | |
| β THE MOST EXPENSIVE THING THIS PRODUCT BUYS, and the reason `commentMetrics` defaults OFF on | |
| both networks: a comments scrape ingests identifiable third parties who never entered anybody's | |
| list (D-24). The mapper already keeps every commenter field in `source_payload` and promotes | |
| none of them to a column; this function adds no new exposure, it just has to be asked for. | |
| """ | |
| urls = [str(u) for u in (post_urls or []) if str(u or "").strip()] | |
| if not urls: | |
| return [], "" | |
| rows, note = bd_scrape(TT_DS_COMMENTS, urls, deferred=deferred) | |
| if note: | |
| log(f"[aios-tt] comments: {note}") | |
| return [], note | |
| out = [r for r in (normalize_comment(n) for n in rows) if r] | |
| return out, "" | |
| #: ββ WAVE 30 Β· D-156 β THE MEDIA DATASETS, AS A SET, SO THE HAND-OFF CAN FILTER ON IDENTITY. | |
| #: `_media_deferrals` uses this to lift ONLY posts/comments snapshots out of the local deferral | |
| #: list. That is what makes it structurally impossible to file a PROFILE snapshot in the engine's | |
| #: metric queue β the defect a draft of T10 shipped and A-39 booked as "the wrong fix is worse | |
| #: than the gap". A membership test cannot be got wrong by a later edit the way `if` order can. | |
| TT_MEDIA_DATASETS = (TT_DS_POSTS, TT_DS_COMMENTS) | |
| def _media_deferrals(deferred): | |
| """The POSTS/COMMENTS entries of a `bd_scrape` deferral list β never the profile's. | |
| β The engine, not this module, decides what a deferral MEANS: it stamps `kind` and the | |
| handle and files it. TikTok needs no `_tag_metric_deferrals` twin because one dataset is one | |
| kind here, so the id already carries everything a mapper choice depends on β and importing | |
| Instagram's tagger is not available anyway (W30-T09 gates ZERO `from connectors_ig` lines). | |
| """ | |
| out = [] | |
| for d in deferred or []: | |
| if isinstance(d, dict) and str(d.get("datasetId") or "") in TT_MEDIA_DATASETS: | |
| out.append(dict(d)) | |
| return out | |
| def pull_profile_tt(url, log=print, pending_profile=None, prefetch=None, | |
| max_posts=0, post_metrics=False, comment_metrics=False): | |
| """ONE TikTok profile from the vendor. Same return contract as `pull_profile`. | |
| `{state, profile, posts, comments, via, note}` with `state β ok | partial | blocked | error`, | |
| so the engine's enrich branch treats every network identically and no caller learns a new | |
| shape. | |
| β WAVE 30 Β· T10 β POSTS AND COMMENTS ARE REAL NOW, AND BOTH DEFAULT OFF, exactly as Instagram's | |
| do. `post_metrics` scrapes the profile row's own `top_videos` permalinks (see `tt_post_urls` β | |
| no discovery call, because both TikTok discovery datasets 404 for our key); `comment_metrics` | |
| then scrapes the comments of the posts that came back. β COMMENTS REQUIRE POSTS by construction | |
| rather than by a rule: their input IS a post permalink, so asking for comments with post capture | |
| off is a request with no subject, and it returns none instead of quietly buying posts nobody | |
| asked for. | |
| β `partial` IS THE SUCCESS STATE WHENEVER NO MEDIA WAS READ, and that is deliberate rather than | |
| pessimistic. The Instagram contract reads `ok` only when identity AND media both landed | |
| (`pull_profile_bd`: *"identity without media is still partial ... a run that wrote a follower | |
| count and no posts must not paint green over a posts table that did not grow"*). So: posts not | |
| ASKED for β `partial`, saying so; posts asked for and landed β `ok`; asked for and none came β | |
| `partial` with the vendor's reason. The state answers "did this pull deliver what it went for", | |
| never "did the function finish". | |
| β **NO FREE RUNG, AND NO FALLBACK CHAIN.** Instagram's `pull_profile` drops to Apify when the | |
| paid rung refuses; `providers.DEFAULT_CHAINS["tt_profile"]` is deliberately single-provider, | |
| with its own note explaining that a multi-provider chain is a promise something walks it and | |
| that nothing walks Instagram's second name today either. So a refusal here is final, and it | |
| says so instead of implying a retry somewhere. | |
| """ | |
| handle = tt_handle(url) | |
| if not handle: | |
| return {"state": "error", "profile": {}, "posts": [], "comments": [], "via": "", | |
| "note": f"{url!r} is not a TikTok profile URL or handle"} | |
| # β THE BATCH FAST PATH, same shape as the Instagram side: `prefetch` is `{handle: node}` from | |
| # one multi-URL scrape covering a whole selection. A hit is a vendor round trip that does not | |
| # happen; a miss falls through to the single-URL call below. | |
| cached = prefetch.get(handle) if isinstance(prefetch, dict) else None | |
| _deferred = [] | |
| if isinstance(cached, dict) and cached: | |
| rows, note = [cached], "" | |
| else: | |
| rows, note = bd_scrape(TT_DS_PROFILES, [tt_profile_url(handle)], deferred=_deferred) | |
| node = rows[0] if rows else {} | |
| profile = normalize_profile(node, handle) if node else {} | |
| # β THE READABILITY TEST IS `followers`/`following`, NOT "did we get a dict". `normalize_profile` | |
| # drops blanks, so an unreadable row still returns `{"platform": β¦, "handle": β¦}` β truthy, and | |
| # carrying nothing anybody asked for. The Instagram rung tests exactly this pair for exactly | |
| # this reason, and answering "0 followers" instead is the failure it exists to prevent. | |
| unreadable = profile.get("followers") is None and profile.get("following") is None | |
| if note or unreadable: | |
| # β THE DEFERRAL IS HANDED OVER RATHER THAN DISCARDED. A snapshot the vendor is still | |
| # building HAS ALREADY BEEN PAID FOR; dropping its id bills again on the next run for the | |
| # same record. That was live on the Instagram profile path until 2026-08-09 β measured on | |
| # nurilab as two runs, two fresh snapshots, both abandoned β and it is not being | |
| # reintroduced here by omission. | |
| if isinstance(pending_profile, list): | |
| for d in _deferred: | |
| pending_profile.append({**d, "kind": "profile", "influencer": handle}) | |
| why = note or ("the scrape answered, but no follower/following counts were readable in it " | |
| "(the field names may have moved - see tiktok-capture.md)") | |
| return {"state": "blocked", "profile": {}, "posts": [], "comments": [], "via": "brightdata", | |
| "deferredProfile": [d.get("snapshotId") for d in _deferred], | |
| "note": why} | |
| # --- W30-T10: THE MEDIA, ONLY WHEN IT WAS ASKED FOR. ------------------------------------ | |
| if not post_metrics: | |
| return {"state": "partial", "profile": profile, "posts": [], "comments": [], | |
| "via": "brightdata", | |
| "note": note or "profile read; post capture is off for this step"} | |
| urls = tt_post_urls(node, limit=max_posts) | |
| if not urls: | |
| # β NOT AN ERROR AND NOT A RETRY. A creator with no `top_videos` has nothing to buy, and | |
| # saying so is what stops the next run paying to be told the same thing. | |
| return {"state": "partial", "profile": profile, "posts": [], "comments": [], | |
| "via": "brightdata", | |
| "note": note or "profile read; this account's row carried no post links"} | |
| posts, p_note = pull_posts_tt(urls, log=log, deferred=_deferred) | |
| comments, c_note = ([], "") | |
| if comment_metrics and posts: | |
| # The comments dataset is keyed on a POST permalink, so it reads the posts we just bought β | |
| # `url` from the mapper, never the profile's raw array, so a post the posts scrape refused | |
| # is not silently asked about again one rung later. | |
| comments, c_note = pull_comments_tt([p.get("url") for p in posts if p.get("url")], | |
| log=log, deferred=_deferred) | |
| # ββ WAVE 30 Β· D-156 β THE MEDIA DEFERRALS ARE HANDED BACK, and the shape of the hand-off is | |
| # the whole lesson. An earlier draft of T10 appended every `_deferred` entry to | |
| # `pending_profile` tagged `kind: "profile"`. By the time control reaches here a PROFILE | |
| # deferral is impossible β the profile branch above returns `blocked` on any note β so **every | |
| # id fanned out that way was a POSTS or COMMENTS snapshot in the PROFILE queue**, whose | |
| # collector writes preset profile cells onto somebody's record from post rows. The engine keeps | |
| # the two queues apart deliberately (`_pending_profile_tasks` vs `_pending_metric_tasks`). | |
| # β So this returns them under their OWN key, filtered by dataset identity | |
| # (`_media_deferrals`), and the engine files them in the metric queue with the handle it | |
| # already holds. Returning rather than appending also keeps the queue's vocabulary out of a | |
| # connector: this module knows which CORPUS deferred, never what the engine calls it. | |
| # β `deferredMedia` rides BOTH returns on purpose. The empty-posts case is the one that | |
| # matters most β that is exactly the run where the vendor took too long, so a caller reading | |
| # the ids only from the success path would lose every batch it actually paid for. | |
| deferred_media = _media_deferrals(_deferred) | |
| if not posts: | |
| return {"state": "partial", "profile": profile, "posts": [], "comments": [], | |
| "via": "brightdata", "deferredMedia": deferred_media, | |
| "note": p_note or note or "profile read; the post source returned nothing"} | |
| return {"state": "ok", "profile": profile, "posts": posts, "comments": comments, | |
| "via": "brightdata", "deferredMedia": deferred_media, | |
| "note": c_note or note or ""} | |