Challenge Parameters
- Opens
- November 9th, 202600:00 UTC
- Submission deadline
- December 20th, 2026end of the day, anywhere on Earth
- You have
- 6 weeks
What you doWork on the hypotheses provided: find the papers, extract the findings
Work on the five APOE4 endosome mechanism hypotheses provided. Find the relevant papers, and extract the relevant experimental findings.
What you submitOne spreadsheet per hypothesis, passing the validator
One spreadsheet per hypothesis, using the template provided (19 columns). Spreadsheets must pass the validator. A submission that fails validation cannot be uploaded.
What you must provideA public README describing your method
A public repository or archive containing a detailed README describing how your submission was produced, and the commit it was built from. Without it the submission does not validate.
How manyOne live submission per hypothesis, replaceable until the deadline
One live submission per hypothesis, which a team may replace as often as it likes until the deadline. The last one before the deadline is the one assessed. You may submit for as many of the five hypotheses as you like.
Limits per submission50 papers. Fewer is not scored down
50 papers. Submitting fewer is not scored down.
Who can take partAnyone, after registering
Anyone. Register once with your Hugging Face account, then create a team or join one with its invitation code. A team of one is fine. Every team names a leader, who submits for the team.
Team membershipSubmissions are made by teams
Each participant must be a member of a team. A team has one or more members. Submissions are made by teams, not individuals. Every team has a leader.
Human, agent, or agent collabAny of these, stated in your README
Your README must include a statement about the type of submission.
Challenge productsAn open-source pipeline, a paper, authorship
The overall goal of the challenge is to produce an open-source pipeline that can be used to find and extract experimental findings from the literature for any mechanism hypothesis. Results will be published in a scientific journal, and all participants in good standing will be invited to be authors.
Time scale to release of resultsAssessment takes time after the challenge closes
Following closure of the challenge, members of the MecCog group will work on assessing the results. This is a new type of challenge, and we anticipate this process may take some time.
What matters mostAccuracy and relevance over volume
Accuracy and relevance over volume. A shorter submission containing only relevant material will score better than one padded with general Alzheimer's literature. Do not exclude findings that provide evidence against the hypothesis.
Select a line for the full rule.
How to Get Started
- 1
Read the hypotheses page
- 2
Register on the submission page with your Hugging Face account, then create a team or join one with its invitation code
- 3
Download submission_template.xlsx and method_README_template.md from shared resources
- 4
Search for papers
- 5
Extract findings and fill the spreadsheet as you go, provenance codes and directness included
- 6
Validate
- 7
Publish your method README, fill in the Method sheet, then the team leader uploads the spreadsheet and keep your acceptance number
Ready? Register, form or join a team, and submit on the submission page.
Register & submit →The official rules apply to every MecCog challenge, and registering or submitting accepts them. Questions go on the Community tab of this Space.
About This Challenge
Welcome to the MecCog Challenges, a collaborative project to critically assess proposed mechanisms by which the APOE4 genetic variant increases the risk of Alzheimer’s disease. This is Step 1 of a multi-phase challenge series. Later phases will ask you to assess evidence, estimate mechanism confidence, and propose discriminatory experiments. For now we focus on finding and documenting what is in the literature.
What You’ll Do
For each hypothesis, you will:
Search the scientific literature (PubMed, preprints, databases, web sources)
Identify papers that bear on that mechanism
Extract experimental findings from those papers (measurements, effects, experimental conditions and so on)
Record everything in a structured spreadsheet.
The Hypotheses
Hypotheses are released in batches, starting November 9th, 2026. The current batch is listed on the hypotheses page.
Work on the hypotheses provided. Do not catalog Alzheimer’s mechanisms in general. The goal is to discover findings that directly test or inform a hypothesis.
The hypotheses are precisely worded and the qualifying clauses are part of them. That does not mean a finding has to match them exactly to belong in your submission. Most experiments are unavoidably done in model systems rather than in living human brain, and those findings are very much relevant. Record the experimental system as precisely as the paper allows and let the specifics speak. Weighing how far an experimental condition sits from the hypothesis is the work of a later challenge, not this one.
Teams
Register first. Sign in on the submission page with your Hugging Face account and register once for the challenge. Registration asks for an institutional or company email address. When your Hugging Face account has verified such an address, you are approved at once. If you do not have one, describe in a sentence or two who you are and what you do, and an organizer reviews it before approving you. This is asked so that the organizers know who produced each submission, not in order to publish it.
Teams. Every submission belongs to a team. After registering, you may work alone as a team of one, create a team, or join an existing one. Creating a team makes you its leader and gives you an invitation code, and anyone joining your team enters that code. There is no maximum team size, and each member of a team must be a registered participant.
The team leader. Every team names a team leader, who is the point of contact for the organizers and is responsible for the team’s submissions. Only the team leader submits, signed in with the account that created the team, and the Team row of the Method sheet carries the team’s name.
You may belong to more than one team, but creating a second team requires a written explanation, for example that your group has two genuinely different methods and wants both assessed. Creating several teams to multiply your submissions is not allowed.
Challenge Requirements
What is Needed
Search widely. PubMed, the preprint servers, Google Scholar, databases, and anything else you think bears on the hypothesis. It is better to look at a source you end up setting aside than to miss one that mattered.
Extract findings precisely. Each one needs:
A one-sentence description of what was done and what was observed
A direct quote from the paper, or N/A if none is available
The experimental system, described as precisely as the paper allows. Not just “astrocytes” but the source of the cells; not just “mouse” but the strain
Where in the paper the finding appears, at panel level. Fig 4C, not Fig 4
The statistics, with their provenance (see below)
The directness class, saying how the experiment lines up with the hypothesis, and the strength of that link
Keep the findings that argue against the hypothesis. A result that contradicts the hypothesis, or that shows no effect, is evidence, and is needed as much as the supporting ones. Record it like any other finding, with the same care over the quote and the statistics.
Leave no cell empty. Every cell carries either a value or the literal N/A. A blank says nothing, and cannot be told apart from an oversight.
Get the metadata right. DOI, PubMed ID where the source has one, and source type, for every paper.
Validate before you submit. Use the Validate button on the submission page, or run the validator yourself beforehand; the download link is on the shared resources page. A submission that fails validation cannot be uploaded, so it is worth running early rather than on the deadline.
Publish your method. Name a public repository or archive holding a README that describes how your submission was produced, and the commit it was built from. Both go on the Method sheet of the spreadsheet, described below. A list of findings without a method is a result without a protocol, which is why a submission without them does not validate.
Five scenarios of how submissions can score badly
Padding the sheet. Include only the papers and findings that genuinely bear on a hypothesis. A submission swollen with general Alzheimer’s literature scores worse than a short, focused one. This is not a license to drop a finding that argues against the hypothesis: relevance is about the subject, not the direction of the result. One submission carries at most 50 papers, and the validator enforces it, so order them by relevance from the start and keep the ones that earn their place.
Fabrication. Every finding must exist in the paper at the location you cite.
An estimate presented as a reported value. Where it is only available that way, you should estimate an effect size off a bar chart, or count the plotted points to get a sample size. A P value should not be estimated. Any estimate has to be identified by a leading ~ on the value and the provenance code estimated_from_figure. An estimate recorded as though the paper printed it is a fabrication.
Invented identifiers. DOIs and PubMed IDs must correspond to real, published sources.
Working ahead. This is Step 1 only. Assessing confidence, designing experiments and building mechanism schemas all come later.
Submission Format
StructureColumnsMethodProvenance
One spreadsheet per hypothesis. Download the official template from the shared resources page.
If something in the format is unclear, ask on the Community tab. Format questions are usually everyone’s.
Each submission is limited to 50 papers.
Use the hypothesis statement exactly as published, including its qualifying clauses, and copy it verbatim into cell A2 of the Submission sheet.
Structure
The template has two sheets that the validator reads, Submission and Method, already named. Do not rename them. A third sheet, How to fill this in, comes last and is there for you; the validator ignores it.
The Submission sheet:
Row 1: column labels, provided in the template. All of them must be present, in the template’s order
Row 2: the hypothesis in column A. The rest of row 2 stays empty
Row 3 onward: one block per paper, a paper row followed by one row per finding from that paper
The Method sheet holds your team, who leads it and how to reach them, and the link to your method, seven labelled rows described under Providing your method.
The columns of the Submission sheet
| Column | Content | Accepted values |
|---|---|---|
| Hypothesis | The hypothesis you selected. Row 2 only | The statement copied verbatim from the hypotheses page. Empty on every other row |
| The paper | ||
| DOI | The source’s DOI | A DOI, or N/A for a source that has none |
| Paper/source type | What kind of source it is | Exactly one of PubMed published, PubMed preprint, Web article, Database, Other |
| PubMed ID | Required for any source cataloged in PubMed | Digits only, or N/A |
| Paper/Finding ID | What the row is: a paper or one of its findings | P1, P2 and so on for papers, in order of relevance, P1 holding the findings most relevant to the hypothesis. P1.F1, P1.F2 and so on for the findings of P1 |
| The finding | ||
| Finding description | Plain English, one sentence | Free text |
| Finding quote | The exact text from the paper | Free text, or N/A |
| Finding summary | Structured notation, used to match findings across submissions | For example (APOE3 -> APOE4) -> (less ABCA1), or N/A where the form does not fit |
| Finding relevance | How relevant the finding is to the hypothesis | A number from 0 to 1 |
| Experimental system | The system and the conditions | Free text, as precise as the paper allows |
| Data location | Where the data are | Fig 4C, Table 2, Supplementary Fig 3, or an exact quoted sentence |
| Statistics and provenance | ||
| Effect size | The size of the change | A percentage (40%, 300%), a fold change (2.5x), a standardized effect size as the paper gives it (d=3.7, r=0.58), any of these with a leading ~ where you estimated it from a figure (~40%), the word qualitative where the paper shows a direction you cannot put a number on, or N/A |
| Effect size provenance | Where that value came from, or why there is none | One of the codes below |
| P value | The reported significance | A number (0.03), an inequality (<0.001, >0.05), or N/A |
| P value provenance | Where that value came from, or why there is none | One of the codes below |
| Sample size | The number of samples | A whole number (12), a range as the paper gives it (8-12), either with a leading ~ where you counted plotted points instead of reading a stated number (~12), or N/A |
| Sample size provenance | Where that value came from, or why there is none | One of the codes below |
| Directness | ||
| Directness class | How directly the finding bears on the hypothesis | A number from 1 to 5, on finding rows only |
| Directness strength | How strong that link is | A number from 1 to 5 where 1 is the strongest, or the number with its word (2 strong), on finding rows only |
Providing your method
A list of findings without a method is a result without a protocol. This challenge is as much about how the literature was read as about what was found, so the method has to be public and it has to be findable.
What to publish. A public repository or archive with a README.md at its root holding six sections:
Who did the work: a person, an agent, a person and an agent, or several agents working together
Model and version, if any was used
How papers were found: sources and queries
How full text was obtained
What a human checked, and on what proportion of the findings
Anything else worth knowing
Copy method_README_template.md from the shared resources page. For work done entirely by hand, six short paragraphs are enough; nobody expects code that does not exist.
What goes in the spreadsheet. The Method sheet carries seven labelled rows. Fill the cell to the right of each label:
The four values for who did the work: person for a human working alone, agent for a single agent, both for a human and an agent, collab for several agents working together. For collab, list every model in Model and version, one per agent, and say in section 1 of your README which agent did which step and whether a person was involved as well.
The two cells have to agree. A submission made entirely by hand is person, and its Model and version is none, or N/A if you prefer the convention used everywhere else in the format. Anything other than person means a model was used, so the model and its version are required. The validator checks the pair in both directions.
Where to publish it. Any public host: GitHub, GitLab, Codeberg, or an archive with a DOI such as Zenodo or OSF. A private repository does not count, and neither does a link that needs an account to open.
Why the commit SHA. A repository can be rewritten, emptied or made private after the deadline, so a bare link is not a record of what produced the submission. The SHA pins it. Run git rev-parse HEAD and put the result in the Commit SHA cell of the Method sheet. An archive with a DOI is already immutable, so it takes N/A instead.
Recording what the paper does not report
A missing number can mean two opposite things. Either the paper never reported it, which is a fact about the literature and one of the things this challenge exists to measure. Or it was reported and the submission did not find it, which is a fact about the submission. Writing N/A for both makes them indistinguishable.
So every statistic carries a provenance code alongside it. Use exactly one of:
a value was foundread from a figureno usable value
| Code | Meaning |
|---|---|
printed_in_text |
The value is printed in the text, a table or a figure legend, and you took it from there |
visual_only |
The paper shows it graphically or as significance stars, with no printed number, and you recorded what it shows without estimating a value |
estimated_from_figure |
There is no printed number and you measured one off the figure, by bar height, axis position or by counting plotted points. The value carries a leading ~ |
supplement_only |
The value is in the supplementary material, and you took it from there |
not_found_in_source |
The paper does not report it anywhere you could find |
full_text_unavailable |
You could not reach the full text, so you cannot say whether it is reported |
ambiguous_panel |
The panel you cite reports several comparisons and it is not clear which one the value belongs to |
Conventions: effect sizes, estimates, P values, significance stars, paywalls, ranges
Report the effect as the paper reports it. A percentage, a fold change and a standardized effect size are all acceptable, and the one to use is the one the authors used. Converting a Cohen’s d into a percentage, or a fold change into a percentage, introduces an assumption that is yours and not theirs. Where the paper gives several, give the one that belongs to the comparison your finding is about.
Estimates are welcome, as long as they say so. Where a figure carries the information but no number is printed, measure it: an effect size from the bars, a sample size by counting the plotted points. Write the value with a leading ~ and use estimated_from_figure. This is better than N/A, because a marked estimate can be used or discarded at will while a blank cannot be recovered. Where you cannot put a defensible number on what the figure shows, use qualitative with visual_only rather than inventing a precision you do not have.
Never estimate a P value. An effect size and a sample size are quantities the figure displays, so measuring them is reading, with an error you have declared by writing the ~. A P value is not displayed. The figure shows the data a test was run on; the test itself happened elsewhere, and its result is either stated or it is not.
So a figure gives you a P value only where the panel or its legend states one. If the panel carries stars, record what the legend says they mean. If it carries nothing, the value is N/A with not_found_in_source. The validator rejects estimated_from_figure on a P value.
Significance stars. Record the inequality the paper’s own legend gives them, with provenance visual_only. The usual convention is * for p<0.05, ** for p<0.01 and *** for p<0.001, but that is only the usual one and it is an example, not the rule: some papers start at ** for p<0.01, some add **** for p<0.0001, and some use the stars for an adjusted P value. Read the legend of the figure you are citing and follow it. Where you can read the key, record the inequality, with provenance visual_only, and do not add a ~: the inequality is what the paper asserts, not something you measured. Where the figure gives no key at all, there is no inequality to record, so the value is N/A with provenance ambiguous_panel. Where you can read the key, N/A would lose what the paper tells you.
Paywalls. full_text_unavailable means the full text was genuinely out of reach. Before using it, try the DOI directly, the publisher’s own page, Europe PMC, PMC, and Unpaywall. “My tool could not fetch it” is not the same as “it is not available”, and the difference matters when submissions are compared.
Ranges and per-group values. Where a paper gives a range (8-12 mice per group), record the range. Do not pick one end of it: a single number that is not in the paper is less faithful than the range that is.
The directness class
Every hypothesis in this challenge has the same form: A change in one system variable A causes a change in another system variable B. For the ABCA1 hypothesis, the system variable A is a change from the reference APOE3 to the APOE4 variant, and B is a change in the abundance of ABCA1 protein in the astrocyte outer membrane.
ABCA1 is used on this page and in the example sheet only as a worked example. It is not one of the five challenge hypotheses.
The directness class says how the experiment behind your finding lines up with that shape. It describes what the experiment did, not what the evidence is worth. Use one of:
1Direct
A itself was perturbed, or varies naturally, and B itself was measured
2Indirect input
A perturbation in A is assumed to have been introduced by a perturbation in surrogate variable A’. For example, a microRNA (A’) is introduced that alters the expression of a protein (A). B itself was measured.
3Indirect output
A itself was perturbed, and B was inferred from measuring a change in a surrogate variable B’ expected to be affected by a change in B. For example, the change in the amount of cholesterol exported from cells (B’) is measured, and assumed to indicate the change in the abundance of a cholesterol exporter protein (B).
4Indirect input and output
There is a perturbation of surrogate variable for A and measurement of a surrogate variable for B.
5Other
None of the classes above fit. Say why in the Finding description.
the variable itself a surrogate another route the perturbation could take
A perturbation does not have to be introduced by the experimenter. Natural variation counts: for example, comparing APOE4 carriers with APOE3 carriers is considered a perturbation of A, the APOE genotype. Do not drop a finding to a lower class because the experiment observed A rather than manipulated it.
The class belongs to the finding, not to the paper. One paper can hold findings of several classes, so the cell is filled on finding rows and left empty on paper rows. The validator rejects a class on a paper row.
The class is relative to the hypothesis you selected. The same experiment may take a different class against a different hypothesis, so do not carry your classes from one hypothesis to another.
The strength of the link. The class describes the form of the experiment. An additional parameter, d, is required, describing strength of the link between that experiment and the hypothesis, on a scale of 1 to 5. 1 is the strongest link and 5 the weakest, so a low number always means the experiment is closer to the hypothesis. The template offers the number with its word, 1 very strong through 5 very weak, and a bare number is accepted too.
‘d’ can be regarded as a measure of how misleading the use of surrogate variables might be. For example, for a class 2 experiment where A’ is measured, how likely is it that A’ might affect the value of B by some other mechanism than causing a perturbation of A. Similarly, for a class 3 experiment, where a change in B’ is measured rather than the change in B, how likely is it that the perturbation of A could cause a perturbation of B’ by some other mechanism than as a consequence of a change in B.
| Strength | When |
|---|---|
| 1 very strong | Usually for a Class 1 experiment, where changes in A and B are directly measured. |
| 2 strong | One or both measurements are of surrogate variables, but there is no plausible way by which the observations could be made other than by a change in A causing a change in B. |
| 3 moderate | A possible alternative route exists but it is unlikely to invalidate the hypothesis that the A change causes the B change. |
| 4 weak | There is a known mechanism by which the surrogate variable(s) changes could be related other than because A causes a change in B. |
| 5 very weak | There are several such alternative mechanisms, or one or both surrogates are weakly coupled to the A and B variables. |
Class 4 has a surrogate variable at both ends, so there are two places an alternative mechanism could be involved, and d is unlikely to be lower than 3. A class 5 finding is judged on the same question and explained in Finding description.
The validator checks the pair for the one error that is easy to make. A class 1 finding with a strength of 4 or 5 is flagged, because class 1 has no stand-in at either end, so that combination usually means the scale was read upside down.
How You’ll Be Evaluated
Unlike most community challenges to date, this one has no ground truth against which to assess the results. Thus, the performance assessment methods break new ground and may evolve as the project progresses. Community challenges differ in the extent to which they predefine assessment criteria. Some, such as those run by Kaggle, predefine a single metric that will be used for ranking. Others, such as CASP, predefine a subset of criteria, but encourage independent assessors to introduce new ones, once the results are in hand. This challenge sets out a set of initial considerations (below), without an explicit ranking formula. These may be modified and added to once the results are analyzed.
Initial assessment will be as follows:
Cross-evaluation by participants to find the subset of strongest performing groups (each participant provides subjective scores for the methods and results submitted by ten other participants, with comments, generating ten assessments per group). Necessary only if there are a large number of submissions.
Comparison of results across groups: which papers were included, which experiments were extracted from those papers. This does not imply that findings reported by one or a few groups will be considered incorrect. Rather, it will help identify ‘easy’ results.
Evaluation of each finding’s relevance using the evidence directness criteria.
Checking of each Finding’s specifics (effect size, P value, experimental system) against the source paper. (Experience with the Hugging Face collab showed that providing these specifics is hard for current methods)
Where necessary, manual inspection.
Submitting
Accepted submissions are given an acceptance number, shown on screen. Keep it, and quote it in any question about the submission.
Each team has one live submission per hypothesis. Submitting again for the same hypothesis replaces it, as often as you like until the deadline, and the last one before the deadline is the one assessed. Only the team leader submits, and replaces, the team’s submissions.
Timeline and Questions
Step 1 opens: 00:00 UTC on November 9th, 2026
First batch of hypotheses released: November 9th, 2026
Step 1 submission deadline: the end of December 20th, 2026, anywhere on Earth
Later phases, evidence assessment, mechanism confidence and experimental design: dates to be announced
Questions go to the Community tab first, then to the organizers. Organizers and the community are here to help.
Thank you for contributing to MecCog. Together we are building a map of APOE4 mechanisms.