# Reproducing the plant promoter audit

Read [study.html](study.html) for the completed analysis and [report.html](report.html) for the discovery history, including rejected ideas and decisions made before validation. [study-draft.txt](study-draft.txt) is an editable plain-text research-note draft. No manuscript has been submitted.

## Inputs and environment

- Jores et al. (2021): DOI [10.1038/s41477-021-00932-y](https://doi.org/10.1038/s41477-021-00932-y), repository revision `ace4ead59de364922fccdecd331554ccf0ac522b`.
- Independent author attachment: [issue 1](https://github.com/tobjores/Synthetic-Promoter-Designs-Enabled-by-a-Comprehensive-Analysis-of-Plant-Core-Promoters/issues/1), posted 2023-08-28; file checksum in `evidence/jores2021_manifest.json`.
- PGB revision `78ec8156c2ffb3e5475277fdb7eb603294224e53`; file hashes independently checked against its recorded LFS metadata.
- Download URLs, byte counts and SHA256 checksums are in the two input manifests under `evidence/`. A checksum mismatch stops the workflow. Cached absolute paths in those historical manifests are remapped to the selected cache directory by filename.
- Analysis used Python 3.12, NumPy 1.26.4, pandas 2.1.4 (Ubuntu `+dfsg` build), SciPy 1.11.4 and pyreadr 0.5.3. Figures used Matplotlib 3.6.3. For a fresh installation, Python 3.11 has wheels for all versions pinned in `requirements-study.txt`.
- Approximately 383 MB of compressed/input downloads, plus generated tables and caches; allow 2 GB disk and about 2 GB RAM. One CPU thread; no GPU. The full analysis takes minutes, depending on machine and downloads. The optional historical R image adds approximately 684 MB compressed download and several GB unpacked storage.

## Main workflow

From an extracted copy of this project:

```bash
python3.11 -m venv .venv
.venv/bin/pip install -r requirements-study.txt
BIO_DISCOVERY_CACHE="$PWD/.cache" STUDY_PYTHON="$PWD/.venv/bin/python" nice -n 19 ./run_study.sh
```

On the original machine, the existing verified environment and cache can be used:

```bash
BIO_DISCOVERY_CACHE=/mnt/data/research/bioinformatics-discovery/cache \
STUDY_PYTHON=/mnt/data/research/bioinformatics-discovery/venv/bin/python \
nice -n 19 ./run_study.sh
```

The workflow checks analytical controls, downloads/verifies inputs, reconstructs historical and robustly parsed measurements, independently checks the 2023 attachment, matches PGB to CNN exports, runs the predeclared validation plus extension, evaluates frozen predictions, and renders the figures and study note. `promoter_full_study.py` intentionally reuses the four validation libraries produced immediately before it, then adds eight libraries. It does not select only successful results.

The exploratory cutoff/aggregation provenance search (`cnn_target_hypotheses.py`) is retained but is not part of the confirmatory workflow; none of its 108 alternatives exactly reconstructed the CNN targets. The unresolved CNN-generation history is explicitly excluded from the main claim.

## Actual historical R parser check

This optional independent verification was executed successfully. It uses no network inside the analysis container and no writable host mount. Run from the project directory after fetching inputs:

```bash
docker run --rm --network none --cpus 1 --memory 1536m \
  --read-only --tmpfs /tmp:rw,size=128m --cap-drop ALL \
  --security-opt no-new-privileges --user "$(id -u):$(id -g)" \
  -v "$BIO_DISCOVERY_CACHE:/cache:ro" -v "$PWD/scripts:/scripts:ro" \
  --entrypoint Rscript \
  rocker/tidyverse@sha256:470ba4d09f861cddef6e0a6e4d07849463f1a1ab817f05357a89027932513bfc \
  --vanilla /scripts/historical_readr_check.R \
  > results/013_historical_r_parser.csv
```

Set `BIO_DISCOVERY_CACHE` to an absolute directory first. Base R `scan()` is the whitespace reference; the historical `read_table2()` was rejected as a reference after it stopped early on one input. The old fixed-width parser reports no parsing problems despite truncations. The actual authors' execution environment is unknown; exact reconstruction is a numerical fingerprint, not an environment log from their machine.

## Outputs and interpretation

- `014_full_depth_metrics.csv`: every library × DNA cutoff × RNA fraction × seed × estimator, with same-cohort comparison, retained denominator and dropout.
- `014_independent_replicates.csv`: all six biological RNA replicate pairs at both DNA cutoffs. At protoplast pairs share their DNA input.
- `014_full_estimators.csv.gz` / `014_quarter_estimators.csv.gz`: per-replicate measurements, including DNA-supported and RNA-retained barcode counts. These are alternate measurements, **not authorized replacements for PGB labels**.
- `014_frozen_cnn_metrics.csv` / `014_prediction_mapping.csv`: selected test-set scoring and explicit exclusions. Three hundred seventy leaf and 329 protoplast predictions have ambiguous target-value-to-gene matches and are excluded before coverage filters. Species-specific and pooled metrics are reported.
- `013_*`: raw-count parser audit, changed values in the 2025 RData snapshot, PGB/CNN provenance and independent 2023 attachment verification. Correct original-estimator values were already public in 2023.
- `figures/`: three standalone SVG/PDF figures. `study.html` embeds SVG images and needs no network connection; adjacent result links require keeping the package files together.
- `evidence_manifest.json`: SHA256 inventory of saved evidence/results at the recorded checkpoint. Generated artifacts can vary in insignificant floating-point formatting across dependency versions; scientific reconstruction tolerance is `1e-10`.

The thinning seeds are reproducible Monte Carlo draws, not biological replicates. No significance claims are based on treating seeds or promoters as independent plants. The study tests robustness of measurement estimates, without claiming an external biological ground truth or an improvement in native gene expression.

## Attribution and sharing

Cite the original experimental paper and PGB when using their data. PGB advertises CC BY-NC-SA 4.0; retain that provenance and its restrictions for artifacts derived from PGB. This package downloads original sequences rather than redistributing them. Input code/data retain their original authorship and terms. The research note is a draft for review; author names and submission venue have not been assigned.