ena_submission

Scripts and XML templates for submitting Oxford Nanopore raw signal data (.pod5) to the European Nucleotide Archive (ENA) from the command line.

ENA submission is a multi-step process in which metadata objects (study, samples, experiments, runs) are registered as XML documents, while the data files themselves are uploaded separately to a private staging area ("drop-box") and then linked to the metadata. This repository collects the pieces that are worth scripting — checksumming, XML generation from tables, and resumable uploads of very large archives — together with templates and instructions for the pieces that are not.

What is and is not automated

This repository deliberately stops short of being a pipeline. There is no Snakemake or Nextflow workflow, because the two most project-specific steps resist useful automation:

Step	Automated?	Why
Registering the study	No — edit a template	One short XML file per project, written once.
Describing the samples	Partly	Biological metadata has to be written by a human; a helper converts a table into XML.
Packaging .pod5 into archives	No — worked example only	Directory layouts, sample-to-barcode mappings and cluster setups differ in every project.
Checksumming and run/experiment XML	Yes	Mechanical, error-prone by hand, and must match the uploaded files exactly.
Uploading tens of GB to ENA	Yes	Needs retries, resume and non-interactive authentication.
Final metadata submission	No — one curl call	Copy, paste, read the receipt.

Treat the scripts as building blocks you invoke by hand, in order, checking the result of each step before moving to the next.

Overview of the submission process

  ENA Webin account
         |
  (1) study / project  ---------------> PRJEB.....  (study accession)
         |
  (2) samples  ---------------------->  ERS..... / SAMEA.....  (sample accessions)
         |
  (3) package raw .pod5 into one .tar.gz per sample        [project-specific]
         |
  (4) upload archives to the Webin drop-box                [Aspera or FTP]
         |
  (5) generate experiment.xml + run.xml, incl. MD5 checksums
         |
  (6) submit experiment + run  ------->  ERX..... / ERR.....  (run accessions)

Steps 1 and 2 can be done at any time, and are independent of the data files. Step 5 must happen after step 3, because the checksums have to describe the exact archives that were uploaded in step 4. Uploaded files that are never referenced by a run are purged from the drop-box after about two months, so do not leave a submission half-finished.

Before you start

An ENA Webin account

Register at Webin if your group does not already have an account. You will receive a username of the form Webin-NNNNN; this is what you authenticate with for uploads and submissions. Note the centre name registered for the account (for example Biozentrum, Unibas) — it goes into every XML document.

Software environment

conda env create -f environment.yml
conda activate ena_submission

This provides pandas (table handling), aspera-cli (the ascp high-speed uploader), lftp (the FTP fallback) and pigz (parallel gzip). curl is assumed to be present system-wide.

Credentials

Every script and command below reads the password from the ENA_PASSWORD environment variable. Set it without leaving the password in your shell history by executing the following command. You will be prompted to type-in the password in a hidden mode:

read -rsp "ENA password: " ENA_PASSWORD && export ENA_PASSWORD && echo

On a shared system such as an HPC cluster, note that anything passed on a command line is visible to other users via ps. This is why curl -u user:password is avoided below in favour of a permission-restricted .netrc file. Modify the following commands to have your actual Webin login, e.g Webin-76146. Password will be uptaken from the $ENA_password variable that was set up above. Execute commands one-by-one.

umask 077
cat > ~/.ena_netrc <<EOF
machine www.ebi.ac.uk login Webin-NNNNN password $ENA_PASSWORD
machine wwwdev.ebi.ac.uk login Webin-NNNNN password $ENA_PASSWORD
EOF

Never commit credentials. .gitignore already excludes .netrc, *.env and receipt*.xml, but the safest habit is to keep working files outside the clone entirely (see Submission Directory layout below).

Test against the development server first

ENA runs a full test instance that accepts submissions and returns real-looking receipts without creating public records. Use it for XML validating:

	Endpoint
Test	https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/
Production	https://www.ebi.ac.uk/ena/submit/drop-box/submit/

The test service is reset periodically, and accessions issued there are meaningless. Actual File uploads (step 4) go to the real drop-box; only the metadata submission has a test endpoint.

Submission Directory layout

Keep this git repository as a read-only toolbox and create a separate submission directory, so that multi-gigabyte tar.gz archives and filled-in metadata never end up in git. By the end of this instruction, the submission directory will have the following layout:

/path/to/my_project/ena_submission_wd/       <- your created submission working directory
├── project.xml                              <- copied from templates/ and edited
├── sample_metadata.tsv                      <- your sample table
├── sample.xml                               <- generated automatically from the table "sample_metadata.tsv"
├── archive_manifest.tsv                     <- sample_id -> .tar.gz paths corresponding to biological samples
├── experiment.xml, run.xml                  <- generated manually
├── receipts/                                <- every ENA receipt, kept
└── archives/                                <- the .tar.gz files that will be uploaded

Throughout this guide, $ENA_TOOLS refers to your clone of this git repository:

export ENA_TOOLS=/path/to/ena_submission

In the following steps, we assume the user is located at the submission working directory ena_submission_wd.

Step 1 — Register the study

Copy and edit the project template:

cp $ENA_TOOLS/templates/project.xml .
cp $ENA_TOOLS/templates/submission_hold.xml submission.xml

In project.xml, set alias (a short unique label of your choosing), center_name, TITLE and DESCRIPTION. The title and description become public, so write them as you would for a paper abstract.

submission.xml carries the actions for a submission rather than any content of its own. The HOLD action keeps everything private until a chosen date:

<ACTION><HOLD HoldUntilDate="2027-12-31"/></ACTION>

The hold date may be at most two years ahead and can be extended or brought forward later. Data is released automatically on that date, or earlier if you publish the accession. Not specifying a HoldUntilDate will drop it to default two months after the data of submission.

Submit.

Tip: copy the multi-line command to another text editor, modify and then copy-paste to the bash terminal

curl --netrc-file ~/.ena_netrc \
     -F "SUBMISSION=@submission.xml" \
     -F "PROJECT=@project.xml" \
     "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/"

Read the printed message. success="true" plus a PROJECT accession="PRJEB....." means you are done; the PRJEB accession is the study identifier you will cite in the paper and pass to later steps. On failure the receipt contains <ERROR> elements that name the offending element — fix and resubmit.

Step 2 — Register the biological samples

This is the step that needs real thought, and the one most often sent back by ENA curators. Each sample needs a taxon, a title and a set of attributes; which attributes are mandatory depends on the sample checklist you submit against. The default checklist requires collection date and geographic location (country and/or sea) even for cell lines, where the honest answer is not applicable — ENA accepts that exact string, and also not collected and restricted access, but not a blank value.

Describe your samples in a tab-separated file sample_metadata.tsv. The first four columns are fixed; every further column becomes a sample attribute, with the column name used verbatim as the attribute tag:

sample_id   title                                       taxon_id   scientific_name           cell line   knock-down
barcode01   HeLa cells PTBP1 siRNA knock-down, rep 1    9606       Homo sapiens              HeLa        PTBP1
barcode02   HeLa cells PTBP1 siRNA knock-down, rep 2    9606       Homo sapiens              HeLa        PTBP1
barcode03   HeLa cells siCTRL, rep 1                    9606       Homo sapiens              HeLa        none
barcode07   D. melanogaster embryos, PTBP1 RNAi, rep 1  7227       Drosophila melanogaster   embryo      PTBP1

Attribute tags must be spelled exactly as the checklist defines them, in lower case with spaces — cell line, not cell_line. Look the exact spellings up in the checklist before inventing your own; free-text tags are allowed but are far less useful to anyone reusing the data. Find taxon IDs via the ENA taxonomy service (9606 = human, 7227 = D. melanogaster).

Convert the sample_metadata.tsv file to XML sample.xml, make sure to appropriately specify -c to match center_name in the project.xml that was created above. See the paragraph below on how to specify --prefix.

python $ENA_TOOLS/scripts/sample_xml_generator.py \
    -i sample_metadata.tsv \
    -c "Biozentrum, Unibas" \
    --prefix "MyProject_" \
    --checklist ERC000011 \
    -o sample.xml

The generated alias for each sample will be <prefix><sample_id> — here MyProject_barcode01. Aliases must be unique across everything your centre has ever submitted, which is what the prefix is for. The script fills in collection date and geographic location (country and/or sea) as not applicable if you have not provided them in the sample_metadata.tsv. Open the generated file sample.xml and read it before submitting; this file is the public description of your experiment.

--checklist is optional; pass a checklist accession to validate against a specific one, or omit it to use ENA's default.

Submit to ENA:

curl --netrc-file ~/.ena_netrc \
     -F "SUBMISSION=@submission.xml" \
     -F "SAMPLE=@sample.xml" \
     "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/"

Upon success the printed message returns one ERS/SAMEA accession pair per sample.

Step 3 — Package the raw data

ENA does not accept bare .pod5 files from ONT's nanopore sequencing. Raw ONT signal data is submitted as tar archives (optionally gzip-compressed) declared with the filetype OxfordNanopore_native. One archive per registered sample is mandatory.

This step is not automated, by design. Where your .pod5 files live, how they map to biological samples, and whether you can run things on a cluster are different in every project. What the scripts downstream need from you is a two-column manifest file where each registered sample_id corresponds to one archive file:

sample_id   archive_path
barcode01   /path/to/archives/barcode01_pod5.tar.gz
barcode02   /path/to/archives/barcode02_pod5.tar.gz

For example, one such archive could be generated with the following command, under assumtion that all .pod5 files constituting barcode01 sample have been present in the subdirectory /path/to/pod5_pass/barcode01/:

tar -czf archives/barcode01_pod5.tar.gz /path/to/pod5_pass/barcode01/*.pod5

Compression of .pod5 gains little, since the .pod5 format is already compressed internally — however, currently .tar.gz is explicitely required for submission.

If your .pod5 files are already listed in a separately prepared sample_id/pod5 table (see the example at examples/pod5_manifest.tsv), the bundled helper does the grouping, tarring and manifest writing for you:

python $ENA_TOOLS/scripts/prepare_archives.py \
    -i pod5_manifest.tsv \
    -o archives \
    -m archive_manifest.tsv \
    --name-prefix "MyProject_" \
    --base-dir /path/to/run \
    --threads 8

To see the explanation of all options, run:

python $ENA_TOOLS/scripts/prepare_archives.py --help

Two flags are worth understanding. --name-prefix prepends a string to every archive name: the drop-box is per account and is reused across projects, so a bare barcode01_pod5.tar.gz can collide with something another submission of yours uploaded last year, and the newer file silently wins. A project-specific prefix avoids that. --base-dir stores paths inside the archive relative to that directory rather than as the absolute paths from the manifest, which keeps the extracted tree shallow.

For a worked example that scales to many samples on a cluster, examples/slurm/ wraps the same script in a SLURM array job, one task per sample, and explains how to merge the per-sample manifests afterwards. It is an example to adapt, not a supported entry point — read it before running it.

Step 4 — Upload the archives

Uploads go to your private Webin drop-box at webin.ebi.ac.uk (final functional drop-box, not a test one). Nothing is public and nothing is linked to your metadata yet; files simply sit there under their basename, waiting to be referenced by a run in step 6. Only the basename matters, so archive names must be unique.

Aspera (recommended for large data)

Aspera's FASP protocol is UDP-based and sustains far higher throughput than FTP over long distances, which matters when a submission is tens or hundreds of GB.

python $ENA_TOOLS/scripts/aspera_uploader.py \
    -u Webin-NNNNN \
    -i archive_manifest.tsv \
    -s 60M \
    --min-speed 10M \
    --log_dir logs/aspera

-s caps the transfer rate; be a good citizen on shared infrastructure and do not exceed what your link can actually carry. The script resolves the bundled ascp binary through ascli, so nothing needs to be on your PATH, and it passes your password to ascp non-interactively via ASPERA_SCP_PASS. Transfers resume rather than restart, so re-running after an interruption is cheap and safe.

Each file is attempted --retries times (3 by default) with a growing pause between attempts, and ascp resumes rather than restarts, so a retry picks up roughly where it left off. A file that still fails is logged and the run continues, with every failure listed again at the end and a non-zero exit status — one flaky transfer in a twelve-hour upload does not cost you the other eleven. Add --dry-run to print the ascp commands without transferring anything, which is the quickest way to confirm your manifest and credentials are sane before committing to a long upload.

ascp itself offers no retry or timeout settings, which is why they live in the wrapper. What it does offer is a transfer policy, and this matters for long uploads: under --policy fixed it transmits at -s regardless of what the network is doing, which is the most likely way to provoke a timeout. The default here is fair, which backs off when the link is congested, and --min-speed stops it backing off to nothing.

FTP (fallback)

Use this when Aspera's UDP ports are blocked by a firewall:

python $ENA_TOOLS/scripts/ftp_uploader.py -u Webin-NNNNN -i archive_manifest.tsv

This one first lists the drop-box and uploads only what is missing, then verifies the listing afterwards and reports anything still absent. FTP connections to ENA drop frequently on large files — just run it again, as many times as needed, until it reports everything present.

Confirming what arrived

You can always inspect the drop-box directly:

lftp -u Webin-NNNNN,"$ENA_PASSWORD" webin.ebi.ac.uk -e "cls -l; bye"

Compare sizes, not just names, against your local files. ascp writes partial data under the final filename unless told otherwise, so an interrupted transfer leaves a truncated file that a listing shows as present. Passing --partial-suffix .partial to aspera_uploader.py (see above) makes in-progress transfers land under a temporary name that is renamed only on completion, so that a correctly-named file does mean a complete one; that is the better habit, but it has not been verified against ENA's endpoint, so if the completion rename is refused, leave it unset and compare sizes instead.

A truncated upload that slips through is not silently accepted — it surfaces at step 6 as a checksum mismatch — but it is much cheaper to catch here.

Resuming a part-finished upload

Uploads of this size rarely all succeed on the first pass. Re-running with the same manifest is safe, but it is easier to reason about a manifest containing only what is left, with no question of whether a finished 60 GiB file is being re-checked or re-sent:

awk -F'\t' 'NR==1 || $1 >= "barcode06"' archive_manifest.tsv > remaining_manifest.tsv
python $ENA_TOOLS/scripts/aspera_uploader.py -u Webin-NNNNN -i remaining_manifest.tsv

Delete any truncated file from the drop-box first, or simply let the re-upload overwrite it.

Step 5 — Generate experiment and run XML

An experiment records how a library was made and on what instrument; a run records which files resulted from it. One of each per sample, and the run carries an MD5 checksum for every file so that ENA can verify the transfer.

python $ENA_TOOLS/scripts/xml_generator.py \
    -i archive_manifest.tsv \
    -p PRJEB123456 \
    -c "Biozentrum, Unibas" \
    --prefix "MyProject_" \
    --instrument PromethION \
    --protocol "SQK-PCB114" \
    --design-description "cDNA sequencing for polyA tail length analysis" \
    --library-strategy RNA-Seq \
    --library-source TRANSCRIPTOMIC \
    --library-selection PolyA \
    -e experiment.xml \
    -r run.xml

--prefix must be the same value used in step 2, since the experiment refers to each sample by the alias <prefix><sample_id>. A mismatch is the most common cause of a rejected submission at step 6.

The library and instrument arguments are controlled vocabularies, not free text; the accepted values are listed in the ENA metadata model documentation. PromethION, GridION, MinION and MinION Mk1C are the usual ONT instrument models.

Hashing is the slow part: the script reads every archive end to end, so expect this to take a while on a large submission and run it somewhere it will not be killed. Checksums are computed from your local files, which is exactly the point — if a file was corrupted or truncated in transit, ENA's checksum of the uploaded copy will not match and the submission is rejected rather than silently accepting bad data.

Step 6 — Submit experiments and runs

This submission should not carry a HOLD action; the hold you set on the study in step 1 already covers everything attached to it. Use the plain ADD template:

cp $ENA_TOOLS/templates/submission_add.xml final_submission.xml

curl --netrc-file ~/.ena_netrc \
     -F "SUBMISSION=@final_submission.xml" \
     -F "EXPERIMENT=@experiment.xml" \
     -F "RUN=@run.xml" \
     "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" \
     -o receipts/run_receipt.xml

A successful receipt contains an ERX accession per experiment and an ERR accession per run. At this point the submission is complete: the files are attached to the archive, and everything stays private until the hold date.

Once you have rehearsed the whole sequence against wwwdev, repeat steps 1, 2 and 6 against https://www.ebi.ac.uk/ena/submit/drop-box/submit/ for the real submission. The files uploaded in step 4 are already in the right place and do not need to be transferred again.

Troubleshooting

ascp prompts for a password on every file, despite ASPERA_SCP_PASS being set. This happens if ascp is invoked with -i <key>. ENA's Aspera endpoint authenticates by password only, and supplying an SSH private key — for instance the bypass key that ascli conf ascp info reports — makes key authentication fail first and drops the session into an interactive prompt, at which point the environment variable is ignored. Do not pass -i for ENA transfers. (Nothing like sshpass is needed either; ascp reads ASPERA_SCP_PASS natively once the key is out of the way.)

Error 35 [Data transfer timeout] partway through a large file. A transient symptom of UDP congestion or packet loss, not a problem with the file: the FASP session stopped receiving acknowledgements for longer than its timeout. It typically shows up hours into an unattended upload, often after the observed throughput has been decaying for a while. Re-run — the retry logic and ascp's resume will usually carry it through on the next attempt. If it keeps happening, lower -s to something the link sustains comfortably rather than its peak, keep --policy fair so congestion slows the transfer instead of killing it, and consider uploading fewer files per invocation so that a session lasts hours rather than most of a day. Institutional firewalls that rate-limit or drop long-lived high-rate UDP flows are a common underlying cause, and worth asking your network admins about if the failures cluster at particular times of day.

Submission fails with an alias or refname error. The --prefix passed to xml_generator.py must match the one used for sample_xml_generator.py. Confirm that the refname in experiment.xml matches an alias in sample.xml exactly.

Checksum mismatch at step 6. The uploaded file differs from the local one that was hashed — usually a truncated transfer. Re-upload the affected archive and resubmit. Do not regenerate the checksums to match what you uploaded.

File not found at step 6. The run references a basename that is not in the drop-box. List the drop-box and check for typos, or for files purged after sitting unreferenced for two months.

Invalid sample attributes. Compare your tags against the checklist. Tags are case- and space-sensitive, and mandatory fields cannot be empty — use not applicable rather than leaving them out.

A submitted object needs changing. Use a MODIFY action in submission.xml in place of ADD, supplying the full, corrected XML for the object. Objects cannot be deleted once submitted; contact ENA support if something is badly wrong.

Repository contents

scripts/
  prepare_archives.py        group a pod5 manifest by sample and tar each group (example)
  sample_xml_generator.py    sample metadata TSV -> sample.xml
  xml_generator.py           archive manifest -> experiment.xml + run.xml, with MD5s
  aspera_uploader.py         resumable high-speed upload via ascp
  ftp_uploader.py            differential upload via lftp, skips what is already there
templates/
  project.xml                study registration
  sample.xml                 single annotated SAMPLE block, for writing XML by hand
  submission_hold.xml        ADD + HOLD, for study and sample submission
  submission_add.xml         ADD only, for experiment and run submission
examples/
  pod5_manifest.tsv          sample_id / pod5
  archive_manifest.tsv       sample_id / archive_path
  sample_metadata.tsv        sample_id / title / taxon_id / scientific_name / attributes...
  slurm/                     SLURM array example for packaging many samples

All scripts take -h.

Manifest formats

Two tab-separated formats are used throughout, both keyed on an arbitrary sample_id that only has to be consistent within your project:

File	Columns	One row per
pod5 manifest	sample_id, pod5	.pod5 file
archive manifest	sample_id, archive_path	.tar.gz archive

A sample may own several archives; each becomes an additional <FILE> within that sample's single run.

Further reading

• ENA submission documentation
• Submitting raw reads
• Sample checklists
• Webin submission portal

License

MIT