Skip to content

Create generic pipeline init script #1

Open
Open
Create generic pipeline init script#1
Assignees
ptrebert

Description

@ptrebert
ptrebert
opened on Apr 23, 2018

A "as simple as possible" init script usable for each DEEP pipeline that identifies itself (WGBS, ChIP-seq etc.) and loads - if necessary - required reference data/configurations from the reference repository and checks/validates the sample annotation table (entries in this table should follow the naming scheme of EGA/ENA). The sample annotation table is a tsv file (not XML, not XLS(X), not anything but tab-separated text...)

Activity

  1. ptrebert
    self-assigned this
    on Apr 23, 2018

  2. ptrebert commented on Apr 25, 2018

    @ptrebert
    ptrebert
    on Apr 25, 2018
    MemberAuthor

    Some thoughts about the standardized sample annotation table that will also be read by the pipeline init script:

    • as discussed, as far as possible, rules derived from EGA/SRA metadata model should be used consistently (except for capitalization; all lowercase is easier)

    • File level: filepath and filename to specify a file location; filepath plus filename gives fullpath

    • Experiment level: use info from SRA 1.5 experiment

      • Example: LibraryStrategy - Bisulfite-seq, ChIP-seq, ATAC-seq, DNase-Hypersensitivity etc
      • @karl616 could you find out how NOMe is handled? I don't see an entry for that...
      • Example: LibraryLayout - single/paired (+ nominal_length => that is insert size in our case)

    • Read level: use info from SRA 1.5 commons

      • Example: read_type - forward, reverse

    The sample annotation could contain an infinite number of fields (columns), but there has to be a required subset per pipeline type (the combination of library strategy and library layout should be sufficient to define that). Given that the init script should be shipped with every pipeline, this config should in turn be part of the reference repository - so I imagine the startup as follows:

    1. clone pipeline repository, say, ChIP-seq
    2. run pipeline init with parameter sample_table and working_directory
    3. the init script shipped with each repository is dumb: it only knows its type, say, ChIP-seq, and where to find more configuration information (in the reference repository)
    4. it downloads the additional configuration (if not already available locally from a previous init run), and then starts validating the sample annotation
    5. last step is to check for reference files, download/prepare if necessary

    Any additional thoughts?

  3. karl616 commented on Apr 25, 2018

    @karl616
    karl616
    on Apr 25, 2018
    Contributor

    I'm not sure about the NOMe data, but we have submitted some to EGA already. I find a way to access the metadata though. I obviously think it deserves its own category, but worst case we call it Bisulfite-seq. And we have to stratify by some additional field. With the different WGBS strategies (tWGBS, TruSeq, EpiGnome, PBAT etc), this might be the case anyhow.

    I posted a request for clarification:
    enasequence/webin-xml#3

    I would consider the inclusion of flowcell, lane and perhaps index as a way to track batch effects when sequencing. I usually include this as read group information in the alignment file. Adapter sequence could also be relevant. To a degree this is probably pipeline-dependent though.

  4. ptrebert commented on Apr 25, 2018

    @ptrebert
    ptrebert
    on Apr 25, 2018
    MemberAuthor

    There is a field LIBRARY_CONSTRUCTION_PROTOCOL as part of the library description, that sounds like a potential candidate to capture tWGBS, EpiGnome etc. - only downside that this is a free form text field, so may cause some trouble.
    Good point about the batch effects - did you find an annotation for flowcell, lane etc in one of the schema files?

  5. karl616 commented on Apr 25, 2018

    @karl616
    karl616
    on Apr 25, 2018
    Contributor

    Yes, that could work... Perhaps we build in our own check?
    I cannot find flowcell or lane explicitly, but as I understand it this is captured with the run file. That keeps the files separated and that would be sufficient... Not sure how to handle naming, but there is surely a way...

  6. ptrebert commented on Apr 27, 2018

    @ptrebert
    ptrebert
    on Apr 27, 2018
    MemberAuthor

    Yeah, sure, the init script accepts a couple of fix values for the lib construction protocol and that's it, no problem.

    Do you already have an example sample annotation for one of your pipelines (WGBS or RNA) that is more or less complete that we could use as a starting point?

  7. karl616 commented on Apr 27, 2018

    @karl616
    karl616
    on Apr 27, 2018 via email
    Contributor

  8. ptrebert commented on Apr 27, 2018

    @ptrebert
    ptrebert
    on Apr 27, 2018
    MemberAuthor

    I mean the "EGA"-style sample annotation (TSV table) plus the necessary references (also TSV? Would have the benefit that you could update this online in github...) so that we have one complete set of starting values for one pipeline.

  9. karl616 commented on May 4, 2018

    @karl616
    karl616
    on May 4, 2018
    Contributor

    I added a draft... As I understand it, this is what would be required for ENA submission (only sequence information so far).

    This being a ChIP-seq sample, I added an extra column for antibody. This is not a general thing, and should probably be specific to the chip-seq pipeline. Perhaps there can be a simple specification of extra information needed for each pipeline.

    ENA/EGA requires insert size for fastq file submission... hence, I added a general number. I'm still of the conviction that we should extract this programatically from the alignment.

  10. ptrebert commented on May 6, 2018

    @ptrebert
    ptrebert
    on May 6, 2018
    MemberAuthor

    Thanks. I agree concerning the insert size/nominal length issue. The way I see it, we would have to combine the initial sample annotation with the final analysis metadata to automatically generate a EGA/ENA submission sheet. If we stick to the naming/terms extracted from the SRA schema files, this should be straightforward, though.

  11. karl616 commented on May 7, 2018

    @karl616
    karl616
    on May 7, 2018
    Contributor

    I agree as well. I saw the vocabulary file.
    Would it be correct to map sample_label to sample_name in SRA.sample?
    And according to the definition of filename it can include a relative path. Maybe that could replace filepath?
    The only thing I'm missing is a reference to the reference. We could use taxon_id, but maybe it is better to explicitly state the target (mm10, hs37, hg19, hs38 etc).
    Looking over the bisulfite pipeline, this set of parameters should be sufficient there as well.

  12. ptrebert
    added a commit that references this issue on May 28, 2018

    changed sample_label to sample_name

    0ce4f72
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions