diff --git a/docs/api/index.md b/docs/api/index.md index f268bb2..1b4086e 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -25,6 +25,9 @@ signature, parameters and returns: `Manipulation`, `Input / output`, …). - **Python API (raven-toolbox)**: one page per package, mirroring the same categories. +- **[MATLAB vs Python](../matlab-vs-python.md)**: a third page pairing every + function that exists in both, generated at build time from both toolboxes' + sources. Every page opens with a *Functions* table you can scan, followed by the full help for each function. @@ -60,4 +63,5 @@ Use the navigation to browse either tree, or start from the matlab/index python/index +../matlab-vs-python ``` diff --git a/docs/data/README.md b/docs/data/README.md index f47aafe..396f01f 100644 --- a/docs/data/README.md +++ b/docs/data/README.md @@ -52,7 +52,7 @@ numbers unreproducible for a reader who takes the model from RAVEN instead. ## Proteomes, for the homology page -`hanpo.faa` and `sce-template.faa` back [18. Reconstruction from +`hanpo.faa` and `sce-template.faa` back [12. Reconstruction from homology](../guide/homology.md). | file | what it is | diff --git a/docs/guide/biomass.md b/docs/guide/biomass.md index 732672e..f7c686b 100644 --- a/docs/guide/biomass.md +++ b/docs/guide/biomass.md @@ -1,4 +1,4 @@ -# 22. Biomass composition and annotation +# 11. Biomass composition and annotation The biomass pseudoreaction is where a model states what a cell is made of, and it sets the units of every growth rate the model reports. This page is about reading @@ -113,7 +113,7 @@ polymerised, two hydrogens or a water, which is what makes a protein weigh less than the sum of its amino acids. `grams` takes the coefficients as already being in g/gDW, which is how the lipid backbone is written. -## 22.1 What is the cell made of? +## 11.1 What is the cell made of? ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -174,7 +174,7 @@ same mass, so counting both would double it; the [lipid curation](../protocol/lipid-curation.md) step in the protocol explains the representation. -## 22.2 Change one component +## 11.2 Change one component Measured a different protein content? Set it, and specify which other component absorbs the change. A biomass that no longer sums to 1 is worse than one with the @@ -219,7 +219,7 @@ protein 0.5000, carbohydrate 0.3384, total 1.0000 `scale_biomass` edits the model **in place** and returns `None`, while `scaleBiomassFraction` returns a new struct and leaves its input alone. That is -the same split as in [16. Combining and simplifying](combining.md), and it is why +the same split as in [20. Combining and simplifying](combining.md), and it is why the Python tab copies the model first. `balanceOut` names the component that absorbs the change, so the total stays @@ -231,7 +231,7 @@ a single pseudoreaction rather than a named component, and rebalance the protons afterwards using the `proton_met` from the configuration. That is what keeps the reaction charge balanced when its coefficients move. -## 22.3 Annotation: SBO terms +## 11.3 Annotation: SBO terms An SBO term says what a reaction or metabolite *is*: a transport reaction, an exchange, a metabolite rather than a pseudo-species. Solvers ignore them; @@ -275,7 +275,7 @@ only by compartment is a transport, and so on. Nothing is inferred that the stoichiometry does not already support, so running it on a model with mislabelled compartments propagates that mistake rather than catching it. -## 22.4 Annotation: cross-references and thermodynamics +## 11.4 Annotation: cross-references and thermodynamics Database cross-references travel in MIRIAM form: a namespace and an identifier, such as `chebi/CHEBI:15589`. RAVEN keeps them in `metMiriams` and `rxnMiriams` @@ -312,5 +312,5 @@ separately from the model, and re-applied after the model changes. - [4. Simulating growth with FBA](fba.md), where the growth rate these units belong to comes from. - [8. Editing an existing model](editing.md), changing a pseudoreaction by hand. -- [Worked protocol: biomass composition](../protocol/biomass.md), building the - pseudoreactions from measurements for a new organism. +- [GEM reconstruction protocol: biomass composition](../protocol/biomass.md), + building the pseudoreactions from measurements for a new organism. diff --git a/docs/guide/combining.md b/docs/guide/combining.md index 0d27c60..91a03e1 100644 --- a/docs/guide/combining.md +++ b/docs/guide/combining.md @@ -1,4 +1,4 @@ -# 16. Combining and simplifying models +# 20. Combining and simplifying models This page covers two operations that look like simple bookkeeping but carry real risk. **Merging** puts models @@ -62,7 +62,7 @@ print(f"{len(model.reactions)} rxns, {len(model.metabolites)} mets, " ::: :::: -## 16.1 Merging +## 20.1 Merging ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -122,7 +122,7 @@ halves that happen to live in the same struct. Merging is a claim that the naming is consistent; check it before you make it, not after. ::: -## 16.2 Merging does not de-duplicate +## 20.2 Merging does not de-duplicate Every reaction from both models survived, so the merged model now describes each conversion twice. Collapsing those is a separate step. @@ -179,7 +179,7 @@ alone. The Python side is not uniform: in Python. Copy the model first to keep the original. ::: -## 16.3 Dropping what cannot carry flux +## 20.3 Dropping what cannot carry flux `smallYeast` ships with its medium shut, so almost nothing in it can carry flux at all, the state [9. Quality control](quality-control.md) measures. That makes @@ -226,7 +226,7 @@ medium, not of the organism. Open the conditions you intend to simulate **before** simplifying, or you will delete the pathways you were about to study; see [5. Growth media and conditions](media.md). -## 16.4 Collapsing compartments +## 20.4 Collapsing compartments Sometimes the compartments are the problem: a draft with unreliable localisation, or a comparison against a model that has none. @@ -313,7 +313,7 @@ Flatten a copy, for a specific question, and keep the original. ## See also - [9. Quality control](quality-control.md), deciding what *should* be removed. -- [13. Gap-filling](gap-filling.md), the opposite operation, and the usual +- [17. Gap-filling](gap-filling.md), the opposite operation, and the usual reason for having a template model to merge from. -- [10. Context-specific models](init.md), cutting a model down by evidence +- [14. Context-specific models](init.md), cutting a model down by evidence rather than by connectivity. diff --git a/docs/guide/comparing.md b/docs/guide/comparing.md index 685f333..1630479 100644 --- a/docs/guide/comparing.md +++ b/docs/guide/comparing.md @@ -1,4 +1,4 @@ -# 17. Comparing models +# 21. Comparing models Curation produces several versions of one model: a draft and a curated version, the model before and after gap-filling, a local build and a @@ -55,7 +55,7 @@ good 53 rxns, bad 54 rxns ::: :::: -## 17.1 What exactly is different? +## 21.1 What exactly is different? `diffModels` matches by identifier and compares what it finds: stoichiometry, bounds, objective coefficients, gene rules, formulas, charges. It answers a yes @@ -107,7 +107,7 @@ This is the comparison to put in a test. `DiffReport` is falsy when the models differ, so `assert diff_models(before, after)` is a working regression test for a curation script, and in MATLAB, `report.equal` does the same job. -## 17.2 How alike are they? +## 21.2 How alike are they? The other question is coarser: across the whole reaction set, how much do two models overlap? `compare_models` builds a presence matrix, one row per @@ -159,13 +159,13 @@ similarity: 0.945 :::{warning} A high similarity does not mean the models agree These two models are **0.945** alike on the reaction set, and one of them is -broken. The differences 17.1 lists are a doubled stoichiometric coefficient +broken. The differences 21.1 lists are a doubled stoichiometric coefficient and a reaction made irreversible, changes that alter what the model *predicts* while barely moving a set-overlap score. Similarity is for grouping models, not for validating one. ::: -## 17.3 Compare what they do, not what they contain +## 21.3 Compare what they do, not what they contain Two models with the same reactions can behave differently, and two models with different reactions can behave identically. The comparison that settles it is of @@ -218,7 +218,7 @@ RAVEN can also run this comparison for you. `compareMultipleModels` takes `funcCompare` and a `taskFile`, and then compares the models by which metabolic tasks each one passes rather than by which reactions each one has. That is the same shift from contents to behaviour made by hand above, and -[12. Metabolic tasks](tasks.md) is where the task file comes from. +[15. Metabolic tasks](tasks.md) is where the task file comes from. `groupVector` labels the models, by tissue or by condition, so the similarity matrix is reported per group. @@ -232,7 +232,7 @@ answer the same question, but you index them in opposite directions. - **Comparing on identifiers alone.** Both functions match by id. Two models from different databases share few ids and will look unrelated even when they describe the same metabolism; see - [16. Combining and simplifying](combining.md), where merging matches on + [20. Combining and simplifying](combining.md), where merging matches on names instead. - **Reading similarity as quality.** It measures overlap, not correctness. - **Forgetting the medium.** A flux comparison compares conditions as much as @@ -247,7 +247,7 @@ answer the same question, but you index them in opposite directions. ## See also - [9. Quality control](quality-control.md), checking one model rather than two. -- [16. Combining and simplifying](combining.md), putting models together, and +- [20. Combining and simplifying](combining.md), putting models together, and what has to line up first. - [4. Simulating growth with FBA](fba.md), the simulation a flux comparison rests on. diff --git a/docs/guide/curation.md b/docs/guide/curation.md index 50123d7..c676393 100644 --- a/docs/guide/curation.md +++ b/docs/guide/curation.md @@ -1,4 +1,4 @@ -# 21. Table-driven curation +# 10. Table-driven curation Curation done one function call at a time is hard to review and harder to repeat. Both toolboxes can instead take the curation as **tables**: one row per @@ -44,7 +44,7 @@ metNames comps formula charge metNotes 3-phospho-D-glycerate c C3H7O7P -3 charge assigned at pH 7.3 ``` -## 21.1 Apply a table +## 10.1 Apply a table ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -137,7 +137,7 @@ the table was applied in full, and note that this is a divergence between the two toolboxes rather than a difference in the tables. ::: -## 21.2 Adding rather than updating +## 10.2 Adding rather than updating A row whose name matches nothing in the model adds a new entity, and the identifier is minted rather than taken from the file. Both toolboxes find the diff --git a/docs/guide/deletions.md b/docs/guide/deletions.md index d35c445..b95c7d7 100644 --- a/docs/guide/deletions.md +++ b/docs/guide/deletions.md @@ -1,4 +1,4 @@ -# 11. Deletions and essentiality +# 16. Deletions and essentiality Which genes can the organism lose and still grow? Knock each one out, re-solve, and compare. The same machinery answers "which reactions are essential", "what @@ -56,7 +56,7 @@ wild type: 0.1222 /h ::: :::: -## 11.1 Knock out one gene +## 16.1 Knock out one gene The question underneath every deletion study: with this gene gone, can the model still reach its objective? A gene knockout is not a reaction knockout: the GPR @@ -102,7 +102,7 @@ block. ::: :::: -## 11.2 Knock out every gene +## 16.2 Knock out every gene ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -149,7 +149,7 @@ because the same function shape serves the double-deletion case. ::: :::: -## 11.3 Essential reactions +## 16.3 Essential reactions Reaction essentiality asks the same question one level down. The two functions below look equivalent but ask different questions, which is why their answers @@ -198,11 +198,11 @@ tab returns none. For essentiality with respect to a *task* rather than the objective, raven-toolbox has `find_task_essential_reactions`; see -[12. Metabolic tasks](tasks.md). +[15. Metabolic tasks](tasks.md). ::: :::: -## 11.4 Two at a time +## 16.4 Two at a time Double deletions find the redundancy single deletions miss: two genes that each look dispensable but cannot both be deleted. The number of pairs to test is @@ -240,7 +240,7 @@ lowest growth: 0.0000 /h ::: :::: -## 11.5 A knockout's fluxes, not just its growth rate +## 16.5 A knockout's fluxes, not just its growth rate FBA assumes the knockout re-optimises perfectly, which a cell that just lost a gene does not do. MOMA instead looks for the flux distribution closest to the @@ -259,7 +259,7 @@ or from the COBRA Toolbox, which keeps its own. What RAVEN does offer for the same *question* (what changed in the mutant, rather than by how much growth fell) is `compareFluxes` on two flux -vectors from the same model, in [17. Comparing models](comparing.md). +vectors from the same model, in [21. Comparing models](comparing.md). ::: :::{tab-item} 🐍 Python :sync: python @@ -298,5 +298,5 @@ MOMA growth: 0.0000 /h ## See also - [4. Simulating growth with FBA](fba.md), the solve underneath all of this. -- [12. Metabolic tasks](tasks.md), essentiality with respect to a task. +- [15. Metabolic tasks](tasks.md), essentiality with respect to a task. - [9. Quality control](quality-control.md), before trusting any of it. diff --git a/docs/guide/editing.md b/docs/guide/editing.md index 2e9ef71..02ffa74 100644 --- a/docs/guide/editing.md +++ b/docs/guide/editing.md @@ -149,7 +149,7 @@ YBR196C or YLR354C model, so the results have to be assigned back into the struct to persist. Disjunctive normal form is a flat `or` of `and` groups, one group per alternative complex, which is the form the scoring in -[10. Context-specific models](init.md) expects. +[14. Context-specific models](init.md) expects. ::: :::{tab-item} 🐍 Python :sync: python @@ -297,7 +297,7 @@ PGI rule: 'YLR354C' `removeGenes` rewrites each affected GPR without the removed gene. Its `removeBlockedRxns` flag, off by default, additionally deletes reactions that lose their last catalyst, which turns a gene deletion into a reaction deletion; -[11. Deletions and essentiality](deletions.md) covers when that is the +[16. Deletions and essentiality](deletions.md) covers when that is the question being asked. `standardizeRules`, on by default, normalises the rewritten rules. diff --git a/docs/guide/engineering.md b/docs/guide/engineering.md index 9b01c87..0b21cbf 100644 --- a/docs/guide/engineering.md +++ b/docs/guide/engineering.md @@ -1,4 +1,4 @@ -# 20. Engineering targets +# 22. Engineering targets Two questions underlie most strain-design work. Which reactions would have to change for the cell to make more of something, and which metabolites are most @@ -53,7 +53,7 @@ growth 0.1222 /h ::: :::: -## 20.1 Which reactions track the product? +## 22.1 Which reactions track the product? FSEOF forces the product exchange to carry progressively more flux, maximising growth at each step, and records what every other reaction does. A reaction whose @@ -131,7 +131,7 @@ zero. ::: :::: -## 20.2 Reading the scan +## 22.2 Reading the scan A target's reliability depends on the trend behind it. The scan matrix is what distinguishes a reaction that rises steadily from one that jumps once and stops. @@ -180,7 +180,7 @@ sign says which way to push, and its magnitude says how hard. Neither says the change is achievable: FSEOF works on the stoichiometry and the bounds, and knows nothing about regulation, enzyme capacity or toxicity. -## 20.3 Reporter metabolites +## 22.3 Reporter metabolites The other direction. Given a differential-expression result, which metabolites have the most transcriptionally-responsive neighbourhood? The algorithm scores @@ -297,7 +297,7 @@ above. ## See also - [4. Simulating growth with FBA](fba.md), the solve each FSEOF step performs. -- [15. Random sampling](sampling.md), the other way to ask what a network can do +- [19. Random sampling](sampling.md), the other way to ask what a network can do rather than what one optimum says. -- [11. Deletions and essentiality](deletions.md), knockouts rather than +- [16. Deletions and essentiality](deletions.md), knockouts rather than amplification. diff --git a/docs/guide/fva.md b/docs/guide/fva.md index d6e5b1f..de5218a 100644 --- a/docs/guide/fva.md +++ b/docs/guide/fva.md @@ -1,4 +1,4 @@ -# 14. Flux variability and alternative optima +# 18. Flux variability and alternative optima An FBA solution is one point in a space of equally optimal answers. Flux variability asks the more accurate question: **given the objective, how much can @@ -56,7 +56,7 @@ growth: 0.1222 /h ::: :::: -## 14.1 The range of every reaction +## 18.1 The range of every reaction With no further constraint, this asks how far each flux can move anywhere in the feasible space; the objective is free. @@ -104,7 +104,7 @@ asks a different question; see the next section. ::: :::: -## 14.2 The range *at* the optimum +## 18.2 The range *at* the optimum The useful question for interpreting a result: holding growth at its maximum (or at 90 % of it), which fluxes are still free to move? @@ -147,11 +147,11 @@ no choice about. Those are the predictions the model determines; a flux with a wide range at the optimum is an artefact of which vertex the solver happened to land on. -## 14.3 A wide range is not always a real one +## 18.3 A wide range is not always a real one A reaction can show a wide range purely because it is part of a thermodynamically infeasible cycle, flux going round a loop with no net driving force. The widest -span in 14.1 was 1000, the model's default bound. That span is such a cycle. +span in 18.1 was 1000, the model's default bound. That span is such a cycle. ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -220,7 +220,7 @@ means `"cycleFreeFlux"`, but it is deprecated; name the algorithm. `find_good_reactions` uses the same idea to decide which reactions are worth sampling over, keeping a reaction only if its **loopless** range is non-trivial. -## 14.4 One representative solution +## 18.4 One representative solution When a single flux distribution is needed (for a figure, or to compare two conditions), take a parsimonious one rather than whatever the solver returns @@ -262,7 +262,7 @@ pFBA 20.9 ``` All three agree here: this particular optimum happens to carry no loop flux, -even though 14.3 showed the cycle is there. Nothing guaranteed that: the +even though 18.3 showed the cycle is there. Nothing guaranteed that: the solver could as easily have returned a vertex with 1000 units going round `FRDS2` and `SDH`, which is exactly the failure `loopless_solution` exists to prevent. @@ -277,7 +277,7 @@ prevent. - **Reporting a flux with a wide range.** If the range at the optimum is wide, the number in your table is one of many equally good answers. - **Mistaking a loop for capacity.** Wide ranges on internal cycles are a - property of the stoichiometry, not of the organism; see 14.3. + property of the stoichiometry, not of the organism; see 18.3. - **Loopless FVA is a MILP.** `loopless="fastSNP"` adds binary variables, so it is far slower than plain FVA and wants a good solver on anything larger than a toy model. @@ -289,7 +289,7 @@ prevent. ## See also - [4. Simulating growth with FBA](fba.md), the single solve this qualifies. -- [11. Deletions and essentiality](deletions.md), the other way to ask what the +- [16. Deletions and essentiality](deletions.md), the other way to ask what the model depends on. - [9. Quality control](quality-control.md), blocked reactions, the degenerate case of a zero range. diff --git a/docs/guide/gap-filling.md b/docs/guide/gap-filling.md index 791b771..53ccec7 100644 --- a/docs/guide/gap-filling.md +++ b/docs/guide/gap-filling.md @@ -1,4 +1,4 @@ -# 13. Gap-filling +# 17. Gap-filling A draft model always has holes: reactions that cannot carry flux because something upstream is missing, metabolites nothing produces, a biomass component @@ -14,7 +14,7 @@ holes close, and the judgement is in deciding which holes are worth closing. | `gapFillFastLP` | `fill_gaps_fast_lp` | the LP formulation (fastGapFill / swiftGapFill) | | `gapFillMILP` | `fill_gaps_kumar_milp` | the MILP formulation, when the LP is not enough | | `gapFillTopological` | `analyse_topology` | connectivity, without solving an LP | -| `fitTasks` | `fill_tasks` | fill until a task passes; see [12. Metabolic tasks](tasks.md) | +| `fitTasks` | `fill_tasks` | fill until a task passes; see [15. Metabolic tasks](tasks.md) | ## Setup @@ -68,7 +68,7 @@ a model in that state and you are asking the wrong question entirely. ::: :::: -## 13.1 Find the holes before filling them +## 17.1 Find the holes before filling them Gap-filling without looking first is how a model acquires reactions nobody can justify. Start from what cannot carry flux. @@ -104,7 +104,7 @@ print(sorted(blocked)[:5]) ::: :::: -## 13.2 Connectivity gap-filling +## 17.2 Connectivity gap-filling The question here is structural: which template reactions, added to the draft, would let a blocked reaction carry flux at all? No objective, no growth, just @@ -161,7 +161,7 @@ the harder question and usually adds more reactions. ::: :::: -## 13.3 The LP formulation +## 17.3 The LP formulation Note the argument: `fillGaps` takes a **cell array** of template models, while `gapFillFastLP` and `gapFillMILP` take a **single struct**. Passing a cell to @@ -218,7 +218,7 @@ the gap between the counts closes. ::: :::: -## 13.4 The MILP formulation +## 17.4 The MILP formulation When the LP relaxation adds too much, or you want the provably smallest set, the mixed-integer formulation is the alternative. It needs a MILP solver, and it @@ -272,7 +272,7 @@ fill_gaps_kumar_milp: reversed 0 reaction(s), added 0 template reaction(s). ::: :::: -## 13.5 Filling towards a task, not a hole +## 17.5 Filling towards a task, not a hole Connectivity gap-filling asks "can this reaction carry flux?". The other question ("can the model still do *this*?") is answered by filling against a task list, @@ -301,8 +301,8 @@ tasks = parseTaskList('tasks.txt'); from raven_toolbox.init import fill_tasks ``` -See [12. Metabolic tasks](tasks.md) for the task list itself, and -[10. Context-specific models](init.md) for `fill_tasks` in its usual role, +See [15. Metabolic tasks](tasks.md) for the task list itself, and +[14. Context-specific models](init.md) for `fill_tasks` in its usual role, repairing a model that ftINIT has just cut down. `fitTasks` fills one task at a time rather than all of them at once, so @@ -337,6 +337,6 @@ already contains the draft, which is the form ftINIT passes. - [9. Quality control](quality-control.md), finding the holes, and deciding which matter. -- [12. Metabolic tasks](tasks.md), the other way to say what the model must do. +- [15. Metabolic tasks](tasks.md), the other way to say what the model must do. - [3. Reading and writing models](io.md), saving the filled model with a record of what was added. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index b1c2ecc..8550e5c 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -105,7 +105,7 @@ compartment its metabolites touch; a transport reaction appears under both. And *unique* means biochemically unique, ignoring compartments: 52 metabolites collapse to 45 because seven of them exist in both the cytosol and the mitochondrion. The same 45 is what -[16. Combining and simplifying](combining.md) arrives at when it flattens the +[20. Combining and simplifying](combining.md) arrives at when it flattens the compartments away. ::: :::{tab-item} 🐍 Python diff --git a/docs/guide/homology.md b/docs/guide/homology.md index 55c02a4..2e6c17d 100644 --- a/docs/guide/homology.md +++ b/docs/guide/homology.md @@ -1,4 +1,4 @@ -# 18. Reconstruction from homology +# 12. Reconstruction from homology Homology-based reconstruction builds a draft model for an organism that has none, using a curated model of a related organism as the source of reactions. Genes in @@ -64,7 +64,7 @@ template: 53 rxns, 61 genes ::: :::: -## 18.1 BLAST, in both directions +## 12.1 BLAST, in both directions Both toolboxes run BLASTP twice: the new organism's proteome against the template's, and the template's against the new organism's. @@ -146,7 +146,7 @@ If you already have orthology assignments from another source (OrthoFinder, OMA, a published table), `makeFakeBlastStructure` and `make_ortholog_hits` wrap them in the structure the transfer step expects, so no search is run. -## 18.2 From hits to a draft +## 12.2 From hits to a draft ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -208,7 +208,7 @@ accepted counterpart are not carried over. That is the intended behaviour, and also the main source of error: a reaction left out because no hit passed the cut-offs looks exactly like a reaction the organism genuinely lacks. -## 18.3 The cut-offs decide the model +## 12.3 The cut-offs decide the model Three cut-offs control which hits are accepted, and tightening any of them shrinks the draft. @@ -278,7 +278,7 @@ can produce a different draft under RAVEN 3. Set them explicitly, or see [Migrating from RAVEN 2](../raven3-migration.md). ::: -## 18.4 A draft is not a model +## 12.4 A draft is not a model What comes out of this step has reactions, metabolites and genes, and nothing else. There is no biomass reaction unless a template reaction happened to carry @@ -332,8 +332,8 @@ the reconstruction, and will only become informative once a medium is defined. A homology draft is a set of claims about which reactions the organism has. Turning it into a model means giving it a medium ([5. Growth media and conditions](media.md)), closing the gaps that stop it -producing biomass ([13. Gap-filling](gap-filling.md)), and checking it against -what the organism is known to do ([12. Metabolic tasks](tasks.md)). The +producing biomass ([17. Gap-filling](gap-filling.md)), and checking it against +what the organism is known to do ([15. Metabolic tasks](tasks.md)). The [GEM reconstruction protocol](../protocol/index.md) follows that path for *H. polymorpha* at full scale. @@ -355,9 +355,9 @@ what the organism is known to do ([12. Metabolic tasks](tasks.md)). The ## See also -- [13. Gap-filling](gap-filling.md), the usual next step, and the one that +- [17. Gap-filling](gap-filling.md), the usual next step, and the one that decides what the draft is missing. -- [10. Context-specific models](init.md), cutting a model down by evidence +- [14. Context-specific models](init.md), cutting a model down by evidence instead of building one up from homology. -- [17. Comparing models](comparing.md), checking a draft against a curated +- [21. Comparing models](comparing.md), checking a draft against a curated model of the same organism. diff --git a/docs/guide/index.md b/docs/guide/index.md index 610f2f6..9e0d501 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -27,8 +27,8 @@ the cobrapy documentation. The :class: info Every snippet on the nineteen numbered pages is run on each commit and checked against the output shown beneath it, in both languages. Two pages -cannot be: [10. Context-specific models](init.md), whose steps take hours on -Human-GEM, and [19. Reconstruction from KEGG](kegg.md), which downloads +cannot be: [14. Context-specific models](init.md), whose steps take hours on +Human-GEM, and [13. Reconstruction from KEGG](kegg.md), which downloads tens of megabytes and takes minutes. Both say so at the top and quote the wall-clock of the run their numbers came from. @@ -74,45 +74,48 @@ tutorials at the end of this section are MATLAB-only and are not executed. deletions. 9. [Quality control](quality-control.md), structure, mass balance, blocked reactions, mass from nothing. +10. [Table-driven curation](curation.md), applying curation as reviewable + tables rather than a script of function calls. +11. [Biomass composition and annotation](biomass.md), what the cell is made + of, rescaling it, and the annotation that stays useful after the solve. **Reconstruction** -10. [Context-specific models (tINIT / ftINIT)](init.md), omics integration and +12. [Reconstruction from homology](homology.md), BLAST a proteome against a + template model's genes, and what the cut-offs decide for you. +13. [Reconstruction from KEGG](kegg.md), a draft with no template model at all, + and what such a draft is missing. +14. [Context-specific models (ftINIT)](init.md), omics integration and extracting a cell-line model from Human-GEM and RNA-seq, end to end. **Analysis and repair** -11. [Deletions and essentiality](deletions.md), knockouts, essential genes and - reactions, MOMA. -12. [Metabolic tasks](tasks.md), saying what a model must be able to do, and +15. [Metabolic tasks](tasks.md), saying what a model must be able to do, and checking that it still can. -13. [Gap-filling](gap-filling.md), closing the holes a draft model has. -14. [Flux variability](fva.md), how much each flux can still move, and +16. [Deletions and essentiality](deletions.md), knockouts, essential genes and + reactions, MOMA. +17. [Gap-filling](gap-filling.md), closing the holes a draft model has. +18. [Flux variability](fva.md), how much each flux can still move, and which predictions the model has no choice about. -15. [Random sampling](sampling.md), the distribution behind those ranges, and +19. [Random sampling](sampling.md), the distribution behind those ranges, and how to condition it on a particular state. -16. [Combining and simplifying models](combining.md), merging models on + +**Comparing and engineering** + +20. [Combining and simplifying models](combining.md), merging models on metabolite names, and taking one apart again. -17. [Comparing models](comparing.md), what changed between two models, and what +21. [Comparing models](comparing.md), what changed between two models, and what effect that difference has. -18. [Reconstruction from homology](homology.md), BLAST a proteome against a - template model's genes, and what the cut-offs decide for you. -19. [Reconstruction from KEGG](kegg.md), a draft with no template model at all, - and what such a draft is missing. -20. [Engineering targets](engineering.md), FSEOF for over-production targets +22. [Engineering targets](engineering.md), FSEOF for over-production targets and reporter metabolites for the metabolites a transcriptional response surrounds. -21. [Table-driven curation](curation.md), applying curation as reviewable - tables rather than a script of function calls. -22. [Biomass composition and annotation](biomass.md), what the cell is made - of, rescaling it, and the annotation that stays useful after the solve. ## Also in this section | | | |---|---| -| [Worked protocol, *H. polymorpha*](../protocol/index.md) | homology-based reconstruction of `hanpo-GEM`, end to end, from template models to a growing methylotrophic draft. A published pipeline followed from start to finish. MATLAB only, not executed here. | +| [GEM reconstruction protocol, *H. polymorpha*](../protocol/index.md) | homology-based reconstruction of `hanpo-GEM`, end to end, from template models to a growing methylotrophic draft. A published pipeline followed from start to finish. MATLAB only, not executed here. | | [Legacy tutorials](../tutorials/index.md) | five exercises from the original RAVEN paper (Agren et al., 2013), updated to run with current RAVEN but otherwise unchanged. MATLAB only. | Use the numbered pages to look something up; use the protocol to see a whole @@ -141,31 +144,25 @@ solvers building editing quality-control +curation +biomass +homology +kegg init -deletions tasks +deletions gap-filling fva sampling combining comparing -homology -kegg engineering -curation -biomass ``` ```{toctree} :hidden: -:caption: Legacy protocol (hanpo-GEM) +:caption: Legacy ../protocol/index -``` - -```{toctree} -:hidden: -:caption: Legacy tutorials - ../tutorials/index ``` diff --git a/docs/guide/init.md b/docs/guide/init.md index b4a26ea..760640e 100644 --- a/docs/guide/init.md +++ b/docs/guide/init.md @@ -1,4 +1,4 @@ -# 10. Context-specific models with tINIT and ftINIT +# 14. Context-specific models with ftINIT A genome-scale model describes what an organism *can* do. tINIT and ftINIT cut it down to what a particular sample (a tissue, a cell line, a condition) appears to @@ -102,7 +102,7 @@ git clone --depth=1 https://github.com/SysBioChalmers/Human-GEM.git Both extractions are mixed-integer problems, so **GLPK will not do**: set Gurobi up first ([6. Solvers and configuration](solvers.md)). -## 10.1 Prepare the template: once +## 14.1 Prepare the template: once The preparation finds the task-essential reactions, classifies every reaction into omics-independent categories, merges linear stretches, and rescales the @@ -176,7 +176,7 @@ same machine and solver. Two things affect the run: ::: :::: -## 10.2 Bring in the expression data +## 14.2 Bring in the expression data RAVEN wants one struct: the genes, the sample names, and a genes × samples matrix of levels. `threshold` is the level above which a gene counts as expressed; leave @@ -385,11 +385,11 @@ question does not arise. `parseHPArna` and `parse_hpa_rna` read the RNA-seq dump instead, whose header is `Gene`, `Gene name`, `Tissue` followed by the TPM columns. Those are quantities -already, so `rna_gene_scores` applies the same logarithmic rule as 10.2 rather +already, so `rna_gene_scores` applies the same logarithmic rule as 14.2 rather than a level mapping. Both routes end at a gene-to-score mapping, which is what `score_reactions_from_genes` walks the GPRs with. -## 10.3 Extract a model for one sample +## 14.3 Extract a model for one sample ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -440,7 +440,7 @@ takes 30–60 s per sample; `1+1` adds a second optimisation over those reaction takes two to three times longer, and gives a smaller model. The run above is `1+0`: **12 931 → 9 595 reactions and 2 848 → 1 761 genes, in 70 seconds.** -## 10.4 Check what came out +## 14.4 Check what came out An extracted model is a hypothesis. The tasks it was built to satisfy are the first thing to re-check, and the least expensive. diff --git a/docs/guide/kegg.md b/docs/guide/kegg.md index 3d2b31b..ba4ccf1 100644 --- a/docs/guide/kegg.md +++ b/docs/guide/kegg.md @@ -1,8 +1,8 @@ -# 19. Reconstruction from KEGG +# 13. Reconstruction from KEGG -[18. Reconstruction from homology](homology.md) needs a template model of a +[12. Reconstruction from homology](homology.md) needs a template model of a related organism. KEGG needs none: its orthology groups (KOs) are already tied to reactions, so annotating a genome with KOs gives you a draft directly. Which route you take depends on one thing: whether your organism is already in KEGG. @@ -24,7 +24,7 @@ their timings, measured on an ordinary laptop. | `getModelFromKEGG` | `build_reference_model` | assemble and cache the global KEGG model | | `getPhylDist` | `PhylDist` | phylogenetic distance, used to weight the search | -## 19.1 When the organism is already in KEGG +## 13.1 When the organism is already in KEGG If KEGG has your species (`sce` for *S. cerevisiae*), its gene-to-KO assignments are already made, and no sequence search is needed. This is the fast route, and @@ -71,14 +71,14 @@ print(f"{len(model.reactions)} rxns, {len(model.metabolites)} mets, " The artefacts (a reference model and three tables, about 47 MB) are fetched from the `kegg118` raven-data release on first use and cached; see -[Downloaded data and binaries](../installation/data-and-binaries.md) for +[Download data and binaries](../installation/data-and-binaries.md) for where, and how to fetch them ahead of a batch run. Expect around five minutes for the first run and much the same afterwards: the download is not the slow part, assembling the draft from the tables is. ::: :::: -## 19.2 When it is not +## 13.2 When it is not For an organism KEGG has never seen, the KO assignments have to be made from sequence. Both toolboxes search your proteins against a library of profile HMMs, @@ -111,7 +111,7 @@ compressed**, and `hmmsearch` against every KO takes tens of minutes to hours fo a full proteome, which is why `outDir` exists in the MATLAB version: results are kept per-KO so an interrupted run can resume rather than start again. It also needs HMMER, which has no bundle for every platform; -[Downloaded data and binaries](../installation/data-and-binaries.md) has the +[Download data and binaries](../installation/data-and-binaries.md) has the matrix. The organism id still matters even here. It sets the phylogenetic distance used @@ -127,7 +127,7 @@ defaults were measured rather than inherited, and the [KEGG HMM cut-off study](../parameter-tuning/studies/kegg-hmm-cutoff-calibration.md) records which of them actually moves the result. -## 19.3 What a KEGG draft is +## 13.3 What a KEGG draft is A KEGG draft is genome-scale in size, and unlike a working model in every other respect. @@ -144,7 +144,7 @@ objective: none **There are no compartments.** KEGG describes reactions, not cell biology, so every metabolite lands in one undifferentiated space. A KEGG draft cannot distinguish mitochondrial from cytosolic anything until you localise it; -see [16. Combining and simplifying](combining.md) for the reverse operation, and +see [20. Combining and simplifying](combining.md) for the reverse operation, and what is lost by it. The flags decide how inclusive the draft is. `keepIncomplete` / @@ -179,9 +179,9 @@ it can. ## See also -- [18. Reconstruction from homology](homology.md), the same goal from a template +- [12. Reconstruction from homology](homology.md), the same goal from a template model instead of an orthology database. -- [13. Gap-filling](gap-filling.md), what to do with 1357 reactions that cannot +- [17. Gap-filling](gap-filling.md), what to do with 1357 reactions that cannot yet carry flux. - [Legacy tutorial 5](../tutorials/tutorial5.md), the original KEGG reconstruction exercise from the RAVEN paper. diff --git a/docs/guide/quality-control.md b/docs/guide/quality-control.md index 99d47da..5fc2284 100644 --- a/docs/guide/quality-control.md +++ b/docs/guide/quality-control.md @@ -349,7 +349,7 @@ Models differ (11 differences): `diffModels` compares two models entry by entry: which reactions are unique to each, and where a shared reaction differs in stoichiometry, bounds, objective coefficient, gene rule or metabolite properties. It is the same function -[17. Comparing models](comparing.md) uses, applied here to a model and its +[21. Comparing models](comparing.md) uses, applied here to a model and its last good version. :::{warning} What can go wrong diff --git a/docs/guide/sampling.md b/docs/guide/sampling.md index 5ce5775..0c6f5b0 100644 --- a/docs/guide/sampling.md +++ b/docs/guide/sampling.md @@ -1,4 +1,4 @@ -# 15. Random sampling +# 19. Random sampling Flux variability gives the extremes a reaction can reach. Sampling gives the **distribution** in between: draw many flux vectors from the feasible space and @@ -17,7 +17,7 @@ is usually 0, usually 10, or evenly spread; a sample does. ## Setup -The same glucose-limited `smallYeast.yml` as [14. Flux variability](fva.md), so +The same glucose-limited `smallYeast.yml` as [18. Flux variability](fva.md), so the two pages can be read against each other. ::::{tab-set} @@ -53,7 +53,7 @@ model.objective = "biomassOUT" ::: :::: -## 15.1 A first sample +## 19.1 A first sample Both toolboxes take a `seed`, and you should always set one: without it a chain is different every run and nothing you report can be reproduced. @@ -104,9 +104,9 @@ and correlation you compute has to pick the right axis, and getting it wrong is silent; you get numbers, just not the ones you meant. ::: -## 15.2 What a distribution says that a range does not +## 19.2 What a distribution says that a range does not -[14. Flux variability](fva.md) found that `FRDS2` and `SDH` have the widest +[18. Flux variability](fva.md) found that `FRDS2` and `SDH` have the widest range in this model, 1000 units, and that all of it is a thermodynamically infeasible cycle. Sampling does not remove that; it shows how much of the space the cycle occupies. @@ -161,7 +161,7 @@ That is the distinction this page turns on. A sampled mean is a statement about the *shape of the feasible space*, not a prediction: this model's mean sampled growth is less than half its optimum, and no biology changed to make it so. -## 15.3 Choosing a method +## 19.3 Choosing a method `achr` and `chrr` both draw the near-uniform interior distribution; `randomObjective` (`random_objective` in Python) instead maximises a small @@ -217,7 +217,7 @@ and the samples are usable, but on a polytope this elongated mixing is slower than the defaults assume, and more thinning helps if the distribution is uneven. -## 15.4 Sample a state, not a model +## 19.4 Sample a state, not a model Sampling an unconstrained model answers "what could this network do?", which is rarely the question. The useful version is to constrain first (hold growth near @@ -257,12 +257,12 @@ ethanol: free 0.72, at 90% growth 0.04 ::: :::: -## 15.5 Which reactions to sample over +## 19.5 Which reactions to sample over The `randomObjective` method needs reactions that can carry real flux to use as objectives. Reactions that move only through a loop are useless for that, so both toolboxes screen them out with a loopless FVA first, the same test -[14. Flux variability](fva.md) used by hand. +[18. Flux variability](fva.md) used by hand. ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -299,7 +299,7 @@ which on a genome-scale model skips the expensive part of the run. :::{warning} What can go wrong - **No seed.** The numbers change every run. Set one, and report it, and do not expect a seed alone to reproduce a MATLAB chain on another machine - (see 15.1). Report the distribution, not the draws. + (see 19.1). Report the distribution, not the draws. - **Too little thinning.** Consecutive MCMC steps are correlated; the default of 100 steps between recorded samples exists for that reason. Lowering it gains speed and loses independence. @@ -309,14 +309,14 @@ which on a genome-scale model skips the expensive part of the run. - **Sampling a model that is wide open.** With an unconstrained medium the space is enormous and the distribution says nothing. Constrain first. - **Loops.** They inflate the space being sampled, and every sample drawn - inside a cycle is wasted. See [14. Flux variability](fva.md). + inside a cycle is wasted. See [18. Flux variability](fva.md). - **Genome-scale runtime.** Sampling is many LPs per recorded sample. Start with a few hundred samples to see the shape, not tens of thousands. ::: ## See also -- [14. Flux variability](fva.md), the ranges this page fills in. +- [18. Flux variability](fva.md), the ranges this page fills in. - [4. Simulating growth with FBA](fba.md), the single point sampling surrounds. -- [11. Deletions and essentiality](deletions.md), the other way to ask what the +- [16. Deletions and essentiality](deletions.md), the other way to ask what the model depends on. diff --git a/docs/guide/solvers.md b/docs/guide/solvers.md index 2c9872a..2e70013 100644 --- a/docs/guide/solvers.md +++ b/docs/guide/solvers.md @@ -9,7 +9,7 @@ read what it gives back. | MATLAB | Python | | |---|---|---| | `setRavenSolver` | `Configuration` {bdg-link-secondary}`cobrapy ` | choose the solver | -| `checkInstallation` | `Configuration` {bdg-link-secondary}`cobrapy ` | check the solver works | +| `checkRaven` | `Configuration` {bdg-link-secondary}`cobrapy ` | check the solver works | | `solveLP` | `Model.optimize` {bdg-link-secondary}`cobrapy ` | solve, and get a solution object | | `optimizeProb` | `Model.solver` {bdg-link-secondary}`cobrapy ` | solve a problem the toolbox built for you | @@ -69,7 +69,7 @@ RAVEN keeps the choice in MATLAB's preferences, so it survives restarts, and the answer is whatever *this* installation was last told, which is why the output above is elided. The preference is global to the installation, not a property of the model, so a script that changes it changes it for everything -that follows. `checkInstallation` prints it along with a test solve. +that follows. `checkRaven` prints it along with a test solve. ::: :::{tab-item} 🐍 Python :sync: python @@ -255,7 +255,7 @@ which is a statement about biology rather than about the constraints. :::{warning} What can go wrong - **A solver that is set but not installed.** `setRavenSolver('gurobi')` stores the preference whether or not Gurobi is there; the failure appears - at the next solve. `checkInstallation` tests it directly. + at the next solve. `checkRaven` tests it directly. - **MILP with GLPK.** GLPK solves LPs only, in RAVEN. Anything mixed-integer (`getMinimalMedium`, some gap-filling) needs Gurobi. - **Reading fluxes after a non-optimal solve.** `sol.stat` of `0` means a diff --git a/docs/guide/tasks.md b/docs/guide/tasks.md index fc163df..d31b5fb 100644 --- a/docs/guide/tasks.md +++ b/docs/guide/tasks.md @@ -1,4 +1,4 @@ -# 12. Metabolic tasks +# 15. Metabolic tasks A metabolic task is a statement of something the model must be able to do: "given glucose and oxygen, produce biomass", "make ATP", "do *not* produce @@ -54,7 +54,7 @@ because a task closes the model's own exchanges and growth has to put its carbon somewhere. Leave CO₂ out and the task is infeasible, which looks like a broken model and is really a broken task. -## 12.1 Read and run the tasks +## 15.1 Read and run the tasks ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -121,7 +121,7 @@ That is why a task list is portable between models in a way a script full of ::: :::: -## 12.2 A task defines its own medium +## 15.2 A task defines its own medium `check_tasks` closes every exchange, sink and demand the model has before applying a task, exactly as RAVEN does. The task's inputs and outputs are then @@ -156,7 +156,7 @@ The only difference between the two is the CO₂ output. ::: :::: -## 12.3 Which reactions does a task depend on? +## 15.3 Which reactions does a task depend on? Essentiality with respect to a task, rather than to the objective: remove each candidate reaction and see whether the task still passes. This is what ftINIT @@ -202,11 +202,11 @@ print(sorted(result.reactions)[:5]) ::: :::: -## 12.4 When a task fails +## 15.4 When a task fails A failing task is a gap: something the model should be able to do and cannot. Both toolboxes can add reactions from a template until the task passes, which is -[13. Gap-filling](gap-filling.md), approached from the task side. +[17. Gap-filling](gap-filling.md), approached from the task side. ::::{tab-set} :::{tab-item} Ⓜ️ MATLAB @@ -246,8 +246,8 @@ every task feasible again. ## See also -- [13. Gap-filling](gap-filling.md), making a failing task pass. -- [10. Context-specific models](init.md), tasks as the thing an extraction must +- [17. Gap-filling](gap-filling.md), making a failing task pass. +- [14. Context-specific models](init.md), tasks as the thing an extraction must preserve. - [9. Quality control](quality-control.md), the checks that do not need a task list. diff --git a/docs/index.md b/docs/index.md index 90dd592..8e30a55 100644 --- a/docs/index.md +++ b/docs/index.md @@ -19,13 +19,13 @@ pip install --pre raven-toolbox ::: :::{tab-item} MATLAB (Add-Ons) -Home → Add-Ons → Get Add-Ons → search "RAVEN Toolbox", then run `checkRaven` -to verify. +Home → Add-Ons → Get Add-Ons → search "RAVEN Toolbox". +Then run `checkRaven` to verify. ::: :::{tab-item} Python (git) ```bash -git clone https://github.com/SysBioChalmers/raven-toolbox.git +git clone --depth=1 https://github.com/SysBioChalmers/raven-toolbox.git pip install -e raven-toolbox/ ``` ::: @@ -143,7 +143,7 @@ Complete function reference for both MATLAB and Python. Set up RAVEN in MATLAB or raven-toolbox in Python with a solver. ::: -:::{grid-item-card} RAVEN 3 and raven-toolbox +:::{grid-item-card} RAVEN and raven-toolbox :link: raven3-vs-raven-toolbox :link-type: doc diff --git a/docs/installation/data-and-binaries.md b/docs/installation/data-and-binaries.md index 373e1eb..26d5adf 100644 --- a/docs/installation/data-and-binaries.md +++ b/docs/installation/data-and-binaries.md @@ -1,4 +1,4 @@ -# Downloaded data and binaries +# Download data and binaries Neither toolbox ships the large files it needs. The KEGG reference data, the profile-HMM libraries, and the BLAST+, DIAMOND and HMMER executables are all @@ -13,10 +13,10 @@ files RAVEN downloads and the files raven-toolbox downloads are the same files. | Artefact | Size | Needed for | |---|---|---| -| KEGG core bundle (reference model plus the KO, reaction and organism-gene tables) | about 47 MB | [19. Reconstruction from KEGG](../guide/kegg.md), either route | +| KEGG core bundle (reference model plus the KO, reaction and organism-gene tables) | about 47 MB | [13. Reconstruction from KEGG](../guide/kegg.md), either route | | KEGG HMM library, one per domain | 129 MB compressed, eukaryotes | a KEGG draft for an organism KEGG has never seen | | KEGG taxonomy | small | phylogenetic weighting of KO assignments | -| BLAST+ (`blastp`, `makeblastdb`) | small | [18. Reconstruction from homology](../guide/homology.md) | +| BLAST+ (`blastp`, `makeblastdb`) | small | [12. Reconstruction from homology](../guide/homology.md) | | DIAMOND | small | the faster homology route | | HMMER (`hmmsearch`) | small | searching the KEGG HMM library | @@ -25,19 +25,43 @@ comparing models require none of it. ## Where it goes -raven-toolbox caches under `$XDG_CACHE_HOME`, or `~/.cache` when that is unset: +raven-toolbox caches under `$XDG_CACHE_HOME`, or `~/.cache` when that is +unset, using the same rule on every platform: + +::::{tab-set} +:::{tab-item} Linux + +| | Path | +|---|---| +| Data artefacts | `/home//.cache/raven_toolbox/data/-/` | +| Binaries | `/home//.cache/raven_toolbox/binaries/` | +::: +:::{tab-item} macOS + +| | Path | +|---|---| +| Data artefacts | `/Users//.cache/raven_toolbox/data/-/` | +| Binaries | `/Users//.cache/raven_toolbox/binaries/` | +::: +:::{tab-item} Windows | | Path | |---|---| -| Data artefacts | `~/.cache/raven_toolbox/data/-/` | -| Binaries | `~/.cache/raven_toolbox/binaries/` | +| Data artefacts | `C:\Users\\.cache\raven_toolbox\data\-\` | +| Binaries | `C:\Users\\.cache\raven_toolbox\binaries\` | +::: +:::: + +raven-toolbox does not use each platform's own convention (`%LOCALAPPDATA%` +on Windows, `~/Library/Caches` on macOS); `~/.cache` is literal everywhere. The version is part of the path, so two KEGG releases coexist and a pinned run keeps fetching the release it was pinned to. In MATLAB the KEGG artefacts go wherever `dataDir` points, which `getKEGGModelForOrganism` requires you to pass, and the executables live in -RAVEN's own `software/` directory. +`software/` inside wherever RAVEN itself is installed, the same on every +platform. ## How a tool is found @@ -127,35 +151,6 @@ not, because that needs MAFFT and CD-HIT, which have no Windows builds. Use WSL2 calls the resolved executable directly and does not translate paths between Windows and WSL. -## Integrity and versions - -Every file carries a SHA256 that is checked after download, so a truncated or -substituted file raises an error instead of being used. - -The two toolboxes pin differently, and both are deliberate. A raven-toolbox -release carries a baked snapshot of the artefact registry, so a given version -always fetches the exact files it was tested against; setting -`RAVEN_PYTHON_MANIFEST` to another manifest's URL or path overrides that. RAVEN -resolves from the published release each time instead, having no baked registry -to keep in step. - -Artefacts are versioned by their upstream version, not by a toolbox release: -`kegg118` for a KEGG release, `diamond-2.1.17` for a DIAMOND build. An asset is -published once under its own tag and never replaced, so a run that pins -`kegg118` keeps getting the same bytes after a `kegg119` appears. - -## Licences - -The bundles carry their upstream licence text, and the terms differ: - -- **BLAST+** is produced by the NCBI and is in the public domain. -- **DIAMOND** is GPL-3.0. It is redistributed unmodified, as a separate download - rather than inside any package, which is what keeps the toolbox's own licence - unaffected. -- **HMMER** is BSD-3-Clause. -- The **KEGG** artefacts are derived from a licensed KEGG dump and redistributed - with permission. Using them in published work means citing KEGG. - :::{warning} What can go wrong - **A machine with no network.** The first reconstruction fails at the download. Fetch on a connected machine, copy `~/.cache/raven_toolbox`, and @@ -175,8 +170,8 @@ The bundles carry their upstream licence text, and the terms differ: ## See also -- [19. Reconstruction from KEGG](../guide/kegg.md), the workflow that pulls the +- [13. Reconstruction from KEGG](../guide/kegg.md), the workflow that pulls the KEGG artefacts. -- [18. Reconstruction from homology](../guide/homology.md), the one that needs +- [12. Reconstruction from homology](../guide/homology.md), the one that needs BLAST+ or DIAMOND. - [Installing RAVEN](raven.md) and [Installing raven-toolbox](python.md). diff --git a/docs/installation/python.md b/docs/installation/python.md index d7c74ee..e655412 100644 --- a/docs/installation/python.md +++ b/docs/installation/python.md @@ -18,6 +18,12 @@ source .venv/bin/activate # Windows: .venv\Scripts\activate ``` ::: +:::{note} Working offline +raven-toolbox fetches KEGG data and reconstruction binaries (BLAST+, DIAMOND, +HMMER) on first use. [Download data and binaries](data-and-binaries.md) +covers preparing a machine with no network access ahead of time. +::: + --- ## Install diff --git a/docs/installation/raven.md b/docs/installation/raven.md index 5370c26..d425ca2 100644 --- a/docs/installation/raven.md +++ b/docs/installation/raven.md @@ -19,9 +19,14 @@ canonical reference is the build for your platform on first use, so the reconstruction functions need internet access the first time they run. `downloadRavenBinaries` fetches them ahead of time. See - [Downloaded data and binaries](data-and-binaries.md) for the full set, where + [Download data and binaries](data-and-binaries.md) for the full set, where it is cached, and how to prepare an offline machine. +:::{note} Working offline +[Download data and binaries](data-and-binaries.md) covers preparing a machine +with no network access ahead of time. +::: + --- (choosing-a-solver-matlab)= @@ -124,22 +129,38 @@ A successful run looks like: ```text *** THE RAVEN TOOLBOX *** -Checking if RAVEN is on the MATLAB path... OK -Checking if it is possible to parse a model in Microsoft Excel format... OK -Checking if it is possible to import an SBML model using libSBML... OK -Solver found in preferences... gurobi -Checking if it is possible to solve an LP problem using gurobi... OK -Checking essential binary executables: - BLAST+... OK - DIAMOND... OK - HMMER... OK + > Installation type Advanced (via git) + > Checking RAVEN release 3.0.0 + > Checking MATLAB release R2024b + > Set RAVEN in MATLAB path Pass + > Save MATLAB path Pass + +=== Model import and export === + > Checking libSBML version 5.20.0 + > Checking model import and export + > Import SBML format Pass + > Export SBML format Pass + > Import YAML format Pass + > Export YAML format Pass + > Export Excel format Pass + +=== Model solvers === + > Checking for LP solvers + > glpk Pass + > gurobi Pass + > Set RAVEN solver gurobi + +=== Essential binary executables === + > Checking BLAST+ Pass + > Checking DIAMOND Pass + > Checking HMMER Pass + +=== Compatibility === + > Checking function uniqueness + *** checkRaven complete *** ``` -`checkInstallation` still works too: it is the old name, kept as a deprecated -wrapper that forwards to `checkRaven` after a warning. Use `checkRaven` -directly in new work. - If MATLAB reports that it could not save the path (common on shared or managed installations where you do not have write access to MATLAB's own `pathdef.m`), run `addRavenToUserPath` instead: it writes a `startup.m` that diff --git a/docs/migrate.md b/docs/migrate.md index deff839..00cec1c 100644 --- a/docs/migrate.md +++ b/docs/migrate.md @@ -4,19 +4,17 @@ icon: material/folder-open # Migrate -Three ways of moving between versions or languages, each answering a +Two ways of moving between versions or languages, each answering a different question. | | | |---|---| | [RAVEN 2 → RAVEN 3 (MATLAB)](raven3-migration.md) | Upgrading an existing MATLAB codebase: what was renamed, removed outright, or changed between RAVEN 2 and RAVEN 3. | | [RAVEN 3 → raven-toolbox](raven3-vs-raven-toolbox.md) | Choosing between the two current implementations, or porting a MATLAB workflow to Python: what each toolbox has, and where they answer the same question differently. | -| [MATLAB vs Python function mapping](matlab-vs-python.md) | The full name table pairing every function that exists in both, generated at build time from both toolboxes' sources. | ```{toctree} :hidden: raven3-migration raven3-vs-raven-toolbox -matlab-vs-python ``` diff --git a/docs/protocol/anticipated-results.md b/docs/protocol/anticipated-results.md deleted file mode 100644 index 92c99fb..0000000 --- a/docs/protocol/anticipated-results.md +++ /dev/null @@ -1,48 +0,0 @@ -# Anticipated results - -Following this protocol yields a **draft** genome-scale metabolic model for -*Hansenula polymorpha* (`hanpo-GEM`) that: - -- is reconstructed by homology from the *S. cerevisiae* (yeast-GEM) and - *R. toruloides* (rhto-GEM) templates; -- has an organism-specific biomass composition (DNA, RNA, protein, carbohydrate - and SLIME-based lipids); -- is gap-filled so it can **produce biomass** on glycerol; and -- after manual curation can also **grow on methanol**, consistent with - *H. polymorpha* being a methylotroph. - -The published hanpo-GEM (the curated result distributed in the repository) -contains on the order of **2,370 reactions, 2,118 metabolites and 984 genes**, -your draft will be similar in size but is expected to differ, since it is a -starting point for the open-ended process of manual curation. - -## What "done" looks like - -A successful draft returns a positive growth rate from FBA: - -```matlab -sol = solveLP(model, 1); -printFluxes(model, sol.x, false); -``` - -and, after the methanol curation, still grows when glycerol/glucose uptake is -blocked and only methanol uptake is allowed. - -## Beyond the draft - -Manual curation is open-ended. Typical next steps include constraining the model -with measured uptake/excretion rates, validating predicted gene essentiality -against experimental data, expanding organism-specific pathways, and improving -annotations. The model is version-controlled (`newCommit`, `newRelease`) so each -curation is tracked and the model can be distributed. - -## The complete protocol script - -The full, runnable reconstruction (the authoritative version of every command -on these pages) is the -[`reconstructionProtocol.m`](https://github.com/SysBioChalmers/hanpo-GEM/blob/main/code/reconstructionProtocol.m) -script in the hanpo-GEM repository, reproduced here: - -```{literalinclude} ../../hanpo-GEM/code/reconstructionProtocol.m -:language: matlab -``` diff --git a/docs/protocol/biomass.md b/docs/protocol/biomass.md index dbf3aef..3181e62 100644 --- a/docs/protocol/biomass.md +++ b/docs/protocol/biomass.md @@ -1,4 +1,4 @@ -# 3.4 Define the biomass composition +# Define the biomass composition Metabolism must be able to synthesise the macromolecules that make up biomass. GEMs encode this in **biomass pseudoreactions** that specify which macromolecules diff --git a/docs/protocol/gap-filling.md b/docs/protocol/gap-filling.md index 0e97e3c..4588594 100644 --- a/docs/protocol/gap-filling.md +++ b/docs/protocol/gap-filling.md @@ -1,4 +1,4 @@ -# 3.6 Perform gap-filling +# Perform gap-filling Homology-based draft models tend to have many **gaps**: missing reactions that break a pathway needed to produce biomass macromolecules. The pathway is likely diff --git a/docs/protocol/homology.md b/docs/protocol/homology.md index 5ec1ecd..6564438 100644 --- a/docs/protocol/homology.md +++ b/docs/protocol/homology.md @@ -1,11 +1,11 @@ -# 3.3 Generate a draft model from homology +# Generate a draft model from homology The protein FASTA files of the target and template organisms are compared to build a draft *H. polymorpha* model based on homology. Two RAVEN functions do the work: `getBlast` aligns the protein sequences, and `getModelFromHomology` uses those results to assemble the first draft. -## 3.3.1 Clean and match protein identifiers +## Clean and match protein identifiers The gene identifiers in each template model **must match** those in its protein FASTA (compare against, e.g., *S. cerevisiae* `YML001W`). For the target @@ -13,7 +13,7 @@ organism it is also convenient to shorten the identifiers to a standard format such as `Hanpo2_12345`. The *H. polymorpha* FASTA from JGI needs some editing to strip annotations; see [Note 2](#notes) below. -## 3.3.2 Determine homology by BLAST +## Determine homology by BLAST Evidence of homology is determined by **bidirectional** BLAST of the target proteome against both template proteomes (this can take some minutes): diff --git a/docs/protocol/index.md b/docs/protocol/index.md index 3ac43ff..b4ee42e 100644 --- a/docs/protocol/index.md +++ b/docs/protocol/index.md @@ -2,7 +2,7 @@ icon: material/folder-open --- -# Worked protocol: *Hansenula polymorpha* +# GEM reconstruction protocol: *Hansenula polymorpha* A complete homology-based reconstruction of `hanpo-GEM`, followed from start to finish: it takes template models as input and produces a growing methylotrophic @@ -27,19 +27,18 @@ with this site. | [Introduction](reconstruction.md) | what is being built, and from what | | [Materials and installation](materials.md) | software, databases and input files | | [Import template models](template-models.md) | the *S. cerevisiae* and *R. toruloides* models the draft is built from | -| [Draft from homology](homology.md) | BLAST and `getModelFromHomology`; see also [18. Reconstruction from homology](../guide/homology.md) | +| [Draft from homology](homology.md) | BLAST and `getModelFromHomology`; see also [12. Reconstruction from homology](../guide/homology.md) | | [Biomass composition](biomass.md) | measuring and assembling the biomass equation | | [Curation of lipid reactions](lipid-curation.md) | the part that never generalises | -| [Gap-filling](gap-filling.md) | closing the holes; see also [13. Gap-filling](../guide/gap-filling.md) | +| [Gap-filling](gap-filling.md) | closing the holes; see also [17. Gap-filling](../guide/gap-filling.md) | | [Save and simulate](simulation.md) | writing the model out and checking it grows | | [Manual curation](manual-curation.md) | what no function does for you | -| [Anticipated results](anticipated-results.md) | what the finished model should look like | ## Related - [Legacy tutorials](../tutorials/index.md), five shorter exercises from the original RAVEN paper. -- [18. Reconstruction from homology](../guide/homology.md), the same technique +- [12. Reconstruction from homology](../guide/homology.md), the same technique at small scale, in both languages, with every example executed. ```{toctree} @@ -54,5 +53,4 @@ lipid-curation gap-filling simulation manual-curation -anticipated-results ``` diff --git a/docs/protocol/lipid-curation.md b/docs/protocol/lipid-curation.md index c2c37e2..f7693f0 100644 --- a/docs/protocol/lipid-curation.md +++ b/docs/protocol/lipid-curation.md @@ -1,4 +1,4 @@ -# 3.5 Curation of lipid reactions +# Curation of lipid reactions After adjusting the biomass composition it is useful to curate reactions already known to differ in the target organism. Using the **SLIME** formalism (*Split diff --git a/docs/protocol/manual-curation.md b/docs/protocol/manual-curation.md index fe0f910..5ba7e0f 100644 --- a/docs/protocol/manual-curation.md +++ b/docs/protocol/manual-curation.md @@ -1,4 +1,4 @@ -# 3.9 Manual curation +# Manual curation A draft GEM always requires manual curation to become a high-quality model. Curation can mean correcting gene associations, adding or removing reactions, and @@ -80,6 +80,5 @@ newCommit(model); The draft model is now methylotrophic, in agreement with the literature. -See [Anticipated results](anticipated-results.md) for what the finished draft -should look like, and the full runnable script in +The full runnable script for the whole protocol is [`code/reconstructionProtocol.m`](https://github.com/SysBioChalmers/hanpo-GEM/blob/main/code/reconstructionProtocol.m). diff --git a/docs/protocol/materials.md b/docs/protocol/materials.md index f4d4006..8f68e2d 100644 --- a/docs/protocol/materials.md +++ b/docs/protocol/materials.md @@ -61,37 +61,22 @@ protein, lipid and carbohydrate content for *H. polymorpha*, used to set the biomass stoichiometric coefficients ([Biomass composition](biomass.md)). Where organism-specific data is missing, values from related organisms can be used. -## 3.1 Install RAVEN +## Install RAVEN -After obtaining the software and files, install RAVEN (see the -[RAVEN Wiki](https://github.com/SysBioChalmers/RAVEN/wiki/Installation)). Use -`pathtool` to add the RAVEN, libSBML and Gurobi subfolders to the MATLAB path, -then verify with: +After obtaining the software and files, install RAVEN (see +[Installing RAVEN](../installation/raven.md)). Use `pathtool` to add the +RAVEN, libSBML and Gurobi subfolders to the MATLAB path, then verify with: ```matlab -checkInstallation +checkRaven ``` -A successful run looks like: - -```text -*** THE RAVEN TOOLBOX *** - -Checking if RAVEN is on the MATLAB path... OK -Checking if it is possible to parse a model in Microsoft Excel format... OK -Checking if it is possible to import an SBML model using libSBML... OK -Solver found in preferences... gurobi -Checking if it is possible to solve an LP problem using gurobi... OK -Checking essential binary executables: - BLAST+... OK - DIAMOND... OK - HMMER... OK -*** checkInstallation complete *** -``` +See [Installing RAVEN](../installation/raven.md#verify) for what a +successful run looks like. :::{warning} -If `checkInstallation` reports that parsing Excel format **FAILED**, -uninstall MATLAB's **Text Analytics Toolbox**, which conflicts with RAVEN's -Excel parser. For support, see the +If `checkRaven` reports that importing SBML format **failed**, uninstall +MATLAB's **Text Analytics Toolbox**, which conflicts with RAVEN's Excel +parser. For support, see the [RAVEN issues](https://github.com/SysBioChalmers/RAVEN/issues) page. ::: diff --git a/docs/protocol/reconstruction.md b/docs/protocol/reconstruction.md index 23b8a46..93d2f02 100644 --- a/docs/protocol/reconstruction.md +++ b/docs/protocol/reconstruction.md @@ -17,6 +17,12 @@ the section numbering there matches the page titles here. The commands shown in these pages are occasionally more concise than in the script (typically around file locations). +:::{note} Written for RAVEN 2 +The original, published protocol was written for RAVEN 2. This site applies +minor modifications to the commands shown so the same protocol also works with +RAVEN 3. +::: + ## What are genome-scale metabolic models? Cellular metabolism is a complex network of hundreds to thousands of reactions @@ -68,6 +74,3 @@ section): 6. [Gap-filling](gap-filling.md), make the draft able to produce biomass. 7. [Save and simulate](simulation.md), version control and FBA. 8. [Manual curation](manual-curation.md), fix gene associations, add methanol metabolism. - -See [Anticipated results](anticipated-results.md) for what the finished draft -should look like. diff --git a/docs/protocol/simulation.md b/docs/protocol/simulation.md index 0459111..b5d23ef 100644 --- a/docs/protocol/simulation.md +++ b/docs/protocol/simulation.md @@ -1,6 +1,6 @@ -# 3.7–3.8 Save to GitHub and simulate +# Save to GitHub and simulate -## 3.7 Save to GitHub +## Save to GitHub The model is tracked and distributed through a GitHub repository. Before saving, add some metadata to the model structure: @@ -43,7 +43,7 @@ several formats, while `code/` and `data/` contain everything needed to regenerate the model and run analyses. ::: -## 3.8 Perform simulations +## Perform simulations The draft model can now simulate biomass production. Set biomass formation as the objective, run FBA with `solveLP`, and inspect the active fluxes with diff --git a/docs/protocol/template-models.md b/docs/protocol/template-models.md index 9d6ca7f..83a2d83 100644 --- a/docs/protocol/template-models.md +++ b/docs/protocol/template-models.md @@ -1,4 +1,4 @@ -# 3.2 Import template models +# Import template models RAVEN creates a draft GEM by using models of phylogenetically related organisms as templates. For *H. polymorpha*, the *S. cerevisiae* model (yeast-GEM) is the diff --git a/docs/raven3-migration.md b/docs/raven3-migration.md index 9ed1842..4e359a7 100644 --- a/docs/raven3-migration.md +++ b/docs/raven3-migration.md @@ -1,4 +1,4 @@ -# Migrating from RAVEN 2 to RAVEN 3 +# RAVEN 2 to RAVEN 3 :::{admonition} If you read nothing else :class: important @@ -36,7 +36,7 @@ mostly the same. :::{note} This is the MATLAB-to-MATLAB axis This page is about RAVEN 2 → RAVEN 3, both MATLAB. How the Python package differs from the MATLAB one is a separate question, answered in -[RAVEN 3 and raven-toolbox](raven3-vs-raven-toolbox.md). +[RAVEN and raven-toolbox](raven3-vs-raven-toolbox.md). ::: Start with the [upgrade checklist](#upgrade-checklist) below, then read diff --git a/docs/raven3-vs-raven-toolbox.md b/docs/raven3-vs-raven-toolbox.md index 419b305..c2ad747 100644 --- a/docs/raven3-vs-raven-toolbox.md +++ b/docs/raven3-vs-raven-toolbox.md @@ -1,4 +1,10 @@ -# RAVEN 3 and raven-toolbox +# RAVEN and raven-toolbox + +:::{note} Looking for a specific function? +The [MATLAB vs Python](matlab-vs-python.md) table pairs every function that +exists in both, and links straight to the reference entry for each. This +page is the narrative comparison; that page is the lookup. +::: RAVEN exists as two independent implementations: @@ -90,7 +96,7 @@ rather than against whichever artefact is currently distributed. same release. What is Python-only is the baked registry that pins a given release to the exact assets it was tested against; RAVEN resolves from the published release each time. See - [Downloaded data and binaries](installation/data-and-binaries.md). + [Download data and binaries](installation/data-and-binaries.md). ## What only RAVEN has @@ -100,7 +106,7 @@ rather than against whichever artefact is currently distributed. them for the models already built with them; raven-toolbox, a new implementation with no such installed base, carries ftINIT alone. A tINIT model has to be reproduced in MATLAB; see -[10. Context-specific models](guide/init.md). +[14. Context-specific models](guide/init.md). ### COBRA Toolbox conversion @@ -223,42 +229,3 @@ Genome-scale MILP work (ftINIT extraction in particular) is where the choice matters most; see [Installation](installation/index.md) for the full solver matrix. -## Moving a model between them - -Models move through **SBML** (`.xml`) and **YAML** (`.yml`). raven-toolbox's -YAML follows the cobrapy layout plus RAVEN's own per-entry fields, so a model -written by either side round-trips through the other; see -[the YAML format reference](yaml-format.md) for the field-by-field spec and -interoperability matrix. - -Excel is **export-only** on both sides: raven-toolbox has never had a reader, -and RAVEN's `importExcelModel` was removed in the RAVEN 3 refactor. - -## Coming from RAVEN 2.0 to Python - -raven-toolbox is not a port of RAVEN 2.0; it is a new implementation that -made different design choices where RAVEN 2.0's design had become outdated. If you are -moving a RAVEN 2.0 workflow straight to Python rather than to RAVEN 3, three -differences matter beyond everything above. - -**The model data structure.** RAVEN 2.0 represents a model as a MATLAB struct -with `.rxns`, `.mets`, `.S`, `.lb`, `.ub`. raven-toolbox loads directly into a -`cobra.Model`; the struct format is not used internally at all, and solver calls -go through cobrapy's unified interface. - -**Packaging and typing.** raven-toolbox installs from PyPI -(`pip install raven-toolbox`), so environments are reproducible and CI is -straightforward. Every public function carries type annotations and passes -`mypy`. - -**MetaCyc reconstruction is gone, on both sides.** RAVEN 2.0 shipped -`getMetaCycModelForOrganism`; RAVEN 3 removed the whole `external/metacyc` -folder, and raven-toolbox never had it. The reason is not neglect: MetaCyc -provides a single representative sequence per enzyme, which gives intrinsically -low gene-calling precision, measured at roughly two-thirds of reaction -assignments wrong at the default cutoff, with no cutoff value that fixes the -problem. Use the KEGG or homology routes. - -What stayed the same are the algorithms: homology search, gap-filling and KEGG -reconstruction follow the same published methods, and models -move between all three through SBML and YAML. diff --git a/docs/references.md b/docs/references.md index fca8446..ce3e737 100644 --- a/docs/references.md +++ b/docs/references.md @@ -23,15 +23,7 @@ ## Context-specific extraction (INIT / ftINIT) -INIT is the method both `runINIT`/`getINITModel` (RAVEN's MATLAB -implementation, called tINIT) and `ftinit` (raven-toolbox, and RAVEN's -`ftINIT`) build on; tINIT has no citation of its own separate from the INIT -paper below. - -> Agren R, Bordel S, Mardinoglu A, Pornputtapong N, Nookaew I, Nielsen J -> (2012). **Reconstruction of genome-scale active metabolic networks for 69 -> human cell types and 16 cancer types using INIT.** *PLoS Computational -> Biology* 8(5): e1002518. +### ftINIT > Gustafsson J, Anton M, Roshanzamir F, Jörnsten R, Kerkhoven EJ, Robinson JL, > Nielsen J (2023). **Generation and analysis of context-specific genome-scale @@ -39,6 +31,17 @@ paper below. > the National Academy of Sciences* 120(6): e2217868120. > +### tINIT + +`runINIT`/`getINITModel` (RAVEN's MATLAB implementation of tINIT) build on +the INIT method described in the paper below; tINIT has no citation of its +own separate from it. + +> Agren R, Bordel S, Mardinoglu A, Pornputtapong N, Nookaew I, Nielsen J +> (2012). **Reconstruction of genome-scale active metabolic networks for 69 +> human cell types and 16 cancer types using INIT.** *PLoS Computational +> Biology* 8(5): e1002518. + ## Random sampling > Bordel S, Agren R, Nielsen J (2010). **Sampling the solution space in diff --git a/docs/tuned-parameters.md b/docs/tuned-parameters.md index 2513647..ea900e2 100644 --- a/docs/tuned-parameters.md +++ b/docs/tuned-parameters.md @@ -68,7 +68,7 @@ RAVEN MATLAB: `runINIT`, `scoreModel`, `getINITModel`, `ftINIT`, `prepINITModel` Only the MATLAB side has tINIT (`runINIT`, `getINITModel`); raven-toolbox implements ftINIT alone; see -[10. Context-specific models](guide/init.md). Rows below that were measured +[14. Context-specific models](guide/init.md). Rows below that were measured across both algorithms say so. | Parameter | Default | How determined | diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 8e503c7..6a60e42 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -24,10 +24,12 @@ folder of the RAVEN repository. ## Before you start - These tutorials use the **MATLAB** toolbox. Make sure RAVEN is installed and - `checkInstallation` passes; see [Installation](../installation/raven.md). -- Tutorials 2–4 involve editing models in **RAVEN-compatible Excel format**, so - you need a working `importExcelModel` (i.e. the Excel parser must pass - `checkInstallation`). + `checkRaven` passes; see [Installation](../installation/raven.md). +- Tutorials 2–4 involve editing models in **RAVEN-compatible Excel format**, + using `importExcelModel`. RAVEN 3 removed `importExcelModel`, so this step + does not run as written on RAVEN 3; see [Excel I/O](../raven3-migration.md#excel-io) + for the replacement, `curateModelFromTables`, which curates an existing + model from `.tsv` files rather than importing a full model from `.xlsx`. - To run a section of a script in MATLAB, highlight it, right-click, and choose *"Evaluate selection"*. - Tutorials 2, 3 and 4 ship with a `*_solutions.m` companion script containing