Hi everyone,I work at a university institute of computational biology. Besides doing research, we also teach quite a few courses for biology students, many of which include an introduction to programming (mostly with R, but also Python). We have a long-standing debate as to what the best approach to this is, and I would like to hear some of your opinions on the matter.
Basically, we have two factions: the “tools-first” and the “fundamentals-first” approach. The supporters of “tools-first” argue that we are teaching biologists, not software developers. They like to teach the specific tools (languages, libraries, functions, etc.) that our students are actually going to need as quickly as possible. To cover as much ground as possible, they are willing to sacrifice a deeper understanding of programming.
Ask HN: Teaching programming in computational biology?
Obey the Testing Goat!
Source: The Book
Athanassios Fokas, a mathematician from the Department of Applied Mathematics and Theoretical Physics of the University of Cambridge and visiting professor in the Ming Hsieh Department of Electrical Engineering at the USC Viterbi School of Engineering has announced the solution of one of the long-standing problems in the history of mathematics, the Lindelöf Hypothesis.
The solution, first published in arXiv, has far reaching implications for fields like quantum computing, number theory, and encryption which forms the basis for cybersecurity.
In order to make our time together as effective as possible, you’ll need to do a bit of homework before coming to the workshop session: download a data bundle and get GATK4 installed on your laptop. To be clear, you will not have time to do this at the start of the session so it’s imperative that you do this ahead of time.
1) Download the “gatk_bundle.zip” data bundle containing data that we will use in the hands-on exercises:
Direct link: https://drive.google.com/open?id=1ixEFNgWQBf79eRVqhKv9OGpXzGzcrbn2
Enclosing folder (containing additional material for further reading): https://drive.google.com/drive/folders/1U6Zm_tYn_3yeEgrD1bdxye4SXf5OseIt?usp=sharing
2) Install Docker on your laptop and download the GATK4 container image, which contains all the system dependencies needed to run GATK4. Please follow the instructions provided here:
If for whatever reason you are unable to follow the docker installation instructions, the recommended alternative is to use the Conda environment that we provide to manage dependencies, as described in the github repository README:
And if that doesn’t work for you, for the purposes of this workshop you can just get the GATK package, as long as you make sure you have Java 8 installed on your laptop: https://github.com/broadinstitute/gatk/releases/download/18.104.22.168/gatk-22.214.171.124.zip
Thank you and see you in Portland!
GATK Haplotype Called
CNN, gCNV Germline CNV Calling
Probabilistic Graphical Models
THIS IS THE GATK WORKSHOP BUNDLE FOR MARCH 2018
The materials are now ready for download. The gatk_bundle.zip package contains the data that is sed in the hands-on exercises. The “worksheets” directory contains the exercise instructions. The “dayX” directories contain all the presentation slide decks from the workshop.
== LINKS FOR SHARING ==
PDFs and gatk_bundle: http://broad.io/gatk-1803
Installation prep: https://broad.io/gatk-w-prep
PairHMM depends on the machine you are running on.
15k genomes in 2 weeks.
76k genomes WGS processing
GenomicsDB gives 100k genomes, but still need some work for doing more than that.
docker run -v /path/gatk_data
Somatic Variant Analysis
Call Variants per Sample
Haplotype Caller in GVCF mode
Google Cloud Platform
Not only of for the cloud
docker run -v /home/raony/gccbosc/gatk/gatk_bundle/2-germline/:/gatk/gatk_data -it broadinstitute/gatk:126.96.36.199
gatk HaplotypeCaller -R /gatk/gatk_data/ref/ref.fasta -I /gatk/gatk_data/bams/mother.bam -O /gatk/gatk_data/sandbox/variants.vcf
Using GATK jar /gatk/build/libs/gatk-package-188.8.131.52-local.jar
gatk ValidateSamFile -I bams/mother.bam -MODE SUMMARY
gatk –java-options “-Xmx4G” MarkDuplicatesSpark -R ref/ref.fasta -I bams/mother.bam -O sandbox/mother_dedup.bam -M sandbox/metrics.txt — –spark-master local[*]
Using GATK jar /gatk/build/libs/gatk-package-184.108.40.206-local.jar
gatk –java-options “-Xmx4G” HaplotypeCaller -R /gatk/gatk_da/ref/ref.fasta -I /gatk/gatk_data/bams/mother.bam -O /gatk/gatk_data/sandbox/mother.g.vcf -ERC GVCF
gatk –java-options “-Xmx4G” HaplotypeCaller -R /gatk/gatk_da/ref/ref.fasta -I /gatk/gatk_data/bams/father.bam -O /gatk/gatk_data/sandbox/father.g.vcf -ERC GVCF
10reads of difference beetwen markduplicates, markduplicatesspark, they are trying to explain that.
7 different levels of certification
Stringent Options Available
gatk MarkDuplicatesSpark -R gs://gatk-workshops/GCCBOSC2018/ref/ref.fasta -I gs://gatk-workshops/GCCBOSC2018/ref/ref.fasta -O mother_dedup.bam -M metrics.txt — –spark-runner GCS –cluster aardvark-01
Galaxy Release Schedule
3 releases per year: January, May and September
Install Galaxy using Ansible
sudo pip install ansible
git clone https://github.com/ARTbio/GalaxyKickStart
git checkout 2018-gccbosc
ansible-galaxy install -r requirements_roles.yml -p roles –force
$ sudo su galaxy $ vi /srv/galaxy/config/galaxy.yml # Add the following line under galaxy: section admin_users: email@example.com $ exit # change back to ubuntu user $ sudo supervisorctl restart galaxy:
|09:00||Welcome and introduction||Slides||(Č)|
|09:15||Deployment and platform options||Slides||(Č)|
|9:30||Using Ansible to deploy Galaxy||Slides, Exercise||(E)(G)|
|10:20||Extending installation||Slides, Exercise||(G)|
|10:40||Defining and importing genomes, Data Managers||Slides, Exercise||(E)|
|11:15||Web Servers nginx/Apache||Slides||(M)(N)|
|11:30||Close Morning Session|
Galaxy admin -> local data: Create DBKey and Reference Genome – fetching
Install dbkey from saccer2 data_manager_fetch_genome_dbkeys_all_fasta
Install BWA data_manager_bwa_mem_index_builder
Admin -> create bwa index
ubuntu@2018-gcc-training-0:~⟫ sudo vim /srv/galaxy/config/galaxy.yml
#nginx_x_accel_redirect_base: False and change it to
nginx_x_accel_redirect_base: /_x_accel_redirect. Remember, this file is owned by the galaxy user so be sure to use
sudo -u galaxy when editing it.
sudo supervisorctl restart nginx galaxy:
Google’s PageSpeed Tools can identify any compression or caching improvements you can make.
If configuring SSL (out of scope for this training), out-of-the-box SSL settings are often insecure!
$ planemo test –no-container –engine toil seqtk_seq.cwl
#this will open the browser
Correspond to job runner plugins in lib/galaxy/jobs/runners
- Slurm (DRMAA subclass)
- DRMAA: SGE, PBS Pro, LSF, Torque
- Torque: Using the
- Pulsar: Galaxy’s own remote job management system
- Command Line Interface (CLI) via SSH
Need a shared file system, nfs, ceph and etc.
Exception is Pulsar!
sudo cat job_conf.xml.sample_basic <?xml version="1.0"?> <!-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). --> <job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/> </plugins> <destinations> <destination id="local" runner="local"/> </destinations> </job_conf>
--- tools: multi: rules: - rule_type: file_size lower_bound: 16 upper_bound: Infinity destination: slurm-2c default_destination: slurm_cluster default_destination: local_no_container verbose: True
I think that the time has come to start talking seriously about bringing async functionality into Django itself, and so I have been working on a draft “roadmap” for what I think this might look like. I’ve run this past a few people – some of who were Django core members, and some who weren’t – but I’m now posting it up for public feedback (see the end for where to discuss this).
Machine learning offers a fantastically powerful toolkit for building complex systems quickly. This paper argues that it is dangerous to think of these quick wins as coming for free. Using the framework of technical debt, we note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning. The goal of this paper is highlight several machine learning specific risk factors and design patterns to be avoided or refactored where possible. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, changes in the external world, and a variety of system-level anti-patterns.
Machine Learning: The High Interest Credit Card of Technical Debt
Biology needs revolution. All my adult life, I have been lost with admirationfor the achievements in molecular biology and genetics, and I have come toknow many of the main proponents. Yet there is an alternative aspect: instudying the minutiae, we have lost sight of the whole cell as organism.Living cells within the body are modelled in this paper as coordinated butessentially autonomous entities. We shall see how independent cells innature have remarkable abilities to make decisions and take constructiveaction, which correlate with the definitions of intelligence.
An influential committee of the European Parliament is due to vote this month on changes to copyright regulations in the European Union, but the latest drafts of the rules have triggered a wave of criticism from open-science advocates. They say that the proposals will stifle research and scholarly communication.