FAQ

You will find here frequently asked questions, organized by thematics

FAQ

migale

Author

Migale team

Published

March 9, 2023

Modified

November 19, 2024

Access

How to connect to front server?

Once your account is activated, you can log in different ways.

Terminal

ssh -X login@front.migale.inrae.fr

Windows

For Windows, you need to use a terminal emulator tool such as MoBaXterm.

How to change my password?

The only way to change your password is to fill the dedicated form.

If you lost your password, you can use the “Mail” button in the form.. Upon completion of your Migale login and submission of the form, it will send you a link to reinitialize your password to the mail you provide us at the account creation.

If none of the above works, send a mail to help-migale@inrae.fr. We will attribute you a new password and then you can change it easily with the previous link.

How to copy a file from my computer to front server?

Imagine you want to copy the file myfile.txt to the directory /projet/tmp/ on the migale server.

Terminal

scp myfile.txt stage01@front.migale.inrae.fr:/projet/tmp/

The password will be asked.

For copying a directory, add the -r option after scp.

scp -r mydir stage01@front.migale.inrae.fr:/projet/tmp/

Windows

Conda

How to show all conda environments?

conda info --envs
# conda environments:
#
base                  *  /usr/local/genome/Anaconda2-5.1.0
BasicQC                  /usr/local/genome/Anaconda2-5.1.0/envs/BasicQC
abyss-2.2.1              /usr/local/genome/Anaconda2-5.1.0/envs/abyss-2.2.1
admixture-1.3.0          /usr/local/genome/Anaconda2-5.1.0/envs/admixture-1.3.0
amos-3.1.0               /usr/local/genome/Anaconda2-5.1.0/envs/amos-3.1.0
...

How to find a specific conda environment name?

Imagine I want to know the name of the bwa conda environment. I can find it by typing:

conda info --envs | grep bwa
bwa-0.7.17 /usr/local/genome/Anaconda2-5.1.0/envs/bwa-0.7.17

How to find a specific conda environment name?

To use bwa with conda, you need to activate the bwa environment

stage01@front:~$ conda activate bwa-0.7.17
(bwa-0.7.17) stage01@front:~$ bwa

Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.17-r1188
Contact: Heng Li

Usage:   bwa command [options]

Command: index         index sequences in the FASTA format
mem           BWA-MEM algorithm
fastmap       identify super-maximal exact matches
pemerge       merge overlapping paired ends (EXPERIMENTAL)
aln           gapped/ungapped alignment
samse         generate alignment (single ended)
sampe         generate alignment (paired ended)
bwasw         BWA-SW for long queries

shm           manage indices in shared memory
fa2pac        convert FASTA to PAC format
pac2bwt       generate BWT from PAC
pac2bwtgen    alternative algorithm for generating BWT
bwtupdate     update .bwt to the new format
bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.
There are three alignment algorithms in BWA: `mem', `bwasw', and
`aln/samse/sampe'. If you are not sure which to use, try `bwa mem'
first. Please `man ./bwa.1' for the manual.

and to leave conda environment:

conda deactivate

How to use a conda environment within my snakemake workflow?

To use a conda environment, it is necessary to tell snakemake to load the profile.d/conda.sh. To do this, add these lines at the beginning of your Snakefile:

shell.executable("/bin/bash")
shell.prefix("source /usr/local/genome/Anaconda3/etc/profile.d/conda.sh;")

In your snakemake rules, you can now use the conda environments with: conda activate my_env. As a reminder, the list of available environments is accessible with:

conda info --envs

Moreover, if you launch snakemake inside a shell script, you need add the following line at the beginning of your script:

source /usr/local/genome/Anaconda3/etc/profile.d/conda.sh

This is a minimal example for a rule that launches fastqc:

shell.executable("/bin/bash")
shell.prefix("source /usr/local/genome/Anaconda3/etc/profile.d/conda.sh;")

rule all:
    input:
        expand("qc/fastqc/{sample}_fastqc.html", sample = ["A", "B", "C"])

rule fastqc:
    input:
        "reads/{sample}.fastq.gz"
    output:
        html = "qc/fastqc/{sample}_fastqc.html",
        zip = "qc/fastqc/{sample}_fastqc.zip"
    params: "--quiet"
    log: "logs/fastqc/{sample}.log"
    threads: 1
    shell:
        """
        conda activate fastqc-0.11.9
        fastqc {params} --threads {threads} --outdir qc/fastqc/ {input} > {log}
        conda deactivate
        """

More information in the Snakemake FAQ.

If you obtain this error message:

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init 

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

check if you have a section called conda initialize in the file ~/.bashrc and remove it.

How to use a conda environment within my Python script ?

The simplest way of using a conda environement in a Python script is by using conda runinstaead of conda activate. conda activate``is designed to work with inetractive environment likebashshell,conda run` is meant to be used in non inetractive env as Python scripts.

An example

#! /bin/env python
import os
return_message = os.system("conda run -n samtools-1.14  samtools version")
print("`conda` ran with exit code %d" % return_message)

How to use multiple threads?

To use multiple threads, you have to specify it by adding -pe thread to your submission command.

Example:

qsub -pe thread 8 myscript.sh

Warning

The number of threads has to be coherent with the usage of the jobs. If the tool is not using multiple threads, you don’t need to use this option!

What is the meaning of the Eqw state of my job?

The Eqw state indicates that your job is in error state. To obtain the reason why, use the qstat command with your job-ID.

stage01@front:~$ qsub -cwd -V -N myblast -o /out/myblast.stdout -b y "blastn -db subject.fasta -query query.fasta -out /out/out.blast"
Your job 1995192 ("myblast") has been submitted
stage01@front:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
1995192 0.00000 myblast    stage01         Eqw    10/04/2019 17:11:46                                    1        

stage01@front:~$ qstat -j 1995192
...
error reason          1:      10/16/2019 14:35:39 [46342:142027]: error: can't open output file "/out/myblast.stdout": No such file or directory
...

Why my job never starts?

login@front:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
2039350 0.60500 myblast    stage01         qw    10/16/2019 14:44:34                                   100

In some cases, jobs are never started and will never be started. In this example, the number of reserved threads is higher than what the nodes offer. As a result, SGE waits until it has a 100-slot node at its disposal to start your job. To check if the waiting is normal, use the qstat command with the job number to check that there is no warning as in this example. You will find here the details about our resources.

login@front:~$ qstat -j 2039350
...
cannot run in PE "thread" because it only offers 0 slots
...

How do I know when my jobs are finished?

If you leave your session, you can be informed by mail when your jobs are finished using the specific options offered by SGE (command qsub):

-m ea will send email when job ends or aborts
-M <emailaddress> email address to send email to

If you keep working on front, you can use the command watch. watch is used to run any arbitrary command at regular intervals and displays the output of the command on the terminal window. It is useful when you have to execute a command repeatedly and watch the command output change over time. In this case, the output of qstat will be empty.

watch -n 5 "qstat"
# Will run qstat every 5 seconds

How to call SGE directly from R

The purpose of the future package is to provide a very simple and uniform way of evaluating R expressions asynchronously using various resources available to the user. A full documentation is available here: https://future.futureverse.org/.

To work on the front server and use the computing farm, you need to create a file called ~/.batchtools.sge.tmpl with this content:

#!/bin/bash

## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>

## Combining output/error messages into one file
#$ -j y

## Giving the name of the output log file
#$ -o <%= log.file %>

## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd

## Use environment variables
#$ -V

## Use correct queue
#$ -q <%= resources$queue %>

## Use threads
#$ -pe thread <%= resources$threads %>

## Reserve resources
#$ -R y

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

and here is a small example:

library(future)
library(future.batchtools)
library(listenv)

plan(batchtools_sge, 
     workers = 10,     
     template = "~/.batchtools.sge.tmpl",
     resources = list(queue = "short.q", threads = 1)
)

results <- listenv()
create <- function(mu = 0) { rnorm(1000, mean = mu) }
analyze <- function(x) { mean(x) }
for (i in 1:10) { 
  results[[i]] %<-% analyze(create())
}
results <- unlist(as.list(results))

My quota does not decrease despite the deletion of datasets!

When you delete a dataset, it is not really deleted but hidden. It is a security in case of misuse. You can restore them. To delete them permanently, click on the button Delete permanently from disk. The quota will then be updated.

Why can’t I build a workflow from my history?

The most common reason why Galaxy cannot build a workflow from a history is that one of the tools used has been updated and parameters have changed. The only solution is to build a worflow from scratch.

There is an error message at Galaxy’s address!

If you get an error message by connecting to Galaxy’s address such as Proxy Error or Service is unavailable or the navigation is very slow, contact us at help-migale@inrae.fr Galaxy has probably crashed, we may need to restart it.

How to archive and compress files?

The tar command allows you to archive several files or the entire contents of a folder in a single file. The z parameter requires the tar file to be compressed so that it takes up less disk space.

Create the .tar.gz archive file:

tar cvzf archive_name.tar.gz folder_to_archive

Untar and uncompress tar.gz archive file

tar xvfz archive_name.tar.gz

Warning

As a matter of habit, we note .tar to indicate that it is an archive and .gz to indicate that the file has been compressed with gzip.

tar usage and options

c: create a archive file
x: extract a archive file
v: show the progress of archive file
f: filename of archive file
t: viewing content of archive file
j: filter archive through bzip2
z: filter archive through gzip

How to Fix the ‘No Space Left on Device’ Error?

It means you have no space left on the directory you want to write. You need to check your quotas with command get_quota:

stage01@front:~$ get_quota
Information for stage01 at mar. 01 févr. 2022 09:45:22 CET
Home  :  110.8MB  /  1GB    (11%)
Save  :  0B       /  50GB   (0%)
Work  :  8KB      /  150GB  (0%)

And to see details by folder you can use the du command:

stage01@front:~$ du -sh ~/myfolder/*
1,7G    /home/stage01/myfolder/checkm-genome-1.0.18
8,0K    /home/stage01/myfolder/checkm_test
8,0K    /home/stage01/myfolder/checkm_test_results
4,0K    /home/stage01/myfolder/dir1
...

I can not connect to Rstudio, what happens?

Problems with Rstudio are usually due to its cache taking up all the available space on your home directory on the front server. You can find out how much disk space you’re using with the get_quota command on the front server

get_quota userid
Information for userid at Mon Jun 5 11:46:18 CEST 2023
Home: 1.03GB / 1GB (103%)
Save: 33.48GB / 50GB (67%)
Work: 14.86GB / 150GB (10%)

You can delete the directory /home/userid/.local/share/rstudio which contains the cache of your sessions, then try to connect again.

If you are not familiar with command line, send an email to help-migale@inrae.fr.

A work by Migale Bioinformatics Facility
Université Paris-Saclay, INRAE, MaIAGE, 78350, Jouy-en-Josas, France
Université Paris-Saclay, INRAE, BioinfOmics, MIGALE bioinformatics facility, 78350, Jouy-en-Josas, France