Massive Eiger data sets (i.e. > 100k frames) - some practical recommendations

The vast majority of Eiger data sets are sensible i.e. measured as would be with a Pilatus, but with a different file format. There is however the perfectly sensible approach of putting a crystal on, spinning with a relatively low transmission beam and collecting until dead, then deciding post mortem when to have finished the experiment. This could however end up with very large data sets, with one master file and (say) over 1,000 HDF5 data files.

A clue: this may not end well - for a start you could have > 200GB of data....

Anyhow, some hints.

1. ulimit -n 4096 
2. run on a machine with a lot of memory or
3. xia2 image=/path/to/master.h5:1:100800:3600

First increases the OS limit on the allowed number of file handles a process is entitled to. You will probably need a lot of RAM for e.g. indexing (because a lot of reflections) or split into evenly sized chunks with the syntax above (e.g. 1:360000:3600 if you did 100 turns, into one turn chunks) - this will make the processing much more swift though the scaling will still take a little while.

Whatever happens, with xia2 this is unlikely to be fast... if using 3d/i/ii pipeline, the use of the neggia plugin with XDS will appear in the next release with DIALS 1.7, which is due real soon now...

Eiger data sets > 10k frames

Turns out xia2 would potentially fail with this - see

https://github.com/xia2/xia2/issues/155

Now fixed - will be updated in next release of DIALS (1.6.4 probably)

Meanwhile will probably set out to replace other jiffies used in DIALS with CCTBX code [link] to make the system more future proof... if anyone has a student in need of a project, please get in touch!

Processing data to higher resolution than diffraction

If you are using xia2.small_molecule the default behaviour is to process every reflection and report the resolution limits observed, but include all reflections however weak in the output data. If you subsequently look at the I/sig(I) statistics and you have data to a lower limit than all reflections, you may see a substantial number of reflections with low or negative I/sig(I) i.e.


This is to be expected - these are essentially a population of noisy zero values, the noise a result of statistical errors from background subtraction. These should have no impact on refinement.

If this is not the behavior you want, set "keep_all_reflections=False" on the command line and only those data considered to be present will be included in the output.

xia2 command syntax: now changed

As per previous post the command syntax change branch has now been merged, if you get your xia2 from git your next git pull will be a big one and you may find e.g. -resolution / -atom etc don't work any more... The functionality is still there however, you just need to use atom=Se (say) or anomalous=true, d_min=1.6... full documentation will follow of the changes.

The pipeline options -3d. etc remain however are now deprecated but still work, you will get a note saying "please use pipeline=3d"

Please if you find any bugs as a side-effect of this contact xia2.support@gmail.com or add an issue at

https://github.com/xia2/xia2/issues

Please be aware: changing command syntax

At the moment we use a mix of command syntax, some using Phil (i.e. unit_cell=a,b,c,al,be,ga) and some using legacy arguments: we plan to completely remove the latter in favour of the former, see

https://github.com/xia2/xia2/issues/42

with the intention that using xia2 in an automated environment is more straightforward and making it much simpler to automatically maintain much more complete documentation of all the options.

This will be a feature of the 1.3 release of DIALS.

Eiger HDF5 data, with xia2

A common question over the past couple of months is when xia2 will work with Eiger HDF5 data natively. Well, in the dials 1.2 release now available from dials.github.io it now does: your mileage may however vary, there is a certain amount of variation in the file formats still, but I think we have tested with all of the machines running out there in the wild.

This is the result of a substantial amount of hard work from the dials and xia2 development teams, thank you to you all.

xia2 -dials spotfinding output

The latest xia2/dials nightly builds contain new output for the spotfinding step of xia2 -dials:

-------------------- Spotfinding SWEEP1 --------------------

102684 spots found on 1800 images (max 1854 / bin)

*****   **                             * * * ******  *     *

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

1                         image                         1800

These show the number of strong spots found on images during the spot finding and may provide insight into data processing problems due e.g. to sample misalignment. This is rather a nice example, you may see rather more variation than this with more typical data, but if you see a clear minimum anywhere in the set (i.e. results tend to 0) do not be surprised if the processing fails. 

For Diamond Light Source users, this is very closely related to the per-image-analysis performed during data collection and should provide a similar level of insight. It is no accident that similar output is now visible in the summary from DIALS spot finding:

Histogram of per-image spot count for imageset 0:

102684 spots found on 1800 images (max 1854 / bin)

*****   **                             * * * ******  *     *

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

************************************************************

1                         image                         1800

Processing compressed data

Even with modern machines today, X-ray diffraction data can still be big. However, pixel array data also compress well with e.g. gzip - say 20:1 on CBF files from a Pilatus detector. If only they could be processed compressed...

... well they can be. Actually xia2 can run just fine with compressed images:

xia2 -atom Zn image=/Volumes/DATA/data/thermc_1_0001.cbf.gz:1:1800

and works with XDS as well. In the case of this thermolysin data, which were kept on an external USB3 drive, the whole xia2 job was about 20% faster using the compressed data rather than the raw. Worth considering as it also saved about 95% of the storage space as well.

The effects for even bigger data sets could be more substantial, as the 1800 images were able to fit in the cache. If this was not the case the time saving would be greater as the compressed data could be cached but the raw no...

Worth thinking next time you complain about the amount of storage X-ray data takes up.

YMMV, not tested on non-CBF images, some software may not support this, ...

xia2: citing software xia2 has used

If you find xia2 useful for processing your data of course the developers will welcome any citations - however please also cite the software that xia2 has used on your behalf - the developers of these packages also want to be recognised!

xia2 will help you with this by including in the log file the appropriate list of citations e.g.

XIA2 used...  ccp4 distl labelit pointless xds xia2

Here are the appropriate citations (BIBTeX in xia-citations.bib.)

(1994) Acta Crystallogr. D 50, 760--763

Evans, Philip (2006) Acta Crystallographica Section D 62, 72--82

Kabsch, Wolfgang (2010) Acta Crystallographica Section D 66, 125--132

Sauter, Nicholas K. and Grosse-Kunstleve, Ralf W. and Adams, Paul D. (2004) Journal of Applied Crystallography 37, 399--409

Winter, G. (2010) Journal of Applied Crystallography 43

Zhang, Z. and Sauter, N.K. and van den Bedem, H. and Snell, G. and Deacon, A.M. (2006) J. Appl. Cryst 39, 112--119

if you run xia2 -3d - please do add these to the citations in the associated publications and to the PDB file.

New XDS: things not quite right

Now debugging xia2 for Oct 2015 version of XDS; not right yet please watch this space!

Update: XDS is just fine and a separate bug has crept into the development / nightly version, sorry for the noise.

With ongoing sf.net outage...

We are looking to move xia2 to github, so for those of you who pull xia2 from subversion this will be a significant change. Also the "home" of xia2 will change as well - though this blog will remain, so watch this space...

sf.net outage

You may have noticed that xia2 things are not working - the svn is dead; the nightly builds are empty etc. - this is because the hosting service I use (sourceforge.net) is currently broken and has been for getting on for four days. They're in the process of restoring services (web now back, but svn is not) and normal service should resume soon.

See here for more details.

Windows...

Does anyone use xia2 / DIALS on MS Windows? Would you want to? Was pointed out to me that the changes referred to in the previous post would not work gracefully on MS Windows due to the c:\ type file paths... this should probably be fixed anyhow. Shout if you have an opinion.

New logs in the LogFiles directory; more image= options

If you look in LogFiles and you use xia2 -dials you will find some new stuff in there:

Graemes-MacBook-Pro-2:image_range graeme$ ls LogFiles/

AUTOMATIC_DEFAULT_NATIVE_truncate.log AUTOMATIC_DEFAULT_pointless.log

AUTOMATIC_DEFAULT_aimless.log SWEEP1_INDEX.log

AUTOMATIC_DEFAULT_aimless_xml.xml SWEEP1_LATTICE.log

This now includes the indexing log file and the lattice determination one; these can be useful to see how xia2 made it's decisions.

Also, if you want to combine some data from several sweeps, but only use limited image ranges, you can with:

xia2 unify_setting=true -atom Ti -small_molecule -dials image=/Users/graeme/data/i04-KTP/1/KTP_1_0001.cbf:1:100 image=/Users/graeme/data/i04-KTP/2/KTP_2_0001.cbf:1:100

so this will use the first 100 images of two sweeps and will make this xinfo file:

BEGIN PROJECT AUTOMATIC

BEGIN CRYSTAL DEFAULT

BEGIN HA_INFO

ATOM ti

!NUMBER_PER_MONOMER N

!NUMBER_TOTAL M

END HA_INFO

BEGIN WAVELENGTH SAD

WAVELENGTH 0.729320

END WAVELENGTH SAD

BEGIN SWEEP SWEEP1

WAVELENGTH SAD

DIRECTORY /Users/graeme/data/i04-KTP/1

IMAGE KTP_1_0001.cbf

START_END 1 100

END SWEEP SWEEP1

BEGIN SWEEP SWEEP2

WAVELENGTH SAD

DIRECTORY /Users/graeme/data/i04-KTP/2

IMAGE KTP_2_0001.cbf

START_END 1 100

END SWEEP SWEEP2

END CRYSTAL DEFAULT

END PROJECT AUTOMATIC

Word of warning though that this will currently (as of noon today) only cope with one subset per sweep; should be easy enough to add multiple image ranges for this though need to think what syntax to use... as always feedback welcome to xia2.support@gmail.com; to get this you will need to pull a new DIALS nightly build later today or tomorrow.

More output: how much is too much?

Request of additional output from xia2 e.g. in the indexing table:

------------------- Autoindexing SWEEP1 --------------------

All possible indexing solutions:

oP   6.39  10.59  12.80  90.00  90.00  90.00

mP   6.39  12.80  10.59  90.00  89.99  90.00

aP   6.39  10.59  12.80  90.00  90.00  89.99

Indexing solution:

oP   6.39  10.59  12.80  90.00  90.00  90.00

adding the R.M.S.D., metric penalty etc. from for example dials.index.  This will however make for more "junk" in the output and will also depend on the package being run underneath. Also requested was number of spots indexed vs. total number of spots. This will make for more information in the output which could make it cluttered and will also have the potential to "break" other software which uses this.

Thoughts anyone?

Free set reflections

Following a question on the CCP4 BB about how many reflections to set aside for the free set (i.e. the FreeR_flag column xia2 adds to the MTZ file) it looks like the conclusion is "data processing packages should not do this" - OK, no problem. That makes the original question go away; always good...

Am keen to now remove this capability from xia2; comments to xia2.support@gmail.com please!

New ability to specify multiple directories or images via command line

As of the latest DIALS/xia2 nightly build (2015-04-24 onwards, available from http://dials.diamond.ac.uk/diamond_builds/), xia2 now supports passing multiple directories or images via the command line, e.g.:

xia2 -dials /path/to/directory1 /path/to/directory2
xia2 -dials image=/path/to/directory1/image_1_0001.cbf image=/path/to/directory2/image_2_0001.cbf

Nasty subtle bug with latest XDS

The default behaviour of XDS with REFINE= in the CORRECT step appears to have changed, which highlights a long-standing bug in xia2. The bug is that the REFINE= card was incorrectly formatted, which some of you have noticed in that the summary from the second pass of integration looks much more nasty than the first pass. This apparently was not a problem with the older version of XDS (before 1mar2015). I had initially (incorrectly) blamed the new version of XDS but Richard Gildea found the underlying bug.

A new version of xia2 will be available already with these fixes in as part of the DIALS nightly builds. The fix is however a one-liner:

In

xia2/Wrappers/XDS/XDSCorrect.py

change

'REFINE(CORRECT)=%s\n' %self._params.refine)

to 

'REFINE(CORRECT)=%s\n' %' '.join(self._params.refine))

(amazing how much effect these little changes can make!) We are in the process if instigating a more thorough testing procedure which will ensure errors like this do not occur again.

Experimental xia2/DIALS installers

We are currently testing new xia2/DIALS installers for mac/linux which provide a self-contained xia2/cctbx/DIALS installation. Instructions on downloading and installing these bundles can be found at http://dials.diamond.ac.uk/doc/installation.html. Once installed, simply sourcing the dials_env.sh script in the installation directory will make available all xia2 and DIALS commands in the current terminal. You will also need CCP4 available - but make sure to source the dials_env.sh script after the CCP4 setup script. You can also try out the new xia2 -dials option, which will use the new software DIALS to index and integrate your data, followed by scaling and merging with Aimless. As always, please send any feedback to xia2.support@gmail.com.

New wavelength_tolerance parameter

A common problem when processing multiple datasets with xia2 is when a small difference in the wavelength recorded in the image headers causes the sweeps to be grouped into more than one wavelength, when the desired outcome was for them to be processed as a single wavelength. Previously it was necessary to manually edit the .xinfo file to tell xia2 to process them as a single wavelength. Now we have added a new parameter wavelength_tolerance (default=0.0002 Å) which you can use to control whether or not different wavelengths get grouped together as a single wavelength.

Example usage:

xia2 -3dii wavelength_tolerance=0.001 /path/to/images/