Processing compressed data

Even with modern machines today, X-ray diffraction data can still be big. However, pixel array data also compress well with e.g. gzip - say 20:1 on CBF files from a Pilatus detector. If only they could be processed compressed...

... well they can be. Actually xia2 can run just fine with compressed images:

xia2 -atom Zn image=/Volumes/DATA/data/thermc_1_0001.cbf.gz:1:1800

and works with XDS as well. In the case of this thermolysin data, which were kept on an external USB3 drive, the whole xia2 job was about 20% faster using the compressed data rather than the raw. Worth considering as it also saved about 95% of the storage space as well.

The effects for even bigger data sets could be more substantial, as the 1800 images were able to fit in the cache. If this was not the case the time saving would be greater as the compressed data could be cached but the raw no...

Worth thinking next time you complain about the amount of storage X-ray data takes up.

YMMV, not tested on non-CBF images, some software may not support this, ...

xia2: citing software xia2 has used

If you find xia2 useful for processing your data of course the developers will welcome any citations - however please also cite the software that xia2 has used on your behalf - the developers of these packages also want to be recognised!

xia2 will help you with this by including in the log file the appropriate list of citations e.g.

XIA2 used...  ccp4 distl labelit pointless xds xia2

Here are the appropriate citations (BIBTeX in xia-citations.bib.)

(1994) Acta Crystallogr. D 50, 760--763

Evans, Philip (2006) Acta Crystallographica Section D 62, 72--82

Kabsch, Wolfgang (2010) Acta Crystallographica Section D 66, 125--132

Sauter, Nicholas K. and Grosse-Kunstleve, Ralf W. and Adams, Paul D. (2004) Journal of Applied Crystallography 37, 399--409

Winter, G. (2010) Journal of Applied Crystallography 43

Zhang, Z. and Sauter, N.K. and van den Bedem, H. and Snell, G. and Deacon, A.M. (2006) J. Appl. Cryst 39, 112--119

if you run xia2 -3d - please do add these to the citations in the associated publications and to the PDB file.

New XDS: things not quite right

Now debugging xia2 for Oct 2015 version of XDS; not right yet please watch this space!

Update: XDS is just fine and a separate bug has crept into the development / nightly version, sorry for the noise.

With ongoing sf.net outage...

We are looking to move xia2 to github, so for those of you who pull xia2 from subversion this will be a significant change. Also the "home" of xia2 will change as well - though this blog will remain, so watch this space...

sf.net outage

You may have noticed that xia2 things are not working - the svn is dead; the nightly builds are empty etc. - this is because the hosting service I use (sourceforge.net) is currently broken and has been for getting on for four days. They're in the process of restoring services (web now back, but svn is not) and normal service should resume soon.

See here for more details.

Windows...

Does anyone use xia2 / DIALS on MS Windows? Would you want to? Was pointed out to me that the changes referred to in the previous post would not work gracefully on MS Windows due to the c:\ type file paths... this should probably be fixed anyhow. Shout if you have an opinion.

New logs in the LogFiles directory; more image= options

If you look in LogFiles and you use xia2 -dials you will find some new stuff in there:

Graemes-MacBook-Pro-2:image_range graeme$ ls LogFiles/

AUTOMATIC_DEFAULT_NATIVE_truncate.log AUTOMATIC_DEFAULT_pointless.log

AUTOMATIC_DEFAULT_aimless.log SWEEP1_INDEX.log

AUTOMATIC_DEFAULT_aimless_xml.xml SWEEP1_LATTICE.log

This now includes the indexing log file and the lattice determination one; these can be useful to see how xia2 made it's decisions.

Also, if you want to combine some data from several sweeps, but only use limited image ranges, you can with:

xia2 unify_setting=true -atom Ti -small_molecule -dials image=/Users/graeme/data/i04-KTP/1/KTP_1_0001.cbf:1:100 image=/Users/graeme/data/i04-KTP/2/KTP_2_0001.cbf:1:100

so this will use the first 100 images of two sweeps and will make this xinfo file:

BEGIN PROJECT AUTOMATIC

BEGIN CRYSTAL DEFAULT

BEGIN HA_INFO

ATOM ti

!NUMBER_PER_MONOMER N

!NUMBER_TOTAL M

END HA_INFO

BEGIN WAVELENGTH SAD

WAVELENGTH 0.729320

END WAVELENGTH SAD

BEGIN SWEEP SWEEP1

WAVELENGTH SAD

DIRECTORY /Users/graeme/data/i04-KTP/1

IMAGE KTP_1_0001.cbf

START_END 1 100

END SWEEP SWEEP1

BEGIN SWEEP SWEEP2

WAVELENGTH SAD

DIRECTORY /Users/graeme/data/i04-KTP/2

IMAGE KTP_2_0001.cbf

START_END 1 100

END SWEEP SWEEP2

END CRYSTAL DEFAULT

END PROJECT AUTOMATIC

Word of warning though that this will currently (as of noon today) only cope with one subset per sweep; should be easy enough to add multiple image ranges for this though need to think what syntax to use... as always feedback welcome to xia2.support@gmail.com; to get this you will need to pull a new DIALS nightly build later today or tomorrow.