CCP4 Study Weekend 2012

I'll be at the CCP4 study weekend in the new year to talk about xia2 - in the main session as well as the lunchtime bytes. If you've any ideas of things you'd like to know about in the lunchtime bytes please make a note below or drop me a line at the usual address...

See you there!

Aimless for merging data from XDS (N.B. not yet scaling!) now working

Bloody edge of svn has this working, which is useful if like me you have better things to be doing than recompiling Scala a lot :o)

Will be in forthcoming 0.3.4 release. Any time now.

Aimless

Have been working on including Aimless from Phil Evans (his "Pointless" replacement for Scala) and if you would like to try this out, the code is now in the trunk. Usual svn checkout, download aimless 0.1.4 or later from Phil's FTP site, then run -2da.

Early results show that it makes the overall xia2 pipeline quite a bit faster, while maintaining the quality of results. Worth looking at.

xia2 0.3.3.3 release

Hi Folks,

0.3.3.3 release of xia2 is now available - which includes a bunch of useful fixes but fundamentally no new functionality:

Changes since 0.3.3.2

Many bug fixes as a result of implementing a comprehensive testing scheme.
Support for full cbf format, making use of cbflib Python bindings pycbf now included in cctbx releases.
Merged functionality from XDSIntegraterR into XDSIntegrater, removed duplicate code. Will make for more maintainable suite in the future.
Improved 2d pipeline scaling, now tests 8 permutations.
Improved handling of twinned data for multiple sweeps.
Fixed occasional bugs with multiple sweeps, reindexing trigonal data sets with XDS.
Removed "old" scaling modes, 2d and 3d, as these were unhelpful.
Changes since 0.3.3.1

-xparallel -1 option added which will split your data into 30 degree chunks and process with forkintegrate Note well you are responsible for setting up forkintegrate correctly!

Extensive testing

So, finally I have set up a system for doing bulk tests, which is surprisingly illuminating. Results of this will follow, but if you have a data set where xia2 fails but manual processing works please get in touch - I will be happy to add it to the test system and work on making it work.

CCP4 6.2.0

If you try running xia2 with CCP4 6.2.0 and XDS it probably won't work, and will complain about something like "End of file".

If you get this, please get a new pointless from

http://www.ccp4.ac.uk/problems.php#6.2.0-programs

and everything should be fine.

Addendum:

My bad: this version is still buggy - better to get a version from ftp://ftp.mrc-lmb.cam.ac.uk/pub/pre and overwrite the ccp4 one with this.

Does anyone use xia2-summary.dat?

Just found this:

[gw56@ws050 3]$ cat xia2-summary.dat
Project: AUTOMATIC
Crystal: DEFAULT
Sequence length: 0
Wavelength: WAVE1 (0.97966)
Sweep: SWEEP1
Files /data2/gw56/jcsg/12287_2L/12287_1_E1_###.img
Images: 1 to 60
Beam 105.10 101.05 => 108.95 105.04
Distance 170.00 => 169.89
Date: Sun Sep 26 14:13:49 2004
Wavelength: WAVE2 (1.00000)
Sweep: SWEEP2
Files /data2/gw56/jcsg/12287_2L/12287_1_E2_###.img
Images: 1 to 60
Beam 105.10 101.05 => 108.93 105.04
Distance 170.00 => 170.09
Date: Sun Sep 26 14:17:56 2004
For AUTOMATIC/DEFAULT/WAVE1:
High resolution limit                           1.66    7.41    1.66
Low resolution limit                            36.8    36.8    1.7
Completeness                                    97.9    96.4    85.7
Multiplicity                                    4.3     3.5     2.9
I/sigma                                         16.5    33.9    2.2
Rmerge                                          0.043   0.023   0.372
Anomalous completeness                          94.3    92.4    78.2
Anomalous multiplicity                          2.4     2.3     1.6
For AUTOMATIC/DEFAULT/WAVE2:
High resolution limit                           1.65    7.39    1.65
Low resolution limit                            36.8    36.8    1.7
Completeness                                    96.3    96.4    74.5
Multiplicity                                    4.2     3.5     2.7
I/sigma                                         17.4    35.6    2.2
Rmerge                                          0.04    0.021   0.369
Anomalous completeness                          92.1    94.2    69.9
Anomalous multiplicity                          2.3     2.3     1.5
Cell:  51.570  51.570 157.600  90.000  90.000  90.000
Spacegroup: P 41 21 2

which I remember from a long time ago - just wondering, does anyone use this? Is it worth keeping (as I clear up...)

Issues with xia2 / 3d / pointless 1.6.1

Many thanks to bug reporters. If you see:

Status: error "End of File "

at the end of a xia2 job which looks like it should have worked, and you have recently updated your pointless version, you have hit a feature resulting from a small change in pointless. Essentially when copying reflections from XDS to MTZ format there is no real orientation matrix, so pointless makes one up. It used to be [1] and was changed to [0] as it is meaningless - however reindex relies on this and therefore explodes.

Phil has fixed this in 1.6.2 (thanks Phil!) and is quite reasonably of the opinion that this is a bug in reindex...

A subsequent question to this is why I still use reindex inside xia2 now pointless does it all. "Historical reasons" would be the fair answer i.e. I haven't got around to fixing this :o)

For followers of the bleeding edge...

You may see an error like this:

ImportError: No module named pycbf

which would mean you need to get an up-to-date build of cctbx - sorry. I have started to use pycbf (clearly) and I had not thought to mention it as no released versions include this dependency. I neglected to think that some people use the svn...

http://cctbx.sf.net is where to get an updated build from! These now include cbflib and pycbf which will give added awesomeness in the future.

Lots of frames? Then things to recompile

In $CPROG/scala_/parameter.fh reset parameter maxbat to (say) 15000

integer maxbat
parameter (maxbat=15000)

Likewise in $CPROG/reindex.f

integer mbatch, mcols
parameter (mbatch=15000, mcols=500)

There could be more...

xia2 at the ACA

So, I will be at the ACA this year, and have a presentation in the multi-crystals session - however if you have any questions or comments feel free to catch me. I will aim to be at the CCP4 stand between sessions.

See you there!

Polycrystal data reduction errors?

If you see:

Status: error "could not find status"

When you are running data reduction from lots of crystals with xia2, you probably need to recompile Scala. Particularly, if you see:

 Rmerge against resolution for each run (deviation from mean I+ or I-)

 $TABLE: Rmerge for each run, WAVE1 :
At line 17263 of file /usr/local/xtal/ccp4-6.1.3/src/scala_/scala.f
Fortran runtime error: End of file

at the end of the last Scala log file mentioned in xia2-debug.txt, you will need to edit around that location in $CPROG/scala_/scala.f and make

 6142  format (
     $     ' $TABLE: Rmerge for each run',a,' :')
      write (line, 6141) (i+3,i=1,nrunp1)
 6141 format(
     $     ' $GRAPHS:Rmerge v. resolution:N: 2,',(20(i2,',')))
      l = lenstr(line)

the format statement 6141 have a bigger number than 20 at the end.

xia2 0.3.3.1 release - finally!

In response to a number of bugs which needed squashing and changes to CCTBX have bundled a new release of xia2. These are the improvements:


Changes since 0.3.3.0

• Handle division in updated CCTBX.
• Squash bug with -3d and -resolution.
• Squash bug with resolution and -2d, result slightly different to that reported.
• Trap weirdness with XDS reindexing, causes problems sometimes trying to process in C2 from P222 not P2 as you would hope!
• Adjustments to cope with excludively narrow sweeps i.e. all 3 image sweeps.

Available from the usual places (ccp4.ac.uk/xia and xia2.sf.net)

As usual, any problems please get in touch!

Cheerio,

Graeme

Trying again at twitter.

Sometimes would be nice to be able to make mention of something without the full blog posting, so giving it a go again with twitter. @xia2stuff I believe is the way you phrase these things.

http://twitter.com/#!/xia2stuff

Major things like releases will always find their way to the blog!

Pro-processing

Saw these advertised on the ccp4 BB -

http://bl831.als.lbl.gov/example_data_sets/ACA2011/DPWTP-website/

So am working on making sure that they all work correctly with xia2... One issue was the naming convention for the high resolution P1 lysozyme data - prefix.number.extension. Sorted now in latest revision.

Bug: -quick command line option killed...

Looks like I forgot to include -quick command-line option in the recognized list, oops. Sorry. I don't know if it still works but it should do and will be fixed in the next build!

[update: it seems that it does still work, though not quite right since the scaling resolution changes were made...]

Much belated xia2 0.3.3.0 release

In the usual place, changes since 0.3.2.0

• Support for new XDS build.
• Small bug fix / improvement in XDS indexer implementation.
• Capability to ptovide refined experimental geometry via XDS GXPARM.XDS - useful for polycrystal data reduction.
• Now check what you tell the program, viz:

  gw56@ws050 nonsense]$ xia2 -3d -nonsense /data/gw56/dl/Cowan/Insulin/insulin/
  Traceback (most recent call last):
    File "/home/gw56/svn/xia2/Applications/xia2.py", line 45, in 
      from xia2setup import write_xinfo
    File "/home/gw56/svn/xia2/Applications/xia2setup.py", line 35, in 
      from Handlers.CommandLine import CommandLine
    File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 1513, in 
      CommandLine.setup()
    File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 349, in setup
      raise RuntimeError, nonsense
  RuntimeError: Unknown command-line options: -nonsense
  

  and complain if you type in something xia2 does not understand.
• Removed a lot of cruft from the xia2 code base in preparation for some more substantial refactoring.
• Found bug which meant that the labelit beam centre computed in the setup phase was not used - mostly this is not important, unless you are using -3dii.
• Added -ice command-line argument, which will exclude measurements from regions which are typically where ice rings land. Will need to add more subtle mechanism which will allow specific regions to be excluded.
• Specifying resolution with 3d(r) pipeline now works correctly.
• Removed XDS version check by default, which was annoying every new year. If you would like to check that the XDS version is explicitly supported, add -check_xds_version to the command-line.
• Added table of scala runs to sweep name in the debug output, helpful for reviewing in polycrystal data reduction cases.
• Mosaic spread used from Mosflm is now average of all images, not the first or last image.
• Fixed header reading for new Pilatus instruments at DLS and elsewhere.
• Added capacity to pass in known globally postrefined experimental setup via -xparm GXPARM.XDS, only useful with XDS processing pipelines.
• Removed use of ccp4 printpeaks tool, which crashes on new pilatus images, replace with Mosflm for the moment. Move to replacement with labelit code at later stage planned.

Underscore weirdness

A weird error I had reported to me last week turned out to be due to the use of underscores_in_sweep_names and (perhaps) to numbering crystals. This is most likely a side-effect of the heuristics I use to understand what is in MTZ files and such. Some recommendations:

* Don't use underscores in your project, crystal, wavelength, sweep names
* Don't use numbers for your crystal names

The second one is a problem as the CCP4 parser can get confused between datasets which are numbered and the number name of data sets. I would prefix it with P/X/W/S if you want to number things...

No observations, I am sure I saw spots!

From time to time, particularly with radiation sensitive samples, you may see:

Status: error "no observations run 1: 1 to 1795"

From xia2. Essentially this means that there are no spots with an I/sigma > 2 or so in the data set and Scala complains. However, you may know that there are strong spots there because you have seen them. So, what's going on?

If your sample is mostly dead some way through the data collection, XDS will continue to process the images. When it gets to the end and scales the values will be pretty poor. To accommodate this
the sig(I) values are inflated, which has the side-effect of reducing the I/sigma for the entire data set. This is what has happened in this example.

So what do you do? As it has not scaled / merged the data properly it is tricky to work out what images are good. Well, here follows a recipe I would suggest.

First, move to the directory where the processing was performed, which could well be DEFAULT/NATIVE/SWEEP1/integrate (say). Then use GNUPLOT to look at the raw measured intensities in INTEGRATE.HKL as follows:

> plot 'INTEGRATE.HKL' using 8:($4/$5) with dots

This will give you a scatter plot:

Which pretty clearly shows that there is something funny going on. In particular, there are very few weak data after frame 1200 ish. So, I would then edit the automatic.xinfo file which the previous xia2 run generated and limit the number of frames. In this case I ended up just using the first 900 frames, before the first gap, though it would be sensible to also consider including the frames 950 - 1250 as a second sweep.

In other cases, particularly when the input beam centre is unreliable, this phenomenon can also result from misindexing the reflections. XDS will in this case inflate the sig(I) estimates to the point where non symmetry related reflections are compatible within errors, which is obviously wrong. I still need to work on an automated tool inside xia2 to spot this!

Getting and running xia2 from subversion

On a couple of occasions lately I have had folk asking about how to run with the xia2 direct from subversion. You should remember that there can always be problems with this (though I try to never commit breaking changes) however it's pretty straightforward. So, here we go:

* Check you have subversion installed!
* Make yourself a xia2-svn directory in wherever you keep your crystallographic software

svn co https://xia2core.svn.sourceforge.net/svnroot/xia2core/trunk/xia2core xia2core
svn co https://xia2.svn.sourceforge.net/svnroot/xia2/trunk/xia2 xia2

* Compose yourself a setup file which will (i) set up ccp4 then (ii) set up cctbx etc and if you like also set up labelit / XDS
* At the end of this file add

export XIA2CORE_ROOT=${HOME}/svn/xia2core
export XIA2_ROOT=${HOME}/svn/xia2
. ${XIA2CORE_ROOT}/setup.sh
. ${XIA2_ROOT}/setup.sh

for the bash shell or

setenv XIA2CORE_ROOT ${HOME}/xia2-svn/xia2core
setenv XIA2_ROOT ${HOME}/xia2-svn/xia2
source ${XIA2CORE_ROOT}/setup.sh
source ${XIA2_ROOT}/setup.sh

for (t)csh. For example. Then once you have sourced this setup file xia2 should work. If you want to update this please go to $XIA2_ROOT and run "svn update".

When reporting problems, please first run svn update in case I have fixed it. If I have not then please get in touch, mentioning that you are getting xia2 from svn.

Big thanks to bug reporters

Hi Folks,

Just like to say thanks for all of the feedback I have been getting recently - many of the changes in the forthcoming 0.3.3 release are directly resulting from this.

Please, keep it coming!

Cheerio, Graeme

So what is xia2 about, then?

Thanks to Wordle, now I know...

Checking up on XDS

So the number of new-year-new-xds-why-doesn't-it-work complaints this year exceeded the energy threshold to remove the XDS version check (that, and the fact that the program has not changed how it behaves for a while now)

So, I have removed the check. I will however post when I think a new XDS build should work, and keep the checked versions up-to-date. If you like to have xia2 check for you, -check_xds_version is there for you.

However, most people don't want this it seems :o)

Contemplations on ice

So, had a request to be able to explicitly exclude ice ring regions from diffraction images in processing. In the first instance I have added a -ice command-line option which will use the following hard-coded ice ring regions, from the XDS documentation:

3.93 3.87 !ice-ring at 3.897 Angstrom
3.70 3.64 !ice-ring at 3.669 Angstrom
3.47 3.41 !ice-ring at 3.441 Angstrom
2.70 2.64 !ice-ring at 2.671 Angstrom
2.28 2.22 !ice-ring at 2.249 Angstrom

These are listed in $XIA2_ROOT/Data/ice-rings.dat, so I guess you could fiddle with them if you wanted. Perhaps a better way would be to specify explicitly regions to exclude. Comments anyone?

Deprecating old versions of Scala

The Scala in CCP4 6.0.2 still appears to be "supported" in xia2 however the version is 5 years old now and has to have special code! I am therefore removing support for this, so you will need to be using CCP4 6.1 or later (which I am 99% sure you need anyway)

If you see this:

                *******************************************
                *    SCALA - continuous scaling program   *
                *                                         *
                *    Version: 3.2.25                      *
                *    Date   : 21/9/2006                   *
                *                                         *
                *   Phil Evans, MRC LMB, Cambridge, UK    *
                *   pre@mrc-lmb.cam.ac.uk                 *
                *                                         *
                *******************************************

when you type "scala" xia2 will probably stop working after a forthcomin update. Need to reduce the effects of code bloat.

Sorry!

Decrufting!

Prior to setting about doing some "real" development work with xia2 which is well past due, I am clearing out some of the cruft of things which really are no longer needed. One thing I identified was:

-ehtpx_xml_out

which produces an XML file fill of not-a-lot. I think that this is superceded by the LogFiles/bioxhit.xml and the optional -ispyb_xml_out which I know are used. I will remove this as I am 99% sure it's redundant - this post is so that anyone googling can find out where it went.

Checking command-line arguments...

A problem people have been reporting for a while is that xia2 does not fail loudly when you tell it nonsense. Well, I have now added something which does:

gw56@ws050 nonsense]$ xia2 -3d -nonsense /data/gw56/dl/Cowan/Insulin/insulin/
Traceback (most recent call last):
  File "/home/gw56/svn/xia2/Applications/xia2.py", line 45, in 
    from xia2setup import write_xinfo
  File "/home/gw56/svn/xia2/Applications/xia2setup.py", line 35, in 
    from Handlers.CommandLine import CommandLine
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 1513, in 
    CommandLine.setup()
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 349, in setup
    raise RuntimeError, nonsense
RuntimeError: Unknown command-line options: -nonsense

however this will only spot nonsense beginning with '-' as the directory stuff is handled elsewhere. Will be included in next release. Note to self: should really move this to a proper option parser.

Anti-alias

So - sometimes people use an alias in their shell to point at programs rather thank tinkering with their path. I have to warn you that xia2 is not smart enough to "get" this - so for instance aliasing xds rather than putting it in your path will stop xia2 -3d from working.

If someone would like to look into fixing this, please feel free to get in touch. Until then, you have been warned ;o)

New year, new XDS

Following Kay's email, the new XDS appears to work fine with xia2. As usual, a little editing will be needed in $XIA2_ROOT/Wrappers/XDS/XDS.py around line 78  -


    supported_versions = ['December 28, 2009',
                          'February 3, 2010',
                          'May 10, 2010',
                          'December 6, 2010']

As always, this will be fixed in the next released build.

Reminder!

http://xia2.blogspot.com/2010/05/summary-of-command-line-options.html

Using -resolution ...

Quick note - not sure that the interaction of -resolution with -3d is working quite correctly, so please run with -3dold if you want to specify the resolution!