Resolution limits in xia2

By default the resolution limits produce what I hope is "something sensible" - however there are cases where for perfectly good reasons people may disagree with these. It's therefore about time these were better explained.

The default resolution limits are:

• I/sigma > 1 at the edge
• Merged(I/sigma) > 2 at the edge

These are determined with my own merging code not scala or XSCALE, so there could be some small differences at the end. These are controlled by the command line options -isigma and -misigma respectively, and whichever one is lower  will be used. Some examples are helpful here, so I will use a data set recorded last week on I24 in 9 seconds (how cool is that)

Default run:

High resolution limit                           1.62    7.23    1.62
Low resolution limit                            31.73   31.73   1.66
Completeness                                    99.8    98.6    99.7
Multiplicity                                    5.7     5.3     5.3
I/sigma                                         14.9    35.9    2.4
Rmerge                                          0.066   0.026   0.695
Rmeas(I)                                        0.073   0.028   0.771
Rmeas(I+/-)                                     0.073   0.028   0.771
Rpim(I)                                         0.03    0.012   0.328
Rpim(I+/-)                                      0.03    0.012   0.328
Wilson B factor                                 19.666
Partial bias                                    0.0     0.0     0.0
Anomalous completeness                          98.1    100.0   95.9
Anomalous multiplicity                          3.0     3.6     2.7
Anomalous correlation                           -0.028  -0.093  -0.015
Anomalous slope                                 0.973   0.0     0.0
Total observations                              193625.0        2465.0  13041.0
Total unique                                    33686.0 465.0   2438.0

Being very cautious - merged and unmerged I/sigma > 2:

-isigma 2 -misigma 2 (unmerged I/sigma will dominate)

High resolution limit                           1.78    7.94    1.78
Low resolution limit                            31.73   31.73   1.82
Completeness                                    99.8    98.1    99.8
Multiplicity                                    5.9     5.2     5.5
I/sigma                                         18.7    35.6    4.8
Rmerge                                          0.058   0.025   0.338
Rmeas(I)                                        0.063   0.028   0.374
Rmeas(I+/-)                                     0.063   0.028   0.374
Rpim(I)                                         0.026   0.011   0.157
Rpim(I+/-)                                      0.026   0.011   0.157
Wilson B factor                                 18.981
Partial bias                                    0.0     0.0     0.0
Anomalous completeness                          98.8    100.0   96.6
Anomalous multiplicity                          3.1     3.6     2.8
Anomalous correlation                           0.004   0.261   -0.001
Anomalous slope                                 0.98    0.0     0.0
Total observations                              150435.0        1872.0  10262.0
Total unique                                    25633.0 362.0   1872.0

Being very generous - unmerged I/sigma > 0.2, merged > 1:

-isigma 0.2 -misigma 1 (merged I/sigma will dominate)

High resolution limit                           1.51    6.75    1.51
Low resolution limit                            31.73   31.73   1.55
Completeness                                    99.8    98.9    99.3
Multiplicity                                    5.5     5.2     4.1
I/sigma                                         12.5    35.4    1.2
Rmerge                                          0.074   0.026   1.095
Rmeas(I)                                        0.081   0.029   1.253
Rmeas(I+/-)                                     0.081   0.029   1.253
Rpim(I)                                         0.034   0.012   0.594
Rpim(I+/-)                                      0.034   0.012   0.594
Wilson B factor                                 19.471
Partial bias                                    0.0     0.0     0.0
Anomalous completeness                          96.9    100.0   88.8
Anomalous multiplicity                          2.9     3.5     2.2
Anomalous correlation                           -0.042  0.072   -0.038
Anomalous slope                                 0.97    0.0     0.0
Total observations                              228772.0        2991.0  12145.0
Total unique                                    41226.0 570.0   2964.0

There are also limits you can apply in terms of Rmerge etc, however these are essentially the same as limiting the unmerged I/sigma as Rmerge ~ 0.8 / (unmerged I/sigma)

pychef error on Mac OS X

Due to a bit of sloppy programming on my part the pychef implementation is broken on Mac OS X - one line fault, no idea how it got through, but this will cause failure on any data set run.

The fix is simple: in $XIA2_ROOT/Modules/PyChef/PyChefMain.py add

import subprocess

anywhere near the top - e.g. after "import sys". Thanks to Jim Naismith for pointing out the problem.

Much delayed documentation begins

Starting at

https://docs.google.com/document/pub?id=1XjiSpNqONSAkPKRgKJOxtBq0SaPohbcKrlTKGtmZWzU

Shout if there are things you would like in there (full definition of .xinfo file structure springs to mind)

ESRF ID29

I have had problems reported when using xia2 to process data from the new Pilatus instrument on ESRF ID29. It seems that there are two problems - reading the image headers for xia2 and Mosflm handling the different image header that they are using. The former I have managed to mend and a patch is available, and Harry is looking to fix the latter in the forthcoming Mosflm release.

Meantimes, with the above patch xia2 -3dii or xia2 -indexer xds -integrater xdsr -scaler xdsr will work.

xia2 0.3.2.0 released

xia2 now updated to 0.3.2.0:

Changes since 0.3.1.7

• Now report the min, mean, max mosaic spread (according to the program definitions after integration of each sweep.
• Resolved problems with overriding environment.
• Added -serial pun for -parallel 1.
• Allow resolution limit to be assigned on a per-sweep basis.
• Handled corner case with images 200 - 250 (say) giving REBATCH error.
• Added full 8-way search for scaling models for 2d pipelines.
• Partial support for two-theta offset data collection (tested with Rigaku X-ray equipment and Diamond I19 data.)
• Added -min_images keyword which is helpful when analysing virus data with e.g. 4 images / sweep.

 Downloads in the usual place - please shout if you have any problems!

Used xia2 in a publication? Please let me know!

At some point I'd like to be in the position to get some help with xia2 developments. If such a situation does arise, it would be much easier if I had an indication of how useful the program has been to people. Now I know I didn't help the cause by taking my time to produce a xia2 paper to cite, to I accept that this is very much may fault :o) However I'd really like to hear from you.

Thanks in advance!

Sweeps starting with image 0: Part Deux

Turns out this may have been self induced, see:

https://sourceforge.net/tracker/index.php?func=detail&aid=3080972&group_id=211879&atid=1019527#

However is now fixed in revision 3306: changes to Experts/FindImages.py

Replacement file:

http://xia2.svn.sourceforge.net/viewvc/xia2/trunk/xia2/Experts/FindImages.py?revision=3306

Sweeps starting with image 0

As some of you will know, if you put in some data into xia2 with images starting from 0 it may well explode. XDS gives an error, as does REBATCH in the CCP4 suite (which you would expect to be able to renumber the batches!)

So, the easiest thing to do I think is to externally renumber the images. I will therefore write a little python program to do this - watch this space.

Bugs in xia2

So as we all know, xia2 has bugs. More are discovered from time to time. I'm moving to the public tracker at sf.net. If you'd like to see what's going on in this department (and maybe even check up on the progress of your bugs) please visit:

https://sourceforge.net/tracker/?group_id=211879&atid=1019527

Downloads now available from sf.net

The files are exactly the same as I have been distributing from http://www.ccp4.ac.uk/xia but now mirrored at

https://sourceforge.net/projects/xia2/files/

I will aim to use this more than ccp4 in the future. Please let me know if you have any problems.

Resolution buglet

So it's become clear that the "fixes" to the resolution limit calculations I made a while back - while useful - broke the ability for you the user to set what you want as a resolution limit, as in essence it ignores you. This was the same from the command line or xinfo file. This is nor repaired in revision #3282 and will be included in the next release.

New mailing list

I have set up a mailing list at sourceforge.net as I left the organisation which host this list 18 months ago - if you are keen to keep subscribing, please go to

https://lists.sourceforge.net/lists/listinfo/xia2-list

and add in your email address. It uses mailman to manage, so it's pretty easy going.

Bug reports

Hi Folks,

I've decided to start using the sourceforge bug tracker in addition to my own internal one, to handle user bug reports & feature requests - this is available from here:

https://sourceforge.net/tracker/?group_id=211879&atid=1019527

You can post anonymous bugs, but if you either sign in or put your email in to the report you can here when things happen...

Cheerio, Graeme

0.3.1.7 release

Somewhat overdue, done in a hurry to get the just mentioned fix out:


Changes since 0.3.1.6

• Changes suitable for working on microcrystals - reachable through a -microcrystal command line flag, and -failover is probably helpful too.
• If lattice or spacegroup specified, do not run tests.
• 2dr and 3dr now the default pipelines - use 2dold or 3dold if you need to get to the old pipelines.
• Fixed selection of known lattice in IDXREF reindexing.

Please get in touch if you would like more information on the microcrystal option - this is definitely work in progress...

Problematic indexing when setting spacegroup

So, following a user problem I have tracked down an issue in the indexing with XDS: if you specify the lattice to use but this has rather a high penalty, the solution will be mangled. This is a bad thing. Now I have tracked down where this was going wrong and fixed it!

About time we had a site file?

So the old chestnut of backstop masking has re-appeared, and this time I figured it would be easiest not to try and code up something clever, but instead let you (or your beamline scientist) tell xia2 where the beamstop etc. are.

So - what I have lined up for the site file so far are:

• a way of locating the beam stop as a function of some distances
• the ADC offset for CCD detectors

If you have a beamline where you run xia2, or a lab source, and would like something else propogated please drop me a line - I will be all ears.

Latest XDS

Just found out that there is a new version of XDS out there - it appears to work fine with xia2, but you will need to do the usual edits of ${XIA2_ROOT}/Wrappers/XDS/XDS.py to add

'May 10, 2010'

to the list of supported versions. Next version of xia2 will include this.

Multi-crystal progress

So - hooda thunkit - it turns out that this is hard. Nevertheless it's going somewhere though there's a hint of the whac-a-mole debugging going on here. At the moment it's working towards handling the trying-to-assemble-a-dataset-from-dozens-of-crystals turf - so assuming a (small) handful of degrees of data from each - along the lines of http://www.nature.com/emboj/journal/v29/n2/abs/emboj2009352a.html.

Most interesting part of this is that all of the program steps need to be hardened against what would usually be fatal errors - negative scales and the like - and just plough on regardless. An interesting paradigm.

Summary of command line options

I realised that the command-line argument documentation is (woefully) out of date. Until I think of a better way to handle the command line (perhaps using Phil from CCTBX) this will be hard to correct. However, here are the current list of "useful" options:

• -debug
• -interactive (for selecting images for autoindexing with Mosflm, Labelit)
• -2d (old mosflm / scala)
• -2dr (new mosflm / scala - recommended)
• -3d (old XDS / XSCALE)
• -3dr (new XDS / XSCALE) 
• -3dii / 3diir (as -3d/3dr, using all frames for autoindex)
• -indexer -integrater -scaler - manually specify the integrater, indexer, scaler to use
• -image /process/this/sweep
• -beam x,y (Mosflm reference frame)
• -atom Se (say) - switch on separation of anomalous pairs
• -project foo
• -crystal bar
• -xinfo project.xinfo
• -parallel N (set to 1 to swtich off parallel processing)
• -xparallel N (use multiple machines for XDS - requires working forkintegrate scripts)
• -spacegroup P212121
• -resolution 2.8
• -freer_file free.mtz
• -reference_reflection_file reference.mtz (to get standard indexing, will copy spacegroup, free column)

That's all for the moment - most of these I never use so I had to consult the code...

xia2 0.3.1.6 released

Much delayed release, now seems to be working well. Also includes support for recent XDS &c.

Sorry for the delay, thankyou for your patience.

Changes since 0.3.1.0

• Fixed bug where the resolution limit was not reset in an indexer solution elimination.
• Added support for XDS from Dec 09.
• Fixed some of the issues found in the resolution limit determination when processing low resolution data.
• Fixed use of I/sigma limit assignment.
• Added command-line control of individual indexer, integrater, scaler.
• Repaired regular expression for image matching, to cope with images where there is e.g. 3.5 in the template. This was previously misinterpreted as an image name of the form setup.NNNN.
• Implemented new resolution limit pipelines, based on new merging code to give more robust control: -2dr and -3dr (recommended.) Fine control over the choices can be made using the -isigma, -misigma, -completeness and -rmerge command line options.
• Added interactive indexing mechanism, where user can assign images to use for autoindexing, e.g.:

  Existing images for indexing: 1, 90, 180
  >1, 60, 120, 180
  New images for indexing: 1, 60, 120, 180
   

  Assign -interactive on the command line (Mosflm and Labelit indexers)

Multi crystal data analysis... started!

Ok, so have finally made a start on the multi-crystal analysis (the resolution limits stuff will be released soon, after some more testing) and crikey it's gonna be fun. Should get the ol' grey cells working though.

Initial plans are to build this through the xinfo hierarchy - deifining multiple crystals within the same project - but if you have a better idea for the interface please shout!

xia2 for small molecule data

Have wondered about extending xia2 to work with data from small molecule crystals - a few of the assumptions would need to change, not least in the indexing, scaling and description of the geometry. However, should be doable.

Interested? If so please drop me a line.

Resolution limit overhaul...

Not before time, have finally overhauled the resolution limit calculations. At the moment they rely on a separate piece of code to merge the reflections after scaling, and some curve fitting to decide some limits. The defaults are:

- I/sigma > 1
- Mn(I/sigma) > 2
- completeness anything
- Rmerge anything

And are accessed through the command line as -isigma -misigma -completeness [0,1] -rmerge. This will form the kernel of the next release 0.3.1.5. N.B. to access the new resolution limit code, use pipelines -2dr -3dr in place of -2d -3d. Worth it.

Also includes: changes to workflow for these pipelines, use of Scala sd correction automation stuff, -interactive mode for autoindexing with Mosflm / Labelit to allow guidance of image selection.

xia2 moves to sourceforge.net

Now seems to be working right, and also appears to have managed to include the history. Access from the command-line via:

svn co https://xia2core.svn.sourceforge.net/svnroot/xia2core/trunk/xia2core xia2core
svn co https://xia2.svn.sourceforge.net/svnroot/xia2/trunk/xia2 xia2

which will give you access to the current state of the trunk - no guarantees! Can also download a tarball of the trunk in a similar way. I will now be using this as the primary repository - only reason to move may be if sourceforge.net proves unreliable - unlikely.

If you'd like to have write access, to contribute to the development, please shout!

xia2 with weak pilatus data

Seems like the latest XDS included a big fix for the processing of weak (i.e. background ~ 0) pilatus data - thanks to Tobias Krojer for the heads-up there. This will involve some slight fixes to the xia2 XDS wrappers, so I will bundle these later today.

N.B. this will require installation of latest XDS!

I should also start to think about how to make fixes like this more responsive... ideas anyone?

xia2 questions...

May be worth posting here too:

http://www.majorgroove.org/questions

in case anyone else would find the answers interesting...

Refactoring resolution calculations

Quick update on this. Phase 1 will be to try and get xia2 to better respect the existing resolution criteria - so when it should be I/sigma ~ 2 at the edge, it is! Then after this, refactor the input a little so that the criteria discussed on the mailing list can be assigned through the input or from a preferences file.

This is quite interesting actually. It's shaking things up a little and may end up with things being a little more efficient...

Weirdness managed

So it seems that this should have been a problem for a lot longer. Thanks to Ralf GWK and Phil Evans for a chat about this which clarified things. Now will pass environment through to the underlying programs correctly (is the intention) though this turned out to be a moderately low level change. Have made this now, and will test, next planned release which should be available sometime soon will contain this fix...

Pointless (latest) and Phenix (latest) slight weirdness

It appears that the combination of the latest (6.1.3) CCP4 and the latest (1.6) Phenix will cause problems for pointless. This will exhibit itself in xia2 as:

 Status: error "no element found: line 27, column 0"

as a result of pointless having an error like:

/scratch/xtal/ccp4/ccp4-6.1.3/bin/pointless: symbol lookup error: /scratch/xtal/ccp4/ccp4-6.1.3/bin/pointless: undefined symbol: _ZNK5cctbx5sgtbx16space_group_type13lookup_symbolEv

I found installing a CCTBX from cctbx.sf.net made the problem "go away", but I have no idea why. Investigating!

Resolution limits ignored? Fix!

Hi Folks,

Got an error report from Chip Lesburg, that the -i_over_sigma_limit was ignored. This was perfectly accurate, and it turns out that yes, it was ignored. Fix is to change the calls to determine_scaled_resolution in Modules/CCP4ScalerImplementation.py and Modules/XDSScaler.py to replace 2.0 and 3.0 respectively to Flags.get_i_over_sigma_limit(). Will be fixed in next version.

Yes, these Python modules are nasty code, refactoring them is a separate problem!

Graeme

Resolution limit things

So, I got a complaint from someone who works with membrane proteins, who complained (accurately) that xia2 falls over at the resolution limit calculation step. This was because it assumes a Wilson plot, but this is only meaningful for better than about 4A.

xia2 should now cope - if the data are not there for a Wilson plot calculation it will fail over to a simple hunt for where I/sigma is about 1. Gets it out of a corner here anyway...

Happy new XDS

Happy new year folks. As is traditional this time of year, XDS changes. Fortunately the new one works (or appears to) just fine with xia2, so all that is necessary is to change the date string in $XIA2_ROOT/Wrappers/XDS/XDS.py. Next update will of course include this change...