xia2

Manual

2012-01-24T07:48:00.001-08:00

One of the most common (and it has to be said, most reasonable) complaints I have heard about xia2 is the absence of the manual. I am now working on this, to be included in the next release of the program.

The first version will no doubt be rather rough, so I will welcome any feedback on it when it becomes available.

Testing in preparation for release

2012-01-19T03:03:00.001-08:00

Finally gearing up for a release, got to run some extensive testing first.

Mosflm / Pilatus

2012-01-17T03:31:00.001-08:00

The beta for mosflm 7.0.8 appears to work much better with Pilatus data than the released 7.0.7 version - if you're using data from a 2M or 6M, I would strongly recommend the upgrade.

CCP4 Study Weekend 2012

2011-12-15T05:33:00.000-08:00

I'll be at the CCP4 study weekend in the new year to talk about xia2 - in the main session as well as the lunchtime bytes. If you've any ideas of things you'd like to know about in the lunchtime bytes please make a note below or drop me a line at the usual address...

See you there!

Aimless for merging data from XDS (N.B. not yet scaling!) now working

2011-11-04T08:48:00.003-07:00

Bloody edge of svn has this working, which is useful if like me you have better things to be doing than recompiling Scala a lot :o)

Will be in forthcoming 0.3.4 release. Any time now.

Aimless

2011-10-14T02:10:00.000-07:00

Have been working on including Aimless from Phil Evans (his "Pointless" replacement for Scala) and if you would like to try this out, the code is now in the trunk. Usual svn checkout, download aimless 0.1.4 or later from Phil's FTP site, then run -2da.

Early results show that it makes the overall xia2 pipeline quite a bit faster, while maintaining the quality of results. Worth looking at.

xia2 0.3.3.3 release

2011-10-03T02:40:00.000-07:00

Hi Folks,

0.3.3.3 release of xia2 is now available - which includes a bunch of useful fixes but fundamentally no new functionality:

Changes since 0.3.3.2

Many bug fixes as a result of implementing a comprehensive testing scheme.
Support for full cbf format, making use of cbflib Python bindings pycbf now included in cctbx releases.
Merged functionality from XDSIntegraterR into XDSIntegrater, removed duplicate code. Will make for more maintainable suite in the future.
Improved 2d pipeline scaling, now tests 8 permutations.
Improved handling of twinned data for multiple sweeps.
Fixed occasional bugs with multiple sweeps, reindexing trigonal data sets with XDS.
Removed "old" scaling modes, 2d and 3d, as these were unhelpful.
Changes since 0.3.3.1

-xparallel -1 option added which will split your data into 30 degree chunks and process with forkintegrate Note well you are responsible for setting up forkintegrate correctly!

Extensive testing

2011-09-15T03:07:00.001-07:00

So, finally I have set up a system for doing bulk tests, which is surprisingly illuminating. Results of this will follow, but if you have a data set where xia2 fails but manual processing works please get in touch - I will be happy to add it to the test system and work on making it work.

CCP4 6.2.0

2011-08-31T01:35:00.001-07:00

If you try running xia2 with CCP4 6.2.0 and XDS it probably won't work, and will complain about something like "End of file".

If you get this, please get a new pointless from

http://www.ccp4.ac.uk/problems.php#6.2.0-programs

and everything should be fine.

Addendum:

My bad: this version is still buggy - better to get a version from ftp://ftp.mrc-lmb.cam.ac.uk/pub/pre and overwrite the ccp4 one with this.

Does anyone use xia2-summary.dat?

2011-08-01T04:01:00.000-07:00

Just found this:

[gw56@ws050 3]$ cat xia2-summary.dat
Project: AUTOMATIC
Crystal: DEFAULT
Sequence length: 0
Wavelength: WAVE1 (0.97966)
Sweep: SWEEP1
Files /data2/gw56/jcsg/12287_2L/12287_1_E1_###.img
Images: 1 to 60
Beam 105.10 101.05 => 108.95 105.04
Distance 170.00 => 169.89
Date: Sun Sep 26 14:13:49 2004
Wavelength: WAVE2 (1.00000)
Sweep: SWEEP2
Files /data2/gw56/jcsg/12287_2L/12287_1_E2_###.img
Images: 1 to 60
Beam 105.10 101.05 => 108.93 105.04
Distance 170.00 => 170.09
Date: Sun Sep 26 14:17:56 2004
For AUTOMATIC/DEFAULT/WAVE1:
High resolution limit                           1.66    7.41    1.66
Low resolution limit                            36.8    36.8    1.7
Completeness                                    97.9    96.4    85.7
Multiplicity                                    4.3     3.5     2.9
I/sigma                                         16.5    33.9    2.2
Rmerge                                          0.043   0.023   0.372
Anomalous completeness                          94.3    92.4    78.2
Anomalous multiplicity                          2.4     2.3     1.6
For AUTOMATIC/DEFAULT/WAVE2:
High resolution limit                           1.65    7.39    1.65
Low resolution limit                            36.8    36.8    1.7
Completeness                                    96.3    96.4    74.5
Multiplicity                                    4.2     3.5     2.7
I/sigma                                         17.4    35.6    2.2
Rmerge                                          0.04    0.021   0.369
Anomalous completeness                          92.1    94.2    69.9
Anomalous multiplicity                          2.3     2.3     1.5
Cell:  51.570  51.570 157.600  90.000  90.000  90.000
Spacegroup: P 41 21 2

which I remember from a long time ago - just wondering, does anyone use this? Is it worth keeping (as I clear up...)

Issues with xia2 / 3d / pointless 1.6.1

2011-08-01T00:57:00.000-07:00

Many thanks to bug reporters. If you see:

Status: error "End of File "

at the end of a xia2 job which looks like it should have worked, and you have recently updated your pointless version, you have hit a feature resulting from a small change in pointless. Essentially when copying reflections from XDS to MTZ format there is no real orientation matrix, so pointless makes one up. It used to be [1] and was changed to [0] as it is meaningless - however reindex relies on this and therefore explodes.

Phil has fixed this in 1.6.2 (thanks Phil!) and is quite reasonably of the opinion that this is a bug in reindex...

A subsequent question to this is why I still use reindex inside xia2 now pointless does it all. "Historical reasons" would be the fair answer i.e. I haven't got around to fixing this :o)

For followers of the bleeding edge...

2011-07-14T03:18:00.000-07:00

You may see an error like this:

ImportError: No module named pycbf

which would mean you need to get an up-to-date build of cctbx - sorry. I have started to use pycbf (clearly) and I had not thought to mention it as no released versions include this dependency. I neglected to think that some people use the svn...

http://cctbx.sf.net is where to get an updated build from! These now include cbflib and pycbf which will give added awesomeness in the future.

Lots of frames? Then things to recompile

2011-06-10T07:02:00.000-07:00

In $CPROG/scala_/parameter.fh reset parameter maxbat to (say) 15000

integer maxbat
parameter (maxbat=15000)

Likewise in $CPROG/reindex.f

integer mbatch, mcols
parameter (mbatch=15000, mcols=500)

There could be more...

xia2 at the ACA

2011-05-16T00:45:00.000-07:00

So, I will be at the ACA this year, and have a presentation in the multi-crystals session - however if you have any questions or comments feel free to catch me. I will aim to be at the CCP4 stand between sessions.

See you there!

Polycrystal data reduction errors?

2011-05-16T00:43:00.000-07:00

If you see:

Status: error "could not find status"

When you are running data reduction from lots of crystals with xia2, you probably need to recompile Scala. Particularly, if you see:

 Rmerge against resolution for each run (deviation from mean I+ or I-)

 $TABLE: Rmerge for each run, WAVE1 :
At line 17263 of file /usr/local/xtal/ccp4-6.1.3/src/scala_/scala.f
Fortran runtime error: End of file

at the end of the last Scala log file mentioned in xia2-debug.txt, you will need to edit around that location in $CPROG/scala_/scala.f and make

 6142  format (
     $     ' $TABLE: Rmerge for each run',a,' :')
      write (line, 6141) (i+3,i=1,nrunp1)
 6141 format(
     $     ' $GRAPHS:Rmerge v. resolution:N: 2,',(20(i2,',')))
      l = lenstr(line)

the format statement 6141 have a bigger number than 20 at the end.

xia2 0.3.3.1 release - finally!

2011-05-12T07:25:00.000-07:00

In response to a number of bugs which needed squashing and changes to CCTBX have bundled a new release of xia2. These are the improvements:

Changes since 0.3.3.0

Handle division in updated CCTBX.
Squash bug with -3d and -resolution.
Squash bug with resolution and -2d, result slightly different to that reported.
Trap weirdness with XDS reindexing, causes problems sometimes trying to process in C2 from P222 not P2 as you would hope!
Adjustments to cope with excludively narrow sweeps i.e. all 3 image sweeps.

Available from the usual places (ccp4.ac.uk/xia and xia2.sf.net)

As usual, any problems please get in touch!

Cheerio,

Graeme

Trying again at twitter.

2011-05-10T07:06:00.000-07:00

Sometimes would be nice to be able to make mention of something without the full blog posting, so giving it a go again with twitter. @xia2stuff I believe is the way you phrase these things.

http://twitter.com/#!/xia2stuff

Major things like releases will always find their way to the blog!

Pro-processing

2011-05-03T01:23:00.000-07:00

Saw these advertised on the ccp4 BB -

http://bl831.als.lbl.gov/example_data_sets/ACA2011/DPWTP-website/

So am working on making sure that they all work correctly with xia2... One issue was the naming convention for the high resolution P1 lysozyme data - prefix.number.extension. Sorted now in latest revision.

Bug: -quick command line option killed...

2011-04-27T01:58:00.001-07:00

Looks like I forgot to include -quick command-line option in the recognized list, oops. Sorry. I don't know if it still works but it should do and will be fixed in the next build!

[update: it seems that it does still work, though not quite right since the scaling resolution changes were made...]

Much belated xia2 0.3.3.0 release

2011-03-15T09:11:00.000-07:00

In the usual place, changes since 0.3.2.0

Support for new XDS build.
Small bug fix / improvement in XDS indexer implementation.
Capability to ptovide refined experimental geometry via XDS GXPARM.XDS - useful for polycrystal data reduction.

Now check what you tell the program, viz:

gw56@ws050 nonsense]$ xia2 -3d -nonsense /data/gw56/dl/Cowan/Insulin/insulin/
Traceback (most recent call last):
  File "/home/gw56/svn/xia2/Applications/xia2.py", line 45, in 
    from xia2setup import write_xinfo
  File "/home/gw56/svn/xia2/Applications/xia2setup.py", line 35, in 
    from Handlers.CommandLine import CommandLine
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 1513, in 
    CommandLine.setup()
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 349, in setup
    raise RuntimeError, nonsense
RuntimeError: Unknown command-line options: -nonsense

and complain if you type in something xia2 does not understand.

Removed a lot of cruft from the xia2 code base in preparation for some more substantial refactoring.
Found bug which meant that the labelit beam centre computed in the setup phase was not used - mostly this is not important, unless you are using -3dii.
Added -ice command-line argument, which will exclude measurements from regions which are typically where ice rings land. Will need to add more subtle mechanism which will allow specific regions to be excluded.
Specifying resolution with 3d(r) pipeline now works correctly.
Removed XDS version check by default, which was annoying every new year. If you would like to check that the XDS version is explicitly supported, add -check_xds_version to the command-line.
Added table of scala runs to sweep name in the debug output, helpful for reviewing in polycrystal data reduction cases.
Mosaic spread used from Mosflm is now average of all images, not the first or last image.
Fixed header reading for new Pilatus instruments at DLS and elsewhere.
Added capacity to pass in known globally postrefined experimental setup via -xparm GXPARM.XDS, only useful with XDS processing pipelines.
Removed use of ccp4 printpeaks tool, which crashes on new pilatus images, replace with Mosflm for the moment. Move to replacement with labelit code at later stage planned.

Underscore weirdness

2011-03-14T08:06:00.000-07:00

A weird error I had reported to me last week turned out to be due to the use of underscores_in_sweep_names and (perhaps) to numbering crystals. This is most likely a side-effect of the heuristics I use to understand what is in MTZ files and such. Some recommendations:

* Don't use underscores in your project, crystal, wavelength, sweep names
* Don't use numbers for your crystal names

The second one is a problem as the CCP4 parser can get confused between datasets which are numbered and the number name of data sets. I would prefix it with P/X/W/S if you want to number things...

No observations, I am sure I saw spots!

2011-03-04T02:40:00.000-08:00

From time to time, particularly with radiation sensitive samples, you may see:

Status: error "no observations run 1: 1 to 1795"

From xia2. Essentially this means that there are no spots with an I/sigma > 2 or so in the data set and Scala complains. However, you may know that there are strong spots there because you have seen them. So, what's going on?

If your sample is mostly dead some way through the data collection, XDS will continue to process the images. When it gets to the end and scales the values will be pretty poor. To accommodate this
the sig(I) values are inflated, which has the side-effect of reducing the I/sigma for the entire data set. This is what has happened in this example.

So what do you do? As it has not scaled / merged the data properly it is tricky to work out what images are good. Well, here follows a recipe I would suggest.

First, move to the directory where the processing was performed, which could well be DEFAULT/NATIVE/SWEEP1/integrate (say). Then use GNUPLOT to look at the raw measured intensities in INTEGRATE.HKL as follows:

> plot 'INTEGRATE.HKL' using 8:($4/$5) with dots

This will give you a scatter plot:

Which pretty clearly shows that there is something funny going on. In particular, there are very few weak data after frame 1200 ish. So, I would then edit the automatic.xinfo file which the previous xia2 run generated and limit the number of frames. In this case I ended up just using the first 900 frames, before the first gap, though it would be sensible to also consider including the frames 950 - 1250 as a second sweep.

In other cases, particularly when the input beam centre is unreliable, this phenomenon can also result from misindexing the reflections. XDS will in this case inflate the sig(I) estimates to the point where non symmetry related reflections are compatible within errors, which is obviously wrong. I still need to work on an automated tool inside xia2 to spot this!

Getting and running xia2 from subversion

2011-02-14T01:16:00.000-08:00

On a couple of occasions lately I have had folk asking about how to run with the xia2 direct from subversion. You should remember that there can always be problems with this (though I try to never commit breaking changes) however it's pretty straightforward. So, here we go:

* Check you have subversion installed!
* Make yourself a xia2-svn directory in wherever you keep your crystallographic software

svn co https://xia2core.svn.sourceforge.net/svnroot/xia2core/trunk/xia2core xia2core
svn co https://xia2.svn.sourceforge.net/svnroot/xia2/trunk/xia2 xia2

* Compose yourself a setup file which will (i) set up ccp4 then (ii) set up cctbx etc and if you like also set up labelit / XDS
* At the end of this file add

export XIA2CORE_ROOT=${HOME}/svn/xia2core
export XIA2_ROOT=${HOME}/svn/xia2
. ${XIA2CORE_ROOT}/setup.sh
. ${XIA2_ROOT}/setup.sh

for the bash shell or

setenv XIA2CORE_ROOT ${HOME}/xia2-svn/xia2core
setenv XIA2_ROOT ${HOME}/xia2-svn/xia2
source ${XIA2CORE_ROOT}/setup.sh
source ${XIA2_ROOT}/setup.sh

for (t)csh. For example. Then once you have sourced this setup file xia2 should work. If you want to update this please go to $XIA2_ROOT and run "svn update".

When reporting problems, please first run svn update in case I have fixed it. If I have not then please get in touch, mentioning that you are getting xia2 from svn.

Big thanks to bug reporters

2011-02-03T01:44:00.001-08:00

Hi Folks,

Just like to say thanks for all of the feedback I have been getting recently - many of the changes in the forthcoming 0.3.3 release are directly resulting from this.

Please, keep it coming!

Cheerio, Graeme

So what is xia2 about, then?

2011-02-02T08:44:00.000-08:00

Thanks to Wordle, now I know...

Checking up on XDS

2011-02-02T01:53:00.001-08:00

So the number of new-year-new-xds-why-doesn't-it-work complaints this year exceeded the energy threshold to remove the XDS version check (that, and the fact that the program has not changed how it behaves for a while now)

So, I have removed the check. I will however post when I think a new XDS build should work, and keep the checked versions up-to-date. If you like to have xia2 check for you, -check_xds_version is there for you.

However, most people don't want this it seems :o)

Contemplations on ice

2011-02-02T01:51:00.000-08:00

So, had a request to be able to explicitly exclude ice ring regions from diffraction images in processing. In the first instance I have added a -ice command-line option which will use the following hard-coded ice ring regions, from the XDS documentation:

3.93 3.87 !ice-ring at 3.897 Angstrom
3.70 3.64 !ice-ring at 3.669 Angstrom
3.47 3.41 !ice-ring at 3.441 Angstrom
2.70 2.64 !ice-ring at 2.671 Angstrom
2.28 2.22 !ice-ring at 2.249 Angstrom

These are listed in $XIA2_ROOT/Data/ice-rings.dat, so I guess you could fiddle with them if you wanted. Perhaps a better way would be to specify explicitly regions to exclude. Comments anyone?

Deprecating old versions of Scala

2011-01-27T01:07:00.000-08:00

The Scala in CCP4 6.0.2 still appears to be "supported" in xia2 however the version is 5 years old now and has to have special code! I am therefore removing support for this, so you will need to be using CCP4 6.1 or later (which I am 99% sure you need anyway)

If you see this:

                *******************************************
                *    SCALA - continuous scaling program   *
                *                                         *
                *    Version: 3.2.25                      *
                *    Date   : 21/9/2006                   *
                *                                         *
                *   Phil Evans, MRC LMB, Cambridge, UK    *
                *   pre@mrc-lmb.cam.ac.uk                 *
                *                                         *
                *******************************************

when you type "scala" xia2 will probably stop working after a forthcomin update. Need to reduce the effects of code bloat.

Sorry!

Decrufting!

2011-01-19T03:27:00.000-08:00

Prior to setting about doing some "real" development work with xia2 which is well past due, I am clearing out some of the cruft of things which really are no longer needed. One thing I identified was:

-ehtpx_xml_out

which produces an XML file fill of not-a-lot. I think that this is superceded by the LogFiles/bioxhit.xml and the optional -ispyb_xml_out which I know are used. I will remove this as I am 99% sure it's redundant - this post is so that anyone googling can find out where it went.

Checking command-line arguments...

2011-01-19T00:50:00.000-08:00

A problem people have been reporting for a while is that xia2 does not fail loudly when you tell it nonsense. Well, I have now added something which does:

gw56@ws050 nonsense]$ xia2 -3d -nonsense /data/gw56/dl/Cowan/Insulin/insulin/
Traceback (most recent call last):
  File "/home/gw56/svn/xia2/Applications/xia2.py", line 45, in 
    from xia2setup import write_xinfo
  File "/home/gw56/svn/xia2/Applications/xia2setup.py", line 35, in 
    from Handlers.CommandLine import CommandLine
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 1513, in 
    CommandLine.setup()
  File "/home/gw56/svn/xia2/Handlers/CommandLine.py", line 349, in setup
    raise RuntimeError, nonsense
RuntimeError: Unknown command-line options: -nonsense

however this will only spot nonsense beginning with '-' as the directory stuff is handled elsewhere. Will be included in next release. Note to self: should really move this to a proper option parser.

Anti-alias

2011-01-14T03:52:00.000-08:00

So - sometimes people use an alias in their shell to point at programs rather thank tinkering with their path. I have to warn you that xia2 is not smart enough to "get" this - so for instance aliasing xds rather than putting it in your path will stop xia2 -3d from working.

If someone would like to look into fixing this, please feel free to get in touch. Until then, you have been warned ;o)

New year, new XDS

2011-01-12T01:35:00.000-08:00

Following Kay's email, the new XDS appears to work fine with xia2. As usual, a little editing will be needed in $XIA2_ROOT/Wrappers/XDS/XDS.py around line 78 -

   supported_versions = ['December 28, 2009',
   'February 3, 2010',
   'May 10, 2010',
   'December 6, 2010']

As always, this will be fixed in the next released build.

Reminder!

2011-01-07T06:35:00.001-08:00

http://xia2.blogspot.com/2010/05/summary-of-command-line-options.html

Using -resolution ...

2011-01-07T06:33:00.001-08:00

Quick note - not sure that the interaction of -resolution with -3d is working quite correctly, so please run with -3dold if you want to specify the resolution!

Resolution limits in xia2

2010-12-16T02:38:00.000-08:00

By default the resolution limits produce what I hope is "something sensible" - however there are cases where for perfectly good reasons people may disagree with these. It's therefore about time these were better explained.

The default resolution limits are:

I/sigma > 1 at the edge
Merged(I/sigma) > 2 at the edge

These are determined with my own merging code not scala or XSCALE, so there could be some small differences at the end. These are controlled by the command line options -isigma and -misigma respectively, and whichever one is lower will be used. Some examples are helpful here, so I will use a data set recorded last week on I24 in 9 seconds (how cool is that)

Default run:

High resolution limit                           1.62    7.23    1.62
Low resolution limit                            31.73   31.73   1.66
Completeness                                    99.8    98.6    99.7
Multiplicity                                    5.7     5.3     5.3
I/sigma                                         14.9    35.9    2.4
Rmerge                                          0.066   0.026   0.695
Rmeas(I)                                        0.073   0.028   0.771
Rmeas(I+/-)                                     0.073   0.028   0.771
Rpim(I)                                         0.03    0.012   0.328
Rpim(I+/-)                                      0.03    0.012   0.328
Wilson B factor                                 19.666
Partial bias                                    0.0     0.0     0.0
Anomalous completeness                          98.1    100.0   95.9
Anomalous multiplicity                          3.0     3.6     2.7
Anomalous correlation                           -0.028  -0.093  -0.015
Anomalous slope                                 0.973   0.0     0.0
Total observations                              193625.0        2465.0  13041.0
Total unique                                    33686.0 465.0   2438.0

Being very cautious - merged and unmerged I/sigma > 2:

-isigma 2 -misigma 2 (unmerged I/sigma will dominate)

High resolution limit                           1.78    7.94    1.78
Low resolution limit                            31.73   31.73   1.82
Completeness                                    99.8    98.1    99.8
Multiplicity                                    5.9     5.2     5.5
I/sigma                                         18.7    35.6    4.8
Rmerge                                          0.058   0.025   0.338
Rmeas(I)                                        0.063   0.028   0.374
Rmeas(I+/-)                                     0.063   0.028   0.374
Rpim(I)                                         0.026   0.011   0.157
Rpim(I+/-)                                      0.026   0.011   0.157
Wilson B factor                                 18.981
Partial bias                                    0.0     0.0     0.0
Anomalous completeness                          98.8    100.0   96.6
Anomalous multiplicity                          3.1     3.6     2.8
Anomalous correlation                           0.004   0.261   -0.001
Anomalous slope                                 0.98    0.0     0.0
Total observations                              150435.0        1872.0 10262.0
Total unique                                    25633.0 362.0   1872.0

Being very generous - unmerged I/sigma > 0.2, merged > 1:

-isigma 0.2 -misigma 1 (merged I/sigma will dominate)

High resolution limit                           1.51    6.75    1.51
Low resolution limit                            31.73   31.73   1.55
Completeness                                    99.8    98.9    99.3
Multiplicity                                    5.5     5.2     4.1
I/sigma                                         12.5    35.4    1.2
Rmerge                                          0.074   0.026   1.095
Rmeas(I)                                        0.081   0.029   1.253
Rmeas(I+/-)                                     0.081   0.029   1.253
Rpim(I)                                         0.034   0.012   0.594
Rpim(I+/-)                                      0.034   0.012   0.594
Wilson B factor                                 19.471
Partial bias                                    0.0     0.0     0.0
Anomalous completeness                          96.9    100.0   88.8
Anomalous multiplicity                          2.9     3.5     2.2
Anomalous correlation                           -0.042 0.072   -0.038
Anomalous slope                                 0.97    0.0     0.0
Total observations                              228772.0        2991.0 12145.0
Total unique                                    41226.0 570.0   2964.0

There are also limits you can apply in terms of Rmerge etc, however these are essentially the same as limiting the unmerged I/sigma as Rmerge ~ 0.8 / (unmerged I/sigma)

pychef error on Mac OS X

2010-12-14T01:33:00.000-08:00

Due to a bit of sloppy programming on my part the pychef implementation is broken on Mac OS X - one line fault, no idea how it got through, but this will cause failure on any data set run.

The fix is simple: in $XIA2_ROOT/Modules/PyChef/PyChefMain.py add

import subprocess

anywhere near the top - e.g. after "import sys". Thanks to Jim Naismith for pointing out the problem.

Much delayed documentation begins

2010-12-06T02:17:00.001-08:00

Starting at

https://docs.google.com/document/pub?id=1XjiSpNqONSAkPKRgKJOxtBq0SaPohbcKrlTKGtmZWzU

Shout if there are things you would like in there (full definition of .xinfo file structure springs to mind)

ESRF ID29

2010-11-25T02:31:00.000-08:00

I have had problems reported when using xia2 to process data from the new Pilatus instrument on ESRF ID29. It seems that there are two problems - reading the image headers for xia2 and Mosflm handling the different image header that they are using. The former I have managed to mend and a patch is available, and Harry is looking to fix the latter in the forthcoming Mosflm release.

Meantimes, with the above patch xia2 -3dii or xia2 -indexer xds -integrater xdsr -scaler xdsr will work.

xia2 0.3.2.0 released

2010-10-21T01:31:00.000-07:00

xia2 now updated to 0.3.2.0:

Changes since 0.3.1.7

Now report the min, mean, max mosaic spread (according to the program definitions after integration of each sweep.
Resolved problems with overriding environment.
Added -serial pun for -parallel 1.
Allow resolution limit to be assigned on a per-sweep basis.
Handled corner case with images 200 - 250 (say) giving REBATCH error.
Added full 8-way search for scaling models for 2d pipelines.
Partial support for two-theta offset data collection (tested with Rigaku X-ray equipment and Diamond I19 data.)
Added -min_images keyword which is helpful when analysing virus data with e.g. 4 images / sweep.

Downloads in the usual place - please shout if you have any problems!

Used xia2 in a publication? Please let me know!

2010-10-11T07:37:00.000-07:00

At some point I'd like to be in the position to get some help with xia2 developments. If such a situation does arise, it would be much easier if I had an indication of how useful the program has been to people. Now I know I didn't help the cause by taking my time to produce a xia2 paper to cite, to I accept that this is very much may fault :o) However I'd really like to hear from you.

Thanks in advance!

Sweeps starting with image 0: Part Deux

2010-10-05T01:22:00.001-07:00

Turns out this may have been self induced, see:

https://sourceforge.net/tracker/index.php?func=detail&aid=3080972&group_id=211879&atid=1019527#

However is now fixed in revision 3306: changes to Experts/FindImages.py

Replacement file:

http://xia2.svn.sourceforge.net/viewvc/xia2/trunk/xia2/Experts/FindImages.py?revision=3306

Sweeps starting with image 0

2010-10-01T02:08:00.000-07:00

As some of you will know, if you put in some data into xia2 with images starting from 0 it may well explode. XDS gives an error, as does REBATCH in the CCP4 suite (which you would expect to be able to renumber the batches!)

So, the easiest thing to do I think is to externally renumber the images. I will therefore write a little python program to do this - watch this space.

Bugs in xia2

2010-09-06T03:20:00.000-07:00

So as we all know, xia2 has bugs. More are discovered from time to time. I'm moving to the public tracker at sf.net. If you'd like to see what's going on in this department (and maybe even check up on the progress of your bugs) please visit:

https://sourceforge.net/tracker/?group_id=211879&atid=1019527

Downloads now available from sf.net

2010-08-24T02:26:00.001-07:00

The files are exactly the same as I have been distributing from http://www.ccp4.ac.uk/xia but now mirrored at

https://sourceforge.net/projects/xia2/files/

I will aim to use this more than ccp4 in the future. Please let me know if you have any problems.

Resolution buglet

2010-08-18T06:50:00.000-07:00

So it's become clear that the "fixes" to the resolution limit calculations I made a while back - while useful - broke the ability for you the user to set what you want as a resolution limit, as in essence it ignores you. This was the same from the command line or xinfo file. This is nor repaired in revision #3282 and will be included in the next release.

New mailing list

2010-08-06T03:13:00.001-07:00

I have set up a mailing list at sourceforge.net as I left the organisation which host this list 18 months ago - if you are keen to keep subscribing, please go to

https://lists.sourceforge.net/lists/listinfo/xia2-list

and add in your email address. It uses mailman to manage, so it's pretty easy going.

Bug reports

2010-08-05T03:53:00.000-07:00

Hi Folks,

I've decided to start using the sourceforge bug tracker in addition to my own internal one, to handle user bug reports & feature requests - this is available from here:

https://sourceforge.net/tracker/?group_id=211879&atid=1019527

You can post anonymous bugs, but if you either sign in or put your email in to the report you can here when things happen...

Cheerio, Graeme

0.3.1.7 release

2010-07-27T02:10:00.000-07:00

Somewhat overdue, done in a hurry to get the just mentioned fix out:

Changes since 0.3.1.6

Changes suitable for working on microcrystals - reachable through a -microcrystal command line flag, and -failover is probably helpful too.
If lattice or spacegroup specified, do not run tests.
2dr and 3dr now the default pipelines - use 2dold or 3dold if you need to get to the old pipelines.
Fixed selection of known lattice in IDXREF reindexing.

Please get in touch if you would like more information on the microcrystal option - this is definitely work in progress...

Problematic indexing when setting spacegroup

2010-07-27T01:57:00.000-07:00

So, following a user problem I have tracked down an issue in the indexing with XDS: if you specify the lattice to use but this has rather a high penalty, the solution will be mangled. This is a bad thing. Now I have tracked down where this was going wrong and fixed it!

About time we had a site file?

2010-06-15T06:09:00.000-07:00

So the old chestnut of backstop masking has re-appeared, and this time I figured it would be easiest not to try and code up something clever, but instead let you (or your beamline scientist) tell xia2 where the beamstop etc. are.

So - what I have lined up for the site file so far are:

a way of locating the beam stop as a function of some distances
the ADC offset for CCD detectors

If you have a beamline where you run xia2, or a lab source, and would like something else propogated please drop me a line - I will be all ears.

Latest XDS

2010-05-23T22:45:00.000-07:00

Just found out that there is a new version of XDS out there - it appears to work fine with xia2, but you will need to do the usual edits of ${XIA2_ROOT}/Wrappers/XDS/XDS.py to add

'May 10, 2010'

to the list of supported versions. Next version of xia2 will include this.

Multi-crystal progress

2010-05-19T07:21:00.000-07:00

So - hooda thunkit - it turns out that this is hard. Nevertheless it's going somewhere though there's a hint of the whac-a-mole debugging going on here. At the moment it's working towards handling the trying-to-assemble-a-dataset-from-dozens-of-crystals turf - so assuming a (small) handful of degrees of data from each - along the lines of http://www.nature.com/emboj/journal/v29/n2/abs/emboj2009352a.html.

Most interesting part of this is that all of the program steps need to be hardened against what would usually be fatal errors - negative scales and the like - and just plough on regardless. An interesting paradigm.

Summary of command line options

2010-05-05T01:50:00.000-07:00

I realised that the command-line argument documentation is (woefully) out of date. Until I think of a better way to handle the command line (perhaps using Phil from CCTBX) this will be hard to correct. However, here are the current list of "useful" options:

-debug
-interactive (for selecting images for autoindexing with Mosflm, Labelit)
-2d (old mosflm / scala)
-2dr (new mosflm / scala - recommended)
-3d (old XDS / XSCALE)
-3dr (new XDS / XSCALE)
-3dii / 3diir (as -3d/3dr, using all frames for autoindex)
-indexer -integrater -scaler - manually specify the integrater, indexer, scaler to use
-image /process/this/sweep
-beam x,y (Mosflm reference frame)
-atom Se (say) - switch on separation of anomalous pairs
-project foo
-crystal bar
-xinfo project.xinfo
-parallel N (set to 1 to swtich off parallel processing)
-xparallel N (use multiple machines for XDS - requires working forkintegrate scripts)
-spacegroup P212121
-resolution 2.8
-freer_file free.mtz
-reference_reflection_file reference.mtz (to get standard indexing, will copy spacegroup, free column)

That's all for the moment - most of these I never use so I had to consult the code...

xia2 0.3.1.6 released

2010-04-21T05:41:00.000-07:00

Much delayed release, now seems to be working well. Also includes support for recent XDS &c.

Sorry for the delay, thankyou for your patience.

Changes since 0.3.1.0

Fixed bug where the resolution limit was not reset in an indexer solution elimination.
Added support for XDS from Dec 09.
Fixed some of the issues found in the resolution limit determination when processing low resolution data.
Fixed use of I/sigma limit assignment.
Added command-line control of individual indexer, integrater, scaler.
Repaired regular expression for image matching, to cope with images where there is e.g. 3.5 in the template. This was previously misinterpreted as an image name of the form setup.NNNN.
Implemented new resolution limit pipelines, based on new merging code to give more robust control: -2dr and -3dr (recommended.) Fine control over the choices can be made using the -isigma, -misigma, -completeness and -rmerge command line options.
Added interactive indexing mechanism, where user can assign images to use for autoindexing, e.g.:
```
Existing images for indexing: 1, 90, 180
>1, 60, 120, 180
New images for indexing: 1, 60, 120, 180
 
```
Assign -interactive on the command line (Mosflm and Labelit indexers)

Multi crystal data analysis... started!

2010-04-07T03:48:00.000-07:00

Ok, so have finally made a start on the multi-crystal analysis (the resolution limits stuff will be released soon, after some more testing) and crikey it's gonna be fun. Should get the ol' grey cells working though.

Initial plans are to build this through the xinfo hierarchy - deifining multiple crystals within the same project - but if you have a better idea for the interface please shout!

xia2 for small molecule data

2010-03-30T07:50:00.001-07:00

Have wondered about extending xia2 to work with data from small molecule crystals - a few of the assumptions would need to change, not least in the indexing, scaling and description of the geometry. However, should be doable.

Interested? If so please drop me a line.

Resolution limit overhaul...

2010-03-30T00:57:00.000-07:00

Not before time, have finally overhauled the resolution limit calculations. At the moment they rely on a separate piece of code to merge the reflections after scaling, and some curve fitting to decide some limits. The defaults are:

- I/sigma > 1
- Mn(I/sigma) > 2
- completeness anything
- Rmerge anything

And are accessed through the command line as -isigma -misigma -completeness [0,1] -rmerge. This will form the kernel of the next release 0.3.1.5. N.B. to access the new resolution limit code, use pipelines -2dr -3dr in place of -2d -3d. Worth it.

Also includes: changes to workflow for these pipelines, use of Scala sd correction automation stuff, -interactive mode for autoindexing with Mosflm / Labelit to allow guidance of image selection.

xia2 moves to sourceforge.net

2010-03-09T01:34:00.000-08:00

Now seems to be working right, and also appears to have managed to include the history. Access from the command-line via:

svn co https://xia2core.svn.sourceforge.net/svnroot/xia2core/trunk/xia2core xia2core
svn co https://xia2.svn.sourceforge.net/svnroot/xia2/trunk/xia2 xia2

which will give you access to the current state of the trunk - no guarantees! Can also download a tarball of the trunk in a similar way. I will now be using this as the primary repository - only reason to move may be if sourceforge.net proves unreliable - unlikely.

If you'd like to have write access, to contribute to the development, please shout!

xia2 with weak pilatus data

2010-02-22T01:41:00.000-08:00

Seems like the latest XDS included a big fix for the processing of weak (i.e. background ~ 0) pilatus data - thanks to Tobias Krojer for the heads-up there. This will involve some slight fixes to the xia2 XDS wrappers, so I will bundle these later today.

N.B. this will require installation of latest XDS!

I should also start to think about how to make fixes like this more responsive... ideas anyone?

xia2 questions...

2010-02-21T02:43:00.001-08:00

May be worth posting here too:

http://www.majorgroove.org/questions

in case anyone else would find the answers interesting...

Refactoring resolution calculations

2010-02-09T02:08:00.000-08:00

Quick update on this. Phase 1 will be to try and get xia2 to better respect the existing resolution criteria - so when it should be I/sigma ~ 2 at the edge, it is! Then after this, refactor the input a little so that the criteria discussed on the mailing list can be assigned through the input or from a preferences file.

This is quite interesting actually. It's shaking things up a little and may end up with things being a little more efficient...

Weirdness managed

2010-01-28T07:49:00.000-08:00

So it seems that this should have been a problem for a lot longer. Thanks to Ralf GWK and Phil Evans for a chat about this which clarified things. Now will pass environment through to the underlying programs correctly (is the intention) though this turned out to be a moderately low level change. Have made this now, and will test, next planned release which should be available sometime soon will contain this fix...

Pointless (latest) and Phenix (latest) slight weirdness

2010-01-27T02:08:00.000-08:00

It appears that the combination of the latest (6.1.3) CCP4 and the latest (1.6) Phenix will cause problems for pointless. This will exhibit itself in xia2 as:

Status: error "no element found: line 27, column 0"

as a result of pointless having an error like:

/scratch/xtal/ccp4/ccp4-6.1.3/bin/pointless: symbol lookup error: /scratch/xtal/ccp4/ccp4-6.1.3/bin/pointless: undefined symbol: _ZNK5cctbx5sgtbx16space_group_type13lookup_symbolEv

I found installing a CCTBX from cctbx.sf.net made the problem "go away", but I have no idea why. Investigating!

Resolution limits ignored? Fix!

2010-01-26T01:49:00.000-08:00

Hi Folks,

Got an error report from Chip Lesburg, that the -i_over_sigma_limit was ignored. This was perfectly accurate, and it turns out that yes, it was ignored. Fix is to change the calls to determine_scaled_resolution in Modules/CCP4ScalerImplementation.py and Modules/XDSScaler.py to replace 2.0 and 3.0 respectively to Flags.get_i_over_sigma_limit(). Will be fixed in next version.

Yes, these Python modules are nasty code, refactoring them is a separate problem!

Graeme

Resolution limit things

2010-01-19T07:15:00.000-08:00

So, I got a complaint from someone who works with membrane proteins, who complained (accurately) that xia2 falls over at the resolution limit calculation step. This was because it assumes a Wilson plot, but this is only meaningful for better than about 4A.

xia2 should now cope - if the data are not there for a Wilson plot calculation it will fail over to a simple hunt for where I/sigma is about 1. Gets it out of a corner here anyway...

Happy new XDS

2010-01-04T00:59:00.000-08:00

Happy new year folks. As is traditional this time of year, XDS changes. Fortunately the new one works (or appears to) just fine with xia2, so all that is necessary is to change the date string in $XIA2_ROOT/Wrappers/XDS/XDS.py. Next update will of course include this change...

xia2 0.3.1.0 release: this is a big one

2009-12-16T02:59:00.000-08:00

Hi Folks,

A new xia2 release for you, this time with added radiation damage analysis and html report generation. Release notes below, download available from the usual place. This isa substantial update over the last release and includes the changes reported below.

Cheers,

Graeme

Changes since 0.3.0.6

Added -xparallel command line option to allow use of forkinitegrate on a cluster with the 3d pipeline.
Allow for xia2html (work by Peter Briggs) and (beta) ISPyB output (latter more useful for synchrotron sources, former handy if you want a report.
Smart scaling mode switched on by default.

Chef now run by default, for radiation damage analysis, e.g.

Group 1: 2-wedge data collection
Group 2: Single wedge
Significant radiation damage detected:
Rd analysis (12287/LREM): 15.17
Rd analysis (12287/INFL): 7.89
Rd analysis (12287/PEAK): 12.82
Conclusion: cut off after DOSE ~ 608.8

Blank images now handled more gracefully.
Overloaded and blank images now correctly reported in the summary output, along with abandoned images for the processing with Mosflm. Also postrefinement results.
Substantially cleaned up the program output.

Finally, a paper

2009-12-01T00:56:00.001-08:00

http://journals.iucr.org/j/issues/2010/01/00/issconts.html

Couldn't have planned it

2009-11-20T01:04:00.000-08:00

Whoda thunkit? Seems that xia can mean "summer" in chinese (according to Natalie Zhao of CCP4) and so the logo could look like this!

Clustery parallelism goodness

2009-11-17T06:27:00.000-08:00

So, I had a request the other day to look at how xia2 could work on a cluster. Back in the day this was a consideration, but it kind of dropped by the wayside what with multi-cpu machines becoming the norm and anyway with Mosflm and Scala, the bottleneck simply moves to scaling. Well, I am reminded that XDS works nice with clusters, so I have added a "secret command line option" which will tell XDS to integrate in parallel across N jobs, using (by default) all of the CPU's in each. -xparallel.

For those cases where you simply have a lot of images, it can be really nice to use XDS like this. Just running a slightly bonkers example just now (1440 degrees, reasonably big unit cell, 1.75A) and it goes from 2h 39m 43s on one eight core machine to 19m 41s over 16.

There's plenty of time to be trimmed from this yet, too - watching top while this happened was just painful! This will be available in the forthcoming 0.3.1.0 release. Just keep in mind you need to configure your forkintegrate scripts to run with your cluster environment - feel free to get in touch for pointers with this.

www.xia2.org now live

2009-11-13T12:19:00.000-08:00

"Proper" project web pages now live at:

http://www.xia2.org/

Broken xia2 cure-all

2009-11-09T07:15:00.000-08:00

A couple of questions came in recently for which the answer appeared to be:

- remove $CCP4/share/XIAROOT (which is obsolete)
- source cctbx setup after CCP4 as the latter pollutes the environment

This will probably fix the problem, particularly if it complains about iotbx :o)

Better output as HTML

2009-11-02T01:55:00.001-08:00

Work has started on the HTMLification of the xia2 output with some success already:

http://www.ccp4.ac.uk/xia/xia2html/

Any comments? Please send to the usual places! Feedback is what drives development.

Bug in CCP4/diffdump with RAXIS on 64 bit

2009-10-29T04:24:00.001-07:00

A bug has been identified in diffdump from CCP4, which came from xia2 and is mostly used therein, when compiled on 64 bit. In particular, in

lib/DiffractionImage/Rigaku/RaxisImage.h, after

// This routine assumes that the size of a long is 4 bytes.

change long to int32_t (sed 's/long/int32_t/g') in RaxisImage.h and
ReadRAXISImages.cpp

Thanks to Takaaki Fukami for pointing this one out. If you use precompiled binaries (32 bit) on the 64 bit system everything should be fine.

Migration of xia2

2009-10-22T07:51:00.001-07:00

So I've been lucky enough to get the URL www.xia2.org, and sourceforge accounts for the different components (for hard-core individuals who want the very latest versions...) so I will start to migrate from the ccp4 servers shortly.

Right now I am trialling moving the xia2 downloads to:

ftp://ftpanon.diamond.ac.uk/xia2/

but for the forseeable, everything that is currently will remain.

As always, shout if you have any problems!

Questions on program output

2009-10-22T06:19:00.000-07:00

Hi Folks,

In an effort to finally tidy up the program output a little, I have made a start on slimming down the default (i.e. not -debug) program output. What I am looking at at the moment is something along the lines of that which follows below (a fixed-width font is probably a good idea here...)

So, here's the question: is there anything you think is currently missing from the program output? Anything that you always look at in a program log file (other than graphs: more on this below) which could be copied over? Anything which should be shouted loudly so you can find it easily?

In other news, Peter Briggs, of CCP4 fame, will now be looking at updating Baubles to take the plain text xia2 output and make it pretty, most likely as a web page, which should allow click-through access to things like the scala and truncate log files with the graphs therein rendered ... as graphs. If you have any interest in this please also shout on what you would like to see... at this end (i.e. at Diamond) we would like to get nice output visible through the web for remote-access users. It's also well over due for review anyway.

So, comments & questions please!

Many thanks,

Graeme


Environment configuration...
XIA2_ROOT => /home/gw56/CVS/xia2
XIA2CORE_ROOT => /home/gw56/CVS/xia2core
CCP4 => /scratch/xtal/ccp4/ccp4-6.1.1
CLIBD => /scratch/xtal/ccp4/ccp4-6.1.1/lib/data
BINSORT_SCR => /tmp/tmppNIYC2
Host: ws050
XIA2 0.3.0.6
------------------- Autoindexing SWEEP1 --------------------
All possible indexing solutions:
tP  51.80  51.80 158.18  90.00  90.00  90.00
oC  73.23  73.34 158.18  90.00  90.00  90.00
oP  51.79  51.84 158.18  90.00  90.00  90.00
mC  73.34  73.23 158.19  90.00  89.95  90.00
mP  51.80 158.18  51.86  90.00  89.91  90.00
aP  51.81  51.86 158.20  89.95  89.98  89.91
Indexing solution:
tP  51.80  51.80 158.18  90.00  90.00  90.00
Sweep epoch: 1096203695
-------------------- Integrating SWEEP1 --------------------
Processed batches 1 to 60
Integration status per image:
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => ok          "%" => iffy rmsd "!" => bad rmsd
"O" => overloaded  "#" => many bad  "." => blank
Need to rerun the integration...
-------------------- Integrating SWEEP1 --------------------
Processed batches 1 to 60
Integration status per image:
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => ok          "%" => iffy rmsd "!" => bad rmsd
"O" => overloaded  "#" => many bad  "." => blank
Stored integration parameters for crystal X12287
------------------- Autoindexing SWEEP2 --------------------
All possible indexing solutions:
tP  51.84  51.84 158.23  90.00  90.00  90.00
oC  73.22  73.51 158.27  90.00  90.00  90.00
oP  51.82  51.89 158.24  90.00  90.00  90.00
mC  73.51  73.22 158.28  90.00  89.93  90.00
mP  51.85 158.28  51.94  90.00  89.77  90.00
aP  51.86  51.95 158.28  89.93  89.97  89.77
Indexing solution:
tP  51.84  51.84 158.23  90.00  90.00  90.00
Using integration parameters for crystal X12287
Sweep epoch: 1096203943
-------------------- Integrating SWEEP2 --------------------
Processed batches 1 to 60
Integration status per image:
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => ok          "%" => iffy rmsd "!" => bad rmsd
"O" => overloaded  "#" => many bad  "." => blank
--------------------- Preparing X12287 ---------------------
Likely spacegroups:
P 41 21 2
P 43 21 2
Reindexing to first spacegroup setting: P 41 21 2 (h,k,l)
---------------------- Scaling X12287 ----------------------
Resolution limit for INFL:  1.64
Resolution limit for LREM:  1.64
Optimised SD corrections (A, B) found to be:
Full:       0.03   12.0
Partial:    0.02   12.0
Computed average unit cell (will use in all files)
51.65  51.65 157.70  90.00  90.00  90.00
Overall twinning score: 2.13
Your data do not appear to be twinned

Inter-wavelength B and R-factor analysis:
INFL   0.0 0.00 (ok)
LREM   0.0 0.08 (ok)

Project: TB0541B
Crystal: X12287
Sequence:
Wavelength name: INFL
Wavelength 0.97966
Sweeps:
SWEEP SWEEP1 [WAVELENGTH INFL]
TEMPLATE 12287_1_E1_###.img
DIRECTORY /scratch/gw56/data/jcsg/als1/8.2.1/20040926/collection/TB0541B/12287
DETECTOR adsc
EXPOSURE TIME 5.100000
PHI WIDTH 1.00
IMAGES (USER) 1 to 60
MTZ file: /scratch/gw56/paper/12287/2d/X12287/INFL/SWEEP1/integrate/12287_1_E1__001.mtz
Wavelength name: LREM
Wavelength 1.00000
Sweeps:
SWEEP SWEEP2 [WAVELENGTH LREM]
TEMPLATE 12287_1_E2_###.img
DIRECTORY /scratch/gw56/data/jcsg/als1/8.2.1/20040926/collection/TB0541B/12287
DETECTOR adsc
EXPOSURE TIME 5.100000
PHI WIDTH 1.00
IMAGES (USER) 1 to 60
MTZ file: /scratch/gw56/paper/12287/2d/X12287/LREM/SWEEP2/integrate/12287_1_E2__001.mtz
For TB0541B/X12287/LREM
High resolution limit                     1.64 7.33 1.64
Low resolution limit                      52.57 52.57 1.68
Completeness                              95.7 98.2 71.8
Multiplicity                              4.2 3.3 2.6
I/sigma                                   13.9 27.2 2.2
Rmerge                                    0.053 0.032 0.354
Rmeas(I)                                  0.067 0.04 0.528
Rmeas(I+/-)                               0.069 0.041 0.489
Rpim(I)                                   0.032 0.021 0.31
Rpim(I+/-)                                0.043 0.026 0.336
Wilson B factor                           18.959
Partial bias                              0.005 -0.028 0.009
Anomalous completeness                    94.3 98.9 67.7
Anomalous multiplicity                    2.3 2.2 1.4
Anomalous correlation                     -0.234 -0.524 -0.021
Anomalous slope                           0.869 0.0 0.0
Total observations                        108108.0 1247.0 3576.0
Total unique                              25995.0 375.0 1383.0

For TB0541B/X12287/INFL
High resolution limit                     1.64 7.33 1.64
Low resolution limit                      52.57 52.57 1.68
Completeness                              97.4 98.0 79.8
Multiplicity                              4.2 3.3 2.8
I/sigma                                   12.8 24.9 2.1
Rmerge                                    0.057 0.031 0.38
Rmeas(I)                                  0.097 0.068 0.582
Rmeas(I+/-)                               0.073 0.041 0.513
Rpim(I)                                   0.045 0.035 0.328
Rpim(I+/-)                                0.046 0.026 0.342
Wilson B factor                           19.244
Partial bias                              -0.007 -0.04 0.011
Anomalous completeness                    96.2 99.4 74.0
Anomalous multiplicity                    2.3 2.2 1.5
Anomalous correlation                     0.654 0.823 0.119
Anomalous slope                           1.558 0.0 0.0
Total observations                        111923.0 1248.0 4218.0
Total unique                              26437.0 374.0 1526.0

Assuming spacegroup: P 41 21 2
Other likely alternatives are:
P 43 21 2
Unit cell:
51.649  51.649 157.697
90.000  90.000  90.000
mtz format:
Scaled reflections:
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_free.mtz
sca format:
Scaled reflections (INFL):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_scaled_INFL.sca
Scaled reflections (LREM):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_scaled_LREM.sca
sca_unmerged format:
Scaled reflections (INFL):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_unmerged_INFL.sca
Scaled reflections (LREM):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_unmerged_LREM.sca
Processing took 00h 16m 48s
XIA2 used...  ccp4 distl labelit mosflm pointless scala
Here are the appropriate citations (BIBTeX in xia-citations.bib.)
(1994) Acta Crystallogr. D 50, 760--763
Evans, P.R. (1997) Proceedings of CCP4 Study Weekend
Evans, Philip (2006) Acta Crystallographica Section D 62, 72--82
Leslie, AGW (1992) Joint CCP4 and ESFEACMB Newsletter on Protein
Crystallography 26
Leslie, Andrew G. W. (2006) Acta Crystallographica Section D 62, 48--57
Sauter, Nicholas K. and Grosse-Kunstleve, Ralf W. and Adams, Paul D.
(2004) Journal of Applied Crystallography 37, 399--409
Zhang, Z. and Sauter, N.K. and van den Bedem, H. and Snell, G. and
Deacon, A.M. (2006) J. Appl. Cryst 39, 112--119
Status: normal termination

xia2 0.3.0.6 bug fix release

2009-10-15T06:35:00.000-07:00

Hi Folks,

Time for some bug fixing & tidying. An unexpected side-effect of the changes allowing the setting of the cell constants etc. in the previous release was that the lattice tests no longer work for XDS. This is now fixed. Also the output has been tidied (fewer .txt files now) so that should help.

Here's the full text of the changes:

Changes since 0.3.0.5

* No lattice test mode added (you can guess the command-line option) for tricky data sets where this perhaps falls over.
* Fixed side effect of changes which allowed setting of unit cell etc. - lattices were no longer eliminated, failed complaining can't eliminate only solution.
* Tidied up the generation of output files and so on - now only have the main log file and the debug trace.

As always, shout if you have problems!

xia2-0.3.0.5: The FreeR release

2009-09-30T03:19:00.000-07:00

New release, pretty minor except the FreeR calculations are much improved:

Changes since 0.3.0.4

* Now able to specify number or fraction of reflections assigned to the free set, rather than the default 5 percent.
* Added trap for sdcorrection not being refined in XDS correct if multiplicity rather low. Unusual case.
* Assigning a freer_file by definition sets this as an indexing reference and also copies the spacegroup assignment. This is what you would expect!

Changes since 0.3.0.3

* User now able to assign cell constants: use with great care, as there is little in the way of nonsense trapping. This may however be used to handle cases where the solution you want is monoclinic with a pseudo-orthorhombic lattice, where the default indexing would select a different setting. This will not currently work with a Mosflm indexer. Usage is:

-cell a,b,c,alpha,beta,gamma

and the correct symmetry should also be assigned. Can also assign USER_CELL a b c alpha beta gamma in the xinfo file.

xia2-0.3.0.4 (quiet) release

2009-09-15T07:42:00.000-07:00

Small change which I would like kinda tested before this becomes a mainstream feature: ability to assign unit cell constants and symmetry to use on the command-line and (untested) xinfo file.

More information at:

http://www.ccp4.ac.uk/xia/xia2-0.3.0.4.html

N.B. this is not linked as the default xia2 page, will be no xia2bb announcement.

xia2 0.3.0.3

2009-09-01T05:14:00.000-07:00

Hi Folks,

A release update, 0.3.0.3 which includes a number of changes and very important bug fixes. Here are the main changes:

Changes since 0.3.0.0

* Now automatically determine the number of available processor cores.
* Implemented check for centring of crystallographic basis from autoindexing.
* Started to use CCTBX - this will be bundled with the release from now on...
* Now runs "smart scaling" i.e. will customise the scaling model used to the data. This can seriously improve the xia2 run time in cases where some of the default scaling models do not converge.
* Now also includes running of CHEF for radiation damage, which will slice and dice your data into dose groups, then run a correctly time sequenced radiation damage analysis. N.B. for data measured in "dose mode" the doses will be scaled to < 1,000,000 to ensure that the output is tidy.
* Corner case of running XDS pipelines from data from a Rigaku setup. The low resolution was previously calculated to be 0.0! D'oh!
* Added access to reference reflection file functionality to the command line - -reference_reflection_file foo.mtz.
* Fixed the use of the -image command-line option. Now works.
* Removed all of the binaries from the distribution, so now you need to be using CCP4 6.1.0 or later!
* Will correctly handle reindexing with the 3d pipeline (XDS/XSCALE) with trigonal spacegroups and multiple sweeps. Many thanks to Wolfgang Kabsch for a lot of help with this.

As things stand I don't include CCTBX - so you will need to have this installed, information at http://cci.lbl.gov.

Parallel mosflm & rotation axis

2009-08-24T03:14:00.000-07:00

If you have a rotation axis which is not perpendicular to the beam (or very close to such) you may have noticed that the parallel integration sometimes falls over: this is because the postrefinement radius of convergence is ~ 1 degree, but the orientation may differ by more than this.

Next version I'll add support for the mosflm AUTOMATCH keyword which helps here, but only for mosflm version 7.0.5 as it was broken beforehand.

xia2 in CCP4 6.1.2

2009-07-23T04:05:00.000-07:00

Just a quick heads-up. It's likely that the xia2 included in CCP4 6.1.2 will not work out-of-the-box with XDS, as it's a little long in the tooth. I would therefore advise if you are going to use 6.1.2 to drop the latest xia2 over the top of the one found in $CCP4/share/XIAROOT (or just delete that one) so as to avoid any conflicts / weirdness.

CCP4 6.1.x dependency

2009-07-22T00:37:00.000-07:00

In order to simplify the distribution of xia2 - and remove all of the binaries - I'm aiming to move to using CCP4 6.1 *only*. Since it's been around for a while and is available for all of the platforms supported by xia2, this should not be a big problem...

Shout if it is!

xia2 moving to sourceforge

2009-07-20T05:49:00.000-07:00

Since I am moving towards using CCTBX for much of the underlying functionality, xia2 will get smaller. This will in turn mean that it can be kept on a public repository, so I am moving it to sourceforge.net to keep it "next door".

As a test, moved the core to

svn co https://xia2core.svn.sourceforge.net/svnroot/xia2core/trunk xia2core

and this appears to work correctly.

-3d / -3dii + RAXIS bug

2009-06-22T06:41:00.000-07:00

Thanks to a bug report a corner case has been identified, between the use of the 3d or 3dii pipeline and RAXIS (general Rigaku?) detectors. in XDSIndexer and XDSIndexerII I compute the low resolution limit from the spot list. For Rigaku detectors however distance is defined to be negative, which causes problems. Adding this small fix:


        if distance < 0.0:
            distance *= -1

After


        distance = values[14]

Will however fix the problem (I think!)

Buried in coding

2009-06-03T01:50:00.000-07:00

Hi Folks,

The release of the first CCTBX-powered version of xia2 is kind-of on hold right now, first because I want to pick Ralf's brains about bundling CCTBX with xia2 when he is over here, and also because I'm trying to do some real catch-up on the coding of new features / enhancements / bug fixes which have been waiting forever to actually go in. Right now I have added the optimization of corrections to apply to the data in scaling, and now I'm working on the inclusion of chef for the analysis of radiation damage. Then a lot of testing / debugging and documentation will follow, hopefully resulting in the release of the new version.

For the first time I will lean towards a "test release" mentality, where new features need to explicitly switched on through a command-line flag. This is to make sure that the results are sensible (indeed, better) before making them the default...

So, watch this space...

Pseudocentring trap #2

2009-05-13T06:59:00.000-07:00

Ok, now trapped for all of the pipelines - woo! This has involved substantial amount of pretty low-level work, so I'd be delighted to hear if anyone finds this useful! Thanks also to Ralf Grosse-Kunstleve and Nick Sauter - without CCTBX and their help this test would never have been implemented.

Now to try the first bundling of a cctbx powered xia2!

Pseudocentring trap #1

2009-05-11T08:16:00.000-07:00

So one example which has been a thorn in my side has been 2PBL, JCSG sample number 45453 - this is really primitive, but indexes as centred because it nearly is (i.e. the lattice points are systematically weak rather than absent as they would be if it were centred) - now nailed this one for -3dii, where I have a lot of information about the diffraction pattern.

Now for -3d.

Then -2d, which will involve me adding my own spot search. Statistics much stronger using the whole data set though! Oh, and since neither Mosflm nor Labelit can index this case correctly, some hacking of orientation matrices.

End of all this though, it'll be another data set defeated by automation. Always good.

CCTBX simpler now

2009-05-11T00:34:00.000-07:00

Following a very sensible suggestion from Ben Eisenbraun, I will aim to provide versions of xia2 with CCTBX bundled in there - this will make installation more straightforward, though will also include a "lite" version which will work as the current one does, and will assume that the cctbx.python in the path is right.

Of course, it'll also complain if it is not ;o)

CCTBX complexities

2009-05-08T08:08:00.000-07:00

CCTBX-using applications are best launched using cctbx.python rather than a standard system python - however, you may have named it something different, right? So what I will do is at the same time change the way that the launcher scripts work - if you have defined the environment variable XIA2_PYTHON this will be used for the execution, else cctbx.python will be - this means that if you have carefully installed one for xia2 to use, you can tell the program by e.g. exporting XIA2_PYTHON=/usr/xtal/cctbx/cctbx_build/bin/cctbx.python, which will then be executed even if it does not occur in the path.

Probably will make for a simpler life.

Just have to figure out how to test for an environment variable being set on Windows :o(

CCTBX, going for it

2009-04-29T07:31:00.000-07:00

Sometime in the next couple of releases I will probably move to using tools from the CCTBX library. This is free and available for all of the xia2 supported platforms, so should not present a problem, and will allow the xia2 program itself to have access to much more in the way of crystallographic functionality.

It can also replace a fair amount of the code I have written already, which can't be a bad thing!

New feature on the way

2009-04-27T08:03:00.000-07:00

A common complaint I get is that xia2 does not use e.g. all 4 cores in a 4 core machine. Well, it does if you tell it with -parallel N where N is e.g. 4. So anyway, xia2 should be clever, so the next release will figure out how many cores (if you don't tell it) and use all of them by default.

Tidy.

xia2 0.3.0.0 released

2009-04-27T02:29:00.000-07:00

Hi Folks,

I have just put 0.3.0.0 on the server - this is the one which includes the following new features:

Changes since 0.2.7.2

- Fixed merging statistics bug - I/sigma output were very slightly different.
- Now extend FreeR column if copying from lower resolution input file.
- Detector limits now correctly specified around the detector centre for Mosflm, rather than the beam centre. Only useful if your beam centre is a fair way from the image centre...
- Parallelised integration with Mosflm - divides the sweeps into an appropriate number of chunks and then sorts together the resulting reflection files. -parallel N, remember.
- Added a putative 'small molecule' mode, which will autoindex with Mosflm from a modest number of images, for smaller molecules where Labelit is unhappy with the number of good Bragg reflections.

As the parallelisation was a major feature, have bumped the version to 0.3!

Thanks as always to everyone who provides feedback - many of the developments above result from bug reports and xia2 development would go nowhere without help from you the users!

All the best,

Graeme

0.3.0.0 Progress

2009-04-20T06:55:00.001-07:00

Quickly hammering out some of the remaining dents which are preventing the release of 0.3.0.0 xia2 - this will be the one which supports parallel integration, has the fix to the statistical summary calculation from XDS, works with Python 2.6 &c.

Obviously there are a few changes, which by extension raises the likelihood of bugs - however as always I will welcome any comments.

Small molecules

2009-04-20T03:00:00.000-07:00

Had a request to get xia2 working with a small molecule (mP 6.18 22.39 11.65 90.00 99.10 90.00) which Labelit did not like as there were too few reflections to be useful. Have accordingly hacked together a putative small molecule mode - this will use quite a few images for autoindex with Mosflm, but is not yet clever enough for small molecule spacegroups (i.e. knows noting of inversions!)

Will be in 0.3.0.0

Parallel Mosflm's

2009-04-03T08:21:00.001-07:00

So, this is on it's way, for those really big data sets you want to process in lots of chunks... it is now done "by magic" - or will be in the next release:


29681 gw56 R 92.1  0.5   1:16.95 ipmosflm-7.0.3
29683 gw56 R 92.1  0.5   1:18.64 ipmosflm-7.0.3
29684 gw56 R 90.3  0.5   1:18.66 ipmosflm-7.0.3
29685 gw56 R 90.3  0.5   1:20.13 ipmosflm-7.0.3
29687 gw56 R 88.5  0.5   1:18.30 ipmosflm-7.0.3
29680 gw56 R 84.9  0.5   1:21.23 ipmosflm-7.0.3
29686 gw56 R 84.9  0.5   1:18.59 ipmosflm-7.0.3
29682 gw56 R 83.0  0.5   1:17.74 ipmosflm-7.0.3

Just benchmarking now, but for big data sets with lots of images it's got to be a fair amount faster. Numbers will follow...

Twitter

2009-04-03T05:37:00.000-07:00

Just for kicks, to keep up with these modern times, have started a twitter feed so people can see what's happening. Are your bugs being fixed? Best way to find out.

http://twitter.com/GraemeWinter

xia2 and CCP4 6.1 reprise

2009-04-03T01:33:00.000-07:00

So, it is clear that the dependence on xia2 being told the version of CCP4 it is using is a pain, and is in no way future proof, so I have now fixed this. No more '7_Sigma' errors, then.

So you no longer need export XIA2_CCP4_61=1 or -ccp4_61...

Any other annoyances like this, please drop me a line!

Side effect of the XDS update

2009-03-20T08:01:00.001-07:00

If:

- you are in the habit of running xia2 where your images are (i.e. xia2 -3d `pwd`)

- you are in the habit of running xia2 more than once

you may notice a couple of odd lines in the output about unmatched parenthesis. This is a side-effect of XDS now using .cbf files for intermediate data, which look a lot like data to xia2 but are not meaningful.

Other than complaining the data reduction goes fine. I will add a trap for this as soon as I figure a reliable way for spotting the difference.

Rebundled 0.2.7.2 with XDS changes in

2009-03-20T04:01:00.003-07:00

Is available from the usual location, www.ccp4.ac.uk/xia, should contain all of the changes needed to get xia2 working again with the latest XDS.