xia2 0.3.1.0 release: this is a big one

Hi Folks,

A new xia2 release for you, this time with added radiation damage analysis and html report generation. Release notes below, download available from the usual place. This isa substantial update over the last release and includes the changes reported below.

Cheers,

Graeme

Changes since 0.3.0.6

• Added -xparallel command line option to allow use of forkinitegrate on a cluster with the 3d pipeline.
• Allow for xia2html (work by Peter Briggs) and (beta) ISPyB output (latter more useful for synchrotron sources, former handy if you want a report.
• Smart scaling mode switched on by default.
• Chef now run by default, for radiation damage analysis, e.g.
  

  Group 1: 2-wedge data collection
  Group 2: Single wedge
  Significant radiation damage detected:
  Rd analysis (12287/LREM): 15.17
  Rd analysis (12287/INFL): 7.89
  Rd analysis (12287/PEAK): 12.82
  Conclusion: cut off after DOSE ~ 608.8
   

  
  
• Blank images now handled more gracefully.
• Overloaded and blank images now correctly reported in the summary output, along with abandoned images for the processing with Mosflm. Also postrefinement results.
• Substantially cleaned up the program output.

Finally, a paper

http://journals.iucr.org/j/issues/2010/01/00/issconts.html

Couldn't have planned it

Whoda thunkit? Seems that xia can mean "summer" in chinese (according to Natalie Zhao of CCP4) and so the logo could look like this!

Clustery parallelism goodness

So, I had a request the other day to look at how xia2 could work on a cluster. Back in the day this was a consideration, but it kind of dropped by the wayside what with multi-cpu machines becoming the norm and anyway with Mosflm and Scala, the bottleneck simply moves to scaling. Well, I am reminded that XDS works nice with clusters, so I have added a "secret command line option" which will tell XDS to integrate in parallel across N jobs, using (by default) all of the CPU's in each. -xparallel.

For those cases where you simply have a lot of images, it can be really nice to use XDS like this. Just running a slightly bonkers example just now (1440 degrees, reasonably big unit cell, 1.75A) and it goes from 2h 39m 43s on one eight core machine to 19m 41s over 16.

There's plenty of time to be trimmed from this yet, too - watching top while this happened was just painful! This will be available in the forthcoming 0.3.1.0 release. Just keep in mind you need to configure your forkintegrate scripts to run with your cluster environment - feel free to get in touch for pointers with this.

www.xia2.org now live

"Proper" project web pages now live at:

http://www.xia2.org/

Broken xia2 cure-all

A couple of questions came in recently for which the answer appeared to be:

- remove $CCP4/share/XIAROOT (which is obsolete)
- source cctbx setup after CCP4 as the latter pollutes the environment

This will probably fix the problem, particularly if it complains about iotbx :o)

Better output as HTML

Work has started on the HTMLification of the xia2 output with some success already:

http://www.ccp4.ac.uk/xia/xia2html/

Any comments? Please send to the usual places! Feedback is what drives development.

Bug in CCP4/diffdump with RAXIS on 64 bit

A bug has been identified in diffdump from CCP4, which came from xia2 and is mostly used therein, when compiled on 64 bit. In particular, in

lib/DiffractionImage/Rigaku/RaxisImage.h, after

  // This routine assumes that the size of a long is 4 bytes.

change long to int32_t (sed 's/long/int32_t/g') in RaxisImage.h and
ReadRAXISImages.cpp

Thanks to Takaaki Fukami for pointing this one out. If you use precompiled binaries (32 bit) on the 64 bit system everything should be fine.

Migration of xia2

So I've been lucky enough to get the URL www.xia2.org, and sourceforge accounts for the different components (for hard-core individuals who want the very latest versions...) so I will start to migrate from the ccp4 servers shortly.

Right now I am trialling moving the xia2 downloads to:

ftp://ftpanon.diamond.ac.uk/xia2/

but for the forseeable, everything that is currently will remain.

As always, shout if you have any problems!

Questions on program output

Hi Folks,

In an effort to finally tidy up the program output a little, I have made a start on slimming down the default (i.e. not -debug) program output. What I am looking at at the moment is something along the lines of that which follows below (a fixed-width font is probably a good idea here...)

So, here's the question: is there anything you think is currently missing from the program output? Anything that you always look at in a program log file (other than graphs: more on this below) which could be copied over? Anything which should be shouted loudly so you can find it easily?

In other news, Peter Briggs, of CCP4 fame, will now be looking at updating Baubles to take the plain text xia2 output and make it pretty, most likely as a web page, which should allow click-through access to things like the scala and truncate log files with the graphs therein rendered ... as graphs. If you have any interest in this please also shout on what you would like to see... at this end (i.e. at Diamond) we would like to get nice output visible through the web for remote-access users. It's also well over due for review anyway.

So, comments & questions please!

Many thanks,

Graeme

Environment configuration...
XIA2_ROOT => /home/gw56/CVS/xia2
XIA2CORE_ROOT => /home/gw56/CVS/xia2core
CCP4 => /scratch/xtal/ccp4/ccp4-6.1.1
CLIBD => /scratch/xtal/ccp4/ccp4-6.1.1/lib/data
BINSORT_SCR => /tmp/tmppNIYC2
Host: ws050
XIA2 0.3.0.6
------------------- Autoindexing SWEEP1 --------------------
All possible indexing solutions:
tP  51.80  51.80 158.18  90.00  90.00  90.00
oC  73.23  73.34 158.18  90.00  90.00  90.00
oP  51.79  51.84 158.18  90.00  90.00  90.00
mC  73.34  73.23 158.19  90.00  89.95  90.00
mP  51.80 158.18  51.86  90.00  89.91  90.00
aP  51.81  51.86 158.20  89.95  89.98  89.91
Indexing solution:
tP  51.80  51.80 158.18  90.00  90.00  90.00
Sweep epoch: 1096203695
-------------------- Integrating SWEEP1 --------------------
Processed batches 1 to 60
Integration status per image:
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => ok          "%" => iffy rmsd "!" => bad rmsd
"O" => overloaded  "#" => many bad  "." => blank
Need to rerun the integration...
-------------------- Integrating SWEEP1 --------------------
Processed batches 1 to 60
Integration status per image:
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => ok          "%" => iffy rmsd "!" => bad rmsd
"O" => overloaded  "#" => many bad  "." => blank
Stored integration parameters for crystal X12287
------------------- Autoindexing SWEEP2 --------------------
All possible indexing solutions:
tP  51.84  51.84 158.23  90.00  90.00  90.00
oC  73.22  73.51 158.27  90.00  90.00  90.00
oP  51.82  51.89 158.24  90.00  90.00  90.00
mC  73.51  73.22 158.28  90.00  89.93  90.00
mP  51.85 158.28  51.94  90.00  89.77  90.00
aP  51.86  51.95 158.28  89.93  89.97  89.77
Indexing solution:
tP  51.84  51.84 158.23  90.00  90.00  90.00
Using integration parameters for crystal X12287
Sweep epoch: 1096203943
-------------------- Integrating SWEEP2 --------------------
Processed batches 1 to 60
Integration status per image:
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => ok          "%" => iffy rmsd "!" => bad rmsd
"O" => overloaded  "#" => many bad  "." => blank
--------------------- Preparing X12287 ---------------------
Likely spacegroups:
P 41 21 2
P 43 21 2
Reindexing to first spacegroup setting: P 41 21 2 (h,k,l)
---------------------- Scaling X12287 ----------------------
Resolution limit for INFL:  1.64
Resolution limit for LREM:  1.64
Optimised SD corrections (A, B) found to be:
Full:       0.03   12.0
Partial:    0.02   12.0
Computed average unit cell (will use in all files)
51.65  51.65 157.70  90.00  90.00  90.00
Overall twinning score: 2.13
Your data do not appear to be twinned

Inter-wavelength B and R-factor analysis:
INFL   0.0 0.00 (ok)
LREM   0.0 0.08 (ok)

Project: TB0541B
Crystal: X12287
Sequence:
Wavelength name: INFL
Wavelength 0.97966
Sweeps:
SWEEP SWEEP1 [WAVELENGTH INFL]
TEMPLATE 12287_1_E1_###.img
DIRECTORY /scratch/gw56/data/jcsg/als1/8.2.1/20040926/collection/TB0541B/12287
DETECTOR adsc
EXPOSURE TIME 5.100000
PHI WIDTH 1.00
IMAGES (USER) 1 to 60
MTZ file: /scratch/gw56/paper/12287/2d/X12287/INFL/SWEEP1/integrate/12287_1_E1__001.mtz
Wavelength name: LREM
Wavelength 1.00000
Sweeps:
SWEEP SWEEP2 [WAVELENGTH LREM]
TEMPLATE 12287_1_E2_###.img
DIRECTORY /scratch/gw56/data/jcsg/als1/8.2.1/20040926/collection/TB0541B/12287
DETECTOR adsc
EXPOSURE TIME 5.100000
PHI WIDTH 1.00
IMAGES (USER) 1 to 60
MTZ file: /scratch/gw56/paper/12287/2d/X12287/LREM/SWEEP2/integrate/12287_1_E2__001.mtz
For TB0541B/X12287/LREM
High resolution limit                     1.64 7.33 1.64
Low resolution limit                      52.57 52.57 1.68
Completeness                              95.7 98.2 71.8
Multiplicity                              4.2 3.3 2.6
I/sigma                                   13.9 27.2 2.2
Rmerge                                    0.053 0.032 0.354
Rmeas(I)                                  0.067 0.04 0.528
Rmeas(I+/-)                               0.069 0.041 0.489
Rpim(I)                                   0.032 0.021 0.31
Rpim(I+/-)                                0.043 0.026 0.336
Wilson B factor                           18.959
Partial bias                              0.005 -0.028 0.009
Anomalous completeness                    94.3 98.9 67.7
Anomalous multiplicity                    2.3 2.2 1.4
Anomalous correlation                     -0.234 -0.524 -0.021
Anomalous slope                           0.869 0.0 0.0
Total observations                        108108.0 1247.0 3576.0
Total unique                              25995.0 375.0 1383.0

For TB0541B/X12287/INFL
High resolution limit                     1.64 7.33 1.64
Low resolution limit                      52.57 52.57 1.68
Completeness                              97.4 98.0 79.8
Multiplicity                              4.2 3.3 2.8
I/sigma                                   12.8 24.9 2.1
Rmerge                                    0.057 0.031 0.38
Rmeas(I)                                  0.097 0.068 0.582
Rmeas(I+/-)                               0.073 0.041 0.513
Rpim(I)                                   0.045 0.035 0.328
Rpim(I+/-)                                0.046 0.026 0.342
Wilson B factor                           19.244
Partial bias                              -0.007 -0.04 0.011
Anomalous completeness                    96.2 99.4 74.0
Anomalous multiplicity                    2.3 2.2 1.5
Anomalous correlation                     0.654 0.823 0.119
Anomalous slope                           1.558 0.0 0.0
Total observations                        111923.0 1248.0 4218.0
Total unique                              26437.0 374.0 1526.0

Assuming spacegroup: P 41 21 2
Other likely alternatives are:
P 43 21 2
Unit cell:
51.649  51.649 157.697
90.000  90.000  90.000
mtz format:
Scaled reflections:
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_free.mtz
sca format:
Scaled reflections (INFL):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_scaled_INFL.sca
Scaled reflections (LREM):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_scaled_LREM.sca
sca_unmerged format:
Scaled reflections (INFL):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_unmerged_INFL.sca
Scaled reflections (LREM):
/scratch/gw56/paper/12287/2d/X12287/scale/TB0541B_X12287_unmerged_LREM.sca
Processing took 00h 16m 48s
XIA2 used...  ccp4 distl labelit mosflm pointless scala
Here are the appropriate citations (BIBTeX in xia-citations.bib.)
(1994) Acta Crystallogr. D 50, 760--763
Evans, P.R. (1997) Proceedings of CCP4 Study Weekend
Evans, Philip (2006) Acta Crystallographica Section D 62, 72--82
Leslie, AGW (1992) Joint CCP4 and ESFEACMB Newsletter on Protein
Crystallography 26
Leslie, Andrew G. W. (2006) Acta Crystallographica Section D 62, 48--57
Sauter, Nicholas K. and Grosse-Kunstleve, Ralf W. and Adams, Paul D.
(2004) Journal of Applied Crystallography 37, 399--409
Zhang, Z. and Sauter, N.K. and van den Bedem, H. and Snell, G. and
Deacon, A.M. (2006) J. Appl. Cryst 39, 112--119
Status: normal termination

xia2 0.3.0.6 bug fix release

Hi Folks,

Time for some bug fixing & tidying. An unexpected side-effect of the changes allowing the setting of the cell constants etc. in the previous release was that the lattice tests no longer work for XDS. This is now fixed. Also the output has been tidied (fewer .txt files now) so that should help.

Here's the full text of the changes:

Changes since 0.3.0.5

* No lattice test mode added (you can guess the command-line option) for tricky data sets where this perhaps falls over.
* Fixed side effect of changes which allowed setting of unit cell etc. - lattices were no longer eliminated, failed complaining can't eliminate only solution.
* Tidied up the generation of output files and so on - now only have the main log file and the debug trace.


As always, shout if you have problems!

xia2-0.3.0.5: The FreeR release

New release, pretty minor except the FreeR calculations are much improved:

Changes since 0.3.0.4

* Now able to specify number or fraction of reflections assigned to the free set, rather than the default 5 percent.
* Added trap for sdcorrection not being refined in XDS correct if multiplicity rather low. Unusual case.
* Assigning a freer_file by definition sets this as an indexing reference and also copies the spacegroup assignment. This is what you would expect!

Changes since 0.3.0.3

* User now able to assign cell constants: use with great care, as there is little in the way of nonsense trapping. This may however be used to handle cases where the solution you want is monoclinic with a pseudo-orthorhombic lattice, where the default indexing would select a different setting. This will not currently work with a Mosflm indexer. Usage is:

-cell a,b,c,alpha,beta,gamma

and the correct symmetry should also be assigned. Can also assign USER_CELL a b c alpha beta gamma in the xinfo file.

xia2-0.3.0.4 (quiet) release

Small change which I would like kinda tested before this becomes a mainstream feature: ability to assign unit cell constants and symmetry to use on the command-line and (untested) xinfo file.

More information at:

http://www.ccp4.ac.uk/xia/xia2-0.3.0.4.html

N.B. this is not linked as the default xia2 page, will be no xia2bb announcement.

xia2 0.3.0.3

Hi Folks,

A release update, 0.3.0.3 which includes a number of changes and very important bug fixes. Here are the main changes:

Changes since 0.3.0.0

* Now automatically determine the number of available processor cores.
* Implemented check for centring of crystallographic basis from autoindexing.
* Started to use CCTBX - this will be bundled with the release from now on...
* Now runs "smart scaling" i.e. will customise the scaling model used to the data. This can seriously improve the xia2 run time in cases where some of the default scaling models do not converge.
* Now also includes running of CHEF for radiation damage, which will slice and dice your data into dose groups, then run a correctly time sequenced radiation damage analysis. N.B. for data measured in "dose mode" the doses will be scaled to < 1,000,000 to ensure that the output is tidy.
* Corner case of running XDS pipelines from data from a Rigaku setup. The low resolution was previously calculated to be 0.0! D'oh!
* Added access to reference reflection file functionality to the command line - -reference_reflection_file foo.mtz.
* Fixed the use of the -image command-line option. Now works.
* Removed all of the binaries from the distribution, so now you need to be using CCP4 6.1.0 or later!
* Will correctly handle reindexing with the 3d pipeline (XDS/XSCALE) with trigonal spacegroups and multiple sweeps. Many thanks to Wolfgang Kabsch for a lot of help with this.


As things stand I don't include CCTBX - so you will need to have this installed, information at http://cci.lbl.gov.

Parallel mosflm & rotation axis

If you have a rotation axis which is not perpendicular to the beam (or very close to such) you may have noticed that the parallel integration sometimes falls over: this is because the postrefinement radius of convergence is ~ 1 degree, but the orientation may differ by more than this.

Next version I'll add support for the mosflm AUTOMATCH keyword which helps here, but only for mosflm version 7.0.5 as it was broken beforehand.

xia2 in CCP4 6.1.2

Just a quick heads-up. It's likely that the xia2 included in CCP4 6.1.2 will not work out-of-the-box with XDS, as it's a little long in the tooth. I would therefore advise if you are going to use 6.1.2 to drop the latest xia2 over the top of the one found in $CCP4/share/XIAROOT (or just delete that one) so as to avoid any conflicts / weirdness.

CCP4 6.1.x dependency

In order to simplify the distribution of xia2 - and remove all of the binaries - I'm aiming to move to using CCP4 6.1 *only*. Since it's been around for a while and is available for all of the platforms supported by xia2, this should not be a big problem...

Shout if it is!

xia2 moving to sourceforge

Since I am moving towards using CCTBX for much of the underlying functionality, xia2 will get smaller. This will in turn mean that it can be kept on a public repository, so I am moving it to sourceforge.net to keep it "next door".

As a test, moved the core to

svn co https://xia2core.svn.sourceforge.net/svnroot/xia2core/trunk xia2core

and this appears to work correctly.

-3d / -3dii + RAXIS bug

Thanks to a bug report a corner case has been identified, between the use of the 3d or 3dii pipeline and RAXIS (general Rigaku?) detectors. in XDSIndexer and XDSIndexerII I compute the low resolution limit from the spot list. For Rigaku detectors however distance is defined to be negative, which causes problems. Adding this small fix:

        if distance < 0.0:
            distance *= -1

After

        distance = values[14]

Will however fix the problem (I think!)

Buried in coding

Hi Folks,

The release of the first CCTBX-powered version of xia2 is kind-of on hold right now, first because I want to pick Ralf's brains about bundling CCTBX with xia2 when he is over here, and also because I'm trying to do some real catch-up on the coding of new features / enhancements / bug fixes which have been waiting forever to actually go in. Right now I have added the optimization of corrections to apply to the data in scaling, and now I'm working on the inclusion of chef for the analysis of radiation damage. Then a lot of testing / debugging and documentation will follow, hopefully resulting in the release of the new version.

For the first time I will lean towards a "test release" mentality, where new features need to explicitly switched on through a command-line flag. This is to make sure that the results are sensible (indeed, better) before making them the default...

So, watch this space...

Pseudocentring trap #2

Ok, now trapped for all of the pipelines - woo! This has involved substantial amount of pretty low-level work, so I'd be delighted to hear if anyone finds this useful! Thanks also to Ralf Grosse-Kunstleve and Nick Sauter - without CCTBX and their help this test would never have been implemented.

Now to try the first bundling of a cctbx powered xia2!

Pseudocentring trap #1

So one example which has been a thorn in my side has been 2PBL, JCSG sample number 45453 - this is really primitive, but indexes as centred because it nearly is (i.e. the lattice points are systematically weak rather than absent as they would be if it were centred) - now nailed this one for -3dii, where I have a lot of information about the diffraction pattern.

Now for -3d.

Then -2d, which will involve me adding my own spot search. Statistics much stronger using the whole data set though! Oh, and since neither Mosflm nor Labelit can index this case correctly, some hacking of orientation matrices.

End of all this though, it'll be another data set defeated by automation. Always good.

CCTBX simpler now

Following a very sensible suggestion from Ben Eisenbraun, I will aim to provide versions of xia2 with CCTBX bundled in there - this will make installation more straightforward, though will also include a "lite" version which will work as the current one does, and will assume that the cctbx.python in the path is right.

Of course, it'll also complain if it is not ;o)

CCTBX complexities

CCTBX-using applications are best launched using cctbx.python rather than a standard system python - however, you may have named it something different, right? So what I will do is at the same time change the way that the launcher scripts work - if you have defined the environment variable XIA2_PYTHON this will be used for the execution, else cctbx.python will be - this means that if you have carefully installed one for xia2 to use, you can tell the program by e.g. exporting XIA2_PYTHON=/usr/xtal/cctbx/cctbx_build/bin/cctbx.python, which will then be executed even if it does not occur in the path.

Probably will make for a simpler life.

Just have to figure out how to test for an environment variable being set on Windows :o(

CCTBX, going for it

Sometime in the next couple of releases I will probably move to using tools from the CCTBX library. This is free and available for all of the xia2 supported platforms, so should not present a problem, and will allow the xia2 program itself to have access to much more in the way of crystallographic functionality.

It can also replace a fair amount of the code I have written already, which can't be a bad thing!

New feature on the way

A common complaint I get is that xia2 does not use e.g. all 4 cores in a 4 core machine. Well, it does if you tell it with -parallel N where N is e.g. 4. So anyway, xia2 should be clever, so the next release will figure out how many cores (if you don't tell it) and use all of them by default.

Tidy.

xia2 0.3.0.0 released

Hi Folks,

I have just put 0.3.0.0 on the server - this is the one which includes the following new features:

Changes since 0.2.7.2

- Fixed merging statistics bug - I/sigma output were very slightly different.
- Now extend FreeR column if copying from lower resolution input file.
- Detector limits now correctly specified around the detector centre for Mosflm, rather than the beam centre. Only useful if your beam centre is a fair way from the image centre...
- Parallelised integration with Mosflm - divides the sweeps into an appropriate number of chunks and then sorts together the resulting reflection files. -parallel N, remember.
- Added a putative 'small molecule' mode, which will autoindex with Mosflm from a modest number of images, for smaller molecules where Labelit is unhappy with the number of good Bragg reflections.

As the parallelisation was a major feature, have bumped the version to 0.3!

Thanks as always to everyone who provides feedback - many of the developments above result from bug reports and xia2 development would go nowhere without help from you the users!

All the best,

Graeme

0.3.0.0 Progress

Quickly hammering out some of the remaining dents which are preventing the release of 0.3.0.0 xia2 - this will be the one which supports parallel integration, has the fix to the statistical summary calculation from XDS, works with Python 2.6 &c.

Obviously there are a few changes, which by extension raises the likelihood of bugs - however as always I will welcome any comments.

Small molecules

Had a request to get xia2 working with a small molecule (mP 6.18 22.39 11.65 90.00 99.10 90.00) which Labelit did not like as there were too few reflections to be useful. Have accordingly hacked together a putative small molecule mode - this will use quite a few images for autoindex with Mosflm, but is not yet clever enough for small molecule spacegroups (i.e. knows noting of inversions!)

Will be in 0.3.0.0

Parallel Mosflm's

So, this is on it's way, for those really big data sets you want to process in lots of chunks... it is now done "by magic" - or will be in the next release:

29681 gw56 R 92.1  0.5   1:16.95 ipmosflm-7.0.3
29683 gw56 R 92.1  0.5   1:18.64 ipmosflm-7.0.3
29684 gw56 R 90.3  0.5   1:18.66 ipmosflm-7.0.3
29685 gw56 R 90.3  0.5   1:20.13 ipmosflm-7.0.3
29687 gw56 R 88.5  0.5   1:18.30 ipmosflm-7.0.3
29680 gw56 R 84.9  0.5   1:21.23 ipmosflm-7.0.3
29686 gw56 R 84.9  0.5   1:18.59 ipmosflm-7.0.3
29682 gw56 R 83.0  0.5   1:17.74 ipmosflm-7.0.3

Just benchmarking now, but for big data sets with lots of images it's got to be a fair amount faster. Numbers will follow...

Twitter

Just for kicks, to keep up with these modern times, have started a twitter feed so people can see what's happening. Are your bugs being fixed? Best way to find out.

http://twitter.com/GraemeWinter

xia2 and CCP4 6.1 reprise

So, it is clear that the dependence on xia2 being told the version of CCP4 it is using is a pain, and is in no way future proof, so I have now fixed this. No more '7_Sigma' errors, then.

So you no longer need export XIA2_CCP4_61=1 or -ccp4_61...

Any other annoyances like this, please drop me a line!

Side effect of the XDS update

If:

- you are in the habit of running xia2 where your images are (i.e. xia2 -3d `pwd`)

- you are in the habit of running xia2 more than once

you may notice a couple of odd lines in the output about unmatched parenthesis. This is a side-effect of XDS now using .cbf files for intermediate data, which look a lot like data to xia2 but are not meaningful.

Other than complaining the data reduction goes fine. I will add a trap for this as soon as I figure a reliable way for spotting the difference.

Rebundled 0.2.7.2 with XDS changes in

Is available from the usual location, www.ccp4.ac.uk/xia, should contain all of the changes needed to get xia2 working again with the latest XDS.

XDS patch

Hi Folks,

As it seems that the XDS problem is becoming more of a ... problem, I have put together a file containing the changes from 0.2.7.2 to bring it up-to-date with respect to the XDS changes, but hopefully nothing else as there are a couple of developments in flight.

If you download

http://www.ccp4.ac.uk/xia/XDS-patch.tar.bz2

Then unpack it as follows:

tar xvfj XDS-patch.tar.bz2 -C $XIA2_ROOT

xia2 should be updated to work with the latest version! The next proper release will contain this, but I have a lot of other things to go in there too...

New XDS version: Jan 2009

Dear xia2 -3d/3dii users,

Anyone who has tried this will know that it does not work. Unlike earlier changes to XDS which involved just adding the new version number to XDS.py, this one is less trivial. I am working on a change at the moment which will probably precipitate a new release before the old XDS dies.

This will not be backwards compatible, i.e. you will need to update XDS to use this version.

xia2 is moving to Diamond

Just a quick note that I'm moving to Diamond Light Source to a job which will give me much more time to work with xia2. The intention is that the web page, mailing lists and what have you won't change, though my contact details will. I'll post an update here when this happens, though the xi2.support email address is unaffected.

Hey! Why do I have one extra batch?

Just had an interesting user query which caused some confusion but relates to a perfectly reasonable cause - a data set with N frames in it had N+1 batches in the Scala log file using the XDS/XSCALE processing. Cause is very simple - reflections which are measured on the last image but have a centroid on the "next" image which was not measured could get assigned to batch N+1 by pointless, during the reconstruction of the orientation matrix.

So, if you see this it is not really something wrong, just an artefact. As far as I can tell this made no difference to the quality of the data set.

xia2 updates / ccp4 6.1+

So an important question came in yesterday - if I want to update xia2 and I have ccp4 6.1 installed, what do I do? There is a version of xia2 in ccp4 6.1. As it happens you can just delete the xiaroot directory and proceed as usual, or you can drop the new xia2 in place and just fix the setup.sh scripts.

XIA2_CCP4_61=1 is set in there, which should be maintained to tell xia2 about the updated Scala version. In the future this will be fixed to be determined dynamically.

xia2 web page

Hi Folks,

I have also updated the web page to make the downloads more obvious and updated the documentation on how to run xia2 - if you have any comments on this please get in touch or leave a comment on the blog...

Thanks,

Graeme

xia2 release 0.2.7.2

Hi Folks,

New release of xia2 is now available, with the following changes:

Changes since 0.2.7.0

- Fixed to work correctly with XDS and Rigaku Saturn/RAXIS detectors.
- Better determination of resolution limits.
- Correct merging of data from XSCALE (needs to invert the I=F*F scale factor...)
- Fixed output of scalepack unmerged output using 3d pipeline.
- Use fewer frames for background calculation with 3dii pipeline - speeds things up a fair amount.
- Write chef output files with correct SD correction parameters.
- Print SD correction factors for scala runs - if these are bigger than about 2 there is something properly wrong. Also print the same from XDS CORRECT step.

Release available from the usual place: http://www.ccp4.ac.uk/xia

xia2: updates

A couple of updates and notes. Firstly the patch release which includes unscaling of XDS data (i.e. converting back to photon counts from F^2 ~ I scaling) has been delayed by the CCP4 release and the holidays - I need to finish testing and release! Sorry! If you are using xia2 -3d with -parallel and xds_par bombs out, it may be that the stack size is not big enough (I have just found this) so it could be worth making it larger.

Release soon, I promise!