Monday 16 December 2013

Interactive control of indexing images with XDS processing


Now incoming in a nightly build...

[gw56@ws133 interactive]$ xia2 -3dii -interactive -image /data1/gw56/i04-bag-training/th_8_2_0001.cbf
Environment configuration...
XIA2_ROOT => /home/gw56/svn/xia2.trunk
XIA2CORE_ROOT => /home/gw56/svn/xia2.trunk/core
CCP4 => /dls_sw/apps/ccp4/x86_64/6.4.0/6nov2013/ccp4-6.4.0
CLIBD => /dls_sw/apps/ccp4/x86_64/6.4.0/6nov2013/ccp4-6.4.0/lib/data
CCP4_SCR => /tmp/tmpOOG0z9
Working directory: /tmp/interactive
Free space:        15.02 GB
Host: ws133
Build: 4468
Contact: xia2.support@gmail.com
XIA2 0.3.6.3
Command line: xia2 -3dii -interactive -image /data1/gw56/i04-bag-training/th_8_2_0001.cbf
Existing images for indexing th_8_2_0###.cbf: 1-540
>1-100, 441-540
New images for indexing: 1-100, 441-540

Essentially allows you to assign which images to use for XDS indexing for the case where you have a "good bit" you want to use.


Tuesday 3 December 2013

Move to git?

In the spirit of not breaking xia2 for users while wanting to make some substantial changes, have been thinking of moving from subversion to git: this would affect you if you like the most bleeding edge xia2 versions and pull xia2 from svn.

So: has anyone an opinion on this?

Monday 25 November 2013

Incoming: parameters to control xds integration / spot profile

Adding more Phil parameters this morning to control reflecting range, beam divergence etc. however these are not for common use: only when the automatic determination of parameters has failed are these useful.

Please see XDS documentation for details:

http://xds.mpimf-heidelberg.mpg.de/html_doc/xds_parameters.html#REFLECTING_RANGE=

In build 4480 & later having phil file containing

xds.integrate.delphi = 10.0
xds.integrate.reflecting_range = 0.31
xds.integrate.reflecting_range_esd = 0.045
xds.integrate.beam_divergence = 0.25
xds.integrate.beam_divergence_esd = 0.025

will allow you to control these via -phil file.phil

Thursday 21 November 2013

Controlling XDS Refinement in Integration with Phil

By default in xia2 the REFINE= for integration refines ORIENTATION and CELL - however if you want to override this you can do so by using Phil parameters. Create a file

whatever.phil

containing

xds.integrate.refine=CELL
xds.integrate.refine_final=CELL

for example - and run with -phil whatever.phil. The former is the first pass, the second the pass when a refined orientation is used in case you want these to be different. You can select from ORIENTATION CELL BEAM DISTANCE AXIS - best to look at the XDS documentation if you want to change these.

In general you don't need to do this but some people would like the choice...

Thursday 7 November 2013

Very big but not functional commit coming

Warning to anyone who uses xia2 from subversion - forthcoming commit will be very large as I change the indentation from 4 spaces to 2 spaces, to make the code more consistent with the cctbx way of things. I have run a bunch of tests and manually inspected at least 1000 lines of the changes to make sure that they are sensible (and they appear to be) but if you have any local modifications in your code you may / will find updating via svn causes conflicts.

Any problems please contact me at the usual address xia2.support@gmail.com

Wednesday 6 November 2013

xia2 broken with very latest cctbx

If you have a cctbx about 3 days old or younger you will have found errors like

Traceback (most recent call last):
  File "/Users/graeme/svn/xia2/Wrappers/XIA/Diffdump.py", line 648, in readheader
    self._header = failover_dxtbx(self._image)
  File "/Users/graeme/svn/xia2/Wrappers/XIA/Diffdump.py", line 471, in failover_dxtbx
    fast, slow = map(int, d.get_image_size())
AttributeError: 'Detector' object has no attribute 'get_image_size'

This relates to recent changes in cctbx; a fix for xia2 will follow later today.

Update: fix was sent to svn yesterday morning, so nightly build from last night should be fine if you hit this problem... however any problems please contact xia2.support@gmail.com as usual.

Monday 14 October 2013

0.3.6.3 quick release

Enough things are broken by the new CCP4 to make a new version worthwhile:

Changes since 0.3.6.2: Cope with new CCP4.
  • Mend xia2 working on Windows.
  • Cope with new version of CTRUNCATE in CCP4 6.4.0.
  • xia2ccp4i added - thanks to David Waterman.
  • Improved handling of nasty image names.
  • Mend -reversephi functionality.
  • Take spacegroup from reference reflection file correctly.
In the usual place(s)


CCP4 / Windows / xia2 fix

Missing detectors.lib from the CCP4 cctbx / dxtbx installation was the problem - easily fixed:

C:\>cd \CCP4\6.4\lib\python2.7\site-packages\dxtbx
C:\CCP4\6.4\lib\python2.7\site-packages\dxtbx>mkdir data
C:\CCP4\6.4\lib\python2.7\site-packages\dxtbx>cd data
C:\CCP4\6.4\lib\python2.7\site-packages\dxtbx\data>cp \CCP4\6.4\share\xia2\Data\detectors.lib .

This should be mended in the next update.

xia2 / Windows / CCP4 6.4

Looks like the xia2 in CCP4 6.4 is not entirely working fine on Windows - one error is my bad:

NotAvailableError: executable C:\CCP4\6.4\share\xia2\Modules\Resolutionizer.py does not exist in PATH

which I have just fixed (an update will go in a CCP4 update soon) and other errors have been found regarding the reading of image headers which CCP4 are looking into. If you find any problems do please get in touch on the usual xia2.support@gmail.com address.

Friday 11 October 2013

New CCP4 / ctruncate bug

Seems a three line change fixes this:

Index: Wrappers/CCP4/Ctruncate.py
===================================================================
--- Wrappers/CCP4/Ctruncate.py (revision 4432)
+++ Wrappers/CCP4/Ctruncate.py (working copy)
@@ -142,6 +142,9 @@
             if 'Acentric moments of E using Truncate method' in results:
                 moments = transpose_loggraph(
                     results['Acentric moments of E using Truncate method'])
+            elif 'Acentric moments of I' in results:
+                moments = transpose_loggraph(
+                    results['Acentric moments of I'])
             elif 'Acentric moments of E' in results:
                 moments = transpose_loggraph(
                     results['Acentric moments of E'])

Will include this in next release (and as a CCP4 update)

New CCP4

Just installed the new CCP4 - looks like the ctruncate output has changed so xia2 crashes. I will make a patch today.

Thursday 3 October 2013

Reversephi: error(s) found

Reversephi from .xinfo file should be respected now in the committed version or the following change can be made:

[gw56@ws133 xia2.trunk]$ svn diff -r 4428 Schema/XWavelength.py
Index: Schema/XWavelength.py
===================================================================
--- Schema/XWavelength.py       (revision 4428)
+++ Schema/XWavelength.py       (working copy)
@@ -170,7 +170,7 @@
                                    directory = directory,
                                    image = image,
                                    beam = beam,
-                                   reversephi = False,
+                                   reversephi = reversephi,
                                    distance = distance,
                                    gain = gain,
                                    dmin = dmin,


(dumb typo which should have been flagged a long time ago)

More complex situation was found with data where the dxtbx format system reads the image headers, which involved slightly deeper changes. These are now made but N.B. that if the dxtbx header reading class correctly identifies it as reversed phi (which should be the case) then -reversephi will do exactly the opposite to what you want. 

Short answer: you should not need -reversephi or REVERSEPHI in the xinfo file: if you do please get in touch and email me an example image.

Updated xia2 release to follow, though new format classes will need to be added to your cctbx installation - instructions for this will follow.

Reversephi broken

It seems that for beamlines where the phi axis is reversed xia2 is currently broken, even when you set -reversephi on the command-line or REVERSEPHI in the sweep block in the xinfo file. Currently working on this, will release an update when this is fixed.

Thursday 26 September 2013

SBGrid webinar on xia2

A while back I did an SBGrid webinar on xia2 - turns out it ended up on youtube. For new users may prove to be useful.

http://www.youtube.com/watch?v=NFTshcOynCo

Friday 20 September 2013

Release finally made

xia2 0.3.6.2 finally released with features as specified before. A couple of sneaky bug fixes (which have now been resolved) have been holding up proceedings.

Usual place:

http://xia2.sf.net


Thursday 25 July 2013

Incoming features

Following requests being worked on:

  • configurable cctbx.python via env variable
  • remove need for env variable XIA2_ROOT - just put directory in PATH
  • user configurable background range via phil parameter / XDS processing
Plan to have these ready early next week.

Monday 8 July 2013

Bug found in 0.3.6.1 / 2d with P1

Due to bug in Mtzdump replacement Python code this error happens:


-------------------- Integrating SWEEP1 --------------------
Processed batches 1 to 180
Weighted RMSD: 0.69 (0.17)
Integration status per image (60/record):
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => good        "%" => ok        "!" => bad rmsd
"O" => overloaded  "#" => many bad  "." => blank
"@" => abandoned
Mosaic spread: 0.680 < 0.817 < 0.990
-------------------- Preparing DEFAULT ---------------------
Status: error "'P'"


Just testing fix for this now, will be 0.3.6.2 tonight hopefully...

New CCTBX build available...

at http://cci.lbl.gov/cctbx_build/ - if you get errors in xia2 following an update (perhaps following an update to XDS) it may be worth installing this. If you have an unreleased cctbx build or a recent phenix nightly you don't need this. Thanks to Nat Echols for making a release with all the new bits xia2 needs.

Friday 5 July 2013

xia2 0.3.6.1 released

Mainly this is a bug fix release to cope with changes in CCP4 6.3.0 latest update and the new XDS. The only other major change is that the indexing via XDS (i.e. processing with 3di or 3dii) now uses a smarter lattice determination method as previously posted. A minor change has also been that the 2d processing using Mosflm is now more reliable if your beam centre in the image headers is off...

Available in the usual place: http://xia2.sf.net

Please note you will need a recent Phenix nightly build or an unreleased cctbx build to make this work... as always any queries please contact xia2.support@gmail.com

Shameless plug

http://journals.iucr.org/d/issues/2013/07/00/ba5195/ba5195.pdf

If you were wondering about some of the things which go on under the hood of xia2, they are described in here. Worth looking at the supplementary materials, as some of the interesting stuff got pulled as the paper was too long :o(

Friday 28 June 2013

Changes in XDS indexing...

Historically the XDS indexing used rather arbitrary criteria for the selection of the "best" solution - I was however prompted today to look at making this a little more robust, and leaning on the tools available in cctbx. Now the indexing is done in P1 as always and the P1 cell passed to iotbx.lattice_symmetry (see http://journals.iucr.org/a/issues/2004/01/00/sh5006/sh5006.pdf for more details) and only the solutions proposed from this analysis are considered from indexing.

Currently the behaviour is as before if you specify the symmetry on the command line. If however you do not it is less likely to try and process the data with too high a symmetry and generally make a mess of things. I will welcome any feedback on how effective this is.

Clearing clutter...

I have been clearing a lot of clutter from the xia2 repository over the last few days with the idea of simplifying future xia2 maintenance. Obviously I am endeavoring to not break anything however please svn update with care, if that is your usual way of getting xia2 code.


Tuesday 25 June 2013

Nightly "build"

I have set up a nightly checkout and tar up system to push builds to

ftp://ftp.ccp4.ac.uk/xia/nightly

If you would like an updated version - which may be broken - from last night do feel free to test these out. Installation should work essentially as usual.


Monday 24 June 2013

Status of xia2 with freshly installed and updated: XDS, CCP4, Phenix

Since there have been some issues with updating CCP4, XDS and so on reported I am just testing the current svn version with all of the newest packages: on Linux everything seems fine, testing OS X now. Everything worked fine there as well - if you get any issues with what look like pointless errors reading XDS files, please try updating CCP4 / XDS.

Thursday 20 June 2013

xia2 with new new XDS broken again (sorry)

Newest new version of XDS (which still claims to be the 30th March 2013 version) has once again broken xia2, due to unexpected changes to file formats (nothing to do with XDSCONV which xia2 does not use)

Update: looks like a one-liner to fix; just committed fix in revision 4329.

Tuesday 21 May 2013

SVN xia2 now working with newest XDS

At least, in most circumstances - any problems do please get in touch!

Moving targets

Two recent issues to arise:

 - using xia2 with new phenix builds: nightly builds as of last night should now work - dxtbx was not being compiled as a module. Using updated cctbx recent build was a useful workaround though that was also briefly broken for Rigaku A200 instruments (now fixed)

 - new XDS - format for XPARM file has changed so xia2 now chokes - this one may not be as straightforward to fix, but work ongoing


Wednesday 8 May 2013

Making xia2 from svn actually *work*

This requires a relatively recent version of cctbx (some bits of xia2 were moved into cctbx.)

Go to

http://cci.lbl.gov/cctbx_build/

Select "show unreleased builds"; select *green* link like YEAR_MO_DY_NNNN (e.,g. 2013_05_07_0005 today) then download the appropriate build for your machine.

This should be automated as part of a xia2 download script: I will look into it. Please be sure to source the cctbx setup file after CCP4 as CCP4 distribute an old version...

Problems / questions to the usual place: xia2.support@gmail.com


Tuesday 7 May 2013

Installing xia2 from subversion (latest update)

Assumes you have xia2, cctbx, optionally Labelit and XDS set up in your PATH... here is a bash script which will install xia2 and make you a setup file:


#!/bin/sh

svn checkout https://svn.code.sf.net/p/xia2/code/trunk/xia2
cd xia2
echo "#!/bin/bash" > ../xia2.sh
echo "export XIA2_ROOT=`pwd`" >> ../xia2.sh
echo ". \${XIA2_ROOT}/setup.sh" >> ../xia2.sh

just copy this to a file of your choice (say install-xia2.sh) and then run bash install-xia2.sh where you want xia2 to be installed. Sourcing xia2.sh (which is auto generated) will then set up xia2.

Friday 3 May 2013

Nightly build needed: will be on the way

In many cases people wait a while before features become available in xia2 because of the slow release schedule. I am looking into producing a nightly build downloadable from the usual CCP4 home, with test logs.

This may be a better way to keep up with the "bleeding edge" of xia2 than using svn...

Tuesday 26 March 2013

dxtbx side effects: new cctbx needed (if you use svn xia2)

So from previous posts it is clear that some fairly major "under the hood" changes relating to how the image headers etc. are read have been made: this will sometimes (for example with Rigaku instruments) use the dxtbx, which was in xia2 and has been moved to cctbx to allow use in other projects.

Thanks to some work from Nat Echols and James Parkhurst this is now working correctly in the nightly build from cctbx.sf.net: N.B. you need to get a recent (i.e. last night) "unreleased build" for example from

http://cci.lbl.gov/cctbx_build/show_results.cgi?build_tag=2013_03_26_0005

If you download and run the installer for this you should find that xia2 from subversion will work: this can be used for processing data from a Rigaku A200, for those of you who want this. The next release of xia2 will need an updated cctbx but if you are using the current release these changes should not affect you.

Tuesday 19 March 2013

dxtbx starting to pay off: Rigaku A200

So, as of now if you have a Rigaku A200 and use xia2 (and cctbx) from subversion, the latter can be used for the former! However today this is a little technical to use - I will try to figure out an updated release mechanism tomorrow. Unless I can figure out a compiled code free way of bundling dxtbx into xia2...

Certainly there will be a dependency on downloading a cctbx nightly build. I will now migrate all of the image handling code over to dxtbx which will make maintaining the whole system a whole lot better.


Monday 18 March 2013

dxtbx for image header reading

Over the next week I will pull out the old way of reading image file headers and add in a new one based on dxtbx - two comments:

 - please send me an image from your detector so I can be sure xia2 will still work - to xia2.support@gmail.com ideally
 - the subversion version of xia2 may appear broken

I will send an all clear when this is done. It should make xia2 much more useful and easier to maintain in the future.


Monday 4 March 2013

dxtbx: the diffraction experiment toolbox

The diffraction experiment toolbox in xia2 has now been migrated to cctbx. This is a two edged sword: it is a good thing as it can be used for other projects, but it is a problem as well as from the next release you will need a cctbx with the dxtbx contained within it and correctly configured.

This is necessarily a breaking change as the API for the dxtbx has changed since import into the cctbx, so I will need to modify xia2 to use the new API, and will remove the old dxtbx.

From later on today the svn version of xia2 is likely to be "broken" therefore if you have an older version of cctbx.

Friday 1 March 2013

-migrate_data

Anyone use -migrate_data or even know what it does? Wondering about culling some ancient and probably useless command-line options to make the code base cleaner, and this is one on the hit list. If you want to keep it then please send a comment to xia2.support@gmail.com otherwise I will clear it soon...


Thursday 28 February 2013

Phenix update: simple fix

There is a simpler fix to the phenix update issue mentioned earlier today: in $XIA2_ROOT/Applications/xia2.py change


sys.path.append(os.environ['XIA2_ROOT'])

to

sys.path.insert(0, os.environ['XIA2_ROOT'])

close to the top of the file. This will override the discovery process for where dxtbx can be found. This is included in the 0.3.6.0 update also.

xia2 0.3.6.0 release

Now available from xia2.sf.net:

xia2 0.3.6.0


Changes since 0.3.5.2: dealing with huge data sets
  • If single sweep, do not rebatch.
  • Replace CCP4 Mtzdump with CCTBX-powered equivalent.
  • If reindex for changing spacegroup name but not indices (i.e. no REINDEX= card) then use CCTBX-powered replacement.
  • Cleaned up repository by pulling xia2core into xia2/core.
  • Patched against dxtbx being included in Phenix. (temporary)
Changes since 0.3.5.1: CCP4 patch release
  • Resolution limit calculation now delegated to resolutionizer.
  • Twinning test now uses CCTBX code not sfcheck.
  • PythonDriver added to support execution of Python code as separate process.
N.B. the installation procedure has changed slightly reflecting the simplification of the code structure:

Since everything is Python using cctbx, the only requirement is to export XIA2_ROOT to point to the directory where xia2 was unpacked (including the xia2-0.3.6.0 bit) and then source $XIA2_ROOT/setup.(c)sh.

I am working on making even this go away... as always, comments / questions to xia2.support@gmail.com.

SVN users: xia2core now imported into xia2

If you use xia2 from subversion, you will see a fairly big update where xia2core is pulled into xia2/core - this was proposed a couple of weeks ago and no objections raised. The xia2core directory you have checked out is no longer needed.

Set up is now simpler:

 - export XIA2_ROOT=/path/to/xia2
 - source $XIA2_ROOT/setup.sh

or the bash / csh equivalent.  A corresponding release will be made available later today, once I have slimmed down the bundle.

This will also fix the phenix issue mentioned in the previous post...

Phenix nightly builds and xia2

The most recent Phenix nightly builds are including some xia2 components within them via cctbx - which is deliberate. They are however not correctly built as of last night, though Nat Echols may well have fixed this now.

Anyhow, if you update Phenix you may see errors like:

/local/phenix-1.8.2-1309/build/intel-linux-2.6-x86_64/bin/phenix
the same run of xia2 as above gave me the following:
  File "/local/xia2-0.3.5.0/xia2-0.3.5.0//Applications/xia2.py", line 32, in
    from xia2setup import write_xinfo
  File "/local/xia2-0.3.5.0/xia2-0.3.5.0/Applications/xia2setup.py", line 34, in
    from Schema.Sweep import SweepFactory
  File "/local/xia2-0.3.5.0/xia2-0.3.5.0/Schema/Sweep.py", line 36, in
    from Wrappers.XIA.Diffdump import Diffdump
  File "/local/xia2-0.3.5.0/xia2-0.3.5.0/Wrappers/XIA/Diffdump.py", line 70, in
    from dxtbx.format.Registry import Registry
  File "/local/phenix-1.8.2-1309/cctbx_project/dxtbx/__init__.py", line 8, in
    ext = boost.python.import_ext("dxtbx_ext", optional = False)
  File "/local/phenix-1.8.2-1309/cctbx_project/boost_adaptbx/boost/python.py", line 36, in import_ext
    + ["  "+p for p in sys.path]))
ImportError: __import__("dxtbx_ext"): No module named dxtbx_ext
  sys.path:
    /local/xia2-0.3.5.0/xia2-0.3.5.0/Applications
    /local/phenix-1.8.2-1309/phenix
    /local/phenix-1.8.2-1309/reel
...
    /local/phenix-1.8.2-1309/build/intel-linux-2.6-x86_64/base/lib/python2.7/site-packages/wx-2.8-gtk2-ansi
    /local/xia2-0.3.5.0/xia2-0.3.5.0/
    /local/xia2-0.3.5.0/xia2-0.3.5.0/Interfaces
    /local/xia2-0.3.5.0/xia2core-0.3.5.0/Python

If you get this there are a few solutions:

 - just rename the dxtbx directory in phenix/cctbx_project to dxtbx_ignore (which should not break Phenix)
 - recompile the module (which takes a moment as *all* of phenix is recompiled) as follows, thanks to Nat Echols:

cd  /local/phenix-1.8.2-1309/build/intel-linux-2.6-x86_64
./bin/libtbx.configure dxtbx
./bin/libtbx.scons -j X

(replace 'X' with the number of processor cores you want to use)

  - go back to using your old phenix for a moment while I add a fix to xia2 to work around this.

I should have seen this coming, but it did take me by surprise... sorry for any inconvenience.

Graeme

Tuesday 19 February 2013

Merging xia2core into xia2

Had a request for xia2core to be merged into xia2 from CCP4: this seems reasonable. I don't think anyone uses it in isolation as it was designed for, so this should only make things easier.

I will go ahead and do this in advance of the 0.3.6.0 release.

Tuesday 5 February 2013

-reference_reflection_file bug...

Just fixed a one-liner bug which stopped reference_reflection_file working (in one example anyhow)

Testing needs more work. What options do you use? Please drop me a line at xia2.support@gmail.com to offer your suggestions for test cases I need!

Full list of command-line options

Just gathered these to add to the manual, thought I should copy them here:


-2d   Use Mosflm and Scala
-2da   Use Mosflm and Aimless
-2dai   Use Mosflm (inc. for indexing) and Aimless
-2di   Use Mosflm (inc. for indexing) and Scala
-2dia   Pun of -2dai
-3d   Use XDS and XSCALE, Scala to merge
-3da   Use XDS and XSCALE, Aimless to merge
-3dai   Use XDS and XSCALE, Aimless to merge, XDS to index
-3daii       Use XDS and XSCALE, Aimless to merge, XDS to index (allframes)
-3di   Use XDS and XSCALE, Scala to merge, XDS to index
-3dii   Use XDS and XSCALE, Scala to merge, XDS to index (allframes)
-3diia       Pun for -3daii
-atom X Switch on separating anomalous pairs  
-batch_scale   Use batch rather than smooth scaling
-beam   Beam coordinates, in Mosflm convention
-blend       Save files for BLEND
-cell   Set the input cell constants (needs symmetry)
-spacegroup   Set the spacegroup
-crystal       Assign the crystal ID
-project   Assign the project ID
-debug   Switch on debugging output (a bad idea)
-executable   Override location of executable (deprecated)
-failover   Continue to process if one sweep fails
-image image Process only sweep containing image
-start_end   Assign start and end of sweep, needs -image
-free_fraction  Fraction of reflections to use in free set
-free_total   Total number of reflections in free set
-freer_file   Copy FreeR_flags from this file
-hdr_in       Input header information (developer tool)
-hdr_out   Output header information (developer tool)
-ice   Exclude ice ring regions
-indexer   Assign indexer to use (developer tool)
-integrater   Assign integrater to use (developer tool)
-scaler         Assign scaler to use (development tool)
-interactive   Manual selection of images for indexing (developertool)
-isigma         Unmerged limit for resolution limit calculations
-misigma   Merged limit for resolution limit calculations
-mask   Additional mask for detector e.g. for backstop.
-microcrystal  Options for microcrystal processing
-min_images   Min no. of images to make a sweep
-no_lattice_test  Switch off Bravais lattice tests
-parallel N   Number of cores to use
-phil     Pass in Phil override file (developer tool)
-quick       Cut corners to go quicker
-reference_reflection_file ref.mtz Assign reference reflection file
-resolution   Set the resolution limit
-reversephi   Warn of backwards phi axis
-scale_model decay,absorption,... Manually set the scale model
-serial   Non-parallel (-parallel 1)
-small_molecule   Settings for processing small molecule data
-trust_timestamps   If no timestamp in header, trust file timestamps
-xinfo   Pass xinfo file in
-xparallel N (XDS) use forkintegrate with N machines
-xparm GXPARM.XDS   Pass in refined coordinate description from calibration set
-xparm_ub GXPARM.XDS   Also use UB matrix from ... (developer tool)
-zero_dose   Switch on zero-dose extrapolation

Aimless wonderings

So, Aimless has been "out in the wild" for a while now, and works well. It has been supported for a while in xia2, but I intend to make this the default in the next release (which I plan for the end of the week.)

This will mean it needs a relatively recent CCP4 installation (i.e. 6.3.x) which should not present a problem.

As always, any problems with this please shout.

Tuesday 29 January 2013

Ctruncate issues

Ctruncate was updated in a manner which breaks xia2 in Update 13 - this has been fixed in the subversion-version but this has not been circulated. Until I get a chance to test the current state of code for release, please shout if you need an updated version!

As usual xia2.support@gmail.com.

Tuesday 22 January 2013

Remerging xia2 data to a lower resolution (technical)

A fairly common question I get from people is how to remerge data from xia2 to a lower resolution, without rerunning the whole xia2 process. There are three answers to this. The first is that xia2 does not do this, which is not so helpful. The second is that this will be in the pipeline at some point along with being able to retrospectively change other decisions and have xia2 just run with those. The third, which follows, is that there is a relatively straightforward way of doing this with tools included with xia2 and CCP4.

The Example

The example to be considered here is some myoglobin data, where diffraction went past the edge of the detector, and xia2 gave:


For AUTOMATIC/DEFAULT/SAD
High resolution limit                             1.20    5.39    1.20
Low resolution limit                             39.15   39.15    1.24
Completeness                                     87.8    96.0    18.7
Multiplicity                                      7.4     9.3     1.6
I/sigma                                          43.3   108.4     2.9
Rmerge                                          0.025   0.013   0.206
Rmeas(I)                                        0.031   0.018   0.314
Rmeas(I+/-)                                     0.028   0.014   0.284
Rpim(I)                                         0.011   0.006   0.214
Rpim(I+/-)                                      0.013   0.007   0.195
Wilson B factor                                 6.648
Partial bias                                    0.000   0.000   0.000
Anomalous completeness                           79.4    95.7     8.3
Anomalous multiplicity                            3.6     5.0     1.1
Anomalous correlation                            0.418   0.506  -0.119
Anomalous slope                                 1.345   0.000   0.000
dF/F                                            0.037
dI/s(dI)                                        1.104
Total observations                              424674  6953    1428
Total unique                                    57065   746     879

From processing with xia2 -3d ... Clearly it would be nice to have somewhere close to complete data in the outer shell, so a limit of around 1.35A would be appropriate (looking at the Scala log file.) So, this is what we need to do:

mkdir remerge
cd remerge/
get_ccp4_commands ../LogFiles/AUTOMATIC_DEFAULT_scala.log > runit

The last command reads the scala log file and essentially reconstructs an input file to rerun the job:

HKLIN /tmp/example/DEFAULT/scale/AUTOMATIC_DEFAULT_sorted.mtz
/tmp/example/DEFAULT/scale/AUTOMATIC_DEFAULT_scaled.mtz /tmp/example/DEFAULT/scale/AUTOMATIC_DEFAULT_scaled.mtz
SYMINFO /dls_sw/apps/ccp4/64/6.3.0/13nov2012/ccp4-6.3.0/lib/data/syminfo.lib
bins 20
resolution 1.200000
run 1 batch 1 to 321
resolution run 1 high 1.200000
name run 1 project AUTOMATIC crystal DEFAULT dataset SAD
scales constant
exclude sdmin 2.0
sdcorrection fixsdb noadjust norefine both 1.0 0.0
anomalous on

So we can edit this to do what we want:

scala \
HKLIN /tmp/example/DEFAULT/scale/AUTOMATIC_DEFAULT_sorted.mtz \
HKLOUT remerge.mtz << eof
bins 20
resolution 1.35
run 1 batch 1 to 321
name run 1 project AUTOMATIC crystal DEFAULT dataset SAD
scales constant
exclude sdmin 2.0
sdcorrection fixsdb noadjust norefine both 1.0 0.0
anomalous on
eof

and run:

bash runit

to get

...

                                           Overall  InnerShell  OuterShell
  Low resolution limit                       39.15     39.15      1.39
  High resolution limit                       1.35      6.04      1.35

  Rmerge                                     0.024     0.013     0.108
  Rmerge in top intensity bin                0.014        -         -
  Rmeas (within I+/I-)                       0.027     0.014     0.139
  Rmeas (all I+ & I-)                        0.030     0.018     0.140
  Rpim (within I+/I-)                        0.012     0.007     0.086
  Rpim (all I+ & I-)                         0.010     0.006     0.070
  Fractional partial bias                    0.000     0.000     0.000
  Total number of observations              397951      4729     12624
  Total number unique                        46162       521      3333
  Mean((I)/sd(I))                             52.1     107.9      10.5
  Completeness                                99.7      94.4      97.9
  Multiplicity                                 8.6       9.1       3.8

  Anomalous completeness                      96.6      93.9      78.8
  Anomalous multiplicity                       3.9       4.9       1.9
  DelAnom correlation between half-sets      0.410     0.587     0.115
  Mid-Slope of Anom Normal Probability       1.422       -         -
...

i.e. essentially complete data. To get this output in scalepack format, adjust the script to:

scala \
HKLIN /tmp/example/DEFAULT/scale/AUTOMATIC_DEFAULT_sorted.mtz \
SCALEPACK unmerged.sca \
HKLOUT remerge.mtz << eof
bins 20
output polish unmerged
resolution 1.35
run 1 batch 1 to 321
name run 1 project AUTOMATIC crystal DEFAULT dataset SAD
scales constant
exclude sdmin 2.0
sdcorrection fixsdb noadjust norefine both 1.0 0.0
anomalous on
eof

and run again - now you can see why I usually just advise running xia2 again with the -resolution flag :o)