Thursday, 28 February 2013

Phenix update: simple fix

There is a simpler fix to the phenix update issue mentioned earlier today: in $XIA2_ROOT/Applications/ change



sys.path.insert(0, os.environ['XIA2_ROOT'])

close to the top of the file. This will override the discovery process for where dxtbx can be found. This is included in the update also.

xia2 release

Now available from


Changes since dealing with huge data sets
  • If single sweep, do not rebatch.
  • Replace CCP4 Mtzdump with CCTBX-powered equivalent.
  • If reindex for changing spacegroup name but not indices (i.e. no REINDEX= card) then use CCTBX-powered replacement.
  • Cleaned up repository by pulling xia2core into xia2/core.
  • Patched against dxtbx being included in Phenix. (temporary)
Changes since CCP4 patch release
  • Resolution limit calculation now delegated to resolutionizer.
  • Twinning test now uses CCTBX code not sfcheck.
  • PythonDriver added to support execution of Python code as separate process.
N.B. the installation procedure has changed slightly reflecting the simplification of the code structure:

Since everything is Python using cctbx, the only requirement is to export XIA2_ROOT to point to the directory where xia2 was unpacked (including the xia2- bit) and then source $XIA2_ROOT/setup.(c)sh.

I am working on making even this go away... as always, comments / questions to

SVN users: xia2core now imported into xia2

If you use xia2 from subversion, you will see a fairly big update where xia2core is pulled into xia2/core - this was proposed a couple of weeks ago and no objections raised. The xia2core directory you have checked out is no longer needed.

Set up is now simpler:

 - export XIA2_ROOT=/path/to/xia2
 - source $XIA2_ROOT/

or the bash / csh equivalent.  A corresponding release will be made available later today, once I have slimmed down the bundle.

This will also fix the phenix issue mentioned in the previous post...

Phenix nightly builds and xia2

The most recent Phenix nightly builds are including some xia2 components within them via cctbx - which is deliberate. They are however not correctly built as of last night, though Nat Echols may well have fixed this now.

Anyhow, if you update Phenix you may see errors like:

the same run of xia2 as above gave me the following:
  File "/local/xia2-", line 32, in
    from xia2setup import write_xinfo
  File "/local/xia2-", line 34, in
    from Schema.Sweep import SweepFactory
  File "/local/xia2-", line 36, in
    from Wrappers.XIA.Diffdump import Diffdump
  File "/local/xia2-", line 70, in
    from dxtbx.format.Registry import Registry
  File "/local/phenix-1.8.2-1309/cctbx_project/dxtbx/", line 8, in
    ext = boost.python.import_ext("dxtbx_ext", optional = False)
  File "/local/phenix-1.8.2-1309/cctbx_project/boost_adaptbx/boost/", line 36, in import_ext
    + ["  "+p for p in sys.path]))
ImportError: __import__("dxtbx_ext"): No module named dxtbx_ext

If you get this there are a few solutions:

 - just rename the dxtbx directory in phenix/cctbx_project to dxtbx_ignore (which should not break Phenix)
 - recompile the module (which takes a moment as *all* of phenix is recompiled) as follows, thanks to Nat Echols:

cd  /local/phenix-1.8.2-1309/build/intel-linux-2.6-x86_64
./bin/libtbx.configure dxtbx
./bin/libtbx.scons -j X

(replace 'X' with the number of processor cores you want to use)

  - go back to using your old phenix for a moment while I add a fix to xia2 to work around this.

I should have seen this coming, but it did take me by surprise... sorry for any inconvenience.


Tuesday, 19 February 2013

Merging xia2core into xia2

Had a request for xia2core to be merged into xia2 from CCP4: this seems reasonable. I don't think anyone uses it in isolation as it was designed for, so this should only make things easier.

I will go ahead and do this in advance of the release.

Tuesday, 5 February 2013

-reference_reflection_file bug...

Just fixed a one-liner bug which stopped reference_reflection_file working (in one example anyhow)

Testing needs more work. What options do you use? Please drop me a line at to offer your suggestions for test cases I need!

Full list of command-line options

Just gathered these to add to the manual, thought I should copy them here:

-2d   Use Mosflm and Scala
-2da   Use Mosflm and Aimless
-2dai   Use Mosflm (inc. for indexing) and Aimless
-2di   Use Mosflm (inc. for indexing) and Scala
-2dia   Pun of -2dai
-3d   Use XDS and XSCALE, Scala to merge
-3da   Use XDS and XSCALE, Aimless to merge
-3dai   Use XDS and XSCALE, Aimless to merge, XDS to index
-3daii       Use XDS and XSCALE, Aimless to merge, XDS to index (allframes)
-3di   Use XDS and XSCALE, Scala to merge, XDS to index
-3dii   Use XDS and XSCALE, Scala to merge, XDS to index (allframes)
-3diia       Pun for -3daii
-atom X Switch on separating anomalous pairs  
-batch_scale   Use batch rather than smooth scaling
-beam   Beam coordinates, in Mosflm convention
-blend       Save files for BLEND
-cell   Set the input cell constants (needs symmetry)
-spacegroup   Set the spacegroup
-crystal       Assign the crystal ID
-project   Assign the project ID
-debug   Switch on debugging output (a bad idea)
-executable   Override location of executable (deprecated)
-failover   Continue to process if one sweep fails
-image image Process only sweep containing image
-start_end   Assign start and end of sweep, needs -image
-free_fraction  Fraction of reflections to use in free set
-free_total   Total number of reflections in free set
-freer_file   Copy FreeR_flags from this file
-hdr_in       Input header information (developer tool)
-hdr_out   Output header information (developer tool)
-ice   Exclude ice ring regions
-indexer   Assign indexer to use (developer tool)
-integrater   Assign integrater to use (developer tool)
-scaler         Assign scaler to use (development tool)
-interactive   Manual selection of images for indexing (developertool)
-isigma         Unmerged limit for resolution limit calculations
-misigma   Merged limit for resolution limit calculations
-mask   Additional mask for detector e.g. for backstop.
-microcrystal  Options for microcrystal processing
-min_images   Min no. of images to make a sweep
-no_lattice_test  Switch off Bravais lattice tests
-parallel N   Number of cores to use
-phil     Pass in Phil override file (developer tool)
-quick       Cut corners to go quicker
-reference_reflection_file ref.mtz Assign reference reflection file
-resolution   Set the resolution limit
-reversephi   Warn of backwards phi axis
-scale_model decay,absorption,... Manually set the scale model
-serial   Non-parallel (-parallel 1)
-small_molecule   Settings for processing small molecule data
-trust_timestamps   If no timestamp in header, trust file timestamps
-xinfo   Pass xinfo file in
-xparallel N (XDS) use forkintegrate with N machines
-xparm GXPARM.XDS   Pass in refined coordinate description from calibration set
-xparm_ub GXPARM.XDS   Also use UB matrix from ... (developer tool)
-zero_dose   Switch on zero-dose extrapolation

Aimless wonderings

So, Aimless has been "out in the wild" for a while now, and works well. It has been supported for a while in xia2, but I intend to make this the default in the next release (which I plan for the end of the week.)

This will mean it needs a relatively recent CCP4 installation (i.e. 6.3.x) which should not present a problem.

As always, any problems with this please shout.