tag:blogger.com,1999:blog-31625917528472804152024-02-18T17:42:24.989-08:00xia2What's going on in xia2 land?Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.comBlogger242125tag:blogger.com,1999:blog-3162591752847280415.post-7796087193633833912017-08-02T23:51:00.000-07:002017-08-02T23:51:44.567-07:00Massive Eiger data sets (i.e. > 100k frames) - some practical recommendationsThe <i>vast majority</i> of Eiger data sets are sensible i.e. measured as would be with a Pilatus, but with a different file format. There is however the perfectly sensible approach of putting a crystal on, spinning with a relatively low transmission beam and collecting until dead, then deciding <i>post mortem</i> when to have finished the experiment. This could however end up with very large data sets, with one master file and (say) over 1,000 HDF5 data files.<br />
<br />
A clue: this may not end well - for a start you could have > 200GB of data....<br />
<br />
Anyhow, some hints.<br />
<br />
<ol>
<li>ulimit -n 4096 </li>
<li>run on a machine with a lot of memory <i>or</i></li>
<li>xia2 image=/path/to/master.h5:1:100800:3600</li>
</ol>
<div>
First increases the OS limit on the allowed number of file handles a process is entitled to. You will probably need a lot of RAM for e.g. indexing (because a lot of reflections) or split into evenly sized chunks with the syntax above (e.g. 1:360000:3600 if you did 100 turns, into one turn chunks) - this will make the processing much more swift though the scaling will still take a little while.</div>
<div>
<br /></div>
<div>
Whatever happens, with xia2 this is unlikely to be fast... if using 3d/i/ii pipeline, the use of the neggia plugin with XDS will appear in the next release with DIALS 1.7, which is due real soon now...</div>
Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-65200545129512939502017-07-07T12:00:00.001-07:002017-08-02T23:41:25.464-07:00Eiger data sets > 10k framesTurns out xia2 would potentially fail with this - see<br />
<br />
<a href="https://github.com/xia2/xia2/issues/155">https://github.com/xia2/xia2/issues/155</a><br />
<br />
Now fixed - will be updated in next release of DIALS (1.6.4 probably)<br />
<br />
Meanwhile will probably set out to replace other jiffies used in DIALS with CCTBX code <a href="https://github.com/xia2/xia2/projects/1" target="_blank">[link]</a> to make the system more future proof... if anyone has a student in need of a project, please get in touch!Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-59608558669780595272017-05-12T02:33:00.002-07:002017-05-12T02:33:37.032-07:00Processing data to higher resolution than diffractionIf you are using xia2.small_molecule the default behaviour is to process every reflection and report the resolution limits observed, but include all reflections however weak in the output data. If you subsequently look at the I/sig(I) statistics and you have data to a lower limit than all reflections, you may see a substantial number of reflections with low or negative I/sig(I) i.e.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8OLmKQsNGsoM8ZQpZdyt9hCwNj-8UmE1yWGu8DR9cSt-pcv2YLYYflk7kvYcKfhSjSOM5gzBWVU244fZENlHLcXwGW8g80ArGaL1mlz6q7BSuqxA1DH2NFDIU8pkY4YXlkkeJdEzj0ek-/s1600/zeros.png" imageanchor="1"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8OLmKQsNGsoM8ZQpZdyt9hCwNj-8UmE1yWGu8DR9cSt-pcv2YLYYflk7kvYcKfhSjSOM5gzBWVU244fZENlHLcXwGW8g80ArGaL1mlz6q7BSuqxA1DH2NFDIU8pkY4YXlkkeJdEzj0ek-/s320/zeros.png" width="320" /></a><br />
This is to be expected - these are essentially a population of noisy zero values, the noise a result of statistical errors from background subtraction. These should have no impact on refinement.<br />
<br />
If this is not the behavior you want, set "keep_all_reflections=False" on the command line and only those data considered to be present will be included in the output.Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-60067625267197659122016-07-08T01:10:00.001-07:002016-07-08T01:10:28.091-07:00xia2 command syntax: now changedAs per previous post the command syntax change branch has now been merged, if you get your xia2 from git your next git pull will be a big one and you may find e.g. -resolution / -atom etc don't work any more... The functionality is still there however, you just need to use atom=Se (say) or anomalous=true, d_min=1.6... full documentation will follow of the changes.<br />
<br />
The pipeline options -3d. etc remain however are now deprecated but still work, you will get a note saying "please use pipeline=3d"<br />
<br />
Please if you find any bugs as a side-effect of this contact xia2.support@gmail.com or add an issue at<br />
<br />
https://github.com/xia2/xia2/issuesGraeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-4135411477586476052016-07-04T05:20:00.000-07:002016-07-04T05:20:28.751-07:00Please be aware: changing command syntaxAt the moment we use a mix of command syntax, some using Phil (i.e. unit_cell=a,b,c,al,be,ga) and some using legacy arguments: we plan to <i>completely remove</i> the latter in favour of the former, see<br />
<br />
https://github.com/xia2/xia2/issues/42<br />
<br />
with the intention that using xia2 in an automated environment is more straightforward and making it much simpler to automatically maintain much more complete documentation of all the options.<br />
<br />
This will be a feature of the 1.3 release of DIALS.Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-38469673057572646082016-05-05T13:59:00.001-07:002016-05-05T14:00:28.615-07:00Eiger HDF5 data, with xia2A common question over the past couple of months is when xia2 will work with Eiger HDF5 data natively. Well, in the dials 1.2 release now available from <a href="http://dials.github.io/">dials.github.io</a> it now does: your mileage may however vary, there is a certain amount of variation in the file formats still, but I think we have tested with all of the machines running out there in the wild.<br />
<br />
This is the result of a substantial amount of hard work from the dials and xia2 development teams, thank you to you all.Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-75951513301563069912016-03-18T05:59:00.001-07:002016-03-18T05:59:09.492-07:00xia2 -dials spotfinding outputThe latest xia2/dials nightly builds contain new output for the spotfinding step of xia2 -dials:<br />
<div>
<br /></div>
<div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">-------------------- Spotfinding SWEEP1 --------------------</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">102684 spots found on 1800 images (max 1854 / bin)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">***** ** * * * ****** * *</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">1 image 1800</span></div>
<div class="p1">
<br /></div>
<div class="p1">
These show the number of strong spots found on images during the spot finding and may provide insight into data processing problems due e.g. to sample misalignment. This is rather a nice example, you may see rather more variation than this with more typical data, but if you see a clear minimum anywhere in the set (i.e. results tend to 0) do not be surprised if the processing fails. </div>
<div class="p1">
<br /></div>
<div class="p1">
For Diamond Light Source users, this is very closely related to the per-image-analysis performed during data collection and should provide a similar level of insight. It is no accident that similar output is now visible in the summary from DIALS spot finding:</div>
<div class="p1">
<span class="s1"></span><br /></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Histogram of per-image spot count for imageset 0:</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">102684 spots found on 1800 images (max 1854 / bin)</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">***** ** * * * ****** * *</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">************************************************************</span></span></div>
<div class="p1">
</div>
<div class="p2">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">1 image 1800</span></span></div>
</div>
rgildeahttp://www.blogger.com/profile/11613815337638531033noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-43477479872316157132016-03-18T05:55:00.000-07:002016-03-18T05:55:09.941-07:00Processing compressed dataEven with modern machines today, X-ray diffraction data can still be <i>big</i>. However, pixel array data also compress well with e.g. gzip - say 20:1 on CBF files from a Pilatus detector. If only they could be processed compressed...<div>
<br /></div>
<div>
... well they can be. Actually xia2 can run just fine with compressed images:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">xia2 -atom Zn image=/Volumes/DATA/data/thermc_1_0001.cbf.gz:1:1800</span></div>
<div>
<br /></div>
<div>
and works with XDS as well. In the case of this thermolysin data, which were kept on an external USB3 drive, the whole xia2 job was about 20% <i>faster</i> using the compressed data rather than the raw. Worth considering as it also saved about 95% of the storage space as well.</div>
<div>
<br /></div>
<div>
The effects for even bigger data sets could be more substantial, as the 1800 images were able to fit in the cache. If this was not the case the time saving would be greater as the compressed data could be cached but the raw no...</div>
<div>
<br /></div>
<div>
Worth thinking next time you complain about the amount of storage X-ray data takes up.</div>
<div>
<br /></div>
<div>
YMMV, not tested on non-CBF images, some software may not support this, ...</div>
Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-2980279322540576792015-11-04T13:09:00.003-08:002015-11-04T13:09:48.020-08:00xia2: citing software xia2 has usedIf you find xia2 useful for processing your data of course the developers will welcome any citations - however please also cite the software that xia2 has used on your behalf - the developers of these packages also want to be recognised!<br />
<br />
xia2 will help you with this by including in the log file the appropriate list of citations e.g.<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">XIA2 used... ccp4 distl labelit pointless xds xia2</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Here are the appropriate citations (BIBTeX in xia-citations.bib.)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">(1994) Acta Crystallogr. D 50, 760--763</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Evans, Philip (2006) Acta Crystallographica Section D 62, 72--82</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Kabsch, Wolfgang (2010) Acta Crystallographica Section D 66, 125--132</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Sauter, Nicholas K. and Grosse-Kunstleve, Ralf W. and Adams, Paul D. (2004) Journal of Applied Crystallography 37, 399--409</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Winter, G. (2010) Journal of Applied Crystallography 43</span></span></div>
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Zhang, Z. and Sauter, N.K. and van den Bedem, H. and Snell, G. and Deacon, A.M. (2006) J. Appl. Cryst 39, 112--119</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">if you run xia2 -3d - please do add these to the citations in the associated publications and to the PDB file.</span></div>
Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com2tag:blogger.com,1999:blog-3162591752847280415.post-28393742465917432532015-10-20T02:58:00.001-07:002015-11-04T13:10:37.705-08:00New XDS: things not quite rightNow debugging xia2 for Oct 2015 version of XDS; not right yet please watch this space!<br />
<br />
Update: XDS is just fine and a separate bug has crept into the development / nightly version, sorry for the noise.<br />
<br />Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-64677058861952066972015-07-21T01:08:00.001-07:002015-07-21T01:08:13.255-07:00With ongoing sf.net outage...We are looking to move xia2 to github, so for those of you who pull xia2 from subversion this will be a significant change. Also the "home" of xia2 will change as well - though this blog will remain, so watch this space...Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-44866959607400224552015-07-19T22:46:00.003-07:002015-07-19T22:46:20.705-07:00sf.net outageYou may have noticed that xia2 things are not working - the svn is dead; the nightly builds are empty etc. - this is because the hosting service I use (sourceforge.net) is currently broken and has been for getting on for four days. They're in the process of restoring services (web now back, but svn is not) and normal service should resume soon.<br />
<br />
See <a href="http://sourceforge.net/blog/sourceforge-infrastructure-and-service-restoration/" target="_blank">here</a> for more details.Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-91153591351350811822015-07-16T05:34:00.002-07:002015-07-16T05:34:44.587-07:00Windows...Does anyone use xia2 / DIALS on MS Windows? Would you want to? Was pointed out to me that the changes referred to in the previous post would not work gracefully on MS Windows due to the c:\ type file paths... this should probably be fixed anyhow. Shout if you have an opinion.Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com1tag:blogger.com,1999:blog-3162591752847280415.post-77058148151532447232015-07-16T03:42:00.000-07:002015-07-16T03:42:04.045-07:00New logs in the LogFiles directory; more image= optionsIf you look in LogFiles and you use xia2 -dials you will find some new stuff in there:<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">Graemes-MacBook-Pro-2:image_range graeme$ ls LogFiles/</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">AUTOMATIC_DEFAULT_NATIVE_truncate.log<span class="Apple-tab-span"> </span>AUTOMATIC_DEFAULT_pointless.log</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">AUTOMATIC_DEFAULT_aimless.log<span class="Apple-tab-span"> </span><span class="Apple-tab-span"> </span>SWEEP1_INDEX.log</span></span></div>
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">AUTOMATIC_DEFAULT_aimless_xml.xml<span class="Apple-tab-span"> </span>SWEEP1_LATTICE.log</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">This now includes the indexing log file and the lattice determination one; these can be useful to see how xia2 made it's decisions.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Also, if you want to combine some data from several sweeps, but only use limited image ranges, you can with:</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">xia2 unify_setting=true -atom Ti -small_molecule -dials image=/Users/graeme/data/i04-KTP/1/KTP_1_0001.cbf:1:100 image=/Users/graeme/data/i04-KTP/2/KTP_2_0001.cbf:1:100</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">so this will use the first 100 images of two sweeps and will make this xinfo file:</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">BEGIN PROJECT AUTOMATIC</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">BEGIN CRYSTAL DEFAULT</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">BEGIN HA_INFO</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">ATOM ti</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">!NUMBER_PER_MONOMER N</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">!NUMBER_TOTAL M</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">END HA_INFO</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">BEGIN WAVELENGTH SAD</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">WAVELENGTH 0.729320</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">END WAVELENGTH SAD</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">BEGIN SWEEP SWEEP1</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">WAVELENGTH SAD</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">DIRECTORY /Users/graeme/data/i04-KTP/1</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">IMAGE KTP_1_0001.cbf</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">START_END 1 100</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">END SWEEP SWEEP1</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">BEGIN SWEEP SWEEP2</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">WAVELENGTH SAD</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">DIRECTORY /Users/graeme/data/i04-KTP/2</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">IMAGE KTP_2_0001.cbf</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">START_END 1 100</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">END SWEEP SWEEP2</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">END CRYSTAL DEFAULT</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">
</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">END PROJECT AUTOMATIC</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Word of warning though that this will currently (as of noon today) only cope with one subset per sweep; should be easy enough to add multiple image ranges for this though need to think what syntax to use... as always feedback welcome to xia2.support@gmail.com; to get this you will need to pull a new DIALS nightly build later today or tomorrow.</span></div>
Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-80420178689537922122015-07-13T05:10:00.002-07:002015-07-13T05:10:50.919-07:00More output: how much is too much?Request of additional output from xia2 e.g. in the indexing table:<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">------------------- Autoindexing SWEEP1 --------------------</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">All possible indexing solutions:</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">oP 6.39 10.59 12.80 90.00 90.00 90.00</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">mP 6.39 12.80 10.59 90.00 89.99 90.00</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">aP 6.39 10.59 12.80 90.00 90.00 89.99</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">Indexing solution:</span></span></div>
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">oP 6.39 10.59 12.80 90.00 90.00 90.00</span></span></div>
<br />
adding the R.M.S.D., metric penalty etc. from for example <span style="font-family: Courier New, Courier, monospace;">dials.index</span>. This will however make for more "junk" in the output and will also depend on the package being run underneath. Also requested was number of spots indexed vs. total number of spots. This will make for more information in the output which could make it cluttered and will also have the potential to "break" other software which uses this.<br />
<br />
Thoughts anyone?<br />
<br />Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-58925474384953863002015-06-05T00:22:00.000-07:002015-06-05T00:22:06.676-07:00Free set reflectionsFollowing a question on the CCP4 BB about how many reflections to set aside for the free set (i.e. the FreeR_flag column xia2 adds to the MTZ file) it looks like the conclusion is "data processing packages should not do this" - OK, no problem. That makes the original question go away; always good...<br />
<br />
Am keen to now remove this capability from xia2; comments to xia2.support@gmail.com please!Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-17047590529096742302015-04-24T04:08:00.002-07:002015-04-24T04:08:17.318-07:00New ability to specify multiple directories or images via command lineAs of the latest DIALS/xia2 nightly build (2015-04-24 onwards, available from <a href="http://dials.diamond.ac.uk/diamond_builds/">http://dials.diamond.ac.uk/diamond_builds/</a>), xia2 now supports passing multiple directories or images via the command line, e.g.:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">xia2 -dials /path/to/directory1 /path/to/directory2</span><br />
<span style="font-family: Courier New, Courier, monospace;">xia2 -dials image=/path/to/directory1/image_1_0001.cbf image=/path/to/directory2/image_2_0001.cbf</span>rgildeahttp://www.blogger.com/profile/11613815337638531033noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-8693984197261446962015-03-27T01:24:00.001-07:002015-03-27T02:01:43.351-07:00Nasty subtle bug with latest XDSThe default behaviour of XDS with REFINE= in the CORRECT step appears to have changed, which highlights a long-standing bug in xia2. The bug is that the REFINE= card was incorrectly formatted, which some of you have noticed in that the summary from the second pass of integration looks much more nasty than the first pass. This apparently was not a problem with the older version of XDS (before 1mar2015). I had initially (incorrectly) blamed the new version of XDS but Richard Gildea found the underlying bug.<br />
<br />
A new version of xia2 will be available already with these fixes in as part of the DIALS nightly builds. The fix is however a one-liner:<br />
<br />
In<br />
<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">xia2/Wrappers/XDS/XDSCorrect.py</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">change</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">'REFINE(CORRECT)=%s\n' %self._params.refine)</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">to </span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">
</span></div>
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace;">'REFINE(CORRECT)=%s\n' %' '.join(self._params.refine))</span></span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">(amazing how much effect these little changes can make!) We are in the process if instigating a more thorough testing procedure which will ensure errors like this do not occur again.</span></div>
Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-14999201569136703652015-03-10T04:26:00.004-07:002015-03-10T04:26:53.131-07:00Experimental xia2/DIALS installersWe are currently testing new xia2/DIALS installers for mac/linux which provide a self-contained xia2/cctbx/DIALS installation. Instructions on downloading and installing these bundles can be found at <a href="http://dials.diamond.ac.uk/doc/installation.html">http://dials.diamond.ac.uk/doc/installation.html</a>. Once installed, simply sourcing the <span style="font-family: Courier New, Courier, monospace;">dials_env.sh</span> script in the installation directory will make available all xia2 and DIALS commands in the current terminal. You will also need CCP4 available - but make sure to source the <span style="font-family: Courier New, Courier, monospace;">dials_env.sh</span> script <i>after</i> the CCP4 setup script. You can also try out the new <span style="font-family: Courier New, Courier, monospace;">xia2 -dials</span> option, which will use the new software <a href="http://dials.diamond.ac.uk/">DIALS</a> to index and integrate your data, followed by scaling and merging with Aimless. As always, please send any feedback to <a href="mailto:xia2.support@gmail.com">xia2.support@gmail.com</a>.rgildeahttp://www.blogger.com/profile/11613815337638531033noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-2446639960834500192015-02-24T02:20:00.002-08:002015-02-24T02:20:45.265-08:00New wavelength_tolerance parameterA common problem when processing multiple datasets with xia2 is when a small difference in the wavelength recorded in the image headers causes the sweeps to be grouped into more than one wavelength, when the desired outcome was for them to be processed as a single wavelength. Previously it was necessary to manually edit the .xinfo file to tell xia2 to process them as a single wavelength. Now we have added a new parameter <span style="font-family: Courier New, Courier, monospace;">wavelength_tolerance</span> (default=0.0002 Å) which you can use to control whether or not different wavelengths get grouped together as a single wavelength.<br />
<br />
Example usage:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">xia2 -3dii wavelength_tolerance=0.001 /path/to/images/</span>rgildeahttp://www.blogger.com/profile/11613815337638531033noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-35815000387567985382015-02-11T06:31:00.001-08:002015-02-11T06:31:05.390-08:00xia2 CCTBX challenges? Try installing DIALS!If you are having problems with xia2, particularly from subversion, you may find it useful to pull a DIALS build from e.g. http://dials.diamond.ac.uk/builds and install this so that xia2 picks up CCTBX from here. This will be up to date and should work, and also includes a collection of very useful tools. It will generally include support files for all of the most recent detector installations.<br />
<br />
The DIALS project (see e.g. http://dials.diamond.ac.uk) is a collaborative project to develop a new integration package and is supported by xia2 <i>via</i> xia2 -dials: if you are feeling brave you can try using this for processing your data in place of e.g. XDS.<br />
<br />
Any problems with xia2 please contact xia2.support@gmail.com. For issues with DIALS please contact dials-support@lists.sourceforge.net.<br />
<br />
<br />Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-54766113162792634342014-11-17T02:21:00.001-08:002014-11-17T02:21:17.407-08:00Bug found: error parsing Aimless XML file for MAD dataFound a bug in the parsing code for Aimless XML which only tries to extract the first data set (not enough testing, fail) only for MAD data sets... This will give an error something like:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">Reindexing to first spacegroup setting: P 41 21 2 (h,k,l)</span><br />
<span style="font-family: Courier New, Courier, monospace;">Status: error "('AUTOMATIC', 'DEFAULT', 'WAVE2')"</span><br />
<div>
<br /></div>
I have now fixed this in revision 4879, now testing prior to updated release... sorry for any inconvenience.<br />
<br />
<br />
<br />Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-58931473854263121042014-11-11T02:01:00.001-08:002014-11-11T02:01:47.955-08:00Warning: Deprecating ThingsTrying to clean up the interface which will in the long run help the documentation, but in the mean time will make some warnings... if you see<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Warning: -spacegroup option deprecated: please use space_group='P422' instead</span></span></div>
<br />
<div class="p1">
<span class="s1"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Warning: -resolution option deprecated: please use d_min=1.5 instead</span></span></div>
<br />
Don't panic - this is to be expected - but you may want to change the way you run xia2 or your scripts. More of a warning for beamline / automation people! The outcome of this should however be automated generation of command-line documentation and the ability to keep "recipes" for running xia2 in tidy files....<br />
<br />Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-49644311245852840522014-10-24T07:39:00.000-07:002014-10-24T07:39:18.204-07:00Corner case bug spotted (thanks Luca Jovine)If you<br />
<br />
<ul>
<li>use xinfo files</li>
<li>use -2d</li>
<li>do not set START_END</li>
</ul>
<div>
you may find you get a weird complaint about detector_class.</div>
<div>
<br /></div>
<div>
This is easily resolved for the moment by setting START_END to correspond to the first and last image you want to process... I need to fix this ;o) it did not used to work like this...</div>
<div>
<br /></div>
<div>
Thanks to Luca for pointing this out and patience in debugging...</div>
Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0tag:blogger.com,1999:blog-3162591752847280415.post-91875328800448818232014-10-20T05:15:00.001-07:002014-10-21T07:39:34.599-07:00xia2 0.3.8.0 releasedMajor changes:<br />
<br />
Changes since 0.3.7.0: Added DIALS (mostly)<br />
<ul>
<li>Added support for <a href="http://dials.sourceforge.net/">DIALS</a> software. Thanks to Richard Gildea at Diamond for most of the work.</li>
<li>Create AIMLESS xml output; useful for displaying results through alternative interfaces.</li>
<li>Squashed nasty bug where input spacegroup set to e.g. I222 but pointgroup derived to be P222 (think this one was reported a while back but I could not find the source at the time... sorry!)</li>
<li>Squashed some other bugs which have appeared in recent history.</li>
</ul>
Changes since 0.3.6.3: Bug fixes, add CC-half, removing Scala.<br />
<ul>
<li>-cc_half 0.5 (say) resolution limit criterion available - suggest set -misigma 0.5 -isigma 0.5 or similar with this.</li>
<li>-freer_file where free flag != FreeR_flag fixed.</li>
<li>Massively trimmed all of the command line options for the pipelines and removed the use of Scala in all pipelines, replaced with Aimless (not before time!) This means that -3dr and so forth <i>will no longer work</i> so please use -2d, -3d, -3di etc.</li>
<li>-phil command line option now deprecated: phil parameters passed on the command-line will be respected, also phil files, without this clue</li>
<li>Parallelisation options: can now cleanly process sweeps from one wavelength in parallel (to aid with multi-crystal data sets) use Phil parameters multiprocessing.njob=4 multiprocessing.nproc=12 multiprocessing.mode=parallel (say) to allow processing four sweeps at a time each using 12 cores. This multi-sweep parallelisation is probably most useful in conjunction with the Phil parameter multiprocessing.type=qsub, which tells xia2 to submit jobs to a cluster, with each job using nproc cores.</li>
</ul>
Graeme Winterhttp://www.blogger.com/profile/15656538905087532935noreply@blogger.com0