Recent Changes - Quixote project on QC databases

Front Page

2013-04-28 10:32:57

(quick edit)

Differences for Front Page

Deletions are marked with - .	Additions are marked with +.
Line 5:	Line 5:
- An example of the type of architecture that we are developing is shown below.	+ Quixote is an international collaboration heavily relying on web connectivity and voluntary work by motivated researchers. The main objective/vision of the Quixote project is to design, test and deploy a modular, open source system of tools that allow computational chemistry data (now sitting in the darkness of individual hard-disks) to be organized, shared, and queried. This is to be achieved by using lightweight interdependent applications, semantic analysis of the data and linkability.

Front Page

2013-04-28 10:32:31

(quick edit)

Differences for Front Page

Deletions are marked with - .	Additions are marked with +.
Line 3:	Line 3:
- Quixote is an international collaboration heavily relying on web connectivity and voluntary work by motivated researchers. The main objective/vision of the Quixote project is to design, test and deploy a modular, open source system of tools that allow computational chemistry data (now sitting in the darkness of individual hard-disks) to be organized, shared, and queried. This is to be achieved by using lightweight interdependent applications, semantic analysis of the data and linkability.	+

Tutorials and problems

tcnsc — 2013-03-05 14:36:15

Revert to version 61 (Removed spam.).

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 86:	Line 86:
- This will download the java jar files required by the converters into your local maven repository (~/.m2/repository on unix), compile, test and install the code into your local maven repository. Everything will then be ready for using the JUMBO-Converters software. [https://www.whiteknightcasino.com/ Online Casino No Download]	+ This will download the java jar files required by the converters into your local maven repository (~/.m2/repository on unix), compile, test and install the code into your local maven repository. Everything will then be ready for using the JUMBO-Converters software.

Tutorials and problems

2013-03-05 09:19:47

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 86:	Line 86:
- This will download the java jar files required by the converters into your local maven repository (~/.m2/repository on unix), compile, test and install the code into your local maven repository. Everything will then be ready for using the JUMBO-Converters software.	+ This will download the java jar files required by the converters into your local maven repository (~/.m2/repository on unix), compile, test and install the code into your local maven repository. Everything will then be ready for using the JUMBO-Converters software. [https://www.whiteknightcasino.com/ Online Casino No Download]

Tutorials and problems

tcnsc — 2013-03-04 12:25:15

Revert to version 59 (Removed spam.).

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 19:	Line 19:
- * [http://mercurial.selenic.com/ Mercurial] for interacting with software repositories. Check ["Mercurial"] for a basic tutorial and known problems. [https://www.whiteknightcasino.com/ Online Casino No Download]	+ * [http://mercurial.selenic.com/ Mercurial] for interacting with software repositories. Check ["Mercurial"] for a basic tutorial and known problems.

Tutorials and problems

2013-03-04 12:16:38

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 19:	Line 19:
- * [http://mercurial.selenic.com/ Mercurial] for interacting with software repositories. Check ["Mercurial"] for a basic tutorial and known problems.	+ * [http://mercurial.selenic.com/ Mercurial] for interacting with software repositories. Check ["Mercurial"] for a basic tutorial and known problems. [https://www.whiteknightcasino.com/ Online Casino No Download]

Tutorials and problems

tcnsc — 2013-03-01 14:42:35

Revert to version 57 (Removed spam.).

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 17:	Line 17:
- If that gives a '''command not found''' type error, you need to install the Java Development Kit. * [http://www.telsgroup.com.de/ Internationale Spedition]	+ If that gives a '''command not found''' type error, you need to install the Java Development Kit.

Tutorials and problems

2013-03-01 08:17:26

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 17:	Line 17:
- If that gives a '''command not found''' type error, you need to install the Java Development Kit.	+ If that gives a '''command not found''' type error, you need to install the Java Development Kit. * [http://www.telsgroup.com.de/ Internationale Spedition]

Tutorials and problems

tcnsc — 2013-02-20 13:57:27

Revert to version 54 (Removed spam.).

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 38:	Line 38:
- [http://www.imcredo.com/services/ppc/ PPC Agency]

Tutorials and problems

2013-02-20 13:14:26

Differences for Tutorials and problems

No differences found!

Tutorials and problems

2013-02-20 13:14:07

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 38:	Line 38:
	+ [http://www.imcredo.com/services/ppc/ PPC Agency]

Resources and technology

JensThomas — 2012-09-20 17:00:34

Differences for Resources and technology

Deletions are marked with - .	Additions are marked with +.
Line 49:	Line 49:
- ==Chunking== - - The idea is implemented in a particular code (["GAMESS-UK"]) and it uses ["JUMBO-Converters"] but the approach is rather general. - - * ["Chunkers" First divide the file in chunks or blocks] - * ["Block" What is a block?] - * ["Parsing Blocks"] - - ==Converting to CML using ["JUMBO-Converters"]== - - The idea is implemented in a particular code (["GAMESS-UK"]) but the approach is rather general. - - * ["How converters work"] - * ["creating rawCML" Creating raw CML] - * ["creating completeCML" Creating complete CML] - - See also ["Creating a converter"]	+ + The current approach adopted by the Quixote Project is to use the ["JUMBO-Converters"].

Chempound

JensThomas — 2012-08-14 17:59:59

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 699:	Line 699:
	+ + === Debugging Chempound running under Jetty === + + A simple way to debug chempound when running under jetty, is to add the following lines to the jetty '''start.ini''' file, which is used to prepend command-line arguments to jetty (the arguments can also be added to the command-line when starting jetty, or indeed to any java program that supports log4j): + + {{{ + -Dlog4j.debug + -Dlog4j.configuration=file:/Users/jmht/Documents/quixote/jetty-hightide-8.1.4.v20120524/quixote/qc-log4j.properties + }}} + + The first line turns on debugging for log4j itself - this is useful as it causes log4j to print which configuration file it is using. The second file gives the path to a log4j configuration file, which should contain the directives as described above.

Chempound

JensThomas — 2012-08-07 09:46:49

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 642:	Line 642:
	+ + == Debugging Chempound == + + If Chempound is not working as expected, the logging facility can be used to increase the amount of information printed, which is useful for tracking down the causes of problems. + + The logging subsystem consists of the interface [http://slf4j.org/ SLF4J] 1.6.1 (Simple Logging Facade for Java): and the implementation [http://logging.apache.org/log4j/1.2/index.html LOG4J] 1.2. + + The included configuration for DepositNWChem is: + + {{{ + log4j.rootLogger = WARN, A + + log4j.appender.A = org.apache.log4j.ConsoleAppender + log4j.appender.A.layout = org.apache.log4j.PatternLayout + log4j.appender.A.layout.ConversionPattern = %-4r [%t] %-5p %c %x - %m%n + log4j.appender.A.target = System.err + }}} + + However, you can change the logging behavior of the application by adding your own log4j.properties file to the classpath. For instance, the following configuration file will set the general log level to INFO and, for class uk.ac.cam.ch.wwmm.chempound.compchem.CmlComp2RdfConverter, the level will be DEBUG. + + {{{ + log4j.rootLogger = INFO, A + + log4j.appender.A = org.apache.log4j.ConsoleAppender + log4j.appender.A.layout = org.apache.log4j.PatternLayout + log4j.appender.A.layout.ConversionPattern = %-4r [%t] %-5p %c %x - %m%n + log4j.appender.A.target = System.err + + log4j.logger.uk.ac.cam.ch.wwmm.chempound.compchem.CmlComp2RdfConverter=DEBUG + }}} + + For example, if you place the log4j.properties configuration file in your current working directory and you run DepositNWChem from there, you can add the current directory to the classpath as follows (it assumes you have the jar file of DepositNWChem with its dependencies in a target subdirectory): + + {{{ + $ java -cp .:target/quixote-utils-0.1-SNAPSHOT-jar-with-dependencies.jar net.quixote.utils.DepositNWChem http://localhost:8080/sword/collection/ n2.out + }}} + + You can use logging anywhere in the code. You will need to grab a Logger object to pass the logging messages. Simply import the Logger and LoggerFactory classes, and call LoggerFactory.getLogger to obtain a Logger object. Then call any of the debug, info, warn or error methods to log your message at the appropriate log level. + + The following code snippet shows how to get the root Logger as well as another child Logger (identified with the uk.ac.cam.ch.wwmm.chempound.compchem.CmlComp2RdfConverter class name) and how to emit a INFO level message. + + {{{ + import org.slf4j.Logger; + import org.slf4j.LoggerFactory; + + class ... { + ... method (...) { + Logger rootL = LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME); + rootL.info("Using root logger."); + + Logger otherL = LoggerFactory.getLogger(uk.ac.cam.ch.wwmm.chempound.compchem.CmlComp2RdfConverter.class); + otherL.info("Using CmlComp2RdfConverter logger."); + + + } + } + }}}

JUMBO-Converters

JensThomas — 2012-06-21 09:53:46

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 5:	Line 5:
- Primary site: [https://bitbucket.org/wwmm/jumbo-converters https://bitbucket.org/wwmm/jumbo-converters]	+ Primary site: [https://bitbucket.org/wwmm/jumboconverters-compchem]
Line 35:	Line 35:
	+ Jumbo converters are written in Java, although the template parsing technology is described entirely in XML, so that once a new parser module has been created, only XML files need to be edited in order to extend and develop the parser. + + The reference parser for computational chemistry is the NWChem parser, so any examples will refer to it. + + The java class that controls the two-stage parsing for the NWChem is [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2CompchemConverter.java NWChemLog2CompchemConverter.java]. + + The first stage (controlled by the [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2XMLConverter.java NWChemLog2XMLConverter.java] class), uses the [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/resources/org/xmlcml/cml/converters/compchem/nwchem/log/templates/topTemplate.xml topTemplate.xml] file to include the various XML templates that parse the different chunks of the logfile. + + The second stage (controlled by the [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLogXML2CompchemConverter.java NWChemLogXML2CompchemConverter.java] class), uses the transforms in the [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/resources/org/xmlcml/cml/converters/compchem/nwchem/log/nwchem2compchem.xml nwchem2compchem.xml] to manipulate the raw XML into a [http://www.xml-cml.org/convention/compchem convention]-compliant form. +
Line 36:	Line 46:
- - == Top-level declarations and parsing hierarchy == - - A '''topTemplate.xml''' file, is a top-level file that creates the parent template and then defines all the templates which it itself contains. - - The definitions of the sub-templates are kept in separate files that are '''''included''''' from a templates subdirectory to aid modularisation. - - Each template has one or more '''pattern'''s, which is a regular expression defining the text in the logfile where the module starts, and one or more '''endPattern'''s, which is a regular expression defining where the module ends. The parser will read through the log file checking each line against the patterns in the list of templates included in the topTemplate. - - If a pattern is matched, all subsequent text is "gobbled" and added to the module until the '''endPattern''' is reached, at which point the module is closed, and the next line of text is searched to see if it matches a pattern. - - Each module is then parsed in turn, either by another template within this template, or by the '''records''' in the template.
Line 107:	Line 105:
-
Line 152:	Line 149:
- This result of the parsing is as follows:	+ This result of this parsing is as follows:
Line 234:	Line 231:
- For the Gaussian logfile templates, the code that runs these tests lives in the file: - - [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/test/java/org/xmlcml/cml/converters/compchem/gaussian/log/TemplateTest.java jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/test/java/org/xmlcml/cml/converters/compchem/gaussian/log/TemplateTest.java] - - To test and develop an individual template (using the l601.fermi template as an example), the following line needs to be added to the TemplateTest.java file. - - {{{ - @Test public void testl601Fermi() {runTemplateTest("l601/", "l601.fermi");} - }}} - - The individual test can be run from within Eclipse, but from the command-line, it only appears possible to run all of the TemplateTests (see not below), using the following command, whilst sat in the '''jumbo-converters/jumbo-converters-compchem/jumbo-converters-compchem-gaussian''' directory: - - {{{ - mvn -Dtest="log.TemplateTest" test	+ For the NWChem logfile templates, the code that runs these tests lives in the file: + + [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/test/java/org/xmlcml/cml/converters/compchem/nwchem/log/TemplateUnitTests.java TemplateUnitTests.java] + + To test and develop an individual template (using the xyz template as an example), the following line needs to be added to the TemplateTest.java file. + + {{{ + @Test public void testXyz() {runTemplateTest("xyz");} + }}} + + The individual test can be run from within Eclipse, but from the command-line, it only appears possible to run all of the TemplateTests (see note below), using the following command, whilst sat in the '''jumboconverters-compchem/jc-compchem-nwchem''' directory: + + {{{ + mvn -Dtest="log.TemplateUnitTests" test
Line 258:	Line 255:
- <module cmlx:templateRef="l601.fermi" xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx">	+ <module cmlx:templateRef="xyz" xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx">
Line 263:	Line 260:
- The chunk of test after the '''------------test---------------------''' line, and excluding the '''<?xml version="1.0" encoding="UTF-8"?>''' line is the output of the test. This should be checked, and if correct, placed in the '''<comment class="example.output" id="l601.fermi">''' tag in the template. Re-running the test should then lead to a successful result.	+ The chunk of test after the '''------------test---------------------''' line, and excluding the '''<?xml version="1.0" encoding="UTF-8"?>''' line is the output of the test. This should be checked, and if correct, placed in the '''<comment class="example.output" id="xyz">''' tag in the template. Re-running the test should then lead to a successful result.
Line 268:	Line 265:
- mvn -Dtest="log.TemplateTest#testl601Fermi" test	+ mvn -Dtest="log.TemplateTest#testXyz" test
Line 279:	Line 276:
- Taking the current implementation of the Gaussian log parser as an example, the code for the [https://bitbucket.org/wwmm/jumbo-converters/src/9aa183a356dd/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/java/org/xmlcml/cml/converters/compchem/gaussian/log/GaussianLog2CompchemConverter.java GaussianLog2CompchemConverter], reads in two files: - - * [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml topTemplate.xml] - as mentioned above, this processes all of the templates. - * [https://bitbucket.org/wwmm/jumbo-converters/src/9aa183a356dd/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/gaussian2compchem.xml gaussian2compchem.xml] - this is run subsequent to the file being parsed and runs transforms on the XML. As with the templates, there are a number of transforms that are carried out, the code for which either resides in this file, or files which are '''included''' by it from the templates directory. -
Line 298:	Line 290:
- A brief overview of the key transformations follows below, however, for those with a strong constitution, a more comprehensive documentation can be found by examining the code in the file: - - * [https://bitbucket.org/wwmm/jumbo-converters/src/9aa183a356dd/jumbo-converters-core/src/main/java/org/xmlcml/cml/converters/text/TransformElement.java jumbo-converters-core/src/main/java/org/xmlcml/cml/converters/text/TransformElement.java] - - The text from ~ line 156, starting with the comment '''// process values''' lists the processes that are available.	+ A brief overview of the key transformations follows below, however, for those with a strong constitution, a more comprehensive documentation can be found by examining the code in the file [https://bitbucket.org/wwmm/jumboconverters-template/src/96f9aa6e4426/src/main/java/org/xmlcml/cml/converters/templates/output/TransformElement.java TransformElement.java] + + The text from ~ line 160, starting with the comment '''// process values''' lists the processes that are available.
Line 743:	Line 733:
- Under - - {{{ - jumbo-converters/ - }}} - - we have - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - }}} - - which is where most of the important stuff for Quixote lies if we do not want to get into too many details.	+ The main folder is + + {{{ + jumboconverters-compchem/ + }}} + + Under this is: + + {{{ + jumboconverters-compchem/ + jc-compchem-nwchem/ + }}}
Line 761:	Line 749:
- jumbo-converters/ - jumbo-converters-compchem/ - src/ - target/	+ jumboconverters-compchem/ + jc-compchem-nwchem/ + src/ + target/
Line 770:	Line 758:
- jumbo-converters/ - jumbo-converters-compchem/	+ jumboconverters-compchem/ + jc-compchem-nwchem/

Declarative parsing syntax

JensThomas — 2012-06-21 09:25:23

Differences for Declarative parsing syntax

Deletions are marked with - .	Additions are marked with +.
Line 70:	Line 70:
- For example, in the case of the list in the previous section, the list includes the file {{{environment.xml}}}, which is located in - - [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601 jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601/] - - together with several other template files related to the environment link.	+ For example, in the case of the list in the previous section, the list includes the file [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/resources/org/xmlcml/cml/converters/compchem/nwchem/log/templates/environment.xml environment.xml], together with several other template files related to the environment link.
Line 157:	Line 153:
- When no further match is found, the parser will proceed to the next item in the list, in this case {{{nwchem.job.xml}}}, which will do the same process but '''without having access''' to the already captured modules.	+ When no further match is found, the parser will proceed to the next item in the list, in this case [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/resources/org/xmlcml/cml/converters/compchem/nwchem/log/templates/nwchem.job.xml nwchem.job.xml], which will do the same process but '''without having access''' to the already captured modules.
Line 159:	Line 155:
- The final important point is that this process can be nested, i.e., there can be modules inside modules inside modules, as you can see in the contents of {{{environment.xml}}}, where a deeper list of templates appear. Of course, its items will only be matched against the chunk that was matched in the parent module.	+ The final important point is that this process can be nested, i.e., there can be modules inside modules inside modules, as you can see in the contents of [https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/resources/org/xmlcml/cml/converters/compchem/nwchem/log/templates/environment.xml environment.xml], where a deeper list of templates appear. Of course, its items will only be matched against the chunk that was matched in the parent module.

Declarative parsing syntax

JensThomas — 2012-06-21 09:22:12

Differences for Declarative parsing syntax

Deletions are marked with - .	Additions are marked with +.
Line 2:	Line 2:
-
Line 8:	Line 7:
- The file called {{{topTemplate.xml}}}, which is located at the top folder for each filetype, e.g.,	+ The file called {{{topTemplate.xml}}}, which is located at the top folder for each filetype, e.g.,[https://bitbucket.org/wwmm/jumboconverters-compchem/src/693ba1b572b8/jc-compchem-nwchem/src/main/resources/org/xmlcml/cml/converters/compchem/nwchem/log/templates/topTemplate.xml topTemplate.xml]
Line 10:	Line 9:
- [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml] - - is the first one which is read by ["JUMBO-Converters"] when the file is to be parsed; a Gaussian logfile in this case.	+ is the first one which is read by ["JUMBO-Converters"] when the file is to be parsed; an NWChem logfile in this case.
Line 18:	Line 15:
- <template id='gaussian.log' output="VERBOSE"> - <templateList id='main' xmlns:xi="http://www.w3.org/2001/XInclude"> - <xi:include href="l1.temp.xml"/> - <xi:include href="l101.temp.xml"/> - <xi:include href="l202.temp.xml"/> - <xi:include href="l301.temp.xml"/> - <xi:include href="l601.temp.xml"/>	+ <!-- The ORDER of the subtemplates may be important. Templates are processed in the order + in this files and the subtemplates. Some of the files are marked with comments + --> + <template id='nwchem.log' + output="VERBOSE" + convention='conventions:compchem' + xmlns:conventions="http://www.xml-cml.org/convention/" + + xmlns:compchem="http://www.xml-cml.org/dictionary/compchem/" + xmlns:cc="http://www.xml-cml.org/dictionary/compchem/" + xmlns:n="http://www.xml-cml.org/dictionary/nwchem/" + xmlns:x="http://www.xml-cml.org/dictionary/cmlx/" + xmlns:h="http://www.w3.org/1999/xhtml" + + xmlns:cmlx="http://www.xml-cml.org/schema/cmlx" + xmlns:cml="http://www.xml-cml.org/schema" + xmlns:xi="http://www.w3.org/2001/XInclude" + > + + <dictionary uri="http://www.xml-cml.org/dictionary/nwchem/" + href="org/xmlcml/cml/converters/compchem/nwchem/nwchemDict.xml"/> + + <templateList id='main'> + <xi:include href="input.user.xml"/> <!-- do this early on --> + <xi:include href="argument.xml"/> + + <!-- errors --> + <xi:include href="error.file.xml"/> + <xi:include href="error.current.xml"/> + <xi:include href="error.mult.xml"/> + <xi:include href="error.nocat.xml"/> + <xi:include href="error.lastsys.xml"/> + + <!-- Environment same for all jobs --> + <xi:include href="environment.xml"/> + + <!-- Parse each job in turn --> + <xi:include href="nwchem.job.xml"/> + + <!-- Below come after job finished --> + <xi:include href="ga.summary.xml"/> + <xi:include href="ga.stats.xml"/> + <xi:include href="citation.xml"/> + <xi:include href="authors.xml"/> + <xi:include href="times.xml"/> +
Line 34:	Line 70:
- For example, in the case of the list in the previous section, the list includes the file {{{l601.popanal.xml}}}, which is located in	+ For example, in the case of the list in the previous section, the list includes the file {{{environment.xml}}}, which is located in
Line 38:	Line 74:
- together with several other template files related to the l601 link.	+ together with several other template files related to the environment link.
Line 43:	Line 79:
- <template id="l601.popanal" name="Population analysis using the SCF density" - repeat="" - pattern="\s\+\s$\s$\sPopulation analysis using the SCF density." - endPattern="\sN\-N\=." endOffset="1" - xmlns:xi="http://www.w3.org/2001/XInclude" - > - <comment class="example.input" id="l601.popanal"> - ... - </comment>	+ <template + id="environment" + name="Environment" + repeat="" + pattern="\sNorthwest Computational Chemistry Package .$\s+\-+." + newline="$" endPattern="\sNWChem Input Module\s$\s+\-+\s*" + offset="0" + endOffset="0">
Line 53:	Line 88:
- <record repeat="5"/> - <record repeat="2"/> - <templateList> - <xi:include href="l601.condensed.xml"/> - <xi:include href="../l401/l4601.virtual.xml"/> - ...	+ <comment class="example.input" id="environment"> + ... + </comment> + + <templateList xmlns:xi="http://www.w3.org/2001/XInclude"> + <xi:include href="nccp.xml"/> + <xi:include href="acknow.xml"/> + <xi:include href="job.info.xml"/> + <xi:include href="memory.xml"/> + <xi:include href="dirinfo.xml"/>
Line 61:	Line 100:
- <comment class="example.output" id="l601.popanal"> - ... - </comment>	+ <comment class="example.output" id="environment"> + ... + </comment>
Line 71:	Line 110:
- pattern"\s\+\s$\s$\sPopulation analysis using the SCF density."	+ pattern="\sNorthwest Computational Chemistry Package .$\s+\-+.*"
Line 77:	Line 116:
- ********************************************************************** - - Population analysis using the SCF density.	+ Northwest Computational Chemistry Package (NWChem) 6.1 + ------------------------------------------------------
Line 87:	Line 125:
- endPattern="\sN\-N\=.*"	+ endPattern="\sNWChem Input Module\s$\s+\-+\s*"
Line 93:	Line 131:
- N-N= 8.247004252289D+02 E-N=-3.066552593713D+03 KE= 6.044489055531D+02	+ NWChem Input Module + -------------------
Line 96:	Line 135:
- Everything between these two lines, '''excluding the last one''', is eaten up by this template and included into the module identified by	+ Everything between these two lines, '''excluding the last one''', is eaten up by this template and included into the module identified by the '''id''':
Line 99:	Line 138:
- l601.popanal	+ environment
Line 105:	Line 144:
- - <module cmlx:templateRef="l601.popanal" xmlns="http... - - </module	+ <module xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx" cmlx:templateRef="environment"> + ... + </module>
Line 119:	Line 157:
- When no further match is found, the parser will proceed to the next item in the list, in this case {{{l601/l601.polariz.xml}}}, which will do the same process but '''without having access''' to the already captured modules.	+ When no further match is found, the parser will proceed to the next item in the list, in this case {{{nwchem.job.xml}}}, which will do the same process but '''without having access''' to the already captured modules.
Line 121:	Line 159:
- The final important point is that this process can be nested, i.e., there can be modules inside modules inside modules, as you can see in the contents of {{{l1.temp.xml}}}, where a deeper list of templates appear. Of course, its items will only be matched against the chunk that was matched in the parent module.	+ The final important point is that this process can be nested, i.e., there can be modules inside modules inside modules, as you can see in the contents of {{{environment.xml}}}, where a deeper list of templates appear. Of course, its items will only be matched against the chunk that was matched in the parent module.

JUMBO-Converters

JensThomas — 2012-06-20 21:03:11

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 43:	Line 43:
- Each template has a '''pattern''', which is a regular expression defining the text in the logfile where the module starts, and an '''endPattern''', which is a regular expression defining where the module ends, and a '''repeatCount''', which describes how many times the module can occur within the file. - - The parser will read through the log file checking each line against the patterns in the list of templates that define the current parser. - - If a pattern is matched, all subsequent text is "gobbled" and added to the module until the '''endPattern''' is reached, at which point the module is closed, and the next line of text is searched to see if it matches a pattern (if there is no end pattern, the template will swallow all the remaining text in it's enclosing template). - - If no pattern is matched, the text is added to the parent module.	+ Each template has one or more '''pattern'''s, which is a regular expression defining the text in the logfile where the module starts, and one or more '''endPattern'''s, which is a regular expression defining where the module ends. The parser will read through the log file checking each line against the patterns in the list of templates included in the topTemplate. + + If a pattern is matched, all subsequent text is "gobbled" and added to the module until the '''endPattern''' is reached, at which point the module is closed, and the next line of text is searched to see if it matches a pattern.
Line 53:	Line 49:
- For example, the Gaussian topTemplate.xml file: - - [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml] - - contains: - - {{{ - <?xml version="1.0" encoding="UTF-8"?> - <template id='gaussian.log' - xmlns:xi="http://www.w3.org/2001/XInclude" - > - <comment> - Entering Gaussian System, Link 0=/usr/local/gaussian/g03/g03 - Initial command: - /usr/local/gaussian/g03/l1.exe /tmp/webmo/1/Gau-28330.inp -scrdir=/tmp/webmo/1/ - Entering Link 1 = /usr/local/gaussian/g03/l1.exe PID= 28333. - ... - Job cpu time: 0 days 0 hours 0 minutes 12.7 seconds. - File lengths (MBytes): RWF= 12 Int= 0 D2E= 0 Chk= 7 Scr= 1 - Normal termination of Gaussian 03 at Mon Nov 20 14:40:36 2006. - </comment> - - <templateList> - <template id="job" pattern="\s((Link1\:\s+Proceeding to internal job step number)\|(Entering Gaussian System))." - endPattern="\sNormal termination of." endOffset="1" repeat="*"> - <templateList id='main'> - <xi:include href="l0.entering.xml"/> - <xi:include href="l601/l601.anisospin.xml"/> - <xi:include href="l301.basis.xml"/> - - <!-- Many more templates --> - - </templateList> - </template> - </templateList> - </template> - }}} - - - The first few lines just declare this as a template and declare the include namespaces, so that we can use this to include other templates. - - There is then a comment containing some example text to show what this template parses. - - There is then a template list, which is the container that holds the templates. This first templateList only holds one template, which is the template that parses the various links (modules) within the Gaussian program. This template then contains a further template list, which includes the other templates that will process the text found by the parent template. - - One of the templates the parent references is the file: - - [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l301.basis.xml jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l301.basis.xml] - - which is described below. -
Line 105:	Line 50:
- - EDIT IN PROGRESS...

JUMBO-Converters

JensThomas — 2012-06-20 20:56:59

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 167:	Line 167:
- Records are the machinery used to extract text from a file and mark it up into XML. A record is an XML element, which can have a number of attributes (see below) - - Each record is processed in turn until it fails, at which point the next record is processed until all records in the module have been processed. - - A record can be used to "gobble" lines (which are discarded), or to capture data that will be marked up with XML. - - - - - Now we get to where the file is actually processed. - - {{{	+ Records are the machinery used to extract text from a file and mark it up into XML. + + A record is an XML element, which can have a number of attributes (see below) and which may contain a string, which is a simple regular expression-type language for determining what will be extracted and how it will be marked up. + + Unlike the templates, where each template is tried in turn against each line of the file, records are processed sequentially. Each record is processed in turn until it fails, at which point the next record is processed until all records in the module have been processed. + + An empty record (such as <record repeat="2"/>) can be used to "gobble" lines (which are discarded). + + If the record has content, then the text of the line is parsed into a CML list with a templateRef as specified by the '''id''' of the record. + + A simple example to read the XYZ format geometry printed in an NWChem output is shown below. The text that is to be parsed is: + + {{{ + XYZ format geometry + ------------------- + 11 + geometry + fe 0.00000000 0.00000000 0.00000000 + c 0.00000000 0.00000000 1.80680057 + o 0.77109980 -2.87778364 0.00000000 + }}} + + The records to parse this are: + + {{{ + <!-- Read 2 lines. The record has no content, so the lines are discarded. -->
Line 180:	Line 194:
- <record id="fermi.atom" repeat="" makeArray="true">\s{I,cc:serial}{A,x:elementType}${I,x:isotopeNumber}$\s{F,cc:coupling,u:au}\s{F,cc:coupling,u:mhz}\s{F,cc:coupling,u:gauss}\s{F,cc:coupling,u:ten4cm-1}\s</record> - <record repeat="4"/> - <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xx}{F,g:spindipole.yy}{F,g:spindipole.zz}</record> - <record repeat="3"/> - <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xy}{F,g:spindipole.xz}{F,g:spindipole.yz}</record> - <record repeat="1"/>}}} - - The first line just reads 2 lines from the file. As there is no id on the record, the lines are discarded. - - The next record reads as many lines (repeat="") as match the expression contained within the record into an entity called "fermi.atom". The expression that is expected to be matched is an integer (I), followed by a character string (A), followed by another integer (I) and then four floats (F). - - The first integer will be labelled '''cc:serial''', the first string '''x:elementType''' etc. The "cc" stands for '''Computational Chemistry''', as these are attribute in the Computational Chemistry dictionary, and the "g" stands for an entry in the Gaussian dictionary. - - The makeArray="true" attribute will create an array for each of the integers, floats etc. - - Once this pattern stops being matched, the parser will skip 4 lines (<record repeat="4"/>) and then the next record will process a matching block of Integers and Floats into arrays as described earlier. - - Next comes a comment: - - - - The class of "example.input" means that this block of test will be used in the tests for the parser (see below). We can see how the first line is that which is matched by the template's '''pattern''' and the last one that matched by '''endPattern'''. - - - The output has been parsed into a module with the templateRef "l601.fermi", which is the id of the template. We then have the 5 arrays (I,A,I,F,F,F,F) parsed by the "fermi.atom" record as a list, followed by that for the "fermi.spindipole" record etc.	+ + <!-- Read a line with a single integer. The integer will be placed in a CML scalar with the dictRef "compchem:numAtoms". + The scalar will itself be within a CML list with the templateRef of "atoms". --> + <record id="atoms">\s{I,compchem:numAtoms}\s</record> + + <!-- Read a line with a single character string. The string will be placed in a CML scalar with the dictRef "n:geomtype". + The scalar will itself be within a CML list with the templateRef of "atoms". --> + <record id="geo">\s{A,n:geomtype}\s</record> + + <!-- Keep reading lines while they contain a character string, followed by 3 floats. Make an array of all matching variables. + The arrays will be held in a CML list with the templateRef of "mol". --> + <record makeArray="true" repeat="" + id="mol">\s{A,compchem:elementType}\s{F,compchem:x3}\s{F,compchem:y3}\s{F,compchem:z3}\s</record> + }}} + + This result of the parsing is as follows: + {{{ + <list cmlx:templateRef="atoms"> + <scalar dataType="xsd:integer" dictRef="compchem:numAtoms">11</scalar> + </list> + <list cmlx:templateRef="geo"> + <scalar dataType="xsd:string" dictRef="n:geomtype">geometry</scalar> + </list> + <list cmlx:lineCount="3" cmlx:templateRef="mol"> + <array dataType="xsd:string" dictRef="compchem:elementType" size="3">fe c o</array> + <array dataType="xsd:double" dictRef="compchem:x3" size="3">0.0 0.0 0.7710998</array> + <array dataType="xsd:double" dictRef="compchem:y3" size="3">0.0 0.0 -2.87778364</array> + <array dataType="xsd:double" dictRef="compchem:z3" size="3">0.0 1.80680057 0.0</array> + </list> + }}}

JUMBO-Converters

JensThomas — 2012-06-20 20:09:57

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 115:	Line 115:
- <!-- The templates contain their own unit-testing framework in the comments. See the 'Unit Testing' section below	+ <!-- The templates contain their own unit-testing framework in the comments. See the 'Unit Testing Framework' section below
Line 127:	Line 127:
- <!-- The record is the mechanism to extract text into XML. See the 'records' section below for further details -->	+ <!-- The record is the mechanism to extract text into XML. See the 'Records' section below for further details -->
Line 132:	Line 132:
- <!-- The XML elements created with the records can be manipulated with transforms. See the 'transforms'	+ <!-- The XML elements created with the records can be manipulated with transforms. See the 'Transforming the raw XML'
Line 140:	Line 140:
- EXAMPLE LOGFILE TEXT above. See the 'Unit Testing' section below for more information -->	+ EXAMPLE LOGFILE TEXT above. See the 'Unit Testing Framework' section below for more information -->
Line 150:	Line 150:
- The possible ATTRIBUTES on a template are:	+ The possible ATTRIBUTES on a template are:
Line 165:	Line 165:
- - A template (aside from possibly containing sub-templates) contains a list of '''records''', which are used to parse the text constituting the module.	+ === Records === + + Records are the machinery used to extract text from a file and mark it up into XML. A record is an XML element, which can have a number of attributes (see below)
Line 205:	Line 206:
- == Testing and developing templates ==	+ === Unit Testing Framework ===
Line 281:	Line 282:
- The individual test can be run from within Eclipse, but from the command-line, it only appears possible to run the entire TemplateTests (see not below), using the following command, whilst sat in the '''jumbo-converters/jumbo-converters-compchem/jumbo-converters-compchem-gaussian''' directory:	+ The individual test can be run from within Eclipse, but from the command-line, it only appears possible to run all of the TemplateTests (see not below), using the following command, whilst sat in the '''jumbo-converters/jumbo-converters-compchem/jumbo-converters-compchem-gaussian''' directory:
Line 287:	Line 288:
- The first time this is run, it will fail. However, it will print out the output of running the test, and something like the following:	+ If you are developing a template, the first time this is run, it will fail. However, it will print out the output of running the test, and something like the following:
Line 310:	Line 311:
- == Transforming the raw XML ==	+ === Transforming the raw XML ===
Line 762:	Line 763:
	+

JUMBO-Converters

JensThomas — 2012-06-20 19:16:42

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 108:	Line 108:
- The structure of a typical template is as follows, with comments to explain the various sections.	+ The structure of a typical template is shown below with comments to explain the various sections.
Line 148:	Line 148:
- Template Attributes	+ === Template Attributes ===
Line 172:	Line 172:
- Using '''l601.fermi.xml''' as an example, we will go through the template and describe what each bit does. - - {{{ - <template id="l601.fermi" name="Isotropic Fermi Contact Couplings" - pattern="\sIsotropic Fermi Contact Couplings." - repeat="" - endPattern="\s" - offset="0" - endOffset="1" - > - }}} - - The template starts with the template tag. The name is a descriptive name for the template, and the id serves to define the module that the text will be parsed into, so the module for this template will start with: - - {{{<module cmlx:templateRef="l601.fermi" xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx">}}} - - repeat="*" says that this template can appear multiple times in the file. - - The '''pattern''' is a regular expression stating that this template will start when it matches the line: - - {{{ - Isotropic Fermi Contact Couplings - }}} - - The template will swallow all text until the end pattern is encountered, which is a line containing nothing but spaces. - - '''offset''' indicates where the text made available to the records inside the template starts. The default (0) means that the line that is matched by the '''pattern''' is part of the text within the template. An offset of 1, would mean that the line would not be available to the template, but would become part of the parent template. - - '''endOffset''' indicates where the text made available to the records inside the template stops. The default (0) means that the line that is matched by the '''endPattern''' is not part of the text within the template, but is pushed into the parent template. An offset of 1, would include the line in the template, 2 would mean that the line following the endPattern would also be included in this template -	+ + + + Now we get to where the file is actually processed. + + {{{ + <record repeat="2"/> + <record id="fermi.atom" repeat="" makeArray="true">\s{I,cc:serial}{A,x:elementType}${I,x:isotopeNumber}$\s{F,cc:coupling,u:au}\s{F,cc:coupling,u:mhz}\s{F,cc:coupling,u:gauss}\s{F,cc:coupling,u:ten4cm-1}\s</record> + <record repeat="4"/> + <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xx}{F,g:spindipole.yy}{F,g:spindipole.zz}</record> + <record repeat="3"/> + <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xy}{F,g:spindipole.xz}{F,g:spindipole.yz}</record> + <record repeat="1"/>}}} + + The first line just reads 2 lines from the file. As there is no id on the record, the lines are discarded. + + The next record reads as many lines (repeat="") as match the expression contained within the record into an entity called "fermi.atom". The expression that is expected to be matched is an integer (I), followed by a character string (A), followed by another integer (I) and then four floats (F). + + The first integer will be labelled '''cc:serial''', the first string '''x:elementType''' etc. The "cc" stands for '''Computational Chemistry''', as these are attribute in the Computational Chemistry dictionary, and the "g" stands for an entry in the Gaussian dictionary. + + The makeArray="true" attribute will create an array for each of the integers, floats etc. + + Once this pattern stops being matched, the parser will skip 4 lines (<record repeat="4"/>) and then the next record will process a matching block of Integers and Floats into arrays as described earlier.
Line 204:	Line 197:
	+ + + + The class of "example.input" means that this block of test will be used in the tests for the parser (see below). We can see how the first line is that which is matched by the template's '''pattern''' and the last one that matched by '''endPattern'''. + + + The output has been parsed into a module with the templateRef "l601.fermi", which is the id of the template. We then have the 5 arrays (I,A,I,F,F,F,F) parsed by the "fermi.atom" record as a list, followed by that for the "fermi.spindipole" record etc. + + == Testing and developing templates == + + The templates contain their own internal testing framework, in the form of one or more pairs of comment blocks within them. + + A comment block with the '''class''' attribute "example.input" should contain a small representative chunk of text that the parsers can be tested with. The '''id''' attribute is used to match the example input with the the representative output that should be produced when the template acts on the sample text. + + An input comment is shown below:
Line 230:	Line 238:
- The class of "example.input" means that this block of test will be used in the tests for the parser (see below). We can see how the first line is that which is matched by the template's '''pattern''' and the last one that matched by '''endPattern'''. - - Now we get to where the file is actually processed. - - {{{ - <record repeat="2"/> - <record id="fermi.atom" repeat="" makeArray="true">\s{I,cc:serial}{A,x:elementType}${I,x:isotopeNumber}$\s{F,cc:coupling,u:au}\s{F,cc:coupling,u:mhz}\s{F,cc:coupling,u:gauss}\s{F,cc:coupling,u:ten4cm-1}\s</record> - <record repeat="4"/> - <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xx}{F,g:spindipole.yy}{F,g:spindipole.zz}</record> - <record repeat="3"/> - <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xy}{F,g:spindipole.xz}{F,g:spindipole.yz}</record> - <record repeat="1"/>}}} - - The first line just reads 2 lines from the file. As there is no id on the record, the lines are discarded. - - The next record reads as many lines (repeat="") as match the expression contained within the record into an entity called "fermi.atom". The expression that is expected to be matched is an integer (I), followed by a character string (A), followed by another integer (I) and then four floats (F). - - The first integer will be labelled '''cc:serial''', the first string '''x:elementType''' etc. The "cc" stands for '''Computational Chemistry''', as these are attribute in the Computational Chemistry dictionary, and the "g" stands for an entry in the Gaussian dictionary. - - The makeArray="true" attribute will create an array for each of the integers, floats etc. - - Once this pattern stops being matched, the parser will skip 4 lines (<record repeat="4"/>) and then the next record will process a matching block of Integers and Floats into arrays as described earlier. - - Finally we get to the last block in the template, which is a comment if class "example.output" showing what this template processes when fed the text that was in the "example.input" comment.	+ + The matching example.output comment is below:
Line 283:	Line 269:
- The output has been parsed into a module with the templateRef "l601.fermi", which is the id of the template. We then have the 5 arrays (I,A,I,F,F,F,F) parsed by the "fermi.atom" record as a list, followed by that for the "fermi.spindipole" record etc. - - == Testing and developing templates == - - The templates contain their own internal testing framework, in the form of two of the comment blocks within them. - - The comment block with the class "example.input" should contain a small representative chunk of text that the parsers can be tested with. - - For the Gaussian logfile templates we have been dealing with here, the code that runs these tests lives in the file:	+ It is possible for the templates to contain multiple examples, provided that each pair has matching '''id''' attributes. In this case, each matching pair will be tested in turn and all must pass for the unit test to be successful. + + For the Gaussian logfile templates, the code that runs these tests lives in the file:
Line 295:	Line 275:
- To test and develop an individual template (we will continue using the l601.fermi template as an example), the following line needs to be added to the TemplateTest.java file.	+ To test and develop an individual template (using the l601.fermi template as an example), the following line needs to be added to the TemplateTest.java file.

JUMBO-Converters

JensThomas — 2012-06-20 15:02:29

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 105:	Line 105:
	+ + EDIT IN PROGRESS... + + The structure of a typical template is as follows, with comments to explain the various sections. + + {{{ + <!-- The template is contained in an XML element, with the behaviour controlled by various attributes + of the form ATTRIBUTE="VALUE". See the 'Template Attributes' section below for more information --> + <template id="foo" pattern="…"> + + <!-- The templates contain their own unit-testing framework in the comments. See the 'Unit Testing' section below + for more information --> + <comment class="example.input" id="foo"> + EXAMPLE LOGFILE TEXT + </comment> + + <!-- Templates can themselves include other templates using a templateList. Only templates, + or include directives to include other templates should be in a templateList --> + <templateList xmlns:xi="http://www.w3.org/2001/XInclude"> + <xi:include href="basis.summary.xml"/> + </templateList> + + <!-- The record is the mechanism to extract text into XML. See the 'records' section below for further details --> + <record + id="iter" + repeat="">\s{I,compchem:iterationIndex}\s+{F,compchem:totalEnergy}\s+{E,n:gnorm}\s+{E,n:gmax}\s+{F,compchem:wallTime}</record> + + <!-- The XML elements created with the records can be manipulated with transforms. See the 'transforms' + section below for more information --> + <transform process="addUnits" + xpath=".//cml:scalar[@dictRef='compchem:totalEnergy']" + value="nonsi:hartree" + /> + + <!-- This is part of the unit testing framework, and contains the marked-up text that should be created from the + EXAMPLE LOGFILE TEXT above. See the 'Unit Testing' section below for more information --> + <comment class='"example.output" id="foo"> + PARSED OUTPUT + </comment> + + </template> + }}} + + Template Attributes + + The possible ATTRIBUTES on a template are: + + * '''id''' - this should be a unique identifier. The text that is parsed by the template will be extracted into a cml module with a templateRef (in the cmlx namespace) of the id. In other words the text parsed by the template with id="foo" will end up in a module as shown below: + {{{<module xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx" cmlx:templateRef="foo"> + PARSED TEXT + </module>}}} + * '''name''' - used to give a name to the template and is currently unused. + * '''repeat''' - the number of times that a template will be matched within the file. If repeat="1" then the template will only be matched once, regardless how many times the '''pattern''' is matched in the file. repeat="" means the template will be matched as many times as the '''pattern''' is matched. + '''newline''' - the character that is used to indicate a new line in the regular expression used in the pattern or endPattern. The default is the dollar character, i.e. newline="$" + * '''pattern''', '''pattern2''', '''pattern3'''… - the regular expression used to trigger this template to start parsing text. The pattern may extend over more then one line if the '''newline''' character (see above) features in the expression. For example, pattern="\Number One$\sThe Larch\s" would only match the line "Number One" if it were followed by "The Larch". Multiple patterns can be specified using the attributes pattern2, pattern3... + * '''endPattern''', '''endPattern2''', '''endPattern3'''… - as for pattern (see above), but this matches where the template stops parsing. If endPattern="~", then if the end of the text that this template is parsing is reached, the entire text will be included in the template. If the endPattern is anything other then "~", then no text will not be included in the template and the entire text will be available for matching by another template within the parent. + * '''offset''', '''offset2''', '''offset3'''… - the number of lines either side of the match to include within the template. WIth the default of "0", all the text from (and including) the first matched line is included in the template. An offset of "-2" includes the two lines before the match, an offset of "3" excludes the first line of the match, and the two lines following it. If no offset is specified (or only offset is specified) the offset will apply to all matches (i.e. pattern, pattern2, pattern3…). If, for example, offset2 is specified, then this is the offset that will be applied when pattern2 is matched. + * '''endOffset''', '''endOffset2''', '''endOffset3'''… - the number of lines to include in the template after the endPattern match. With the default of "0", the line matched by endPattern is NOT included, and this line is pushed into the containing template, where it may be matched by the pattern of another template. An endOffset of 2, includes the endPattern line, and the one after. An endOffest of "-1" excludes the line preceding the match as well. + +

JUMBO-Converters

JensThomas — 2012-06-20 11:33:47

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 13:	Line 13:
- Most log files can be logically separated into high level units called '''chunks''', which in turn are made up of smaller units called '''blocks'''. - - There may be many repeated chunks within a log file (e.g. if a chunk is an SCF cycle, there will be as many SCF chunks as there were SCF iterations), but a block is only likely to appear once within each chunk. Continuing the SCF example, a block within the SCF chunk could be the piece of text describing the current energy and energy change from the previous cycle.	+ The approach that has been adopted by the parsers is to break the monolithic text block of the logfile into a series of separate '''chunks''' that encapsulate a coherent piece of data. + + There may be many repeated chunks within a log file. For example, if a chunk is an SCF calculation, for a single-point energy calculation there would just be a single ''SCF chunk'', whereas for a geometry optimisation calculation, there would be as many SCF chunks as there were SCF calculations. + + Chunks are often nested, so using the geometry optimisation example, a single geometry optimisation step would itself be a chunk, and this in turn would contain (one or more) SCF chunks. There would then be as many geometry optimisation step chunks as there were geometry optimisation steps.
Line 30:	Line 32:
-

Chempound

JensThomas — 2012-06-18 11:31:00

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 612:	Line 612:
	+ '''NB:''' For general information and examples of contexts file, please see the [http://wiki.eclipse.org/Jetty_Expanded_Webapp_Deploy jetty wiki]. +
Line 624:	Line 626:
	+ + === Security Considerations === + + In order to make chempound available on a standard URL (such as [http://cdsora4.dl.ac.uk/chempound]), the server needs to listen for TCP requests on port 80. + + On unix systems, only processes started by root are permitted to bind to ports numbered less than 1024, which would entail a requirement to run the chempound jetty server as root. However, this is not considered a good security practice, and there is no other reason why the server needs to run as root. + + On debian-based systems, a way around this is the '''authbind''' package, which allows users to bind non-root servers to a low-numbered port. + + Another approach is to start the server under a non-root user, binding to a high-numbered port and to use a firewall to redirect requests from port 80 to the port the server is listening on. If the server was started on port 8080, then the iptables rule to accomplish this would be: + + {{{ + iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to 8080 + }}} + + '''NB:''' if using this method, it is important to remember that the '''chempound.uri''' variable, will need to be set to point at the url as visible externally, and should not include the port number, as otherwise the CSS files will not be found.

Chempound

JensThomas — 2012-06-17 21:31:29

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 408:	Line 408:
	+ + The latest version of the war file for chempound can be downloaded [https://hudson.ch.cam.ac.uk/job/quixote-repository/lastStableBuild/uk.ac.cam.ch.wwmm.quixote.repository$quixote-repository-webapp/ here].

Chempound

2012-06-17 21:12:37

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 407:	Line 407:
- Chempound is a pure java program, so can be run in any java container, but these instructions are specific to installing it into [http://jetty.codehaus.org/jetty/ jetty] and for use on unix systems.	+ Chempound is a pure java program, so can be run in any java container. These instructions are specific to installing it into [http://jetty.codehaus.org/jetty/ jetty] for use on unix systems.
Line 443:	Line 443:
- The file '''start.jar''' is the java file used to start jetty, and the server is started with the command:	+ The file '''start.jar''' is the java file used to start jetty with the command:
Line 578:	Line 578:
- There are two ways that jetty is usually configured to server applications: - - * jetty can monitor a directory (by default the '''webapps''' directory) and any '''.war''' files placed there, will be served at a URL determined by the name of the war file (i.e. '''quixote-repository-webapp-0.1-SNAPSHOT.war''' would be served at the URL '''/quixote-repository-webapp-0.1-SNAPSHOT''' relative to the base server url.	+ There are two ways that jetty is usually configured to serve applications: + + * jetty can monitor a directory (by default the '''webapps''' directory) and any '''.war''' files placed there, will be served at a URL determined from the name of the war file (i.e. '''quixote-repository-webapp-0.1-SNAPSHOT.war''' would be served at the URL '''/quixote-repository-webapp-0.1-SNAPSHOT''' relative to the base server url.
Line 583:	Line 583:
- This example uses the second approach, he final block of XML in the '''jetty.xml''' above, configures jetty to monitor the context directory. The contexts directory contains one file, '''quixote.xml''' the contents of which are shown below (with comments to explain relevant bits of the file):	+ This example uses the second approach, the final block of XML in the '''jetty.xml''' above, configuring jetty to monitor the context directory. The contexts directory contains one file, '''quixote.xml''', the contents of which are shown below (with comments to explain relevant bits):
Line 612:	Line 612:
- The next block sets two important variables that are needed by chempound, as described below: - - * '''chempound.uri''' - this is a string included (via a template mechanism) in the html pages served by chempound and is used to set the url where various files (such as the CSS files) are expected to be found. It should be the full url where the base chempound sever will be found, such as '''http://cdsora4.dl.ac.uk/chempound'''. - * '''chempound.workspace''' - this the path to a locally accessible directory on the server where all the files needed by chempound will be stored. The actual files held by chempound (such as the logfiles, CML file etc, are stored in the this directory in the '''content''' folder.	+ The next block sets two important variables that are needed by chempound: + + * '''chempound.uri''' - this is a string included in the html pages served by chempound and is used to set the url where various files (such as the CSS files) are expected to be found. It should be the full url where the base chempound sever will be found, such as '''http://cdsora4.dl.ac.uk/chempound'''. + * '''chempound.workspace''' - this the path to a locally accessible directory on the server where all the files needed by chempound will be stored. The actual files held by chempound (such as the logfiles, CML file etc, are stored in the this directory in the '''content''' folder).

Chempound

2012-06-17 21:07:26

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 578:	Line 578:
- There are two ways that jetty is usually configured to server applications. - * jetty can monitor a directory (by default the '''webapps''' directory) and any '''.war''' files placed there, will be served at a URL determined by the name of the war file (i.e. '''quixote-repository-webapp-0.1-SNAPSHOT.war''' would be served at the URL '''/quixote-repository-webapp-0.1-SNAPSHOT''' relative to the base server url. - * jetty can monitor a directory (by default the '''contexts''' directory)	+ There are two ways that jetty is usually configured to server applications: + + * jetty can monitor a directory (by default the '''webapps''' directory) and any '''.war''' files placed there, will be served at a URL determined by the name of the war file (i.e. '''quixote-repository-webapp-0.1-SNAPSHOT.war''' would be served at the URL '''/quixote-repository-webapp-0.1-SNAPSHOT''' relative to the base server url. + * jetty can monitor a directory (by default the '''contexts''' directory) for XML files, and these will then be parsed to determine the location of the application's war file and the options required for serving the application. + + This example uses the second approach, he final block of XML in the '''jetty.xml''' above, configures jetty to monitor the context directory. The contexts directory contains one file, '''quixote.xml''' the contents of which are shown below (with comments to explain relevant bits of the file): + + {{{ + <?xml version="1.0" encoding="ISO-8859-1"?> + <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd"> + + <Configure class="org.eclipse.jetty.webapp.WebAppContext"> + <!-- This is the URL where the application will be served from --> + <Set name="contextPath">/chempound</Set> + + <!-- The absolute location of the war file for chempound on the server's filesystem --> + <Set name="war">/home/jens/jetty-hightide-8.1.4.v20120524/quixote/quixote-repository-webapp-0.1-SNAPSHOT.war</Set> + + <!-- Set startup parameters --> + <Get name="ServletContext"> + <Call name="setAttribute"> + <Arg>chempound.uri</Arg> + <Arg>http://cdsora4.dl.ac.uk/chempound</Arg> + </Call> + <Call name="setAttribute"> + <Arg>chempound.workspace</Arg> + <Arg>/home/jens/jetty-hightide-8.1.4.v20120524/quixote/workspace</Arg> + </Call> + </Get> + </Configure> + }}} + + The first two '''Set''' commands should be self-explanatory. + + The next block sets two important variables that are needed by chempound, as described below: + + * '''chempound.uri''' - this is a string included (via a template mechanism) in the html pages served by chempound and is used to set the url where various files (such as the CSS files) are expected to be found. It should be the full url where the base chempound sever will be found, such as '''http://cdsora4.dl.ac.uk/chempound'''. + * '''chempound.workspace''' - this the path to a locally accessible directory on the server where all the files needed by chempound will be stored. The actual files held by chempound (such as the logfiles, CML file etc, are stored in the this directory in the '''content''' folder. + + These two variables can also be set by setting them as environment variables before the server is started, or setting them on the command-line when the server is started as shown below: + + {{{ + java -Dchempound.uri="http://cdsora4.dl.ac.uk/chempound" -Dchempound.workspace="/home/jens/jetty-hightide-8.1.4.v20120524/quixote/workspace" -jar start.jar + }}}

Chempound

2012-06-17 20:49:19

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 409:	Line 409:
	+ There are any number of ways to configure jetty, so this just describes one way, with some pointers to the other possibilities. +
Line 416:	Line 418:
	+ +-start.ini + \|
Line 417:	Line 421:
- \| - +-start.ini
Line 427:	Line 429:
- \| - +-lib/
Line 442:	Line 442:
	+ + The file '''start.jar''' is the java file used to start jetty, and the server is started with the command: + + {{{ + java -jar start.jar + }}} + + By default, on startup, jetty will parse the file '''start.ini''', which contains command-line options for the server, including the list of modules to include, and a list of XML configuration files that determine various options (these are listed one per line in the start.ini files and can be removed by commenting the line out with the '''#''' character). By default, the XML files reside in the '''etc''' directory. In this example, only one configuration file is used, the '''jetty.xml''' file in the '''etc''' directory. + + This file contains the following: + + {{{ + <?xml version="1.0"?> + <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd"> + + <!-- =============================================================== --> + <!-- Configure the Jetty Server --> + <!-- --> + <!-- Documentation of this file format can be found at: --> + <!-- http://wiki.eclipse.org/Jetty/Reference/jetty.xml_syntax --> + <!-- --> + <!-- Additional configuration files are available in $JETTY_HOME/etc --> + <!-- and can be mixed in. For example: --> + <!-- java -jar start.jar etc/jetty-ssl.xml --> + <!-- --> + <!-- See start.ini file for the default configuraton files --> + <!-- =============================================================== --> + + + <Configure id="Server" class="org.eclipse.jetty.server.Server"> + + <!-- =========================================================== --> + <!-- Server Thread Pool --> + <!-- =========================================================== --> + <Set name="ThreadPool"> + <!-- Default queued blocking threadpool --> + <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> + <Set name="minThreads">10</Set> + <Set name="maxThreads">200</Set> + <Set name="detailedDump">false</Set> + </New> + </Set> + + <!-- =========================================================== --> + <!-- Set connectors --> + <!-- =========================================================== --> + + <Call name="addConnector"> + <Arg> + <New class="org.eclipse.jetty.server.nio.SelectChannelConnector"> + <Set name="host"><Property name="jetty.host" /></Set> + <Set name="port"><Property name="jetty.port" default="8181"/></Set> + <Set name="maxIdleTime">300000</Set> + <Set name="Acceptors">2</Set> + <Set name="statsOn">false</Set> + <Set name="confidentialPort">8443</Set> + <Set name="lowResourcesConnections">20000</Set> + <Set name="lowResourcesMaxIdleTime">5000</Set> + </New> + </Arg> + </Call> + + <!-- =========================================================== --> + <!-- Set handler Collection Structure --> + <!-- =========================================================== --> + <Set name="handler"> + <New id="Handlers" class="org.eclipse.jetty.server.handler.HandlerCollection"> + <Set name="handlers"> + <Array type="org.eclipse.jetty.server.Handler"> + <Item> + <New id="Contexts" class="org.eclipse.jetty.server.handler.ContextHandlerCollection"/> + </Item> + <Item> + <New id="DefaultHandler" class="org.eclipse.jetty.server.handler.DefaultHandler"/> + </Item> + </Array> + </Set> + </New> + </Set> + + <!-- =========================================================== --> + <!-- extra options --> + <!-- =========================================================== --> + <Set name="stopAtShutdown">true</Set> + <Set name="sendServerVersion">true</Set> + <Set name="sendDateHeader">true</Set> + <Set name="gracefulShutdown">1000</Set> + <Set name="dumpAfterStart">false</Set> + <Set name="dumpBeforeStop">false</Set> + + <!-- =============================================================== --> + <!-- Create the deployment manager --> + <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --> + <!-- The deplyment manager handles the lifecycle of deploying web --> + <!-- applications. Apps are provided by instances of the --> + <!-- AppProvider interface. Typically these are provided by --> + <!-- one or more of: --> + <!-- jetty-webapps.xml - monitors webapps for wars and dirs --> + <!-- jetty-contexts.xml - monitors contexts for context xml --> + <!-- jetty-templates.xml - monitors contexts and templates --> + <!-- =============================================================== --> + <Call name="addBean"> + <Arg> + <New id="DeploymentManager" class="org.eclipse.jetty.deploy.DeploymentManager"> + <Set name="contexts"> + <Ref id="Contexts" /> + </Set> + <!-- + <Call name="setContextAttribute"> + <Arg>org.eclipse.jetty.server.webapp.ContainerIncludeJarPattern</Arg> + <Arg>./servlet-api-[^/]\.jar$</Arg> + </Call> + --> + </New> + </Arg> + </Call> + + <!-- =============================================================== --> + <!-- Add a ContextProvider to the deployment manager --> + <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --> + <!-- This scans the contexts directory for xml files descrbing an app --> + <!-- =============================================================== --> + <Ref id="DeploymentManager"> + <Call name="addAppProvider"> + <Arg> + <New class="org.eclipse.jetty.deploy.providers.ContextProvider"> + <Set name="monitoredDirName"><Property name="jetty.home" default="." />/contexts</Set> + <Set name="scanInterval">1</Set> + </New> + </Arg> + </Call> + </Ref> + + </Configure> + }}} + + There are two ways that jetty is usually configured to server applications. + * jetty can monitor a directory (by default the '''webapps''' directory) and any '''.war''' files placed there, will be served at a URL determined by the name of the war file (i.e. '''quixote-repository-webapp-0.1-SNAPSHOT.war''' would be served at the URL '''/quixote-repository-webapp-0.1-SNAPSHOT''' relative to the base server url. + * jetty can monitor a directory (by default the '''contexts''' directory)

Chempound

JensThomas — 2012-06-16 11:51:29

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 424:	Line 424:
- +-contexts/ - \| - +-lib/ - \| - +-chempound/	+ +-etc/
Line 431:	Line 427:
	+ \| + +-lib/ + \| + +-webapps/ + \| + +-chempound/
Line 439:	Line 441:
- \| - +-webapps/ - }}}	+ }}}

Chempound

JensThomas — 2012-06-16 10:13:01

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 398:	Line 398:
	+ + == Installing Chempound == + + For installing chempound for personal use on a local machine, the [http://www.chempound.net/getting-started.html getting-started] notes should be sufficient. + + The following instructions apply for installing Chempound on an existing server, for use by an institution or group. + + === Installing Chempound into an existing Jetty server === + + Chempound is a pure java program, so can be run in any java container, but these instructions are specific to installing it into [http://jetty.codehaus.org/jetty/ jetty] and for use on unix systems. + + If you do not already have jetty installed on your server, and it is not available within the package management software for your distribution, a jetty hightide distribution, can be downloaded from [http://dist.codehaus.org/jetty/ codehaus]. + + The following instructions assume that you have a jetty server, with a directory structure similar to the following (only the relevant files and directories are listed). + + {{{ + +-jetty-hightide-8.1.4.v20120524/ + \| + +-start.jar + \| + +-start.ini + \| + +-contexts/ + \| \| + \| +-quixote.xml + \| + +-contexts/ + \| + +-lib/ + \| + +-chempound/ + \| \| + \| +-jetty.xml + \| \| + \| +-workspace/ + \| \| \| + \| \| +-cache/ + \| \| \| + \| \| +-content/ + \| \| \| + \| \| +-tdb/ + \| + +-webapps/ + }}}

Prototype data

JensThomas — 2012-06-16 09:01:25

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 587:	Line 587:
- This is just a temporary place to dump stuff related to the above before it is put somewhere sensible. - - == Conventions == - - The conventions are [http://www.xml-cml.org/convention/ here] - - The text for the conventions currently resides in the the [https://bitbucket.org/cml/xml-cml.org/src/b08186d90a19/convention bitbucket repository] for the xml-cml website. This is where the "human-readable" text lives. - - The rules that the convention text describes is actually implemented in xslt within the [https://bitbucket.org/cml/cmllite-validator-code/overview cml validator]. So for example the rules defining the compchem convention, are implemented in the [https://bitbucket.org/cml/cmllite-validator-code/src/2eaa18f959bb/src/main/resources/org/xmlcml/www/compchem-rules.xsl compchem-rules.xsl] file. - - Any changes or updates to the convention generally requires editing BOTH files. - - === Updates and Extensions to the CompChem convention === - - The [http://www.xml-cml.org/convention/compchem compchem convention] is currently rather loose and abstract, so this is an attempt to attach some definitions and examples to clarify its usage. - - The job and calculation modules are effectively functionally identical, as many of the calculations within a job could be run as separate jobs themselves (e.g. a single-point calculation in an optimisation run could be submitted separately). A job therefore serves to group one or more calculations into a logical unit of work - it can be thought of as the unit of work that would be submitted to a computational resource. - - Calculations can themselves contain calculations and be nested to any degree. Aside from having an initialization module, there is no requirement on a calculation to have any particular attributes (and indeed calculations are optional within jobs, as the job may just define an input, or the workings of the calculations may not be of any interest to the user). - - Calculations can also inherit attributes from their parents, so for example, a basis_set need not be contained within each SCF iteration calculation, as the parent's basis set can be linked to the child. - - However, both calculations and jobs must have an initialization module, which at least contains a task field to identify the role of the calculation, and any important results must go in the finalization module. This allows code developers complete freedom to structure the modules as they see fit, but any software that uses the convention knows to query the initialization module for information on the calculation, and the finalization module for any results. - - === Energies and Results === - - Several energies are output by computational jobs through the course of a run. These may be a guess energy, iteration energy, two-electron energy, correlation energy etc. Labelling each of these separately would lead to a large number of energy terms and difficulty searching for them (e.g. if you search for an MP2 energy, do you want RI-MP2 results too? Is a DFT energy labelled with the functional?). To prevent this confusion, the energies are split into categories (e.g. one-electron energy, correlation energy, total energy etc) and the meaning is determined by the context the energy is found in, i.e. the parameters of the initialization module of the calculation that contains the energy. For this reason, for an SCF calculation, the guess and individual iteration energies, each need to be held within their own calculation module. This leads to some additional verbosity and complexity, but is largely mitigated because the (e.g.) iteration calculations need only contain the information in their initialization modules that cannot be inherited from their parent SCF modules. - - The advantage of this approach is that simple generic xpath queries can be used to extract all the energies from (e.g. a geometry optimisation or mcscf calculation) by any program that wishes to plot the energy trajectory of a calculation, without needing to know anything of the structure or logic of the calculation it is querying. - - ==== Data structures === - - Scalar, array or matrix quantities are already supported in CML and be added to the initialization/finalization modules as parameters or properties. - - Any complex data structures should be held in cml lists (aside from molecules which are an existing fundamental part of CML), with a dictRef that defines what the data contained within the module is. - - The current complex data structures are: - - [http://www.xml-cml.org/convention/molecular molecules] - [http://www.xml-cml.org/dictionary/compchem/#basis_set basis sets] - [http://www.xml-cml.org/dictionary/compchem/#dft_functional DFT functional] - [http://www.xml-cml.org/dictionary/compchem/#molecular_orbital molecular orbital] - - - == Dictionaries == - - The dictionaries are [http://www.xml-cml.org/dictionary/ here]. - - - == Basis sets == - - NB: Most of this information is now in the [http://www.xml-cml.org/dictionary/compchem/ compchem dictionary] - - The [https://bse.pnl.gov/bse EMSL basis set exchange] has already done most of the work to describe basis sets. - - The paper: [http://pubs.acs.org/doi/abs/10.1021/ci600510j Basis Set Exchange: A Community Database for Computational Sciences] describes the infrastructure and the considerations that went into the design. - - The schemas they have developed are available here: https://bse.pnl.gov/bse/docs/schemas/ - - This should serve as the basis for anything that is done with basis sets in CML. They currently use version 2 of the CML schema (http://www.xml-cml.org/schema/cml2/core), but the only things they use are '''cml:matrix''' and '''cml:elementTypeType'''. The following snippet of NWChem output: - - {{{ - - - Basis "ao basis" -> "" (cartesian) - ----- - O (Oxygen) - ---------- - Exponent Coefficients - -------------- --------------------------------------------------------- - 1 S 5.48467170E+03 0.001831 - 1 S 8.25234950E+02 0.013950 - 1 S 1.88046960E+02 0.068445 - 1 S 5.29645000E+01 0.232714 - 1 S 1.68975700E+01 0.470193 - 1 S 5.79963530E+00 0.358521 - - 2 S 1.55396160E+01 -0.110778 - 2 S 3.59993360E+00 -0.148026 - 2 S 1.01376180E+00 1.130767 - - 3 P 1.55396160E+01 0.070874 - 3 P 3.59993360E+00 0.339753 - 3 P 1.01376180E+00 0.727159 - - 4 S 2.70005800E-01 1.000000 - - 5 P 2.70005800E-01 1.000000 - - 6 D 8.00000000E-01 1.000000 - - H (Hydrogen) - ------------ - Exponent Coefficients - -------------- --------------------------------------------------------- - 1 S 1.87311370E+01 0.033495 - 1 S 2.82539370E+00 0.234727 - 1 S 6.40121700E-01 0.813757 - - 2 S 1.61277800E-01 1.000000 - - - - Summary of "ao basis" -> "" (cartesian) - ------------------------------------------------------------------------------ - Tag Description Shells Functions and Types - ---------------- ------------------------------ ------ --------------------- - O 6-31g* 6 15 3s2p1d - H 6-31g* 2 2 2s - }}} - - - Would be marked up as follows: - - {{{ - <?xml version="1.0" encoding="UTF-8"?> - - <module xmlns:bse="http://purl.oclc.org/NET/EMSL/BSE" - xmlns:dc="http://purl.org/dc/elements/1.1/"> - <bse:basisSet> - - <!-- Need to look into MIME type --> - <dc:format>???</dc:format> - - <dc:title>6-31g</dc:title> - - <bse:basisSetType>orbital</bse:basisSetType> - - <bse:harmonicType>cartesian</bse:harmonicType> - - <bse:contractionType>general</bse:contractionType> - - <dc:description>???</dc:description> - - <bse:contractions elementType="O" harmonicType="cartesian"> - - <dc:description>???</dc:description> - - <bse:contraction shell="S"> - <cml:matrix rows="6" columns="2" dataType="xsd:double"> - 5.48467170E+03 0.001831 - 8.25234950E+02 0.013950 - 1.88046960E+02 0.068445 - 5.29645000E+01 0.232714 - 1.68975700E+01 0.470193 - 5.79963530E+00 0.358521 - </cml:matrix> - </bse:contraction> - - <bse:contraction shell="S"> - <cml:matrix rows="3" columns="2" dataType="xsd:double"> - 1.55396160E+01 -0.110778 - 3.59993360E+00 -0.148026 - 1.01376180E+00 1.130767 - </cml:matrix> - </bse:contraction> - - <bse:contraction shell="P"> - <cml:matrix rows="3" columns="2" dataType="xsd:double"> - 1.55396160E+01 0.070874 - 3.59993360E+00 0.339753 - 1.01376180E+00 0.727159 - </cml:matrix> - </bse:contraction> - - <bse:contraction shell="S"> - <cml:matrix rows="1" columns="2" dataType="xsd:double"> - 2.70005800E-01 1.000000 - </cml:matrix> - </bse:contraction> - - <bse:contraction shell="P"> - <cml:matrix rows="1" columns="2" dataType="xsd:double"> - 2.70005800E-01 1.000000 - </cml:matrix> - </bse:contraction> - - <bse:contraction shell="D"> - <cml:matrix rows="1" columns="2" dataType="xsd:double"> - 8.00000000E-01 1.000000 - </cml:matrix> - </bse:contraction> - - </bse:contractions> - - <bse:contractions elementType="H" harmonicType="cartesian"> - - <dc:description>???</dc:description> - - <bse:contraction shell="S"> - <cml:matrix rows="3" columns="3" dataType="xsd:double"> - 1.87311370E+01 0.033495 - 2.82539370E+00 0.234727 - 6.40121700E-01 0.813757 - </cml:matrix> - </bse:contraction> - - <bse:contraction shell="S"> - <cml:matrix rows="1" columns="1" dataType="xsd:double"> - 1.61277800E-01 1.000000 - </cml:matrix> - </bse:contraction> - - </bse:contractions> - - </bse:basisSet> - - </module> - </xml> - }}} - - Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Attributes (e.g. elementType on contractions) will be moved into separate elements, with the convention used to enforce their appearance. This also means that the attributes will appear as dictionary entries with an associated explanation. We need to add an compchem:atom_label entry so that the basis sets can be mapped onto the associated atoms - the "O" printd in the NWChem output is a tag that - in theory - could be anything, it is just used to label the atom and can be used for assigning different basis sets to different atoms of the same element type. Following on from the convention used in the dictionaries, names will be separated with an underscore. - - {{{ - <?xml version="1.0" encoding="UTF-8"?> - - <cml:module - xmlns:compchem="http://www.xml-cml.org/dictionary/compchem/" - xmlns:convention="http://www.xml-cml.org/convention/" - convention="convention:compchem" - > - <cml:list dictRef="compchem:basis_set"> - - <cml:scalar dictRef="compchem:basis_set_title" dataType="xsd:string"> - 6-31g - </cml:scalar> - - <cml:scalar dictRef="compchem:basis_set_type" dataType="xsd:string"> - orbital - </cml:scalar> - - <cml:scalar dictRef="compchem:basis_set_harmonic_type" dataType="xsd:string"> - cartesian - </cml:scalar> - - <cml:scalar dictRef="compchem:basis_set_contraction_type" dataType="xsd:string"> - general - </cml:scalar> - - <cml:scalar dictRef="compchem:basis_set_description" dataType="xsd:string"> - ??? - </cml:scalar> - - <cml:list dictRef="compchem:basis_set_contractions"> - - <!-- added as need to link contractions to atom --> - <cml:scalar dictRef="compchem:atom_label" dataType="xsd:string"> - O - </cml:scalar> - - <!-- was elementType attribute --> - <cml:scalar dictRef="compchem:element_type" dataType="xsd:string"> - O - </cml:scalar> - - <!-- was harmonicType attribute --> - <cml:scalar dictRef="compchem:basis_set_harmonic_type" dataType="xsd:string"> - cartesian - </cml:scalar> - - <cml:scalar dictRef="compchem:basis_set_contractions_description" dataType="xsd:string"> - ??? - </cml:scalar> - - <cml:list dictRef="compchem:basis_set_contraction"> - <!-- was shell attribute --> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - S - </cml:scalar> - - <cml:array size="6" dataType="xsd:double" - dictRef="compchem:basis_set_exponents"> - 5.48467170E+03 8.25234950E+02 1.88046960E+02 5.29645000E+01 1.68975700E+01 5.79963530E+00 - </cml:array> - - <cml:array size="6" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients"> - 0.001831 0.013950 0.068445 0.232714 0.470193 0.358521 - </cml:array> - </cml:list> - - <cml:list dictRef="compchem:basis_set_contraction"> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - S - </cml:scalar> - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:basis_set_exponents"> - 1.55396160E+01 3.59993360E+00 1.01376180E+00 - </cml:array> - - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients"> - -0.110778 -0.148026 1.130767 - </cml:array> - </cml:list> - - <cml:list dictRef="compchem:basis_set_contraction"> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - P - </cml:scalar> - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:basis_set_exponents"> - 1.55396160E+01 3.59993360E+00 1.01376180E+00 - </cml:array> - - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients"> - 0.070874 0.339753 0.727159 - </cml:array> - </cml:list> - - <cml:list dictRef="compchem:basis_set_contraction"> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - S - </cml:scalar> - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_exponents"> - 2.70005800E-01 - </cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients"> - 1.000000 - </cml:array> - </cml:list> - - <cml:list dictRef="compchem:basis_set_contraction" shell="P"> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - P - </cml:scalar> - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_exponents"> - 2.70005800E-01 - </cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients"> - 1.000000 - </cml:array> - </cml:list> - - <cml:list dictRef="compchem:basis_set_contraction"> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - D - </cml:scalar> - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_exponents">8.00000000E-01 - </cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients">1.000000 - </cml:array> - </cml:list> - - </cml:list> - - <cml:list dictRef="compchem:basis_set_contractions"> - - <!-- added as need to link contractions to atom --> - <cml:scalar dictRef="compchem:atom_label" dataType="xsd:string"> - H - </cml:scalar> - - <!-- was elementType attribute --> - <cml:scalar dictRef="compchem:element_type" dataType="xsd:string"> - H - </cml:scalar> - - <!-- was harmonicType attribute --> - <cml:scalar dictRef="compchem:basis_set_harmonic_type" dataType="xsd:string"> - cartesian - </cml:scalar> - - <cml:scalar dictRef="compchem:basis_set_contractions_description" dataType="xsd:string"> - ??? - </cml:scalar> - - <cml:list dictRef="compchem:basis_set_contraction"> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - S - </cml:scalar> - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:basis_set_exponents"> - 1.87311370E+01 2.82539370E+00 6.40121700E-01 - </cml:array> - - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients"> - 0.033495 0.234727 0.813757 - </cml:array> - </cml:list> - - <cml:list dictRef="compchem:basis_set_contraction"> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> - S - </cml:scalar> - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_exponents"> - 1.61277800E-01 - </cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:basis_set_coefficients"> - 1.000000 - </cml:array> - </cml:list> - - </cml:list> - - </cml:list> - - </module> - </xml> - }}}	+ The information that was previously here concerning conventions and dictionaries has now been integrated into the [http://www.xml-cml.org/convention/ convention] and [http://www.xml-cml.org/dictionary/compchem/ dictionary]

Chempound

JensThomas — 2012-04-23 09:18:30

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 191:	Line 191:
- === Discovering what terms can be searched on === - - The data that is extracted int RDF and therefore available for searching in Chempound is determined by the convention and dictionaries that apply to the files in question.	+ === Discovering the available search terms === + + The data that is extracted into RDF and therefore available for searching in Chempound is determined by the convention and dictionaries that apply to the files in question.
Line 219:	Line 219:
- SELECT ?molecule ?value - { - ?molecule cif:cell_measurement_temperature ?temp .	+ SELECT ?entry ?value + { + ?entry cif:cell_measurement_temperature ?temp .
Line 228:	Line 228:
-	+ {{{
Line 238:	Line 238:
	+ }}}

Chempound

JensThomas — 2012-04-23 09:15:00

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 34:	Line 34:
- Chempound uses a [http://en.wikipedia.org/wiki/Representational_state_transfer RESTful] interface, which means that, by going to the url for a particular calculation, depending on how we make the request to the server, we can receive the request in a variety of formats.	+ Chempound uses a [http://en.wikipedia.org/wiki/Representational_state_transfer RESTful] interface, which means that, by going to the url for a particular calculation, depending on how we make the request to the server, we can receive the requested data in a variety of formats.
Line 191:	Line 191:
- EXPLAIN HOW TO USE THE DICTIONARIES TO DISCOVER WHAT TERMS ARE AVAILABLE.	+ === Discovering what terms can be searched on === + + The data that is extracted int RDF and therefore available for searching in Chempound is determined by the convention and dictionaries that apply to the files in question. + + Please follow these links for more information on [http://www.xml-cml.org/convention/ conventions] and [http://www.xml-cml.org/dictionary/ dictionaries]. + + For CIF files, the CIF [http://www.xml-cml.org/dictionary/cif/ dictionary] lists all the terms that are available. + + For Computational Chemistry outputs, the CompChem [http://www.xml-cml.org/dictionary/compchem/ dictionary] lists the indexed terms. + + To determine how best to search for data, it is usually useful to go to the splash page for a representative structure in chempound and download the RDF file. This will show how the form of the RDF and how a structure needs to be constructed. + + For example, if we wish to search on the cell_measurement_temperature, looking at the RDF for a CIF file, we see it is structured as shown below: + + {{{ + <iucr:cell_measurement_temperature rdf:parseType="Resource"> + <rdf:value rdf:datatype="http://www.w3.org/2001/XMLSchema#double">173.0</rdf:value> + <cml:units rdf:resource="http://www.xml-cml.org/unit/sik"/> + <cml:errorValue rdf:datatype="http://www.w3.org/2001/XMLSchema#double">2.0</cml:errorValue> + </iucr:cell_measurement_temperature> + }}} + + If we just search for the cell_measurement_temperature, we will be returned the RDF resource, we therefore further need to extract the value, which is done with the following query: + + {{{ + PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> + PREFIX cif: <http://www.xml-cml.org/dictionary/cif/> + + SELECT ?molecule ?value + { + ?molecule cif:cell_measurement_temperature ?temp . + ?temp rdf:value ?value . + } + }}} + + A similar example for a CompChem file is shown below. This searches on a term in the compchem dictionary, and then filters the value for only those structures with a charge of 0. + + + PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> + PREFIX compchem: <http://www.xml-cml.org/dictionary/compchem/> + + SELECT ?molecule ?charge + { + ?molecule compchem:charge ?chargeR . + ?chargeR rdf:value ?charge + FILTER ( ?charge = 0 ) + }

Chempound

JensThomas — 2012-04-04 17:10:35

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 350:	Line 350:
- If the new terms are to be added to the chemistry search page, then the [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-handler/src/main/java/net/chempound/compchem/search/CompChemSearchProvider.java CompChemSearchProvider.java] file will need to be edited, and suitable tests added to the file [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-test-harness/src/test/java/net/chempound/compchem/CompChemSearchIntegrationTest.java ompChemSearchIntegrationTest.java]	+ If the new terms are to be added to the chemistry search page, then the [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-handler/src/main/java/net/chempound/compchem/search/CompChemSearchProvider.java CompChemSearchProvider.java] file will need to be edited, and suitable tests added to the file [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-test-harness/src/test/java/net/chempound/compchem/CompChemSearchIntegrationTest.java CompChemSearchIntegrationTest.java]

Prototype data

JensThomas — 2012-04-03 14:02:46

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 599:	Line 599:
	+ === Updates and Extensions to the CompChem convention === + + The [http://www.xml-cml.org/convention/compchem compchem convention] is currently rather loose and abstract, so this is an attempt to attach some definitions and examples to clarify its usage. + + The job and calculation modules are effectively functionally identical, as many of the calculations within a job could be run as separate jobs themselves (e.g. a single-point calculation in an optimisation run could be submitted separately). A job therefore serves to group one or more calculations into a logical unit of work - it can be thought of as the unit of work that would be submitted to a computational resource. + + Calculations can themselves contain calculations and be nested to any degree. Aside from having an initialization module, there is no requirement on a calculation to have any particular attributes (and indeed calculations are optional within jobs, as the job may just define an input, or the workings of the calculations may not be of any interest to the user). + + Calculations can also inherit attributes from their parents, so for example, a basis_set need not be contained within each SCF iteration calculation, as the parent's basis set can be linked to the child. + + However, both calculations and jobs must have an initialization module, which at least contains a task field to identify the role of the calculation, and any important results must go in the finalization module. This allows code developers complete freedom to structure the modules as they see fit, but any software that uses the convention knows to query the initialization module for information on the calculation, and the finalization module for any results. + + === Energies and Results === + + Several energies are output by computational jobs through the course of a run. These may be a guess energy, iteration energy, two-electron energy, correlation energy etc. Labelling each of these separately would lead to a large number of energy terms and difficulty searching for them (e.g. if you search for an MP2 energy, do you want RI-MP2 results too? Is a DFT energy labelled with the functional?). To prevent this confusion, the energies are split into categories (e.g. one-electron energy, correlation energy, total energy etc) and the meaning is determined by the context the energy is found in, i.e. the parameters of the initialization module of the calculation that contains the energy. For this reason, for an SCF calculation, the guess and individual iteration energies, each need to be held within their own calculation module. This leads to some additional verbosity and complexity, but is largely mitigated because the (e.g.) iteration calculations need only contain the information in their initialization modules that cannot be inherited from their parent SCF modules. + + The advantage of this approach is that simple generic xpath queries can be used to extract all the energies from (e.g. a geometry optimisation or mcscf calculation) by any program that wishes to plot the energy trajectory of a calculation, without needing to know anything of the structure or logic of the calculation it is querying. + + ==== Data structures === + + Scalar, array or matrix quantities are already supported in CML and be added to the initialization/finalization modules as parameters or properties. + + Any complex data structures should be held in cml lists (aside from molecules which are an existing fundamental part of CML), with a dictRef that defines what the data contained within the module is. + + The current complex data structures are: + + [http://www.xml-cml.org/convention/molecular molecules] + [http://www.xml-cml.org/dictionary/compchem/#basis_set basis sets] + [http://www.xml-cml.org/dictionary/compchem/#dft_functional DFT functional] + [http://www.xml-cml.org/dictionary/compchem/#molecular_orbital molecular orbital] +

Front Page

JensThomas — 2012-04-03 09:17:28

Differences for Front Page

Deletions are marked with - .	Additions are marked with +.
Line 10:	Line 10:
	+ + To see the quixote architecture in operation, visit the Cambridge [http://quixote.ch.cam.ac.uk/ Chempound] server.

Chempound

JensThomas — 2012-04-02 15:18:11

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 297:	Line 297:
- Chempound is actually a very general tool for managing collections of objects and their associated data and metadata, using RDF for the data model. As such, almost all of the chemistry functionality is implemented using plugins, so the code that needs to be modified to change the chemistry behaviour is very localised.	+ Chempound is actually a very general tool for managing collections of objects (collected as [http://www.openarchives.org/ore/1.0/datamodel ORE aggregates]) and their associated data and metadata, using [http://www.w3.org/TR/rdf-primer/ RDF] for the data model. As such, almost all of the chemistry functionality is implemented using plugins, so the code that needs to be modified to change the chemistry behaviour is very localised.
Line 324:	Line 324:
	+ \|\|\|\|\|\|\|\|
Line 328:	Line 329:
	+ + + === Adding New Data and editing the Splash page === + + Chempound extracts data from [http://www.xml-cml.org/schema/ CML] in accordance with the [http://www.xml-cml.org/convention/compchem compchem] convention. Provided that the data is a CML scalar, and is in the job's environment, initialization or finalization modules, with a dictRef (ideally) in the [http://www.xml-cml.org/dictionary/compchem/ compchem dictionary], then the data will already be extracted into RDF. + + If additional data needs to be extracted (such as is currently done for basis sets and dft functionals), then all that may be necessary is to edit the file [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-importer/src/main/java/net/chempound/compchem/CmlComp2RdfConverter.java CmlComp2RdfConverter.java] to add the additional data to the RDF. + + The html pages in chempound are generated using the [http://freemarker.sourceforge.net/ freemarker] template engine. The freemarker template that is used to generate the html page for each individual structure is the file: [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-handler/src/main/resources/net/chempound/compchem/templates/comp.ftl comp.ftl] (other template and css files are in the parent directory). + + In order to facilitate extracting key RDF data for use with the freemarker templates, several classes are used. For adding new terms, the following files needed to be edited: + + * [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-common/src/main/java/net/chempound/compchem/rdf/CompChemCalculation.java CompChemCalculation.java] - this defines the interface that will be used by the freemarker template to access the data. + + * [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-common/src/main/java/net/chempound/compchem/rdf/ont/CompChem.java CompChem.java] - this creates the RDF terms that are used. + + * [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-common/src/main/java/net/chempound/compchem/rdf/impl/CompChemCalculationImpl.java CompChemCalculationImpl.java] - this actually implements the functions to get the data. + + + When the new terms have been added, the tests should be updated, or a new test added in the directory [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-importer/src/test/java/net/chempound/compchem] + + If the new terms are to be added to the chemistry search page, then the [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-handler/src/main/java/net/chempound/compchem/search/CompChemSearchProvider.java CompChemSearchProvider.java] file will need to be edited, and suitable tests added to the file [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-test-harness/src/test/java/net/chempound/compchem/CompChemSearchIntegrationTest.java ompChemSearchIntegrationTest.java]

Chempound

JensThomas — 2012-04-02 13:54:19

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 324:	Line 324:
	+ \|\| [https://bitbucket.org/chempound/compchem/ compchem ] \|\| [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-common compchem-common] \|\| General code related to the compchem RDF data structures. The utility functions used by the freemarker templates to access the compchem data live here. \|\| + \|\|\|\| [https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-handler compchem-handler] \|\| Code to handle the processing of chemical data on the server, such as display of the html pages and the freemarker templates. \|\| + \|\|\|\|[https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-importer compchem-importer] \|\| The classes to handle importing code-specific logfiles (NWChem, Gaussian etc) using the jumbo-classes. These classes are used by the client, not chempound itself. The test cases for checking the imports also live here. \|\| + \|\|\|\|[https://bitbucket.org/chempound/compchem/src/ef32d64ba51b/compchem-test-harness compchem-test-harness] \|\| Code to test the various compchem-specific modules, as most do not contain any test code themselves. \|\|

Chempound

JensThomas — 2012-04-02 13:32:30

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 293:	Line 293:
- This section is for those who may be interested in altering or extending Chempound. It isn't intended to be a programmer's manual, more a brief overview of chempound's current structure and a walk-though on how to add additional CML data to the repository, which is expected to be the reason why most people would want to extend Chempound.	+ This section is for those who may be interested in altering or extending Chempound. It isn't intended to be a programmer's manual, more a brief overview of chempound's current structure and a walk-though on how to add additional CML data to the repository, which is expected to be the reason why most people would currently want to extend Chempound.
Line 314:	Line 314:
	+ + A slightly more detailed view of the chemistry-specific repositories and their modules follows below. + + \|\|'''Repository''' \|\| '''Modules''' \|\| '''Description and important classes''' \|\| + \|\| [https://bitbucket.org/chempound/chemistry chemistry ] \|\| [https://bitbucket.org/chempound/chemistry/src/208dcf643265/chemistry-common chemistry-common] \|\| Classes to handle the generic processing of CML datatypes and the conversion to RDF \|\| + \|\| \|\| \|\| '''* net.chempound.chemistry.cmlChemicalMine.java''' - mime types \|\| + \|\| \|\| \|\| '''* net.chempound.chemistry.Cml2RdfConverter.java''' - code to handle the conversion of generic, simple cml datatypes into RDF. \|\| + \|\| \|\|[https://bitbucket.org/chempound/chemistry/src/208dcf643265/chemistry-importer chemistry-importer] \|\| Base classes for the client-side conversion of files and the generation of images \|\| + \|\| \|\|[https://bitbucket.org/chempound/chemistry/src/208dcf643265/chemistry-jmol-plugin chemistry-jmol-plugin] \|\| Classes to drive jmol to generate the images, and also the jmol code itself \|\| + \|\| \|\|[https://bitbucket.org/chempound/chemistry/src/208dcf643265/chemistry-search-structure chemistry-search-structure] \|\| Classes to handle the chemistry-specific search page - if you want to add more chemistry search boxes, the you'll need to edit things here. \|\|

Chempound

JensThomas — 2012-04-02 13:05:11

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 291:	Line 291:
- - - ##==Download== - ##The source code for chempound is available from [https://bitbucket.org/chempound]. - ## - ##Pre-built binaries are available from the Cambridge hudson [https://hudson.ch.cam.ac.uk server]. - ## - ##To run any of the binaries, you will need to have [http://www.java.com/ java] installed. - ## - ##The latest version of the client (which is used for loading files into chempound), is available [https://hudson.ch.cam.ac.uk/job/quixote-client/lastStableBuild/ here]. The latest standalone version (which includes its own java server), is available [https://hudson.ch.cam.ac.uk/job/quixote-repository-standalone/lastStableBuild/ here]. In both cases, download either the zip or tar.gz file from the '''Build Artifacts''' section towards the top of the page. - ## - ##==Running the server== - ##If you are familiar with running your own java servers, you can run chempound from within tomcat or jetty using the archive available from [https://hudson.ch.cam.ac.uk/job/quixote-repository/lastStableBuild/ here] - ## - ##To run the standalone server (which doesn't require you to already have installed a sever), download the zip or tar.gz file for the [https://hudson.ch.cam.ac.uk/job/quixote-repository-standalone/lastStableBuild/ "standalone server"]. Under Windows use the '''run.bat''' file to start the server, and under unix/osx, use the '''run.sh''' script. These will both start the server on port 8181, which can be accessed by pointing a browser at: [http://localhost:8181] - ## - ##At this point, you will have chempound running on your machine, but it will have no data in it. - ## - ##==Adding files== - ##Files are added using the client, which can be downloaded as a zip or tar.gz file from [https://hudson.ch.cam.ac.uk/job/quixote-client/lastStableBuild/ here]. Once the zip file is unpacked, the client is run from the command-line using the scripts in the '''bin''' directory: '''qc.bat''' file under windows or '''qc.sh file''' under unix/osx. These will need the '''JAVA_HOME''' environment variable to be set and the path to the '''bin''' directory holding the scripts will need to be in your PATH. - ## - ##The first step in using the client is to create a collection within chempound to hold your files. This just serves to group the files together. Create the collection with the following command (replace '''qc.sh''' with '''qc.bat''' under windows): - ## - ##{{{ - ##qc.sh -R http://localhost:8181/content/ create-collection <collection name> - ##}}} - ## - ##where '''<collection name>''' is the desired name of the collection. - ## - ##If you now point your browser at [http://localhost:8181], you should be able to browse to your collection, although it obviously has no files in it. - ## - ##To add a gaussian file, use the command: - ## - ##{{{ - ##qc.sh -R http://localhost:8181/content/<collection name>/ deposit-gaussian <path to gaussian file> - ##}}} - ## - ##The file types that are currently supported are: - ##[http://www.gaussian.com/ Gaussian]: '''deposit-gaussian''' - ##[http://www.nwchem-sw.org/ NWChem]: '''deposit-nwchem''' - ##[http://www.iucr.org/resources/cif CIF]: '''deposit-cif''' - ##	+ == Hacking Chempound == + + This section is for those who may be interested in altering or extending Chempound. It isn't intended to be a programmer's manual, more a brief overview of chempound's current structure and a walk-though on how to add additional CML data to the repository, which is expected to be the reason why most people would want to extend Chempound. + + '''NB:''' additional information can be found in Jorge Estrada's [https://bytebucket.org/jestrada/quixote-docs/wiki/developersGuide/quixote-developersGuide.html repository] + + Chempound is actually a very general tool for managing collections of objects and their associated data and metadata, using RDF for the data model. As such, almost all of the chemistry functionality is implemented using plugins, so the code that needs to be modified to change the chemistry behaviour is very localised. + + === Overview of the repositories === + + The repositories for the chempound packages is hosted on bitbucket: [https://bitbucket.org/chempound] + + Currently, there are 8 repositories as detailed below: + + * [https://bitbucket.org/chempound/] - this contains the main server code. There is almost no chemistry-specific code here, apart from in the chempound-rdf-cml directory, which has a very small class to add some CML data to the RDF model. + * [https://bitbucket.org/chempound/chemistry] - this is where the most general chemistry code lives, and where the general functions to handle the conversion of data from CML are. + * [https://bitbucket.org/chempound/chempound-client] - the base classes for the command-line client (it is the client that actually handles the conversion of logfiles into CML and the generation of the jmol pictures etc) are here, although there is no chemistry-specific code here. + * [https://bitbucket.org/chempound/chempound-parent] - this just contains the central maven pom.xml that is used to configure maven for chempound. + * [https://bitbucket.org/chempound/compchem] - all the code to handle the data associated with computational chemistry calculations (both server and client) lives here. + * [https://bitbucket.org/chempound/crystallography] - all the code to handle the crystallography-specific aspects of the data. + * [https://bitbucket.org/chempound/deposit-client] - TODO - not had to look at this yet. + * [https://bitbucket.org/chempound/quixote-client] - the code to drive the code-specific imports of compchem logfiles. + * [https://bitbucket.org/chempound/quixote-repository] - this is more code to package chempound for use by the quixote project and create the stand-alone chempound server war file.

JUMBO-Converters

JensThomas — 2012-03-30 08:52:19

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 371:	Line 371:
- * '''addUnits''' - TODO	+ * '''addUnits''' - this will add a '''units''' attribute to the element with the value specified in the '''value''' argument. The value should be of the form '''namespace:id''', where namespace refers to one of the [http://www.xml-cml.org/dictionary/ units dictionaries] and the id points to the actual unit. In the example below, the namespace refers to the [http://www.xml-cml.org/unit/nonSi/ non-si unit dictionary] and the id links to the entry for the [http://www.xml-cml.org/unit/nonSi/#hartree hartree].
Line 375:	Line 375:
- xpath=".//cml:scalar" - value="u:jmol-1"/>	+ xpath=".//cml:scalar[@dictRef='compchem:total_energy']" + value="nonsi:hartree" + />

JUMBO-Converters

JensThomas — 2012-03-22 18:55:17

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 663:	Line 663:
- * '''move''' - this takes one or more nodes, and moves them into the node defined by the '''to''' argument. The '''to''' argument is an xpath that must just return a single element.	+ * '''move''' - this takes one or more nodes, and moves them into the node defined by the '''to''' argument. The '''to''' argument is an xpath that must just return a single element. The '''position''' argument indicates where in the children of the target, the element will be moved to. "1" makes the element the first, "2" makes it the second etc.

Prototype data

JensThomas — 2012-03-14 20:56:26

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 589:	Line 589:
	+ == Conventions == + + The conventions are [http://www.xml-cml.org/convention/ here] + + The text for the conventions currently resides in the the [https://bitbucket.org/cml/xml-cml.org/src/b08186d90a19/convention bitbucket repository] for the xml-cml website. This is where the "human-readable" text lives. + + The rules that the convention text describes is actually implemented in xslt within the [https://bitbucket.org/cml/cmllite-validator-code/overview cml validator]. So for example the rules defining the compchem convention, are implemented in the [https://bitbucket.org/cml/cmllite-validator-code/src/2eaa18f959bb/src/main/resources/org/xmlcml/www/compchem-rules.xsl compchem-rules.xsl] file. + + Any changes or updates to the convention generally requires editing BOTH files. + + + == Dictionaries == + + The dictionaries are [http://www.xml-cml.org/dictionary/ here]. +
Line 591:	Line 606:
	+ + NB: Most of this information is now in the [http://www.xml-cml.org/dictionary/compchem/ compchem dictionary]

Prototype data

JensThomas — 2012-03-14 13:21:13

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 585:	Line 585:
- = Changes to CML = - - This is just a temporary place to dump stuff that relates to changes to CML.	+ = Conventions, Dictionaries and CML = + + This is just a temporary place to dump stuff related to the above before it is put somewhere sensible. +

JUMBO-Converters

2012-03-13 22:06:27

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 313:	Line 313:
- * '''addChild''' - this will create a child element of the nodes specified by the '''xpath'''. The only required argument is '''elementName''', which specifies the type of element to create. Other supported arguments are: '''id''', '''dictRef''' and '''value'''. The '''position''' argument specifies where the child will be created in the list of children. position="0" creates it as the first child, "1", the second etc. With no position argument, the child is added as the last child. '''NB:''' the value argument currently doesn't support the $string() syntax.	+ * '''addChild''' - this will create a child element of the nodes specified by the '''xpath'''. The only required argument is '''elementName''', which specifies the type of element to create. Other supported arguments are: '''id''', '''dictRef''' and '''value'''. The '''position''' argument specifies where the child will be created in the list of children. position="0" creates it as the first child, "1", the second etc. With no position argument, the child is added as the last child. If value is a string of the form "$string(XPATH)" or $number(XPATH), where XPATH is a valid XPATH, then the value will be the result of evaluating the XPATH relative to the current node in the nodeset evaluated by '''xpath''', in string or number form.
Line 361:	Line 361:
- * '''addSibling''' - this will add a sibling element to each node in the '''xpath''' nodeset, with the type of element being that specified in '''elementName''' and the elements id attribute as specified by '''id''' argument. The '''position''' argument indicates where the element will be created, "0" creates the element before node, "1" creates it after the current node. If there are multiple siblings to the current node, "-2" would create it 2 nodes down from the current node, "2", one up from it etc.	+ * '''addSibling''' - this will add a sibling element to each node in the '''xpath''' nodeset, with the type of element being that specified in '''elementName''' and the elements id attribute as specified by '''id''' argument. The '''position''' argument indicates where the element will be created, "0" creates the element before node, "1" creates it after the current node. If there are multiple siblings to the current node, "-2" would create it 2 nodes down from the current node, "2", one up from it etc. If value is a string of the form "$string(XPATH)" or $number(XPATH), where XPATH is a valid XPATH, then the value will be the result of evaluating the XPATH relative to the current node in the nodeset evaluated by '''xpath''', in string or number form.

Prototype data

JensThomas — 2012-03-12 13:21:44

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 749:	Line 749:
- Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Attributes (e.g. elementType on contractions) will be moved into separate elements, with the convention used to enforce their appearance. This also means that the attributes will appear as dictionary entries with an associated explanation. We need to add an compchem:atom_type entry so that the basis sets can be mapped onto the associated atoms - the "O" printd in the NWChem output is a tag that - in theory - could be anything, it is just used to label the atom and can be used for assigning different basis sets to different atoms of the same element type. Following on from the convention used in the dictionaries, names will be separated with an underscore.	+ Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Attributes (e.g. elementType on contractions) will be moved into separate elements, with the convention used to enforce their appearance. This also means that the attributes will appear as dictionary entries with an associated explanation. We need to add an compchem:atom_label entry so that the basis sets can be mapped onto the associated atoms - the "O" printd in the NWChem output is a tag that - in theory - could be anything, it is just used to label the atom and can be used for assigning different basis sets to different atoms of the same element type. Following on from the convention used in the dictionaries, names will be separated with an underscore.
Line 784:	Line 784:
- <cml:scalar dictRef="compchem:atom_type" dataType="xsd:string">	+ <cml:scalar dictRef="compchem:atom_label" dataType="xsd:string">
Line 897:	Line 897:
- <cml:scalar dictRef="compchem:atom_type" dataType="xsd:string">	+ <cml:scalar dictRef="compchem:atom_label" dataType="xsd:string">

Prototype data

JensThomas — 2012-03-12 13:13:10

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 749:	Line 749:
- Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Attributes (e.g. elementType on contractions) will be moved into separate elements, with the convention used to enforce their appearance. This also means that the attributes will appear as dictionary entries with an associated explanation. Entries that are to be indexed and made searchable within chempound will be wrapped with a parameter element. Following on from the convention used in the dictionaries, names will be separated with an underscore.	+ Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Attributes (e.g. elementType on contractions) will be moved into separate elements, with the convention used to enforce their appearance. This also means that the attributes will appear as dictionary entries with an associated explanation. We need to add an compchem:atom_type entry so that the basis sets can be mapped onto the associated atoms - the "O" printd in the NWChem output is a tag that - in theory - could be anything, it is just used to label the atom and can be used for assigning different basis sets to different atoms of the same element type. Following on from the convention used in the dictionaries, names will be separated with an underscore.
Line 761:	Line 761:
- <!-- currently only need basis set title - should everything be a parameter? --> - <parameter dictRef="compchem:basis_set_title" > - <cml:scalar dataType="xsd:string">	+ <cml:scalar dictRef="compchem:basis_set_title" dataType="xsd:string">
Line 765:	Line 763:
- </cml:scalar> - </parameter>	+ </cml:scalar>
Line 786:	Line 783:
	+ <!-- added as need to link contractions to atom --> + <cml:scalar dictRef="compchem:atom_type" dataType="xsd:string"> + O + </cml:scalar> +
Line 789:	Line 791:
- </cml:scalar>	+ </cml:scalar>
Line 794:	Line 796:
- </cml:scalar>	+ </cml:scalar>
Line 800:	Line 802:
- <cml:list dictRef="compchem:basis_set_contraction"> - <!-- was shell attribute --> - <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string">	+ <cml:list dictRef="compchem:basis_set_contraction"> + <!-- was shell attribute --> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string">
Line 804:	Line 806:
- </cml:scalar>	+ </cml:scalar> +
Line 814:	Line 817:
- </cml:list>	+ </cml:list>
Line 819:	Line 822:
- </cml:scalar> - <cml:array size="3" dataType="xsd:double"	+ </cml:scalar> + <cml:array size="3" dataType="xsd:double"
Line 823:	Line 826:
- </cml:array> - - <cml:array size="3" dataType="xsd:double"	+ </cml:array> + + <cml:array size="3" dataType="xsd:double"
Line 828:	Line 831:
- </cml:array>	+ </cml:array>
Line 834:	Line 837:
- </cml:scalar> - <cml:array size="3" dataType="xsd:double"	+ </cml:scalar> + <cml:array size="3" dataType="xsd:double"
Line 838:	Line 841:
- </cml:array> - - <cml:array size="3" dataType="xsd:double"	+ </cml:array> + + <cml:array size="3" dataType="xsd:double"
Line 843:	Line 846:
- </cml:array>	+ </cml:array>
Line 849:	Line 852:
- </cml:scalar> - <cml:array size="1" dataType="xsd:double"	+ </cml:scalar> + <cml:array size="1" dataType="xsd:double"
Line 853:	Line 856:
- </cml:array> - - <cml:array size="1" dataType="xsd:double"	+ </cml:array> + + <cml:array size="1" dataType="xsd:double"
Line 858:	Line 861:
- </cml:array>	+ </cml:array>
Line 864:	Line 867:
- </cml:scalar> - <cml:array size="1" dataType="xsd:double"	+ </cml:scalar> + <cml:array size="1" dataType="xsd:double"
Line 868:	Line 871:
- </cml:array> - - <cml:array size="1" dataType="xsd:double"	+ </cml:array> + + <cml:array size="1" dataType="xsd:double"
Line 873:	Line 876:
- </cml:array>	+ </cml:array>
Line 879:	Line 882:
- </cml:scalar> - <cml:array size="1" dataType="xsd:double"	+ </cml:scalar> + <cml:array size="1" dataType="xsd:double"
Line 882:	Line 885:
- </cml:array> - - <cml:array size="1" dataType="xsd:double"	+ </cml:array> + + <cml:array size="1" dataType="xsd:double"
Line 886:	Line 889:
- </cml:array>	+ </cml:array>
Line 892:	Line 895:
	+ + <!-- added as need to link contractions to atom --> + <cml:scalar dictRef="compchem:atom_type" dataType="xsd:string"> + H + </cml:scalar>
Line 896:	Line 904:
- </cml:scalar>	+ </cml:scalar>
Line 901:	Line 909:
- </cml:scalar> - - <cml:scalar dictRef="compchem:basis_set_contractions_description" dataType="xsd:string">	+ </cml:scalar> + + <cml:scalar dictRef="compchem:basis_set_contractions_description" dataType="xsd:string">
Line 905:	Line 913:
- </cml:scalar>	+ </cml:scalar>
Line 910:	Line 918:
- </cml:scalar> - <cml:array size="3" dataType="xsd:double"	+ </cml:scalar> + <cml:array size="3" dataType="xsd:double"
Line 914:	Line 922:
- </cml:array> - - <cml:array size="3" dataType="xsd:double"	+ </cml:array> + + <cml:array size="3" dataType="xsd:double"
Line 919:	Line 927:
- </cml:array>	+ </cml:array>
Line 925:	Line 933:
- </cml:scalar> - <cml:array size="1" dataType="xsd:double"	+ </cml:scalar> + <cml:array size="1" dataType="xsd:double"
Line 929:	Line 937:
- </cml:array> - - <cml:array size="1" dataType="xsd:double"	+ </cml:array> + + <cml:array size="1" dataType="xsd:double"
Line 934:	Line 942:
- </cml:array>	+ </cml:array>

Prototype data

JensThomas — 2012-03-11 19:56:55

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 749:	Line 749:
- Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Entries that are to be indexed and made searchable within chempound will be wrapped with a parameter element. Following on from the convention used in the dictionaries, names will be separated with an underscore.	+ Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Attributes (e.g. elementType on contractions) will be moved into separate elements, with the convention used to enforce their appearance. This also means that the attributes will appear as dictionary entries with an associated explanation. Entries that are to be indexed and made searchable within chempound will be wrapped with a parameter element. Following on from the convention used in the dictionaries, names will be separated with an underscore.
Line 761:	Line 761:
- <cml:scalar dictRef="compchem:basis_set_title" dataType="xsd:string"> - 6-31g*	+ <!-- currently only need basis set title - should everything be a parameter? --> + <parameter dictRef="compchem:basis_set_title" > + <cml:scalar dataType="xsd:string"> + 6-31g* + </cml:scalar> + </parameter> + + <cml:scalar dictRef="compchem:basis_set_type" dataType="xsd:string"> + orbital
Line 765:	Line 772:
- <cml:scalar dictRef="compchem:basis_set_type" dataType="xsd:string"> - orbital - </cml:scalar> -
Line 770:	Line 773:
- cartesian	+ cartesian

Prototype data

JensThomas — 2012-03-11 19:51:28

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 748:	Line 748:
	+ + Within the context of conventions and dictionaries as detailed in the [http://www.jcheminf.com/content/3/1/43 J. Chem. Inf. paper (doi:10.1186/1758-2946-3-43)], this is best dealt with using existing CML datatypes and dictionaries. For the time being, lists will be used to hold all complex data structures, with dictionary entries describing the element. Rather then using matricies to hold the exponents and coefficients, arrays will be used. Entries that are to be indexed and made searchable within chempound will be wrapped with a parameter element. Following on from the convention used in the dictionaries, names will be separated with an underscore. + + {{{ + <?xml version="1.0" encoding="UTF-8"?> + + <cml:module + xmlns:compchem="http://www.xml-cml.org/dictionary/compchem/" + xmlns:convention="http://www.xml-cml.org/convention/" + convention="convention:compchem" + > + <cml:list dictRef="compchem:basis_set"> + + <cml:scalar dictRef="compchem:basis_set_title" dataType="xsd:string"> + 6-31g* + </cml:scalar> + + <cml:scalar dictRef="compchem:basis_set_type" dataType="xsd:string"> + orbital + </cml:scalar> + + <cml:scalar dictRef="compchem:basis_set_harmonic_type" dataType="xsd:string"> + cartesian + </cml:scalar> + + <cml:scalar dictRef="compchem:basis_set_contraction_type" dataType="xsd:string"> + general + </cml:scalar> + + <cml:scalar dictRef="compchem:basis_set_description" dataType="xsd:string"> + ??? + </cml:scalar> + + <cml:list dictRef="compchem:basis_set_contractions"> + + <!-- was elementType attribute --> + <cml:scalar dictRef="compchem:element_type" dataType="xsd:string"> + O + </cml:scalar> + + <!-- was harmonicType attribute --> + <cml:scalar dictRef="compchem:basis_set_harmonic_type" dataType="xsd:string"> + cartesian + </cml:scalar> + + <cml:scalar dictRef="compchem:basis_set_contractions_description" dataType="xsd:string"> + ??? + </cml:scalar> + + <cml:list dictRef="compchem:basis_set_contraction"> + <!-- was shell attribute --> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + S + </cml:scalar> + <cml:array size="6" dataType="xsd:double" + dictRef="compchem:basis_set_exponents"> + 5.48467170E+03 8.25234950E+02 1.88046960E+02 5.29645000E+01 1.68975700E+01 5.79963530E+00 + </cml:array> + + <cml:array size="6" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients"> + 0.001831 0.013950 0.068445 0.232714 0.470193 0.358521 + </cml:array> + </cml:list> + + <cml:list dictRef="compchem:basis_set_contraction"> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + S + </cml:scalar> + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:basis_set_exponents"> + 1.55396160E+01 3.59993360E+00 1.01376180E+00 + </cml:array> + + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients"> + -0.110778 -0.148026 1.130767 + </cml:array> + </cml:list> + + <cml:list dictRef="compchem:basis_set_contraction"> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + P + </cml:scalar> + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:basis_set_exponents"> + 1.55396160E+01 3.59993360E+00 1.01376180E+00 + </cml:array> + + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients"> + 0.070874 0.339753 0.727159 + </cml:array> + </cml:list> + + <cml:list dictRef="compchem:basis_set_contraction"> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + S + </cml:scalar> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_exponents"> + 2.70005800E-01 + </cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients"> + 1.000000 + </cml:array> + </cml:list> + + <cml:list dictRef="compchem:basis_set_contraction" shell="P"> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + P + </cml:scalar> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_exponents"> + 2.70005800E-01 + </cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients"> + 1.000000 + </cml:array> + </cml:list> + + <cml:list dictRef="compchem:basis_set_contraction"> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + D + </cml:scalar> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_exponents">8.00000000E-01 + </cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients">1.000000 + </cml:array> + </cml:list> + + </cml:list> + + <cml:list dictRef="compchem:basis_set_contractions"> + + <!-- was elementType attribute --> + <cml:scalar dictRef="compchem:element_type" dataType="xsd:string"> + H + </cml:scalar> + + <!-- was harmonicType attribute --> + <cml:scalar dictRef="compchem:basis_set_harmonic_type" dataType="xsd:string"> + cartesian + </cml:scalar> + + <cml:scalar dictRef="compchem:basis_set_contractions_description" dataType="xsd:string"> + ??? + </cml:scalar> + + <cml:list dictRef="compchem:basis_set_contraction"> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + S + </cml:scalar> + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:basis_set_exponents"> + 1.87311370E+01 2.82539370E+00 6.40121700E-01 + </cml:array> + + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients"> + 0.033495 0.234727 0.813757 + </cml:array> + </cml:list> + + <cml:list dictRef="compchem:basis_set_contraction"> + <cml:scalar dictRef="compchem:basis_set_shell" dataType="xsd:string"> + S + </cml:scalar> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_exponents"> + 1.61277800E-01 + </cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:basis_set_coefficients"> + 1.000000 + </cml:array> + </cml:list> + + </cml:list> + + </cml:list> + + </module> + </xml> + }}}

Prototype data

JensThomas — 2012-03-11 12:50:24

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 597:	Line 597:
- This should serve as the basis for anything that is done with basis sets in CML. They currently use version 2 of the CML schema (http://www.xml-cml.org/schema/cml2/core), but the only things they use are '''cml:matrix''' and '''cml:elementTypeType'''. The cml:matrix element is used to hold the coefficients and exponents, although it would appear that a cml:array would be more suitable for this. With this single change taken into consideration, the following snippet of NWChem output:	+ This should serve as the basis for anything that is done with basis sets in CML. They currently use version 2 of the CML schema (http://www.xml-cml.org/schema/cml2/core), but the only things they use are '''cml:matrix''' and '''cml:elementTypeType'''. The following snippet of NWChem output:
Line 676:	Line 676:
	+ <bse:contraction shell="S"> + <cml:matrix rows="6" columns="2" dataType="xsd:double"> + 5.48467170E+03 0.001831 + 8.25234950E+02 0.013950 + 1.88046960E+02 0.068445 + 5.29645000E+01 0.232714 + 1.68975700E+01 0.470193 + 5.79963530E+00 0.358521 + </cml:matrix> + </bse:contraction> +
Line 677:	Line 688:
- <cml:array size="6" dataType="xsd:double" - dictRef="compchem:gbasis_exponents"> - 5.48467170E+03 8.25234950E+02 1.88046960E+02 - 5.29645000E+01 1.68975700E+01 - 5.79963530E+00 - </cml:array> - - <cml:array size="6" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients"> - 0.001831 0.013950 0.068445 0.232714 0.470193 - 0.358521 - </cml:array>	+ <cml:matrix rows="3" columns="2" dataType="xsd:double"> + 1.55396160E+01 -0.110778 + 3.59993360E+00 -0.148026 + 1.01376180E+00 1.130767 + </cml:matrix>
Line 691:	Line 695:
	+ <bse:contraction shell="P"> + <cml:matrix rows="3" columns="2" dataType="xsd:double"> + 1.55396160E+01 0.070874 + 3.59993360E+00 0.339753 + 1.01376180E+00 0.727159 + </cml:matrix> + </bse:contraction> +
Line 692:	Line 704:
- <cml:array size="3" dataType="xsd:double" - dictRef="compchem:gbasis_exponents"> - 1.55396160E+01 3.59993360E+00 1.01376180E+00 - </cml:array> - - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients"> - -0.110778 -0.148026 1.130767 - </cml:array>	+ <cml:matrix rows="1" columns="2" dataType="xsd:double"> + 2.70005800E-01 1.000000 + </cml:matrix>
Line 704:	Line 710:
- <cml:array size="3" dataType="xsd:double" - dictRef="compchem:gbasis_exponents"> - 1.55396160E+01 3.59993360E+00 1.01376180E+00 - </cml:array> - - <cml:array size="3" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients"> - 0.070874 0.339753 0.727159 - </cml:array>	+ <cml:matrix rows="1" columns="2" dataType="xsd:double"> + 2.70005800E-01 1.000000 + </cml:matrix>
Line 715:	Line 715:
	+ <bse:contraction shell="D"> + <cml:matrix rows="1" columns="2" dataType="xsd:double"> + 8.00000000E-01 1.000000 + </cml:matrix> + </bse:contraction> + + </bse:contractions> + + <bse:contractions elementType="H" harmonicType="cartesian"> + + <dc:description>???</dc:description> +
Line 716:	Line 728:
- <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_exponents"> - 2.70005800E-01 - </cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients"> - 1.000000 - </cml:array>	+ <cml:matrix rows="3" columns="3" dataType="xsd:double"> + 1.87311370E+01 0.033495 + 2.82539370E+00 0.234727 + 6.40121700E-01 0.813757 + </cml:matrix>
Line 727:	Line 735:
- <bse:contraction shell="P"> - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_exponents"> - 2.70005800E-01 - </cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients"> - 1.000000 - </cml:array> - </bse:contraction> - - <bse:contraction shell="D"> - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_exponents">8.00000000E-01</cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients">1.000000</cml:array> - </bse:contraction> - - </bse:contractions> - - <bse:contractions elementType="H" harmonicType="cartesian"> - - <dc:description>???</dc:description> -
Line 754:	Line 736:
- <cml:array size="3" dataType="xsd:double" - dictRef="compchem:gbasis_exponents"> - 1.87311370E+01 2.82539370E+00 6.40121700E-01 - </cml:array> - - <cml:array size="6" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients"> - 0.033495 0.234727 0.813757 - </cml:array> - </bse:contraction> - - <bse:contraction shell="S"> - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_exponents"> - 1.61277800E-01 - </cml:array> - - <cml:array size="1" dataType="xsd:double" - dictRef="compchem:gbasis_coefficients"> - 1.000000 - </cml:array>	+ <cml:matrix rows="1" columns="1" dataType="xsd:double"> + 1.61277800E-01 1.000000 + </cml:matrix>

JUMBO-Converters

JensThomas — 2012-03-09 18:44:02

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 581:	Line 581:
- * '''createWrapper''' - TODO	+ * '''createWrapper''' - for each node in the '''xpath''' nodeset, this will create an enveloping element of type '''elementName''' that will become the child of the node's parent, and hold the node and all of its children. '''id''' and '''dictRef''' arguments are supported.
Line 589:	Line 589:
- * '''createWrapperMetadata''' - TODO	+ * '''createWrapperMetadata''' - for each node in the '''xpath''' nodelist, if the node is one of a cml:scalar, cml:array, cml:list, cml:table or cml:matrix, and has a "dictRef" attribute, it will remove the dictRef attribute and instead wrap the element in a cml:metadata element, so that, e.g. '''{{{<scalar dataType="xsd:string" dictRef="n:basis_type">ao basis</scalar>}}}''' becomes: + + {{{ + <metadata name="n:basis_type"> + <scalar dataType="xsd:string">ao basis</scalar> + </metadata> + }}}
Line 598:	Line 604:
- * '''createWrapperParameter''' - TODO	+ * '''createWrapperParameter''' - this performs the same operation as createWrapperMetadata, but wraps the element in a cml:parameter with the dictRef of the target element.
Line 608:	Line 614:
- * '''createWrapperProperty''' - TODO	+ * '''createWrapperProperty''' - - this performs the same operation as createWrapperMetadata, but wraps the element in a cml:property with the dictRef of the target element.

JUMBO-Converters

JensThomas — 2012-03-08 14:33:24

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 313:	Line 313:
- * '''addChild''' - this will create a child element of the nodes specified by the '''xpath'''. The only required argument is '''elementName''', which specifies the type of element to create. Other supported arguments are: '''id''', '''dictRef''' and '''value'''. '''NB:''' the value argument currently doesn't support the $string() syntax.	+ * '''addChild''' - this will create a child element of the nodes specified by the '''xpath'''. The only required argument is '''elementName''', which specifies the type of element to create. Other supported arguments are: '''id''', '''dictRef''' and '''value'''. The '''position''' argument specifies where the child will be created in the list of children. position="0" creates it as the first child, "1", the second etc. With no position argument, the child is added as the last child. '''NB:''' the value argument currently doesn't support the $string() syntax.
Line 320:	Line 320:
	+ position="0"

JUMBO-Converters

JensThomas — 2012-03-08 14:28:46

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 313:	Line 313:
- * '''addChild''' - this will create a child element of the nodes specified by the '''xpath'''. The only required argument is '''elementName''', which specifies the type of element to create. Other supported arguments are: '''id''', '''dictRef''' and '''value'''.	+ * '''addChild''' - this will create a child element of the nodes specified by the '''xpath'''. The only required argument is '''elementName''', which specifies the type of element to create. Other supported arguments are: '''id''', '''dictRef''' and '''value'''. '''NB:''' the value argument currently doesn't support the $string() syntax.

Prototype data

2012-03-08 12:05:41

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 597:	Line 597:
- This should serve as the basis for anything that is done with basis sets in CML. They currently use version 2 of the CML schema (http://www.xml-cml.org/schema/cml2/core), but the only things they use are '''cml:matrix''' and '''cml:elementTypeType'''. The cml:matrix element is used to hold the coefficients and exponents, although it would appear that a cml:array would be more suitable for this. With this single change taking into consideration, the following snippet of NWChem output:	+ This should serve as the basis for anything that is done with basis sets in CML. They currently use version 2 of the CML schema (http://www.xml-cml.org/schema/cml2/core), but the only things they use are '''cml:matrix''' and '''cml:elementTypeType'''. The cml:matrix element is used to hold the coefficients and exponents, although it would appear that a cml:array would be more suitable for this. With this single change taken into consideration, the following snippet of NWChem output:

Prototype data

2012-03-08 12:04:50

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 655:	Line 655:
- <module xmlns:bse="http://purl.oclc.org/NET/EMSL/BSE">	+ <module xmlns:bse="http://purl.oclc.org/NET/EMSL/BSE" + xmlns:dc="http://purl.org/dc/elements/1.1/">
Line 659:	Line 660:
- <bse:format>???</bse:format> - - <bse:title>6-31g*</bse:title>	+ <dc:format>???</dc:format> + + <dc:title>6-31g*</dc:title>
Line 669:	Line 670:
- <bse:description>???</bse:description>	+ <dc:description>???</dc:description>
Line 673:	Line 674:
- <bse:description>???</bse:description>	+ <dc:description>???</dc:description>
Line 750:	Line 751:
- <bse:description>???</bse:description>	+ <dc:description>???</dc:description>

Prototype data

2012-03-08 12:01:05

Differences for Prototype data

Deletions are marked with - .	Additions are marked with +.
Line 584:	Line 584:
	+ + = Changes to CML = + + This is just a temporary place to dump stuff that relates to changes to CML. + + == Basis sets == + + The [https://bse.pnl.gov/bse EMSL basis set exchange] has already done most of the work to describe basis sets. + + The paper: [http://pubs.acs.org/doi/abs/10.1021/ci600510j Basis Set Exchange: A Community Database for Computational Sciences] describes the infrastructure and the considerations that went into the design. + + The schemas they have developed are available here: https://bse.pnl.gov/bse/docs/schemas/ + + This should serve as the basis for anything that is done with basis sets in CML. They currently use version 2 of the CML schema (http://www.xml-cml.org/schema/cml2/core), but the only things they use are '''cml:matrix''' and '''cml:elementTypeType'''. The cml:matrix element is used to hold the coefficients and exponents, although it would appear that a cml:array would be more suitable for this. With this single change taking into consideration, the following snippet of NWChem output: + + {{{ + + + Basis "ao basis" -> "" (cartesian) + ----- + O (Oxygen) + ---------- + Exponent Coefficients + -------------- --------------------------------------------------------- + 1 S 5.48467170E+03 0.001831 + 1 S 8.25234950E+02 0.013950 + 1 S 1.88046960E+02 0.068445 + 1 S 5.29645000E+01 0.232714 + 1 S 1.68975700E+01 0.470193 + 1 S 5.79963530E+00 0.358521 + + 2 S 1.55396160E+01 -0.110778 + 2 S 3.59993360E+00 -0.148026 + 2 S 1.01376180E+00 1.130767 + + 3 P 1.55396160E+01 0.070874 + 3 P 3.59993360E+00 0.339753 + 3 P 1.01376180E+00 0.727159 + + 4 S 2.70005800E-01 1.000000 + + 5 P 2.70005800E-01 1.000000 + + 6 D 8.00000000E-01 1.000000 + + H (Hydrogen) + ------------ + Exponent Coefficients + -------------- --------------------------------------------------------- + 1 S 1.87311370E+01 0.033495 + 1 S 2.82539370E+00 0.234727 + 1 S 6.40121700E-01 0.813757 + + 2 S 1.61277800E-01 1.000000 + + + + Summary of "ao basis" -> "" (cartesian) + ------------------------------------------------------------------------------ + Tag Description Shells Functions and Types + ---------------- ------------------------------ ------ --------------------- + O 6-31g* 6 15 3s2p1d + H 6-31g* 2 2 2s + }}} + + + Would be marked up as follows: + + {{{ + <?xml version="1.0" encoding="UTF-8"?> + + <module xmlns:bse="http://purl.oclc.org/NET/EMSL/BSE"> + <bse:basisSet> + + <!-- Need to look into MIME type --> + <bse:format>???</bse:format> + + <bse:title>6-31g*</bse:title> + + <bse:basisSetType>orbital</bse:basisSetType> + + <bse:harmonicType>cartesian</bse:harmonicType> + + <bse:contractionType>general</bse:contractionType> + + <bse:description>???</bse:description> + + <bse:contractions elementType="O" harmonicType="cartesian"> + + <bse:description>???</bse:description> + + <bse:contraction shell="S"> + <cml:array size="6" dataType="xsd:double" + dictRef="compchem:gbasis_exponents"> + 5.48467170E+03 8.25234950E+02 1.88046960E+02 + 5.29645000E+01 1.68975700E+01 + 5.79963530E+00 + </cml:array> + + <cml:array size="6" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients"> + 0.001831 0.013950 0.068445 0.232714 0.470193 + 0.358521 + </cml:array> + </bse:contraction> + + <bse:contraction shell="S"> + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:gbasis_exponents"> + 1.55396160E+01 3.59993360E+00 1.01376180E+00 + </cml:array> + + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients"> + -0.110778 -0.148026 1.130767 + </cml:array> + </bse:contraction> + + <bse:contraction shell="P"> + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:gbasis_exponents"> + 1.55396160E+01 3.59993360E+00 1.01376180E+00 + </cml:array> + + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients"> + 0.070874 0.339753 0.727159 + </cml:array> + </bse:contraction> + + <bse:contraction shell="S"> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_exponents"> + 2.70005800E-01 + </cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients"> + 1.000000 + </cml:array> + </bse:contraction> + + <bse:contraction shell="P"> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_exponents"> + 2.70005800E-01 + </cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients"> + 1.000000 + </cml:array> + </bse:contraction> + + <bse:contraction shell="D"> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_exponents">8.00000000E-01</cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients">1.000000</cml:array> + </bse:contraction> + + </bse:contractions> + + <bse:contractions elementType="H" harmonicType="cartesian"> + + <bse:description>???</bse:description> + + <bse:contraction shell="S"> + <cml:array size="3" dataType="xsd:double" + dictRef="compchem:gbasis_exponents"> + 1.87311370E+01 2.82539370E+00 6.40121700E-01 + </cml:array> + + <cml:array size="6" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients"> + 0.033495 0.234727 0.813757 + </cml:array> + </bse:contraction> + + <bse:contraction shell="S"> + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_exponents"> + 1.61277800E-01 + </cml:array> + + <cml:array size="1" dataType="xsd:double" + dictRef="compchem:gbasis_coefficients"> + 1.000000 + </cml:array> + </bse:contraction> + + </bse:contractions> + + </bse:basisSet> + + </module> + </xml> + }}}

JUMBO-Converters

JensThomas — 2012-03-07 21:17:56

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 304:	Line 304:
- * '''addAttribute''' - TODO	+ * '''addAttribute''' - add an attribute of type '''name''' and value '''value''' to all nodes in the '''xpath''' nodeset. If value is a string of the form "$string(XPATH)" or $number(XPATH), where XPATH is a valid XPATH, then the value will be the result of evaluating the XPATH relative to the current node in the nodeset evaluated by '''xpath''', in string or number form. If '''name''' consists of two strings separated by a colon, SOMETHING EXCITING HAPPENS...

JUMBO-Converters

JensThomas — 2012-03-07 20:14:43

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 549:	Line 549:
- * '''createString''' - if '''xpath''' returns a list arrays, then each array will be converted into a cml:scalar with a dataType xsd:string, with the value of the scalar being the values in the array as strings and separated by whitespace. If '''xpath''' returns a list of cml:scalars, the the first scalar will be converted to type xsd:string, the value of which will be the concatenation of all the values in the remaining scalar nodes. The remaining scalar nodes will then be deleted. If a single node is returned by '''xpath''' and it is of instance text, then a new cml:scalar node will be created in its place with an optional id attribute as specified in the '''id''' argument.	+ * '''createString''' - if '''xpath''' returns a list arrays, then each array will be converted into a cml:scalar with a dataType xsd:string, with the value of the scalar being the values in the array, concatentated as strings and separated by whitespace. If '''xpath''' returns a list of cml:scalars, the the first scalar will be converted to type xsd:string, the value of which will be the concatenation of all the values in the remaining scalar nodes. The remaining scalar nodes will then be deleted. If a single node is returned by '''xpath''' and it is of instance text, then a new cml:scalar node will be created in its place with an optional id attribute as specified in the '''id''' argument.

JUMBO-Converters

JensThomas — 2012-03-07 18:31:06

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 549:	Line 549:
- * '''createString''' - TODO	+ * '''createString''' - if '''xpath''' returns a list arrays, then each array will be converted into a cml:scalar with a dataType xsd:string, with the value of the scalar being the values in the array as strings and separated by whitespace. If '''xpath''' returns a list of cml:scalars, the the first scalar will be converted to type xsd:string, the value of which will be the concatenation of all the values in the remaining scalar nodes. The remaining scalar nodes will then be deleted. If a single node is returned by '''xpath''' and it is of instance text, then a new cml:scalar node will be created in its place with an optional id attribute as specified in the '''id''' argument.

JUMBO-Converters

JensThomas — 2012-03-07 18:13:58

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 360:	Line 360:
- * '''addSibling''' - TODO	+ * '''addSibling''' - this will add a sibling element to each node in the '''xpath''' nodeset, with the type of element being that specified in '''elementName''' and the elements id attribute as specified by '''id''' argument. The '''position''' argument indicates where the element will be created, "0" creates the element before node, "1" creates it after the current node. If there are multiple siblings to the current node, "-2" would create it 2 nodes down from the current node, "2", one up from it etc.
Line 365:	Line 365:
- elementName="cml:module" id="l202.group" position="1" />	+ elementName="cml:module" + id="l202.group" + position="1" />
Line 692:	Line 694:
- * '''setValue''' - TODO	+ * '''setValue''' - with just a simple string as an argument (e.g. value="foo") to the value argument, this will set the value of all nodes in the '''xpath''' to be equal to this string. If value is a string of the form "$string(XPATH)" or $number(XPATH), where XPATH is a valid XPATH, then the value will be the result of evaluating the XPATH relative to the current node in the nodeset evaluated by '''xpath''', in string or number form. With a map argument ...TODO

JUMBO-Converters

2012-03-07 16:56:40

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 647:	Line 647:
- * '''joinArrays''' - TODO	+ * '''joinArrays''' - with a single '''xpath''' argument, this will take the first array in the nodelist and join all the others to it, deleting the other arrays and leaving a single array with the dictRef of the original array. With an additional '''key''' argument, SOMETHING ELSE HAPPENS. With an additional '''from''' argument, SOMETHING ELSE HAPPENS.

JUMBO-Converters

JensThomas — 2012-03-06 22:32:41

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 332:	Line 332:
- * '''addId''' - TODO	+ * '''addId''' - this adds an '''id'''with the value specified by the '''value''' argument to the nodeset specified in the '''xpath'''.
Line 340:	Line 340:
- * '''addMap''' - TODO - - {{{ - <transform process="addMap" xpath="."	+ * '''addMap''' - for every node in the nodeset specified by '''xpath''', this creates a '''cml:map''' with the specified '''id''' that links the '''values''' of the nodes in the '''from''' nodeset to that in the '''to''' nodeset. + + {{{ + <transform process="addMap" + xpath="."
Line 349:	Line 350:
- * '''addNamespace''' - TODO	+ * '''addNamespace''' - this will add a namespace element of the form xmlns:'''name'''="'''value'''" to every element in the nodeset returned by '''xpath'''.

JUMBO-Converters

JensThomas — 2012-03-06 19:09:59

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 282:	Line 282:
	+ The philosophy of the transforms is very similar to the idea of templates in [http://en.wikipedia.org/wiki/XSLT xslt], using the idea of "nodeset" to which operations are applied. +
Line 290:	Line 292:
- The transforms have a '''process''' which defines the operation that will be carried out, almost all have an '''xpath''' that is an xpath expression indicating the elements the process will be applied to, and a variable number of arguments, depending on the process being carried out.	+ The transforms have a '''process''' which defines the operation that will be carried out, almost all have an '''xpath''' that is an xpath expression indicating the elements the process will be applied to (the nodeset), and a variable number of arguments, depending on the process being carried out.
Line 297:	Line 299:
	+ + Various miscellaneous notes will be added in the section below, which will be merged into the documentation in due course.
Line 702:	Line 706:
	+ }}} + + + === Notes on Transforms === + * where possible, '''id''''s should always be added to nodesets to facilitate later operations. + * the < and > symbols should not be used in xpath comparisons, however < and > can be used, as shown below: + {{{ + <transform process="debugNodes" xpath=".//cml:array[position() > 1 and position() < 4]"/> + }}} + + * use "..." for quoting attribute values + * Rather then use relative namespaces (e.g. g:charge), the more reliable namespace-uri syntax can be used: + {{{ + foo[@dictRef[namespace-uri()='http://www.xml-cml.org/dict/gaussian' and .='charge']]

JUMBO-Converters

JensThomas — 2012-03-06 10:20:03

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 302:	Line 302:
	+ {{{ + <transform process="addAttribute" + xpath=".//cml:molecule" + name="formalCharge" + value="$number(.//cml:scalar[@dictRef='g:charge'])" /> + }}} +
Line 309:	Line 316:
- dictRef="cc:jobList"/>	+ dictRef="cc:jobList" />
Line 318:	Line 325:
- value="cc:popanal"/>	+ value="cc:popanal "/>
Line 323:	Line 330:
- * '''addInteger''' - TODO	+ {{{ + <transform process="addId" + value="mol9999" + xpath=".//cml:molecule[starts-with(@id,'a')]" /> + }}}
Line 327:	Line 338:
	+ {{{ + <transform process="addMap" xpath="." + id="variableConstantMap" + from=".//cml:scalar[@dictRef='g:variable' or @dictRef='g:const']" + to=".//cml:scalar[@dictRef='g:value']" /> + }}} +
Line 329:	Line 347:
	+ {{{ + <transform process="addNamespace" + xpath="." + name="convention" + value="http://www.xml-cml.org/convention/" /> + }}} + +
Line 331:	Line 357:
	+ {{{ + <transform process="addSibling" + xpath="./cml:module[@id='calculation']/cml:module[@cmlx:templateRef='l202.rotconst']" + elementName="cml:module" id="l202.group" position="1" /> + }}} +
Line 332:	Line 364:
	+ + {{{ + <transform process="addUnits" + xpath=".//cml:scalar" + value="u:jmol-1"/> + }}}
Line 343:	Line 381:
	+ {{{ + <transform process="createAngle" + xpath=".//cml:list/cml:list[cml:atom]" + atomRefs="$string(cml:scalar[3]) $string(cml:scalar[1]) $string(cml:atom/@id)" value="$string(cml:scalar[4])" /> + }}} +
Line 353:	Line 397:
	+ {{{ + <transform process="createAtom" + xpath=".//cml:scalar[@dictRef='cc:elementType']" /> + }}} +
Line 355:	Line 404:
	+ {{{ + <transform process="createDate" + xpath=".//cml:list[@dictRef='g:archive1']/cml:scalar[9]" + format="dd-MMM-yyyy" + dictRef="cc:date"/> + }}} +
Line 357:	Line 413:
	+ {{{ + <transform process="createDouble" + xpath=".//cml:list[@dictRef='g:archive.namevalue']/cml:scalar[@dictRef='x:HF']" + dictRef="cc:hfenergy" /> + }}} +
Line 359:	Line 421:
	+ {{{ + <transform process="createFormula" + xpath=".//cml:list[@dictRef='g:archive1']/cml:scalar[7]"/> + }}} +
Line 360:	Line 427:
	+ + {{{ + <transform process="createLength" + xpath=".//cml:list/cml:list[cml:atom]" + atomRefs="$string(cml:scalar[1]) $string(cml:atom/@id)" value="$string(cml:scalar[2])"/> + }}}
Line 370:	Line 443:
	+ {{{ + <transform process="createMatrix33" + xpath="." dictRef="g:axis" + from=".//cml:scalar[contains(@dictRef,':x.') or contains(@dictRef,':y.') or contains(@dictRef,':z.')]" /> + }}} +
Line 371:	Line 450:
	+ + {{{ + <transform process="createMatrix33" xpath="." + dictRef="g:axis" + from=".//cml:scalar[contains(@dictRef,':x.') or contains(@dictRef,':y.') or contains(@dictRef,':z.')]" /> + }}}
Line 450:	Line 535:
	+ {{{ + <transform process="createNameValue" + xpath="./cml:list/cml:list" + name=".//cml:scalar[@dictRef='x:name']" + value=".//cml:scalar[@dictRef='x:value']" /> + }}} +
Line 452:	Line 544:
	+ {{{ + <transform process="createString" + xpath="./cml:list/cml:scalar"/> + }}} +
Line 454:	Line 551:
	+ {{{ + <transform process="createTable" + xpath=".//cml:list[@cmlx:templateRef='symmadapt']" /> + }}} +
Line 455:	Line 557:
	+ + {{{ + <transform process="createTorsion" + xpath=".//cml:list/cml:list[cml:atom]" + atomRefs="$string(cml:scalar[5]) $string(cml:scalar[3]) $string(cml:scalar[1]) $string(cml:atom/@id)" value="$string(cml:scalar[6])" /> + }}}
Line 467:	Line 575:
	+ {{{ + <transform process="createWrapper" + xpath=".//cml:module/text()" + elementName="UNPARSED"/> + }}} +
Line 469:	Line 583:
	+ {{{ + <transform process="createWrapperMetadata" + xpath=".//cml:scalar[@dictRef='cc:version' or + @dictRef='cc:date' or + @dictRef='cc:title']"/> + }}} +
Line 471:	Line 592:
	+ {{{ + <transform process="createWrapperParameter" + xpath=".//cml:scalar[@dictRef='cc:hostname' or + @dictRef='cc:jobname' or + @dictRef='cc:method' or + @dictRef='cc:basis' ]"/> + }}} +
Line 473:	Line 602:
	+ {{{ + <transform process="createWrapperProperty" + xpath=".//*[@dictRef='cc:electronicstate' or + @dictRef='cc:hfenergy' or + @dictRef='cc:dipole' or + @dictRef='cc:dipolederiv' or + @dictRef='cc:polarizability' or + @dictRef='cc:pointgroup' or + @dictRef='cc:rmsd' or + @dictRef='cc:rmsf']"/> + }}} +
Line 474:	Line 615:
	+ + {{{ + <transform process="createZMatrix" + xpath="." id="zinitial"/> + }}}
Line 491:	Line 637:
	+ {{{ + <transform process="groupSiblings" + xpath=".//cml:module[@id='l202.group']" /> + }}} +
Line 492:	Line 643:
	+ + {{{ + <transform process="joinArrays" + xpath=".//cml:list[@cmlx:templateRef='atom']/cml:array" /> + }}}
Line 525:	Line 681:
	+ {{{ + <transform process="reparse" + xpath=".//cml:scalar[@id='scraped']" + regexPath=".//record[@id='natoms']"/> + }}} +
Line 526:	Line 688:
	+ + {{{ + <transform process="setValue" + xpath=".//cml:list/cml:scalar[2] \| + .//cml:list/cml:scalar[4] \| + .//cml:list/cml:scalar[6]" + map="//cml:map[@id='variableMap']" + value="$string(.)"/> + }}}

JUMBO-Converters

JensThomas — 2012-03-06 09:29:58

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 300:	Line 300:
	+ * '''addAttribute''' - TODO +
Line 319:	Line 321:
	+ * '''addId''' - TODO + + * '''addInteger''' - TODO + + * '''addMap''' - TODO + + * '''addNamespace''' - TODO + + * '''addSibling''' - TODO + + * '''addUnits''' - TODO +
Line 327:	Line 341:
	+ * '''createAngle''' - TODO +
Line 335:	Line 351:
	+ * '''createAtom''' - TODO + + * '''createDate''' - TODO + + * '''createDouble''' - TODO + + * '''createForumla''' - TODO + + * '''createLength''' - TODO +
Line 341:	Line 367:
	+ + * '''createMatrix''' - TODO + + * '''createMatrix33''' - TODO
Line 418:	Line 448:
	+ * '''createNameValue''' - TODO + + * '''createString''' - TODO + + * '''createTable''' - TODO + + * '''createTorsion''' - TODO
Line 428:	Line 465:
	+ * '''createWrapper''' - TODO + + * '''createWrapperMetadata''' - TODO + + * '''createWrapperParameter''' - TODO + + * '''createWrapperProperty''' - TODO + + * '''createZMatrix''' - TODO
Line 442:	Line 488:
	+ + * '''groupSiblings''' - TODO + + * '''joinArrays''' - TODO
Line 472:	Line 522:
	+ + * '''reparse''' - TODO + + * '''setValue''' - TODO

JUMBO-Converters

JensThomas — 2012-03-05 21:53:40

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 325:	Line 325:
	+ }}} + + * '''createArray''' - this will create a cml:array at each of the nodes in the '''xpath''' query from the cml:scalar nodes generated by the '''from''' xpath query. If only one node is supplied, the contents of the node will be separated by whitespace and the array created from these. Arrays can only be created for integer or double data types. The scalar nodes with then be discarded. + + {{{ + <transform process="createArray" + xpath="." + from="./cml:list[@cmlx:templateRef='length']/cml:scalar[@dictRef='g:symbol']"/>

JUMBO-Converters

JensThomas — 2012-03-05 21:37:56

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 330:	Line 330:
- <transform process="createList" xpath=".//cml:module[@cmlx:templateRef='multipole']"/>	+ <transform process="createList" + xpath=".//cml:module[@cmlx:templateRef='multipole']"/> + }}} + + * '''createMolecule''' - this will create a molecule from the list of cml:arrays generated by the 'xpath' query. The length of the arrays indicates the number of atoms, and the '''dictRef''' attribute of each array determines the property of the atom it will be used for. Supported types are: '''x3''', '''y3''', '''z3''' for the coordinates, '''id''', '''elementType''', '''label''' and '''atomTypeRef'''. The molecule will be created as a child of the parent of the first array, and the arrays will then be discarded. The gaussian template [https://bitbucket.org/wwmm/jumbo-converters/src/9aa183a356dd/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l202/l202.orient.xml l202.orient.xml] is shown below as an example. + + {{{ + <template id="l202.orient" name="input or standard orientation" repeat="" + pattern="\s(Input\|Standard)\sorientation:\s$\s\-+\s" + endPattern="\s\d.$\s\-+\s" endOffset="2"> + + <comment class="example.input" id="l202.orient"> + Input orientation: + --------------------------------------------------------------------- + Center Atomic Atomic Coordinates (Angstroms) + Number Number Type X Y Z + --------------------------------------------------------------------- + 1 6 0 0.000000 0.000000 0.000000 + 2 1 0 0.000000 0.000000 1.093266 + 3 1 0 1.030741 0.000000 -0.364422 + 4 1 0 -0.515370 -0.892648 -0.364422 + 5 1 0 -0.515371 0.892648 -0.364422 + --------------------------------------------------------------------- + </comment> + + <record repeat="5"/> + <record repeat="" makeArray="true" id="atom">{I,cc:serial}{I,cc:elementType}{I,g:atomicType}{F,cc:x3}{F,cc:y3}{F,cc:z3}</record> + <record/> + + <transform process="createMolecule" xpath="./cml:list[@cmlx:templateRef='atom']/cml:array" id="mol.l202.orient"/> + <transform process="pullupSingleton" xpath="./cml:list"/> + + <comment class="example.output" id="l202.orient"> + <module cmlx:templateRef="l202.orient" xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx"> + <molecule id="mol.l202.orient" cmlx:templateRef="atom"> + <atomArray> + <atom id="a1" elementType="C" x3="0.0" y3="0.0" z3="0.0"> + <scalar dataType="xsd:integer" dictRef="cc:serial">1</scalar> + <scalar dataType="xsd:integer" dictRef="g:atomicType">0</scalar> + <scalar dataType="xsd:integer" dictRef="cc:atomicNumber">6</scalar> + </atom> + <atom id="a2" elementType="H" x3="0.0" y3="0.0" z3="1.093266"> + <scalar dataType="xsd:integer" dictRef="cc:serial">2</scalar> + <scalar dataType="xsd:integer" dictRef="g:atomicType">0</scalar> + <scalar dataType="xsd:integer" dictRef="cc:atomicNumber">1</scalar> + </atom> + <atom id="a3" elementType="H" x3="1.030741" y3="0.0" z3="-0.364422"> + <scalar dataType="xsd:integer" dictRef="cc:serial">3</scalar> + <scalar dataType="xsd:integer" dictRef="g:atomicType">0</scalar> + <scalar dataType="xsd:integer" dictRef="cc:atomicNumber">1</scalar> + </atom> + <atom id="a4" elementType="H" x3="-0.51537" y3="-0.892648" z3="-0.364422"> + <scalar dataType="xsd:integer" dictRef="cc:serial">4</scalar> + <scalar dataType="xsd:integer" dictRef="g:atomicType">0</scalar> + <scalar dataType="xsd:integer" dictRef="cc:atomicNumber">1</scalar> + </atom> + <atom id="a5" elementType="H" x3="-0.515371" y3="0.892648" z3="-0.364422"> + <scalar dataType="xsd:integer" dictRef="cc:serial">5</scalar> + <scalar dataType="xsd:integer" dictRef="g:atomicType">0</scalar> + <scalar dataType="xsd:integer" dictRef="cc:atomicNumber">1</scalar> + </atom> + </atomArray> + <formula formalCharge="0" concise="C 1 H 4"> + <atomArray elementType="C H" count="1.0 4.0"/> + </formula> + <bondArray> + <bond atomRefs2="a1 a2" id="a1_a2" order="S"/> + <bond atomRefs2="a1 a3" id="a1_a3" order="S"/> + <bond atomRefs2="a1 a4" id="a1_a4" order="S"/> + <bond atomRefs2="a1 a5" id="a1_a5" order="S"/> + </bondArray> + <property dictRef="cml:molmass"> + <scalar dataType="xsd:double" units="unit:dalton" xmlns:unit="http://www.xml-cml.org/unit/si/">16.04246</scalar> + </property> + </molecule> + </module> + </comment> + </template> + }}} + + + '''createVector3''' - for each node specified in the '''xpath''' this will take the nodes listed in the '''to''' argument and create a cml:vector from them, and give it the specified '''dictRef'''. The '''to''' argument must return 3 cml:scalar nodes for this to work. + + {{{ + <transform process="createVector3" + xpath="." + dictRef="g:coupling.ten" + from="./cml:list/cml:list/cml:scalar[contains(@dictRef,'.a.t') or contains(@dictRef,'.b.t') or contains(@dictRef,'.c.t')]" />

JUMBO-Converters

JensThomas — 2012-03-05 18:59:13

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 327:	Line 327:
	+ * '''createList''' - this will take a list of nodes from the '''xpath''' and, if they are cml modules, it will convert them to cml lists. + + {{{ + <transform process="createList" xpath=".//cml:module[@cmlx:templateRef='multipole']"/> + }}} + +

JUMBO-Converters

JensThomas — 2012-03-05 18:46:31

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 300:	Line 300:
	+ * '''addChild''' - this will create a child element of the nodes specified by the '''xpath'''. The only required argument is '''elementName''', which specifies the type of element to create. Other supported arguments are: '''id''', '''dictRef''' and '''value'''. + + {{{ + <transform process="addChild" + xpath="." + elementName="cml:module" + id="jobList1" + dictRef="cc:jobList"/> + }}} + +
Line 359:	Line 370:
	+ + * '''split''' - this will take the nodes in the '''xpath''' and split them (in places) according to their type - a scalar will be split by whitespace and turned into a list, a 1D array will be split into a cml list, and 2D arrays will be split into a list of separate arrays. + + {{{ + <transform process="split" xpath=".//cml:array[@dictRef='cc:mulliken']"/> + }}} +

JUMBO-Converters

JensThomas — 2012-03-05 17:31:19

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 300:	Line 300:
- * '''addDictRef''' - this will add a '''dictRef''' attribute with the specified value to the nodes defined by the xpath expression.	+ * '''addDictRef''' - this will add a '''dictRef''' attribute with the specified value to the nodes defined by '''xpath'''.
Line 308:	Line 308:
	+ * '''copy''' - this copies the nodes defined by '''xpath''' to the xpath defined by the '''to''' argument, which is relative to the element being copied. e.g. if '''to''' is ".", then the element and its children will be copied to become children of itself. If the element has an '''id''' attribute, this will have the string ".copy" appended to it, if not, an id of "n.copy" will be created, where n is the index of the node in the original xpath. + + {{{ + <transform process="copy" + xpath="(//cml:list[@cmlx:templateRef='l914_excit2'])[1]" + to="."/> + }}} +
Line 312:	Line 320:
	+ xpath="(//cml:list[@cmlx:templateRef='l914_excit2'])[1]"/> + }}} + + * '''debugNodes''' - this just prints out the nodes selected by the xpath and is only useful for developing and debugging the transforms. + + {{{ + <transform process="debugNodes"

JUMBO-Converters

JensThomas — 2012-03-05 16:51:30

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 298:	Line 298:
	+ === Key Transforms === + + * '''addDictRef''' - this will add a '''dictRef''' attribute with the specified value to the nodes defined by the xpath expression. + + {{{ + <transform process="addDictRef" + xpath="//cml:property[cml:module[@cmlx:templateRef='l601.popanal']]" + value="cc:popanal"/> + }}} + + * '''delete''' - this will delete the list of nodes defined by the xpath, along with all of their child nodes. + + {{{ + <transform process="delete" + xpath=".//cml:module[not(cml:array)]"/> + }}} + + * '''move''' - this takes one or more nodes, and moves them into the node defined by the '''to''' argument. The '''to''' argument is an xpath that must just return a single element. + + {{{ + <transform process="move" + to="." + xpath=".//[contains(@dictRef,':serial') or contains(@dictRef,':elementType') or contains(@dictRef,':isotop') or contains(@dictRef,':coupling')]" /> + }}} + + '''moveRelative''' - this is similar to move, but the '''to''' argument is a xpath that is relative to the element being moved, so that if the '''xpath''' returns a list of elements scattered from throughout the document, each will be moved to the '''to''' relative to itself. + + {{{ + <transform process="moveRelative" + xpath="//cml:module[@cmlx:templateRef='l4601.virtual']" + to="parent::/parent::/parent::"/> + }}} + + '''pullup''' - this takes one or more elements defined by an xpath and moves them up out of their current containing element, so that they become children of their current grandparent. The only argument required is the xpath of the nodes to be pulled up. + + {{{ + <transform process="pullup" + xpath=".//cml:module[@cmlx:templateRef='l1.version']/cml:"/> + }}} + + '''pullupSingleton''' - this takes one or more elements defined by an xpath and, if the element only has one child, replaces the element with the child, thereby deleting the original element and "pulling up" the child. + + {{{ + <transform process="pullupSingleton" + xpath=".//cml:list"/> + }}} +

JUMBO-Converters

JensThomas — 2012-03-05 16:08:55

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 269:	Line 269:
	+ == Transforming the raw XML == + + As has been mentioned, the parsing is a two-stage process, consisting of marking up the file with xml and then converting the raw XML to valid CML. In some cases, the raw XML may already be valid CML, but it most cases transforms will need to be applied. + + The transforms can either be applied within the template, after the text has been parsed and marked up, or as an entirely separate step, once the whole file has been parsed. + + Taking the current implementation of the Gaussian log parser as an example, the code for the [https://bitbucket.org/wwmm/jumbo-converters/src/9aa183a356dd/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/java/org/xmlcml/cml/converters/compchem/gaussian/log/GaussianLog2CompchemConverter.java GaussianLog2CompchemConverter], reads in two files: + + * [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml topTemplate.xml] - as mentioned above, this processes all of the templates. + * [https://bitbucket.org/wwmm/jumbo-converters/src/9aa183a356dd/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/gaussian2compchem.xml gaussian2compchem.xml] - this is run subsequent to the file being parsed and runs transforms on the XML. As with the templates, there are a number of transforms that are carried out, the code for which either resides in this file, or files which are '''included''' by it from the templates directory. + + The transformation process relies heavily on the powerful [http://en.wikipedia.org/wiki/XPath XPath] language. A short tutorial on xpath can be found [http://www.w3schools.com/xpath/ here]. + + The transforms are carried out by elements like the following: + + {{{ + <transform process="addAttribute" xpath="./cml:module[@cmlx:templateRef='job']" name="id" value="job" /> + }}} + + In this case, the attribute '''id="job"''' will be added to all cml modules that are direct children of the document, and have the '''templateRef''' "job". + + The transforms have a '''process''' which defines the operation that will be carried out, almost all have an '''xpath''' that is an xpath expression indicating the elements the process will be applied to, and a variable number of arguments, depending on the process being carried out. + + A brief overview of the key transformations follows below, however, for those with a strong constitution, a more comprehensive documentation can be found by examining the code in the file: + + * [https://bitbucket.org/wwmm/jumbo-converters/src/9aa183a356dd/jumbo-converters-core/src/main/java/org/xmlcml/cml/converters/text/TransformElement.java jumbo-converters-core/src/main/java/org/xmlcml/cml/converters/text/TransformElement.java] + + The text from ~ line 156, starting with the comment '''// process values''' lists the processes that are available. +

JUMBO-Converters

JensThomas — 2012-03-05 14:50:48

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 46:	Line 46:
- If a pattern is matched, all subsequent text is "gobbled" and added to the module until the '''endPattern''' is reached, at which point the module is closed, and the next line of text is searched to see if it matches a pattern. - - WHAT HAPPENS IF THERE IS NO endPattern?	+ If a pattern is matched, all subsequent text is "gobbled" and added to the module until the '''endPattern''' is reached, at which point the module is closed, and the next line of text is searched to see if it matches a pattern (if there is no end pattern, the template will swallow all the remaining text in it's enclosing template).
Line 117:	Line 115:
- pattern="\sIsotropic Fermi Contact Couplings." repeat="" - endPattern="\s"> - }}} - - The template starts with the template tag. The id and name define the template and the cml module that this text will end up in. repeat="*" says that this template can appear multiple times in the file.	+ pattern="\sIsotropic Fermi Contact Couplings." + repeat="" + endPattern="\s" + offset="0" + endOffset="1" + > + }}} + + The template starts with the template tag. The name is a descriptive name for the template, and the id serves to define the module that the text will be parsed into, so the module for this template will start with: + + {{{<module cmlx:templateRef="l601.fermi" xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx">}}} + + repeat="*" says that this template can appear multiple times in the file.
Line 129:	Line 135:
- The template will swallow all text until the end pattern is encountered, which is a containing nothing but spaces. - - This examples does not include the "endOffest=N", which indicates what happens to the file pointer within the template i.e. whether it rolls back so that the line that was matched by the template is available for processing within the template or not.	+ The template will swallow all text until the end pattern is encountered, which is a line containing nothing but spaces. + + '''offset''' indicates where the text made available to the records inside the template starts. The default (0) means that the line that is matched by the '''pattern''' is part of the text within the template. An offset of 1, would mean that the line would not be available to the template, but would become part of the parent template. + + '''endOffset''' indicates where the text made available to the records inside the template stops. The default (0) means that the line that is matched by the '''endPattern''' is not part of the text within the template, but is pushed into the parent template. An offset of 1, would include the line in the template, 2 would mean that the line following the endPattern would also be included in this template +
Line 177:	Line 186:
- The first integer will be labelled '''cc:serial''', the first string '''x:elementType''' etc. The "cc" stands for '''Computational Chemistry''', as these are attribute in the Computational Chemisty dictionary, and the "g" stands for an entry in the Gaussian dictionary.	+ The first integer will be labelled '''cc:serial''', the first string '''x:elementType''' etc. The "cc" stands for '''Computational Chemistry''', as these are attribute in the Computational Chemistry dictionary, and the "g" stands for an entry in the Gaussian dictionary.

Tutorials and problems

JensThomas — 2012-03-01 11:30:19

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 91:	Line 91:
- - Now cd into the ''<project-root>/jumbo-converters-compchem'' directory. - - {{{ - cd jumbo-converters-compchem - }}}	+ == Gaussian == + + To use the Gaussian parser, cd into the ''<project-root>/jumbo-converters-compchem/jumbo-converters-compchem-gaussian'' directory. + + {{{ + cd jumbo-converters-compchem/jumbo-converters-compchem-gaussian + }}} + + + and run the converter. With the command: + + {{{ + mvn -e exec:java -Dexec.mainClass="org.xmlcml.cml.converters.compchem.gaussian.log.GaussianLog2CompchemConverter" -Dexec.args="logfile.out" + }}} + + where '''logfile.out''' is the path to a gaussian logfile you wish to convert. This will then create a cml file named '''logfile.cml''' in the directory that the command was executed. + + '''mvn exec:java''' takes care of setting up the Java environment (the classpath) so as to locate all compiled files needed for the execution. For this to happen, the previous command must be executed at the jumbo-converters-compchem-gaussian folder. + + The argument '''-Dexec.mainClass''' states the Java class to execute (in this case, the Gaussian converter), and the '''-Dexec.args''' argument passes the space-separated strings in the argument as arguments to the main class. + + == NWChem ==
Line 110:	Line 126:
- mvn exec:java takes care of setting up the Java environment (the classpath) so as to locate all compiled files needed for the execution. For this to happen, - the previous command must be executed at the jumbo-converters-compchem-nwchem folder. - - The argument '''-Dexec.mainClass''' states the Java class to execute (in this case, the NWChem converter). If you want to pass command-line arguments to the main class, - use the '''-Dexec.args''' argument. - - This will take one of the nwchem output files in '''jumbo-converters-compchem-nwchem/src/test/resources/compchem/nwchem/log/in/''' and create cml file in the directory '''jumbo-converters-compchem-nwchem/test'' folder (for details of what happens, see the file [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-nwchem/src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2CompchemConverter.java src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2CompchemConverter.java]. - - Similarly, to use the Gaussian parser: - - Go into the ''<project-root>/jumbo-converters-compchem/jumbo-converters-compchem-gaussian'' directory. - - {{{ - cd jumbo-converters-compchem/jumbo-converters-compchem-gaussian - }}} - - - and run the converter. With the command: - - {{{ - mvn -e exec:java -Dexec.mainClass=org.xmlcml.cml.converters.compchem.gaussian.log.GaussianLog2CompchemConverter - }}} - - Once changed the source code, recompile (mvn clean install), and run the converter: - - {{{ - mvn -e exec:java -Dexec.mainClass=org.xmlcml.cml.converters.compchem.gaussian.log.GaussianLog2XMLConverter -Dexec.args="./src/test/resources/echeniquep/inputQC/g03_threeSPs/RHF_6-31Gdp_sp_HCO-L-Ala-NH2_hashP.log output.xml" - }}} - - This will parse the input file '''./src/test/resources/echeniquep/inputQC/g03_threeSPs/RHF_6-31Gdp_sp_HCO-L-Ala-NH2_hashP.log''' and it will create the parsed file '''output.xml'''. -	+ This will take one of the nwchem output files in '''jumbo-converters-compchem-nwchem/src/test/resources/compchem/nwchem/log/in/''' and create cml file in the directory '''jumbo-converters-compchem-nwchem/test''' folder (for details of what happens, see the file [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-nwchem/src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2CompchemConverter.java src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2CompchemConverter.java].

Tutorials and problems

JensThomas — 2012-03-01 09:40:22

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 107:	Line 107:
- mvn -e exec:java -Dexec.mainClass=org.xmlcml.cml.converters.compchem.nwchem.log.NWChemLog2XMLConverter	+ mvn -e exec:java -Dexec.mainClass=org.xmlcml.cml.converters.compchem.nwchem.log.NWChemLogCompchemConverter
Line 116:	Line 116:
- Similarly, for the Gaussian parser: - - Go into the ''<project-root>/jumbo-converters-compchem'' directory. - - {{{ - cd jumbo-converters-compchem - }}} - - To use the Gaussian converter, go into the jumbo-converters-compchem-gaussian folder: - - {{{ - cd jumbo-converters-compchem-gaussian - }}} - - and run the converter. This currently does not work. Until it is fixed, you have to change the source code at ./src/main/java/org/xmlcml/cml/converters/compchem/gaussian/log/GaussianLog2XMLConverter.java - to read the name of the input file and the output file from the command-line. Comment out the main method, and write the following instead: - - {{{ - public static void main(String[] args) throws IOException { - CompchemTemplateConverter converter = new GaussianLog2XMLConverter(); - - File in = new File(args[0]); - File out = new File(args[1]); - converter.convert(in, out); - }	+ This will take one of the nwchem output files in '''jumbo-converters-compchem-nwchem/src/test/resources/compchem/nwchem/log/in/''' and create cml file in the directory '''jumbo-converters-compchem-nwchem/test'' folder (for details of what happens, see the file [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-nwchem/src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2CompchemConverter.java src/main/java/org/xmlcml/cml/converters/compchem/nwchem/log/NWChemLog2CompchemConverter.java]. + + Similarly, to use the Gaussian parser: + + Go into the ''<project-root>/jumbo-converters-compchem/jumbo-converters-compchem-gaussian'' directory. + + {{{ + cd jumbo-converters-compchem/jumbo-converters-compchem-gaussian + }}} + + + and run the converter. With the command: + + {{{ + mvn -e exec:java -Dexec.mainClass=org.xmlcml.cml.converters.compchem.gaussian.log.GaussianLog2CompchemConverter

JUMBO-Converters

JensThomas — 2012-02-29 18:33:19

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 259:	Line 259:
	+ + = JUMBO-Converters filesystem structure = + + (I guess some of this is standard for Maven projects but my ignorance forces me to document everything. The bright side is that other newbies like me will feel happy!) + + Under + + {{{ + jumbo-converters/ + }}} + + we have + + {{{ + jumbo-converters/ + jumbo-converters-compchem/ + }}} + + which is where most of the important stuff for Quixote lies if we do not want to get into too many details. + + The two most important subfolders of this are + + {{{ + jumbo-converters/ + jumbo-converters-compchem/ + src/ + target/ + }}} + + The second one is where the final compiled Java classes are located ('''any more stuff?''') and we will not care about it for the moment. The {{{src}}} subfolder, as its name indicates, contains the source code associated to the compchem part of JUMBOconverters (i.e., the one most related to the Quixote project). Inside the {{{src}}} subfolder, we have the following chain of folders, at the bottom of which all Java source code is located: + + {{{ + jumbo-converters/ + jumbo-converters-compchem/ + src/ + main/ + java/ + org/ + xmlcml/ + cml/ + converters/ + }}} + + Inside {{{converters}}}, we have two main subfolders: + + {{{ + jumbo-converters/ + jumbo-converters-compchem/ + src/ + main/ + java/ + org/ + xmlcml/ + cml/ + converters/ + compchem/ + marker/ + }}} + + The most specific compchem code is in {{{compchem}}} (as you might have guessed!) ordered by the name of the compchem package ({{{gamessus}}}, {{{gaussian}}}, {{{nwchem}}}, etc.), and {{{marker}}} contains more general source code to support the former. + + If you are Java-savy, you might want to check these folders and read the code, but one of the great things about the declarative approach that [http://www-pmr.ch.cam.ac.uk/wiki/Main_Page PMR] has created into JUMBOconverters and we describe in this page is that you don't need to! If you know [http://en.wikipedia.org/wiki/Regular_expression regular expressions] and some very basic [http://www.w3.org/TR/xpath/ XPath] (both of which you could even infer from already made examples), that should be sufficient. + + One important thing to remember though, even if you don't plan to read the Java source code, is that the above folders structure translates into the names of the classes that do all the magic stuff, so, if you want to call these classes in the command line, like in + + {{{ + mvn -e exec:java -Dexec.mainClass=org.xmlcml.cml.converters.compchem.nwchem.log.NWChemLog2XMLConverter -Dexec.args="./src/test/resources/compchem/nwchem/log/in/test1.out ./test.cml" + }}} + + you need to have this structure in mind. + + The declarative bits of the parsing infrastructure (i.e., what you, parsers developer, will have to check, understand and probably make a version for your favourite compchem code) are inside a similar folder tree under {{{src/main}}}: + + {{{ + jumbo-converters/ + jumbo-converters-compchem/ + src/ + main/ + resources/ + org/ + xmlcml/ + cml/ + converters/ + compchem/ + amber/ + gamessus/ + gaussian/ + nwchem/ + ... + }}} + + Inside each code folder, one can find subfolders for the different types of file, and inside each one of them a {{{templates}}} subfolder, e.g., + + {{{ + jumbo-converters/ + jumbo-converters-compchem/ + src/ + main/ + resources/ + org/ + xmlcml/ + cml/ + converters/ + compchem/ + gaussian/ + in/ + templates/ + log/ + templates/ + ... + }}} + + In the rest of the sections and in some of the tutorials, we explain in detail how the different bits of declarative parsing are related and how everything works, but let us mention at this point that, at the filetype folders (i.e., at {{{in}}} or {{{log}}}) the top level parsing template list file {{{templateList.xml}}} can be found, while each one of the smaller templates included in this list are located in {{{templates}}}. + + Now, branching out at the same level as {{{main}}}, still inside {{{src}}}, we have a {{{test}}} subfolder, which contains, on the one hand (under {{{java}}}), the Java source code for performing automatic tests, and, on the other hand (under {{{resources}}}), a number of example files produced by the compchem codes that Quixote wants to tackle. The scheme of the folder tree is as follows: + + {{{ + jumbo-converters/ + jumbo-converters-compchem/ + src/ + test/ + java/ + org/ + xmlcml/ + cml/ + converters/ + compchem/ + amber/ + gamessus/ + gaussian/ + nwchem/ + ... + resources/ + compchem/ + amber/ + gamessus/ + gaussian/ + in/ + log/ + ... + nwchem/ + ... + }}} + + A '''general scheme''' summarizing all the details commented above is the following: + + + {{{ + jumbo-converters/ * Main JUMBOconverters folder + jumbo-converters-compchem/ * Compchem JUMBOconverters + src/ * Source code and test files + main/ * Source code for the parsing machinery + java/ * Java source code + org/ + xmlcml/ + cml/ + converters/ + compchem/ + marker/ + resources/ * Declarative parsing source code + org/ + xmlcml/ + cml/ + converters/ + compchem/ + amber/ + gamessus/ + gaussian/ + in/ * Top level parsing directives + templates/ * Subparsers templates + log/ + templates/ + ... + nwchem/ + ... + test/ * Source code for the automatic testing + java/ + org/ + xmlcml/ + cml/ + converters/ + compchem/ + amber/ + gamessus/ + gaussian/ + nwchem/ + ... + resources/ * Example test files + compchem/ + amber/ + gamessus/ + gaussian/ + in/ + log/ + ... + nwchem/ + ... + target/ *** Compiled classes + }}}

JUMBO-Converters

JensThomas — 2012-02-29 18:31:38

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 231:	Line 231:
- The individual test can be run from within Eclipse, but from the command-line, it only appears possible to run the entire TemplateTests, using the following command, whilst sat in the '''jumbo-converters/jumbo-converters-compchem/jumbo-converters-compchem-gaussian''' directory:	+ The individual test can be run from within Eclipse, but from the command-line, it only appears possible to run the entire TemplateTests (see not below), using the following command, whilst sat in the '''jumbo-converters/jumbo-converters-compchem/jumbo-converters-compchem-gaussian''' directory:
Line 239:	Line 239:
- {{{ - ==============template===================	+ {{{==============template===================
Line 242:	Line 241:
- ==========XMLDIFF reference=========	+ XMLDIFF reference
Line 251:	Line 250:
- The chunk of test after the ''''------------test---------------------'''' line, and excluding the '''<?xml version="1.0" encoding="UTF-8"?>''' line is the output of the test. This should be checked, and if correct, placed in the '''<comment class="example.output" id="l601.fermi">''' tag in the template. Re-running the test should then lead to a successful result.	+ The chunk of test after the '''------------test---------------------''' line, and excluding the '''<?xml version="1.0" encoding="UTF-8"?>''' line is the output of the test. This should be checked, and if correct, placed in the '''<comment class="example.output" id="l601.fermi">''' tag in the template. Re-running the test should then lead to a successful result.

JUMBO-Converters

JensThomas — 2012-02-29 18:26:15

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 234:	Line 234:
- mvn -Dtest="log.TemplateTes" test - }}}	+ mvn -Dtest="log.TemplateTest" test + }}} + + The first time this is run, it will fail. However, it will print out the output of running the test, and something like the following: + + {{{ + ==============template=================== + Error: template expected:<3> but was:<4> + ==========XMLDIFF reference========= + + ------------test--------------------- + <?xml version="1.0" encoding="UTF-8"?> + <module cmlx:templateRef="l601.fermi" xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx"> + <list cmlx:lineCount="3" cmlx:templateRef="fermi.atom"> + <array dataType="xsd:integer" dictRef="cc:serial" size="3">1 2 13</array> + }}} + + The chunk of test after the ''''------------test---------------------'''' line, and excluding the '''<?xml version="1.0" encoding="UTF-8"?>''' line is the output of the test. This should be checked, and if correct, placed in the '''<comment class="example.output" id="l601.fermi">''' tag in the template. Re-running the test should then lead to a successful result.
Line 244:	Line 260:
- - - = JUMBO-Converters filesystem structure = - - (I guess some of this is standard for Maven projects but my ignorance forces me to document everything. The bright side is that other newbies like me will feel happy!) - - Under - - {{{ - jumbo-converters/ - }}} - - we have - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - }}} - - which is where most of the important stuff for Quixote lies if we do not want to get into too many details. - - The two most important subfolders of this are - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - src/ - target/ - }}} - - The second one is where the final compiled Java classes are located ('''any more stuff?''') and we will not care about it for the moment. The {{{src}}} subfolder, as its name indicates, contains the source code associated to the compchem part of JUMBOconverters (i.e., the one most related to the Quixote project). Inside the {{{src}}} subfolder, we have the following chain of folders, at the bottom of which all Java source code is located: - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - src/ - main/ - java/ - org/ - xmlcml/ - cml/ - converters/ - }}} - - Inside {{{converters}}}, we have two main subfolders: - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - src/ - main/ - java/ - org/ - xmlcml/ - cml/ - converters/ - compchem/ - marker/ - }}} - - The most specific compchem code is in {{{compchem}}} (as you might have guessed!) ordered by the name of the compchem package ({{{gamessus}}}, {{{gaussian}}}, {{{nwchem}}}, etc.), and {{{marker}}} contains more general source code to support the former. - - If you are Java-savy, you might want to check these folders and read the code, but one of the great things about the declarative approach that [http://www-pmr.ch.cam.ac.uk/wiki/Main_Page PMR] has created into JUMBOconverters and we describe in this page is that you don't need to! If you know [http://en.wikipedia.org/wiki/Regular_expression regular expressions] and some very basic [http://www.w3.org/TR/xpath/ XPath] (both of which you could even infer from already made examples), that should be sufficient. - - One important thing to remember though, even if you don't plan to read the Java source code, is that the above folders structure translates into the names of the classes that do all the magic stuff, so, if you want to call these classes in the command line, like in - - {{{ - mvn -e exec:java -Dexec.mainClass=org.xmlcml.cml.converters.compchem.nwchem.log.NWChemLog2XMLConverter -Dexec.args="./src/test/resources/compchem/nwchem/log/in/test1.out ./test.cml" - }}} - - you need to have this structure in mind. - - The declarative bits of the parsing infrastructure (i.e., what you, parsers developer, will have to check, understand and probably make a version for your favourite compchem code) are inside a similar folder tree under {{{src/main}}}: - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - src/ - main/ - resources/ - org/ - xmlcml/ - cml/ - converters/ - compchem/ - amber/ - gamessus/ - gaussian/ - nwchem/ - ... - }}} - - Inside each code folder, one can find subfolders for the different types of file, and inside each one of them a {{{templates}}} subfolder, e.g., - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - src/ - main/ - resources/ - org/ - xmlcml/ - cml/ - converters/ - compchem/ - gaussian/ - in/ - templates/ - log/ - templates/ - ... - }}} - - In the rest of the sections and in some of the tutorials, we explain in detail how the different bits of declarative parsing are related and how everything works, but let us mention at this point that, at the filetype folders (i.e., at {{{in}}} or {{{log}}}) the top level parsing template list file {{{templateList.xml}}} can be found, while each one of the smaller templates included in this list are located in {{{templates}}}. - - Now, branching out at the same level as {{{main}}}, still inside {{{src}}}, we have a {{{test}}} subfolder, which contains, on the one hand (under {{{java}}}), the Java source code for performing automatic tests, and, on the other hand (under {{{resources}}}), a number of example files produced by the compchem codes that Quixote wants to tackle. The scheme of the folder tree is as follows: - - {{{ - jumbo-converters/ - jumbo-converters-compchem/ - src/ - test/ - java/ - org/ - xmlcml/ - cml/ - converters/ - compchem/ - amber/ - gamessus/ - gaussian/ - nwchem/ - ... - resources/ - compchem/ - amber/ - gamessus/ - gaussian/ - in/ - log/ - ... - nwchem/ - ... - }}} - - A '''general scheme''' summarizing all the details commented above is the following: - - - {{{ - jumbo-converters/ * Main JUMBOconverters folder - jumbo-converters-compchem/ * Compchem JUMBOconverters - src/ * Source code and test files - main/ * Source code for the parsing machinery - java/ * Java source code - org/ - xmlcml/ - cml/ - converters/ - compchem/ - marker/ - resources/ * Declarative parsing source code - org/ - xmlcml/ - cml/ - converters/ - compchem/ - amber/ - gamessus/ - gaussian/ - in/ * Top level parsing directives - templates/ * Subparsers templates - log/ - templates/ - ... - nwchem/ - ... - test/ * Source code for the automatic testing - java/ - org/ - xmlcml/ - cml/ - converters/ - compchem/ - amber/ - gamessus/ - gaussian/ - nwchem/ - ... - resources/ * Example test files - compchem/ - amber/ - gamessus/ - gaussian/ - in/ - log/ - ... - nwchem/ - ... - target/ *** Compiled classes - }}}

JUMBO-Converters

JensThomas — 2012-02-29 18:19:23

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 231:	Line 231:
	+ The individual test can be run from within Eclipse, but from the command-line, it only appears possible to run the entire TemplateTests, using the following command, whilst sat in the '''jumbo-converters/jumbo-converters-compchem/jumbo-converters-compchem-gaussian''' directory: + + {{{ + mvn -Dtest="log.TemplateTes" test + }}} + + '''Note:'''' The discussion at [http://stackoverflow.com/questions/1873995/run-a-single-test-method-with-maven stackoverflow] and [http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html maven documentation] suggests that the following syntax should work: + + {{{ + mvn -Dtest="log.TemplateTest#testl601Fermi" test + }}} + + But this appears not to be the case. Are we using the junit < 4.7?

JUMBO-Converters

JensThomas — 2012-02-29 16:45:33

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 2:	Line 2:
-
Line 214:	Line 213:
	+ The output has been parsed into a module with the templateRef "l601.fermi", which is the id of the template. We then have the 5 arrays (I,A,I,F,F,F,F) parsed by the "fermi.atom" record as a list, followed by that for the "fermi.spindipole" record etc. + + == Testing and developing templates == + + The templates contain their own internal testing framework, in the form of two of the comment blocks within them. + + The comment block with the class "example.input" should contain a small representative chunk of text that the parsers can be tested with. + + For the Gaussian logfile templates we have been dealing with here, the code that runs these tests lives in the file: + + [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/test/java/org/xmlcml/cml/converters/compchem/gaussian/log/TemplateTest.java jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/test/java/org/xmlcml/cml/converters/compchem/gaussian/log/TemplateTest.java] + + To test and develop an individual template (we will continue using the l601.fermi template as an example), the following line needs to be added to the TemplateTest.java file. + + {{{ + @Test public void testl601Fermi() {runTemplateTest("l601/", "l601.fermi");} + }}} + +

JUMBO-Converters

JensThomas — 2012-02-29 16:27:50

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 37:	Line 37:
-
Line 59:	Line 58:
-
Line 73:	Line 71:
- ... - ******************************************* - Gaussian 03: x86-Linux-G03RevB.04 2-Jun-2003 - 20-Nov-2006 - ******************************************* - -------------------------- - #N B3LYP/6-31G(d) OPT FREQ - -------------------------- - 1/14=-1,18=20,26=3,38=1/1,3; - ... - ... - Job cpu time: 0 days 0 hours 0 minutes 16.2 seconds. - File lengths (MBytes): RWF= 12 Int= 0 D2E= 0 Chk= 7 Scr= 1 - Normal termination of Gaussian 03 at Mon Nov 20 14:40:23 2006. - Link1: Proceeding to internal job step number 2. - ------------------------------------------------------------------ - #N Geom=AllCheck Guess=Read SCRF=Check GenChk RB3LYP/6-31G(d) Freq - ------------------------------------------------------------------ - 1/10=4,29=7,30=1,38=1,40=1,46=1/1,3; - ...	+ ...
Line 96:	Line 75:
-
Line 104:	Line 82:
- <xi:include href="l1/l1.legal.xml"/> - <xi:include href="l1/l1.citation.xml"/> - <xi:include href="l1/l1.end.xml"/> - <xi:include href="l716.xml"/>
Line 110:	Line 84:
- <!-- Many more templates -->	+ + <!-- Many more templates --> +
Line 114:	Line 90:
-
Line 121:	Line 96:
- There is then a comment containing some example text to show what this template parses - as will be explained later, this is also used for testing the template. - - There is then a template list, which is the container that holds the templates. The first templatelist only holds one template, which is the template that parses the various links (modules) within the Gaussian program. This template itself then contains a template list, which includes the other templates that will process the text found by the parent template.	+ There is then a comment containing some example text to show what this template parses. + + There is then a template list, which is the container that holds the templates. This first templateList only holds one template, which is the template that parses the various links (modules) within the Gaussian program. This template then contains a further template list, which includes the other templates that will process the text found by the parent template.
Line 129:	Line 104:
-
Line 140:	Line 114:
- Using '''l601.popanal.xml''' as an example, we will go through the template and describe what each bit does. - - {{{ - <template id="l301.basis" name="basis" repeat="" - pattern="\sStandard basis." endPattern="\sNAtoms=." - endOffset="1" - > - }}} - - The template starts with the template tag. The id and name define the template, repeat="" says that it can occur multiple times in the file. The '''pattern''' is a regular expression stating that this template will start when it matches a line that starts with spaces followed by the phrase "Standard basis" followed by any text. - - The template will swallow all text until the end pattern is encountered, which is a line with spaces followed by '''NAtoms=''' followed by some arbitrary text. - - The endOffest indicates what happens to the file pointer within the template i.e. whether it rolls back so that the line that was matched by the template is available for processing within the template or not. -	+ Using '''l601.fermi.xml''' as an example, we will go through the template and describe what each bit does. + + {{{ + <template id="l601.fermi" name="Isotropic Fermi Contact Couplings" + pattern="\sIsotropic Fermi Contact Couplings." repeat="" + endPattern="\s"> + }}} + + The template starts with the template tag. The id and name define the template and the cml module that this text will end up in. repeat="*" says that this template can appear multiple times in the file. + + The '''pattern''' is a regular expression stating that this template will start when it matches the line: + + {{{ + Isotropic Fermi Contact Couplings + }}} + + The template will swallow all text until the end pattern is encountered, which is a containing nothing but spaces. + + This examples does not include the "endOffest=N", which indicates what happens to the file pointer within the template i.e. whether it rolls back so that the line that was matched by the template is available for processing within the template or not.
Line 159:	Line 137:
- <comment class="example.input" id="l301.basis.09"> - Standard basis: 3-21G (6D, 7F) - Ernie: Thresh= 0.10000D-02 Tol= 0.10000D-05 Strict=F. - There are 7 symmetry adapted basis functions of AG symmetry. - There are 0 symmetry adapted basis functions of B1G symmetry. - There are 2 symmetry adapted basis functions of B2G symmetry. - There are 4 symmetry adapted basis functions of B3G symmetry. - There are 0 symmetry adapted basis functions of AU symmetry. - There are 7 symmetry adapted basis functions of B1U symmetry. - There are 4 symmetry adapted basis functions of B2U symmetry. - There are 2 symmetry adapted basis functions of B3U symmetry. - Integral buffers will be 131072 words long. - Raffenetti 1 integral format. - Two-electron integral symmetry is turned on. - 26 basis functions, 42 primitive gaussians, 26 cartesian basis functions - 8 alpha electrons 8 beta electrons - nuclear repulsion energy 33.7515964544 Hartrees. - IExCor= 0 DFT=F Ex=HF Corr=None ExCW=0 ScaHFX= 1.000000 - ScaDFX= 1.000000 1.000000 1.000000 1.000000 ScalE2= 1.000000 1.000000 - IRadAn= 0 IRanWt= -1 IRanGd= 0 ICorTp=0 - NAtoms= 6 NActive= 6 NUniq= 2 SFac= 4.00D+00 NAtFMM= 50 NAOKFM=F Big=F	+ <comment class="example.input" id="l601.fermi"> + Isotropic Fermi Contact Couplings + Atom a.u. MegaHertz Gauss 10(-4) cm-1 + 1 C(13) 0.02539 28.54777 10.18656 9.52251 + 2 C(13) 0.00582 6.54434 2.33518 2.18296 + 13 Cl(35) 0.05688 24.94015 8.89927 8.31914 + -------------------------------------------------------- + Center ---- Spin Dipole Couplings ---- + 3XX-RR 3YY-RR 3ZZ-RR + -------------------------------------------------------- + 1 Atom 0.005300 -0.061839 0.056540 + 2 Atom -0.039723 -0.068059 0.107782 + 13 Atom 0.621221 -2.038530 1.417309 + -------------------------------------------------------- + XY XZ YZ + -------------------------------------------------------- + 1 Atom 0.000010 0.095387 0.000013 + 2 Atom 0.005157 0.081893 0.006262 + 13 Atom 0.000344 3.043747 0.000390 + -------------------------------------------------------- +
Line 188:	Line 166:
- <record id="basis">\sStandard basis: {A,cc:basis} {X,cc:diffuse}</record> - }}} - - The first line reads a record from the file. '''repeat''' is omitted so defaults to one, so this reads a single line that starts with spaces and the phrase "Standard basis:" The terms that follow in the brackets determine what will be marked up into the xml. - - '''{A,cc:basis}''' this will read the data as a string (A) and put it in a term with the '''dictRef''' cc:basis. The cc prefix is the namespace of the computational chemistry dictionary, so this term is a '''basis''' as that term is defined in the computational chemistry dictionary. - * '''{X,cc:diffuse}''' the X is actually used to discard data, so this will swallow the text that comes after the character data. The dictRef is irrelevant in this case and could be anything. - - Other values for data that can be read are: - * F - float - * I - integer - - {{{ - <templateList id="ernie"> - <template pattern="\sErnie." endPattern="." id="ernie"> - <record id="ernie" repeat="">\sErnie: Thresh={E,g:thresh}\s Tol={E,g:tol}\sStrict={A,g:strict}\.\s</record> - </template> - </templateList> - <templateList id="symadnucl"> - <template pattern="\sThere are." endPattern="\snuclear repulsion." id="symaddnuc" endOffset="1"> - <record id="symmadapt" repeat="" makeArray="true">\s There are{I,cc:adapted} symmetry adapted basis functions of{A,cc:symm}symmetry\.\s</record> - <record id="buffer">\sIntegral buffers will be {I,g:buffer}\swords long\.\s</record> - <record id="raff">\s{A,g:raffenetti}\s{I,g:raff}\sintegral format\.\s</record> - <record id="twoe">\s{X,g:twoe} integral symmetry is turned on\.\s</record> - <record id="basiscount">\s{I,cc:basiscount}basis functions,{I,g:primbasis}primitive gaussians,{I,cc:cartesianbasis}cartesian basis functions\s</record> - <record id="alphabeta">\s{I,cc:alphae}alpha electrons\s{I,cc:betae}beta electrons\s</record> - <record id="nucrep">\snuclear repulsion energy\s{F,cc:nucrepener}Hartrees\.\s</record> - </template> - </templateList> - <templateList id="natoms"> - <template pattern="\sNAtoms=." repeat="" endPattern="." id="natoms"> - <record id="natoms" repeat="">\sNAtoms={I,cc:natoms}\sNActive={I,cc:nactiveatoms}\sNUniq={I,cc:uniqatoms}\sSFac={E,g:sfac}\sNAtFMM={I,g:natfmm}.\sBig={A,g:big}\s</record> - </template> - </templateList> - <templateList id="misc"> - <template pattern="\s((IExCor)\|(ScaDFX)\|(IRadAn))." repeat="" endPattern="." id="misc"> - <record id="misc">\s{1_20A,g:misc}\s</record> - </template> - </templateList> - }}} - - This starts reading into a module with a templateRef of '''l202''' (from the id) when it encounters the regular expression ".Enter.l202." and stops when it encounters the regular expression ".Leave Link +202 ." - - The '''l202''' module is then processed by matching each line against the patterns in each of the included templates, which in this case is just '''coord.input.temp.xml'''. - - If we look at '''coord.input.temp.xml''': - {{{ - <template id="gau:coord" name="Coordinates" - pattern=".Input orientation:." > - <comment class="example"> - Input orientation: - --------------------------------------------------------------------- - Center Atomic Atomic Coordinates (Angstroms) - Number Number Type X Y Z - --------------------------------------------------------------------- - 1 1 0 -3.004390 0.907942 -0.042084 - 2 6 0 -2.114020 0.295280 -0.082179 - 3 7 0 -0.993069 1.029184 -0.183798 - --------------------------------------------------------------------- - </comment> - <record repeatCount="5" formatType="REGEX">(.)</record> - <record formatType="REGEX" id="charges" makeArray="true" - repeatCount="">{I,c:ser} {I,c:atnum} {I,gau:type} {F,c:x3} {F,c:y3} {F,c:z3}</record> - <record formatType="REGEX">(.)</record> - </template> - }}} - - This matches a line with the phrase "Input Orientation:" on it and as there is no endPattern, it carries on reading until the end of the module (CORRECT?). - - Each record in the template is then processed in turn. - - The first record reads five lines, and as there is no id, the lines are discarded. - - The next record reads as many lines (repeatCount="") as match the expression contained within the record into an entity called "charges". The expression that is expected to be matched is 3 integers followed by the floats. The first integer will be labelled''' c:ser''', the second '''c:atnum''' etc. The "c" stands for '''cml''', as these are attribute in the CML dictionary, and the "g" stands for an entry in the Gaussian dictionary.	+ <record repeat="2"/> + <record id="fermi.atom" repeat="" makeArray="true">\s{I,cc:serial}{A,x:elementType}${I,x:isotopeNumber}$\s{F,cc:coupling,u:au}\s{F,cc:coupling,u:mhz}\s{F,cc:coupling,u:gauss}\s{F,cc:coupling,u:ten4cm-1}\s</record> + <record repeat="4"/> + <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xx}{F,g:spindipole.yy}{F,g:spindipole.zz}</record> + <record repeat="3"/> + <record id="fermi.spindipole" repeat="" makeArray="true">\s{I,cc:serial}\sAtom\s{F,g:spindipole.xy}{F,g:spindipole.xz}{F,g:spindipole.yz}</record> + <record repeat="1"/>}}} + + The first line just reads 2 lines from the file. As there is no id on the record, the lines are discarded. + + The next record reads as many lines (repeat="") as match the expression contained within the record into an entity called "fermi.atom". The expression that is expected to be matched is an integer (I), followed by a character string (A), followed by another integer (I) and then four floats (F). + + The first integer will be labelled '''cc:serial''', the first string '''x:elementType''' etc. The "cc" stands for '''Computational Chemistry''', as these are attribute in the Computational Chemisty dictionary, and the "g" stands for an entry in the Gaussian dictionary.
Line 265:	Line 182:
- An example result of this, is the following: - - {{{ - <module lineCount="31" templateRef="gau:coord"> - − - <list lineCount="7" templateRef="charges"> - <array dataType="xsd:integer" dictRef="c:ser" size="7" delimiter="">1 2 3 4 5 6 7</array> - <array dataType="xsd:integer" dictRef="c:atnum" size="7" delimiter="">6 1 1 1 6 1 1</array> - <array dataType="xsd:integer" dictRef="gau:type" size="7" delimiter="">0 0 0 0 0 0 0</array> - − - <array dataType="xsd:double" dictRef="c:x3" size="7" delimiter=""> - 0.91615 1.82641 1.00333 0.95414 -0.31877 -0.7203 -0.77165 - </array> - − - <array dataType="xsd:double" dictRef="c:y3" size="7" delimiter=""> - 0.07243 -0.22339 1.14742 -0.44492 -0.26078 0.42201 -1.24177 - </array> - − - <array dataType="xsd:double" dictRef="c:z3" size="7" delimiter=""> - 4.99221 5.53974 4.80723 4.02871 5.75176 6.49042 5.67704 - </array> - </list> - }}} -	+ Once this pattern stops being matched, the parser will skip 4 lines (<record repeat="4"/>) and then the next record will process a matching block of Integers and Floats into arrays as described earlier. + + Finally we get to the last block in the template, which is a comment if class "example.output" showing what this template processes when fed the text that was in the "example.input" comment. + + {{{ + <comment class="example.output" id="l601.fermi"> + <module cmlx:templateRef="l601.fermi" xmlns="http://www.xml-cml.org/schema" xmlns:cmlx="http://www.xml-cml.org/schema/cmlx"> + <list cmlx:lineCount="3" cmlx:templateRef="fermi.atom"> + <array dataType="xsd:integer" dictRef="cc:serial" size="3">1 2 13</array> + <array dataType="xsd:string" dictRef="x:elementType" size="3">C C Cl</array> + <array dataType="xsd:integer" dictRef="x:isotopeNumber" size="3">13 13 35</array> + <array dataType="xsd:double" dictRef="cc:coupling" size="3">0.02539 0.00582 0.05688</array> + <array dataType="xsd:double" dictRef="cc:coupling" size="3">28.54777 6.54434 24.94015</array> + <array dataType="xsd:double" dictRef="cc:coupling" size="3">10.18656 2.33518 8.89927</array> + <array dataType="xsd:double" dictRef="cc:coupling" size="3">9.52251 2.18296 8.31914</array> + </list> + <list cmlx:lineCount="3" cmlx:templateRef="fermi.spindipole"> + <array dataType="xsd:integer" dictRef="cc:serial" size="3">1 2 13</array> + <array dataType="xsd:double" dictRef="g:spindipole.xx" size="3">0.0053 -0.039723 0.621221</array> + <array dataType="xsd:double" dictRef="g:spindipole.yy" size="3">-0.061839 -0.068059 -2.03853</array> + <array dataType="xsd:double" dictRef="g:spindipole.zz" size="3">0.05654 0.107782 1.417309</array> + </list> + <list cmlx:lineCount="3" cmlx:templateRef="fermi.spindipole"> + <array dataType="xsd:integer" dictRef="cc:serial" size="3">1 2 13</array> + <array dataType="xsd:double" dictRef="g:spindipole.xy" size="3">1.0E-5 0.005157 3.44E-4</array> + <array dataType="xsd:double" dictRef="g:spindipole.xz" size="3">0.095387 0.081893 3.043747</array> + <array dataType="xsd:double" dictRef="g:spindipole.yz" size="3">1.3E-5 0.006262 3.9E-4</array> + </list> + </module> + </comment> + }}}

Declarative parsing syntax

JensThomas — 2012-02-29 15:58:35

Differences for Declarative parsing syntax

Deletions are marked with - .	Additions are marked with +.
Line 10:	Line 10:
- {{{ - jumbo-converters/jumbo-converters-compchem/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log - }}}	+ [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml]
Line 36:	Line 34:
- For example, in the case of the list in the previous section, the first item in the list includes the file {{{l1.temp.xml}}}, which is located in	+ For example, in the case of the list in the previous section, the list includes the file {{{l601.popanal.xml}}}, which is located in
Line 38:	Line 36:
- {{{ - jumbo-converters/jumbo-converters-compchem/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates - }}}	+ [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601 jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601/]
Line 42:	Line 38:
- together with all template files except for {{{topTemplate.xml}}}-	+ together with several other template files related to the l601 link.
Line 47:	Line 43:
- <template id="l1" repeatCount="" pattern=".Enter.l1[^\d]." - endPattern=".Leave Link +1[^\d]."> - <comment>	+ <template id="l601.popanal" name="Population analysis using the SCF density" + repeat="" + pattern="\s\+\s$\s$\sPopulation analysis using the SCF density." + endPattern="\sN\-N\=." endOffset="1" + xmlns:xi="http://www.w3.org/2001/XInclude" + > + <comment class="example.input" id="l601.popanal">
Line 51:	Line 51:
- </comment> - <templateList id='l1_list' xmlns:xi="http://www.w3.org/2001/XInclude"> - <xi:include href="l1.programversion.temp.xml"/>	+ </comment> + + <record repeat="5"/> + <record repeat="2"/> + <templateList> + <xi:include href="l601.condensed.xml"/> + <xi:include href="../l401/l4601.virtual.xml"/> + ...
Line 55:	Line 60:
- </template>	+ + <comment class="example.output" id="l601.popanal"> + ... + </comment> + + </template>
Line 61:	Line 71:
- pattern=".Enter.l1[^\d].*"	+ pattern"\s\+\s$\s$\sPopulation analysis using the SCF density."
Line 64:	Line 74:
- which matches the line in the logfile that reads	+ which matches the lines in the logfile that read:
Line 67:	Line 77:
- Entering Link 1 = /apps/apps64/g03/l1.exe PID= 6730.	+ ********************************************************************** + + Population analysis using the SCF density.
Line 75:	Line 87:
- endPattern=".Leave Link +1[^\d]."	+ endPattern="\sN\-N\=.*"
Line 81:	Line 93:
- Leave Link 1 at Thu Feb 17 17:30:26 2011, MaxMem= 117964800 cpu: 0.2	+ N-N= 8.247004252289D+02 E-N=-3.066552593713D+03 KE= 6.044489055531D+02
Line 87:	Line 99:
- id="l1"	+ l601.popanal
Line 93:	Line 105:
- <module lineCount="87" templateRef="l1"> Entering Link 1 = /apps/apps64/g03/l1.exe PID= 6730. - ... - </module> - Leave Link 1 at Thu Feb 17 17:30:26 2011, MaxMem= 117964800 cpu: 0.2 - ...	+ + <module cmlx:templateRef="l601.popanal" xmlns="http... + + </module
Line 108:	Line 119:
- When no further match is found, the parser will proceed to the next item in the list, in this case {{{l101.temp.xml}}}, which will do the same process but '''without having access''' to the already captured modules.	+ When no further match is found, the parser will proceed to the next item in the list, in this case {{{l601/l601.polariz.xml}}}, which will do the same process but '''without having access''' to the already captured modules.

JUMBO-Converters

JensThomas — 2012-02-29 11:49:26

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 193:	Line 193:
- * ''{A,cc:basis}''' this will read the data as a string (A) and put it in a term with the '''dictRef'' cc:basis. The cc prefix is the namespace of the computational chemistry dictionary, so this term is a '''basis''' as that term is defined in the computational chemistry dictionary.	+ * '''{A,cc:basis}''' this will read the data as a string (A) and put it in a term with the '''dictRef''' cc:basis. The cc prefix is the namespace of the computational chemistry dictionary, so this term is a '''basis''' as that term is defined in the computational chemistry dictionary.

JUMBO-Converters

JensThomas — 2012-02-29 11:47:06

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 109:	Line 109:
- <xi:include href="l601/l601.popanal.xml"/> - <xi:include href="l601/l601.polariz.xml"/>	+ <xi:include href="l301.basis.xml"/>
Line 124:	Line 123:
- There is then a template list, which is the container that holds the templates. The first templatelist only holds one template, which is the template that parses the various links (modules) within the Gaussian program. This template then contains a template list, which includes the other templates that will process the text found by the parent template.	+ There is then a template list, which is the container that holds the templates. The first templatelist only holds one template, which is the template that parses the various links (modules) within the Gaussian program. This template itself then contains a template list, which includes the other templates that will process the text found by the parent template.
Line 128:	Line 127:
- [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601/l601.popanal.xm jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601/l601.popanal.xm	+ [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l301.basis.xml jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l301.basis.xml]
Line 141:	Line 140:
- Using '''l601.popanal.xml'' as an example: - - {{{ - <template id="l202" repeatCount="" pattern=".Enter.l202." - endPattern=".Leave Link +202."> - <comment> - ... COMMENT EXPLAINING WHAT THIS SECTION IS ABOUT ... - </comment> - <templateList id='id2' xmlns:xi="http://www.w3.org/2001/XInclude"> - <xi:include href="coord.input.temp.xml"/> - </templateList> - </template>	+ Using '''l601.popanal.xml''' as an example, we will go through the template and describe what each bit does. + + {{{ + <template id="l301.basis" name="basis" repeat="" + pattern="\sStandard basis." endPattern="\sNAtoms=." + endOffset="1" + > + }}} + + The template starts with the template tag. The id and name define the template, repeat="" says that it can occur multiple times in the file. The '''pattern''' is a regular expression stating that this template will start when it matches a line that starts with spaces followed by the phrase "Standard basis" followed by any text. + + The template will swallow all text until the end pattern is encountered, which is a line with spaces followed by '''NAtoms=''' followed by some arbitrary text. + + The endOffest indicates what happens to the file pointer within the template i.e. whether it rolls back so that the line that was matched by the template is available for processing within the template or not. + + + Next comes a comment: + + {{{ + <comment class="example.input" id="l301.basis.09"> + Standard basis: 3-21G (6D, 7F) + Ernie: Thresh= 0.10000D-02 Tol= 0.10000D-05 Strict=F. + There are 7 symmetry adapted basis functions of AG symmetry. + There are 0 symmetry adapted basis functions of B1G symmetry. + There are 2 symmetry adapted basis functions of B2G symmetry. + There are 4 symmetry adapted basis functions of B3G symmetry. + There are 0 symmetry adapted basis functions of AU symmetry. + There are 7 symmetry adapted basis functions of B1U symmetry. + There are 4 symmetry adapted basis functions of B2U symmetry. + There are 2 symmetry adapted basis functions of B3U symmetry. + Integral buffers will be 131072 words long. + Raffenetti 1 integral format. + Two-electron integral symmetry is turned on. + 26 basis functions, 42 primitive gaussians, 26 cartesian basis functions + 8 alpha electrons 8 beta electrons + nuclear repulsion energy 33.7515964544 Hartrees. + IExCor= 0 DFT=F Ex=HF Corr=None ExCW=0 ScaHFX= 1.000000 + ScaDFX= 1.000000 1.000000 1.000000 1.000000 ScalE2= 1.000000 1.000000 + IRadAn= 0 IRanWt= -1 IRanGd= 0 ICorTp=0 + NAtoms= 6 NActive= 6 NUniq= 2 SFac= 4.00D+00 NAtFMM= 50 NAOKFM=F Big=F + </comment> + }}} + + The class of "example.input" means that this block of test will be used in the tests for the parser (see below). We can see how the first line is that which is matched by the template's '''pattern''' and the last one that matched by '''endPattern'''. + + Now we get to where the file is actually processed. + + {{{ + <record id="basis">\sStandard basis: {A,cc:basis} {X,cc:diffuse}</record> + }}} + + The first line reads a record from the file. '''repeat''' is omitted so defaults to one, so this reads a single line that starts with spaces and the phrase "Standard basis:" The terms that follow in the brackets determine what will be marked up into the xml. + + ''{A,cc:basis}''' this will read the data as a string (A) and put it in a term with the '''dictRef'' cc:basis. The cc prefix is the namespace of the computational chemistry dictionary, so this term is a '''basis''' as that term is defined in the computational chemistry dictionary. + * '''{X,cc:diffuse}''' the X is actually used to discard data, so this will swallow the text that comes after the character data. The dictRef is irrelevant in this case and could be anything. + + Other values for data that can be read are: + * F - float + * I - integer + + {{{ + <templateList id="ernie"> + <template pattern="\sErnie." endPattern="." id="ernie"> + <record id="ernie" repeat="">\sErnie: Thresh={E,g:thresh}\s Tol={E,g:tol}\sStrict={A,g:strict}\.\s</record> + </template> + </templateList> + <templateList id="symadnucl"> + <template pattern="\sThere are." endPattern="\snuclear repulsion." id="symaddnuc" endOffset="1"> + <record id="symmadapt" repeat="" makeArray="true">\s There are{I,cc:adapted} symmetry adapted basis functions of{A,cc:symm}symmetry\.\s</record> + <record id="buffer">\sIntegral buffers will be {I,g:buffer}\swords long\.\s</record> + <record id="raff">\s{A,g:raffenetti}\s{I,g:raff}\sintegral format\.\s</record> + <record id="twoe">\s{X,g:twoe} integral symmetry is turned on\.\s</record> + <record id="basiscount">\s{I,cc:basiscount}basis functions,{I,g:primbasis}primitive gaussians,{I,cc:cartesianbasis}cartesian basis functions\s</record> + <record id="alphabeta">\s{I,cc:alphae}alpha electrons\s{I,cc:betae}beta electrons\s</record> + <record id="nucrep">\snuclear repulsion energy\s{F,cc:nucrepener}Hartrees\.\s</record> + </template> + </templateList> + <templateList id="natoms"> + <template pattern="\sNAtoms=." repeat="" endPattern="." id="natoms"> + <record id="natoms" repeat="">\sNAtoms={I,cc:natoms}\sNActive={I,cc:nactiveatoms}\sNUniq={I,cc:uniqatoms}\sSFac={E,g:sfac}\sNAtFMM={I,g:natfmm}.\sBig={A,g:big}\s</record> + </template> + </templateList> + <templateList id="misc"> + <template pattern="\s((IExCor)\|(ScaDFX)\|(IRadAn))." repeat="" endPattern="." id="misc"> + <record id="misc">\s{1_20A,g:misc}\s*</record> + </template> + </templateList>

JUMBO-Converters

JensThomas — 2012-02-29 10:38:03

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 56:	Line 56:
- For example, the Gaussian templateList.xml file:	+ For example, the Gaussian topTemplate.xml file:
Line 119:	Line 119:
- One of the templates it references is the file:	+ + The first few lines just declare this as a template and declare the include namespaces, so that we can use this to include other templates. + + There is then a comment containing some example text to show what this template parses - as will be explained later, this is also used for testing the template. + + There is then a template list, which is the container that holds the templates. The first templatelist only holds one template, which is the template that parses the various links (modules) within the Gaussian program. This template then contains a template list, which includes the other templates that will process the text found by the parent template. + + One of the templates the parent references is the file:

JUMBO-Converters

JensThomas — 2012-02-29 10:23:10

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 40:	Line 40:
- A '''templateList.xml''' file, is a top-level file that creates the parent template and then defines all the templates which it itself contains.	+ A '''topTemplate.xml''' file, is a top-level file that creates the parent template and then defines all the templates which it itself contains.
Line 44:	Line 44:
- Each template has a '''pattern''', which is a regular expression defining the text in the logfile where the module starts, may have an '''endPattern''', which is a regular expression defining where the module ends, and a '''repeatCount''', which describes how many times the module can occur within the file.	+ Each template has a '''pattern''', which is a regular expression defining the text in the logfile where the module starts, and an '''endPattern''', which is a regular expression defining where the module ends, and a '''repeatCount''', which describes how many times the module can occur within the file.
Line 54:	Line 54:
- Each module is then parsed in turn, either by another template, or by the '''records''' in the template.	+ Each module is then parsed in turn, either by another template within this template, or by the '''records''' in the template.
Line 58:	Line 58:
- {{{ - jumbo-converters/jumbo-converters-compchem/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templateList.xml - }}}	+ [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/topTemplate.xml] +
Line 66:	Line 65:
- <template id='gaussian.log' output="VERBOSE"> - <templateList id='main' xmlns:xi="http://www.w3.org/2001/XInclude"> - <xi:include href="l202.temp.xml"/> - <xi:include href="l000.temp.xml"/> - <xi:include href="l1.temp.xml"/> - <xi:include href="l101.temp.xml"/> - ... MORE TEMPLATES ... - <xi:include href="lany.temp.xml"/>	+ <template id='gaussian.log' + xmlns:xi="http://www.w3.org/2001/XInclude" + > + <comment> + Entering Gaussian System, Link 0=/usr/local/gaussian/g03/g03 + Initial command: + /usr/local/gaussian/g03/l1.exe /tmp/webmo/1/Gau-28330.inp -scrdir=/tmp/webmo/1/ + Entering Link 1 = /usr/local/gaussian/g03/l1.exe PID= 28333. + ... + ******************************************* + Gaussian 03: x86-Linux-G03RevB.04 2-Jun-2003 + 20-Nov-2006 + ******************************************* + -------------------------- + #N B3LYP/6-31G(d) OPT FREQ + -------------------------- + 1/14=-1,18=20,26=3,38=1/1,3; + ... + ... + Job cpu time: 0 days 0 hours 0 minutes 16.2 seconds. + File lengths (MBytes): RWF= 12 Int= 0 D2E= 0 Chk= 7 Scr= 1 + Normal termination of Gaussian 03 at Mon Nov 20 14:40:23 2006. + Link1: Proceeding to internal job step number 2. + ------------------------------------------------------------------ + #N Geom=AllCheck Guess=Read SCRF=Check GenChk RB3LYP/6-31G(d) Freq + ------------------------------------------------------------------ + 1/10=4,29=7,30=1,38=1,40=1,46=1/1,3; + ... + Job cpu time: 0 days 0 hours 0 minutes 12.7 seconds. + File lengths (MBytes): RWF= 12 Int= 0 D2E= 0 Chk= 7 Scr= 1 + Normal termination of Gaussian 03 at Mon Nov 20 14:40:36 2006. + + </comment> + + <templateList> + <template id="job" pattern="\s((Link1\:\s+Proceeding to internal job step number)\|(Entering Gaussian System))." + endPattern="\sNormal termination of." endOffset="1" repeat="*"> + <templateList id='main'> + <xi:include href="l0.entering.xml"/> + <xi:include href="l1/l1.legal.xml"/> + <xi:include href="l1/l1.citation.xml"/> + <xi:include href="l1/l1.end.xml"/> + <xi:include href="l716.xml"/> + <xi:include href="l601/l601.anisospin.xml"/> + <xi:include href="l601/l601.popanal.xml"/> + <xi:include href="l601/l601.polariz.xml"/> + <!-- Many more templates --> + </templateList> + </template>
Line 75:	Line 115:
	+
Line 78:	Line 119:
- The first template it references is the file: - - {{{ - jumbo-converters/jumbo-converters-compchem/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l202.temp.xml - }}}	+ One of the templates it references is the file: + + [https://bitbucket.org/wwmm/jumbo-converters/src/c25586883d1b/jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601/l601.popanal.xm jumbo-converters-compchem/jumbo-converters-compchem-gaussian/src/main/resources/org/xmlcml/cml/converters/compchem/gaussian/log/templates/l601/l601.popanal.xm +
Line 94:	Line 134:
- Using '''l202.temp.xml''' as an example:	+ Using '''l601.popanal.xml'' as an example:

Chempound

JensThomas — 2012-02-26 12:23:54

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 133:	Line 133:
- The easiest way to get to grips with SPARQL is to disect a simple query:	+ The easiest way to get to grips with SPARQL is to dissect a simple query:

Chempound

JensThomas — 2012-02-26 12:22:35

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 197:	Line 197:
- As the Chempound SPARQL endpoint exposes a RESTful API , we can query it directly, as the following python script demonstrates.	+ As the Chempound SPARQL endpoint exposes a RESTful API , we can query it directly. The following python script executes a SPARQL query against chempound and then saves the result as a csv (comma-separated variable) file, so that the results of the query can be imported into a spreadsheet program for (e.g.) plotting a graph of the results.
Line 204:	Line 204:
- # The SPARQL query you want to execute	+ # The SPARQL query we want to execute
Line 212:	Line 212:
- # url of the chempound SPARQL endpoint you want to query	+ # url of the chempound SPARQL endpoint we want to query
Line 226:	Line 226:
- #Set up our GET query to the SPARQL endpoint	+ # Set up our GET query to the SPARQL endpoint
Line 234:	Line 234:
- # Accept: text/turtle, application/rdf+xml - #request.add_header('Accept','application/sparql-results+json')
Line 239:	Line 237:
-
Line 253:	Line 250:
- #variables = etree.findall('.//{%s}variable' % namespace)
Line 258:	Line 254:
- # Loop through results adding the relevant bindings to the dictionary - # currently only support uri	+ # Loop through results adding the relevant bindings to the dictionary. + # Currently only support uri
Line 270:	Line 266:
-

Chempound

JensThomas — 2012-02-26 12:17:46

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 202:	Line 202:
- - # url of the chempound SPARQL endpoint - baseurl = "http://quixote.ch.cam.ac.uk/sparql/" - - # The SPARQL query - query="""SELECT ?molecule	+ import xml.etree.ElementTree as ET + + # The SPARQL query you want to execute + query="""SELECT ?molecule ?inchi
Line 211:	Line 209:
	+ ?molecule <http://www.xmlcml.org/rdf-schema#inchi> ?inchi .
Line 213:	Line 212:
	+ # url of the chempound SPARQL endpoint you want to query + baseurl = "http://quixote.ch.cam.ac.uk/sparql/" + + # The "comma-separated variable" file where the results should + # go so they can be imported into a spreadsheet program + csvFile = "/Users/jmht/sparql_results.csv" + + + # The real work starts here! + + # SPARQL namespace - shouldn't need to change this + namespace="http://www.w3.org/2005/sparql-results#" + # NB: results format based on: http://www.w3.org/2001/sw/DataAccess/rf1/ + + #Set up our GET query to the SPARQL endpoint
Line 214:	Line 228:
- urlparam = { "query" : query - "format" : "json", - "output" : "json" }	+ urlparam = { "query" : query }
Line 218:	Line 230:
	+ request = urllib2.Request(baseurl,querystr)
Line 220:	Line 233:
- request = urllib2.Request(baseurl,querystr) - request.add_header('Accept','application/json' ) - request.add_header('Accept','sparql-results/json')	+ request.add_header('Accept','application/sparql-results+xml') + # Accept: text/turtle, application/rdf+xml + #request.add_header('Accept','application/sparql-results+json')
Line 226:	Line 239:
- print response.read()	+ + + # We now have the results in SPARQL xml so we need to turn them into + # a csv file - we use etree to do this: + # http://effbot.org/zone/element-index.htm + + # Parse results to create etree & get root element + etree = ET.parse(response) + root = etree.getroot() + + # Sparql query always returns 2 elements: head and results + head,results = root[:] + + # Get head and create dictionary for variables + #variables = etree.findall('.//{%s}variable' % namespace) + resultsDict = {} + for var in head: + resultsDict[var.get("name")] = [] + + # Loop through results adding the relevant bindings to the dictionary + # currently only support uri + nresults=len(results) + for result in results: + for binding in result: + # One element for each binding of type: uri, literal or label + # currently only deal with uri + if ( len(binding) == 1 and binding[0].tag == "{%s}uri" % namespace ): + resultsDict[binding.get("name")].append(binding[0].text) + else: + raise RuntimeError("Results only supported for uri!") + + + # output as csv file + rfile = open(csvFile,'w') + + # column headers + headers = resultsDict.keys() + rfile.write(",".join(headers)+"\n") + + # data + for i in range(nresults): + newline=[] + for header in headers: + newline.append(resultsDict[header][i]) + rfile.write(",".join(newline)+"\n") + + rfile.close()

Maven

PeterMurrayRust — 2012-02-23 12:27:40

added petermr tips for Maven

Differences for Maven

Deletions are marked with - .	Additions are marked with +.
Line 76:	Line 76:
	+ == If you are offline == + + Although Maven has an offline setting, I (petermr) have never been able to use it reliably. If you have to work offline I'd suggest that you + {{{ + mvn clean install + }}} + everything you need before you get on the train. Then there is a reasonable chance that Maven will use your local repo for the jars you depend on. If it doesn't you'll have to wait till you get back home. + + == Your local repo == + + If something goes wrong in a build you can end up with rubbish in the repo. It's worth cleaning out grot from time to time. Even deleting the whole repo. But only do this with a good connection as Maven downloads an awful lot. + + == Scope for compile and test == + + Classes and jars for tests can be defined as + {{{ + <scope>test</scope> + }}} + and most of the time {{junit}} is so defined. However sometimes the classes with the test routines (e.g. {{org.junit.Assert}}) are used to build test utilities and will need the + {{{ + <scope>compile</scope> + }}} + + == Maven and Eclipse == + + Eclipse can do a lot of "smart" things such as caching classes and rebuilding the {{target}} immediately after deleting it. If Eclipse is giving problems (unresolved classes) I then run Maven to clean things. Maven is much stricter about the order of dependencies and you should always run Maven to verify that your code is clean (Hudson will insist on it anyway). + + If Eclipse is Open, then sometimes it will (at least on Windows) stop classes being deleted and crash Maven. I usually close Eclipse before running Maven. + + == Maven and Hudson == + + Hudson is even stricter than Maven. Maven can resolve relative directories, Hudson cannot. So use the URL method - I think I have abandoned relativeUrl.

Maven

JensThomas — 2012-02-23 11:36:43

Differences for Maven

Deletions are marked with - .	Additions are marked with +.
Line 233:	Line 233:
- * '''-Dexec.mainClass''' defines the variable '''exec.mainClass''', which specifies the main class to execute. In the cmllite-validator code example, the class we wish to execute is the CmlLiteValidator	+ * '''-Dexec.mainClass''' defines the variable '''exec.mainClass''', which specifies the main class to execute. In the cmllite-validator code example, the class we wish to execute is the CmlLiteValidator class, which lives in the directory: <project_root>/src/main/java/org/xmlcml/www/CmlLiteValidator.java In the maven convention, the class path for the file is therefore '''org.xmlcml.www.CmlLiteValidator''', so this is what we pass in using the mainClass argument. The main class of this file will then be executed/ + * -Dexec.args - this specifies a list of space-separated arguments that will be passed to the main method.

Maven

JensThomas — 2012-02-23 11:29:27

Differences for Maven

Deletions are marked with - .	Additions are marked with +.
Line 207:	Line 207:
	+ + + == Remove all project artifacts that have been installed into your local Maven repository == + + The repository is usually found at {{{~/.m2}}}). Run this command from ''<project-root>'': + + {{{ + mvn build-helper:remove-project-artifact + }}} + + == Execute java classes from the command-line == + + It is often useful to be able to execute individual classes from the maven command line - particularly during development. + + An example for the [https://bitbucket.org/cml/cmllite-validator-code] is the following: + + {{{ + mvn \ + -o \ + exec:java \ + -Dexec.mainClass="org.xmlcml.www.CmlLiteValidator" \ + -Dexec.args="/Users/jmht/Documents/quixote/repositories/dictionary-compchem/dictionary-compchem.cml" + }}} + + * The -o flag runs maven in offline mode, so it doesn't check for any updates to dependencies + * [http://mojo.codehaus.org/exec-maven-plugin/java-mojo.html exec:java] sets the maven goal to execute some java code + * '''-Dexec.mainClass''' defines the variable '''exec.mainClass''', which specifies the main class to execute. In the cmllite-validator code example, the class we wish to execute is the CmlLiteValidator

Tutorials and problems

JensThomas — 2012-02-23 11:23:21

Differences for Tutorials and problems

Deletions are marked with - .	Additions are marked with +.
Line 31:	Line 31:
-
Line 38:	Line 37:
- }}} - - * Remove all project artifacts that have been installed into your local Maven repository (usually at {{{~/.m2}}}). Run from ''<project-root>'': - - {{{ - mvn build-helper:remove-project-artifact - }}} - - * Clean your cloned project folder of all compiled files. Run from ''<project-root>'': - - {{{ - mvn clean

JUMBO-Converters

JensThomas — 2012-02-20 14:24:34

Differences for JUMBO-Converters

Deletions are marked with - .	Additions are marked with +.
Line 8:	Line 8:
	+ = Running the converters = + + This page is concerned with the philosophy and design of the converters. If you are just interested in running them, please go the the [http://quixote.wikispot.org/Tutorials_and_problems#head-3f3979152d76625356b090db3fc81300f47cf760 Tutorials and problems] page.

NWChem

JensThomas — 2012-02-20 10:55:53

Differences for NWChem

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
	+ = NWChem = + + The [http://www.nwchem-sw.org/ NWChem homepage] provides more information on NWChem. This page just details some information on building/running NWChem. + + == Building and running on OSX == + + For pre-Lion OSX's (i.e. OSX < 10.7), pre-compiled binaries of NWChem 6.0 are available from the [http://www.nwchem-sw.org/index.php/Download#NWChem_6.0 NWChem website]. + + The tar file will unpack to give a directory called (e.g.) '''nwchem-6.0-binary'''. + + In order to run the serial binary, you will need to set the environment variable NWCHEM_BASIS_LIBRARY to point to path to the basis set library directory, which is the directory '''nwchem-6.0-binary/usr.local.lib.nwchem/libraries/'''. You need the full path and do not omit the trailing slash, e.g.: + + {{{ + export NWCHEM_BASIS_LIBRARY=/User/jmht/nwchem-6.0-binary/usr.local.lib.nwchem/libraries/" + }}} + + The current serial binaries will not work on Lion, for which you will need to build NWChem as described below. + + === Building NWChem on OSX === + + The following steps show how to build NWChem on a mac without using MPI, i.e. suitable for running serially on a laptop/workstation. + + * Install the OSX [https://developer.apple.com/xcode/ developer tools] from Apple + * Install [http://gcc.gnu.org/wiki/GFortranBinaries#MacOS Gfortran] + * Download the latest NWChem source from the [http://www.nwchem-sw.org/index.php/Download website]. + * Unpack the source, and '''cd ''' into the resulting directory (which will be called something like '''nwchem-src-2012-Feb-16'''. + * Create a script called '''build.sh''' with the following in it: + + {{{ + #!/bin/bash + + # Change NWCHEM_TOP to match the source directory on your machine. + export NWCHEM_TOP=/Users/jmht/Documents/quixote/nwchem/nwchem-src-2012-Feb-16 + export NWCHEM_TARGET=MACX64 + export NWCHEM_MODULES=all + + export USE_MPI="" + export BLASOPT=" " + # Below may not be needed + export MSG_COMMS=TCGMSG + + cd $NWCHEM_TOP/src + + # Uncomment the following 2 lines and run the first time you use the script to + # configure nwchem. Once this has been done once, comment them out and rerun + # the script to run the build + #make nwchem_config + #exit + + make \ + FC=gfortran \ + CC=gcc \ + USE_64TO32=y + }}} + + Make the script executable ('''chmod +x ./build.sh''') and run it, and (with luck) it will build NWChem, which you will find in the directory: + + '''nwchem-src-2012-Feb-16/bin/MACX64/nwchem'''

Resources and technology

JensThomas — 2012-02-20 10:22:57

Differences for Resources and technology

Deletions are marked with - .	Additions are marked with +.
Line 24:	Line 24:
	+ + == Optional (Opensource) Components == + * ["NWChem"] - a powerful Opensource electronic structure code. + * [http://avogadro.openmolecules.net/ Avogadro] - an Opensource molecular modelling environment

Creating dictionaries

JensThomas — 2012-02-20 08:30:35

Differences for Creating dictionaries

Deletions are marked with - .	Additions are marked with +.
Line 1:	Line 1:
	+ = CREATING A NEW CML DICTIONARY = + + === Create bitbucket repository === + + The repository should contain a single .cml file in its root directory, containing the dictionary. The dictionary must either be the root element of the document, or be the child of a <cml> that is the root of the document. + + The root directory can contain non-cml files and subdirectories. + + === Create hudson job === + + In hudson click 'New Job' (in left-hand menu). + + Enter job name (e.g. cml-dictionary-compchem-gaussian) - this should be prefixed '''cml-dictionary-'''. + Select copy existing job, and enter 'cml-dictionary-compchem' + Click 'Ok' + + In the 'Source Code Management' section update the URLs to point at the new bitbucket repository (e.g. [https://bitbucket.org/cml/dictionary-compchem-gaussian]). + + In the 'Build Triggers' section tick 'Trigger builds remotely' box, and enter an Authentication Token (e.g. W3y7N3Ha). (This may already be set, in which case you can just keep the existing value). + + Click save. + + + === Get the bitbucket repo to trigger hudson === + + Go back to the bitbucket repository and click 'Admin' on the top menu and then 'Services' in the LH menu. + + Select 'POST' from the drop down menu and click 'Add Service'. + + In the URL box enter the hudson trigger url in the format: + + '''{{{https://hudson.ch.cam.ac.uk/job/<JOB_NAME>/build?token=<AUTH_TOKEN>}}}'''' + + e.g. + + '''{{{https://hudson.ch.cam.ac.uk/job/cml-dictionary-compchem-gaussian/build?token=W3y7N3Ha}}}''' + + And click 'Save Settings' + + Now, when you push to bitbucket a hudson job should trigger immediately. This will validate the dictionary and upload it to the CML website. If validation fails then the job will fail and the dictionary won't get uploaded. + + You can check the status of the jobs on the 'CML Dictionaries' page in hudson: + + [https://hudson.ch.cam.ac.uk/view/cml-dictionaries/]

Resources and technology

JensThomas — 2012-02-20 08:27:22

Differences for Resources and technology

Deletions are marked with - .	Additions are marked with +.
Line 120:	Line 120:
	+ * ["Creating dictionaries"]

Chempound

2012-02-18 01:58:18

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 197:	Line 197:
- As the Chempound SPARQL endpoint exposes a RESTFUL api, we can query it directly, as the following python script demonstrates.	+ As the Chempound SPARQL endpoint exposes a RESTful API , we can query it directly, as the following python script demonstrates.
Line 214:	Line 214:
- urlparam = { "query" : query }	+ urlparam = { "query" : query + "format" : "json", + "output" : "json" }
Line 219:	Line 221:
- request.add_header('Accept','text/html' )	+ request.add_header('Accept','application/json' ) + request.add_header('Accept','sparql-results/json')

Chempound

2012-02-17 23:22:48

Differences for Chempound

Deletions are marked with - .	Additions are marked with +.
Line 191:	Line 191:
	+ EXPLAIN HOW TO USE THE DICTIONARIES TO DISCOVER WHAT TERMS ARE AVAILABLE. + + == Remote Chempound SPARQL queries with Python == + + The Chempound SPARQL page will return the results as html or rdf/xml. The rdf/xml can of course be saved and processed offline, but it is more useful to be able to query and download the results all from within a single script. + + As the Chempound SPARQL endpoint exposes a RESTFUL api, we can query it directly, as the following python script demonstrates. + + {{{ + import urllib + import urllib2 + + # url of the chempound SPARQL endpoint + baseurl = "http://quixote.ch.cam.ac.uk/sparql/" + + # The SPARQL query + query="""SELECT ?molecule + WHERE + { + ?molecule <http://www.xmlcml.org/rdf-schema#formula> "H 2 O 1" . + }""" + + # Encode the parts of the query string into a form suitable for POST + urlparam = { "query" : query } + querystr=urllib.urlencode(urlparam) + + # Add the header to state what we want back + request = urllib2.Request(baseurl,querystr) + request.add_header('Accept','text/html' ) + + # Get the results + response = urllib2.urlopen(request) + print response.read() + }}} +