Accuracy of PHDsec
PHDsec: secondary structure prediction
****************************************************************************
* *
* *
* *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* Secondary structure prediction by PHDsec: *
* a Profile fed neural network system from HeiDelberg *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* Authors: Burkhard Rost & Chris Sander *
* EMBL, Heidelberg, FRG *
* Meyerhofstrasse 1, 69 117 Heidelberg *
* Internet: Predict-Help@EMBL-Heidelberg.DE *
* *
* All rights reserved. *
* *
* *
****************************************************************************
* *
* *
* Please quote *
* ~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* The network procedure is described in detail in: *
* 1) B Rost & C Sander: Prediction of protein structure at better than *
* 70% accuracy. J. Mol. Biol., 1993, 232, 584-599. (Abstract) *
* *
* A brief description is given in: *
* B Rost & C Sander:Improved prediction of protein secondary structure *
* by use of sequence profiles and neural networks. PNAS, 1993, 90, *
* 7558-7562. (Abstract) *
* *
* The PredictProtein mail server is described in: *
* 2) B Rost: PHD: predicting one-dimensional protein structure by pro- *
* file based neural networks. Meth. in Enzym., 1996, 266, 525-539. *
* (Text) *
* B Rost, C Sander & R Schneider: PHD-a mail server for protein secon- *
* dary structure prediction. CABIOS, 1994, 10, 53-60. (Abstract) *
* *
* The latest improvement steps (up to 72%) are explained in: *
* 3) B Rost & C Sander: Combining evolutionary information and neural *
* networks to predict protein secondary struct. Proteins, 1994, 19, *
* 55-77. (Abstract) *
* *
* To be quoted for publications of PHD output: *
* Papers 1-3 for the prediction of secondary structure and the pre- *
* diction server. *
* *
* *
****************************************************************************
* *
* *
* About the input to the network *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* The prediction is performed by a system of neural networks. *
* The input is a multiple sequence alignment. It is taken from an HSSP *
* file (produced by the program MaxHom: *
* Sander, Chris & Schneider, Reinhard: Database of Homology-Derived *
* Structures and the Structural Meaning of Sequence Alignment. *
* Proteins, Vol.9, 1991, pp. 56-68. *
* *
* For optimal results the alignment should contain sequences with varying *
* degrees of sequence similarity relative to the input protein. *
* The following is an ideal situation: *
* *
* +-----------------+----------------------+ *
* | sequence: | sequence identity | *
* +-----------------+----------------------+ *
* | target sequence | 100 % | *
* | aligned seq. 1 | 90 % | *
* | aligned seq. 2 | 80 % | *
* | ... | ... | *
* | aligned seq. 7 | 30 % | *
* +-----------------+----------------------+ *
* *
* *
****************************************************************************
* *
* *
* Estimated Accuracy of Prediction *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* A careful cross validation test on some 250 protein chains (in total *
* about 55,000 residues) with less than 25% pairwise sequence identity *
* gave the following results: *
* *
* ++================++-----------------------------------------+ *
* || Qtotal = 72.1% || ("overall three state accuracy") | *
* ++================++-----------------------------------------+ *
* *
* +----------------------------+-----------------------------+ *
* | Qhelix (% of observed)=70% | Qhelix (% of predicted)=77% | *
* | Qstrand(% of observed)=62% | Qstrand(% of predicted)=64% | *
* | Qloop (% of observed)=79% | Qloop (% of predicted)=72% | *
* +----------------------------+-----------------------------+ *
*..........................................................................*
* *
* These percentages are defined by: *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* | number of correctly predicted residues *
* |Qtotal = --------------------------------------- (*100)*
* | number of all residues *
* | *
* | no of res correctly predicted to be in helix *
* |Qhelix (% of obs) = -------------------------------------------- (*100)*
* | no of all res observed to be in helix *
* | *
* | *
* | no of res correctly predicted to be in helix *
* |Qhelix (% of pred)= -------------------------------------------- (*100)*
* | no of all residues predicted to be in helix *
* *
*..........................................................................*
* *
* Averaging over single chains *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* The most reasonable way to compute the overall accuracies is the above *
* quoted percentage of correctly predicted residues. However, since the *
* user is mainly interested in the expected performance of the prediction *
* for a particular protein, the mean value when averaging over protein *
* chains might be of help as well. Computing first the three state *
* accuracy for each protein chain, and then averaging over 250 chains *
* yields the following average: *
* *
* +-------------------------------====--+ *
* | Qtotal/averaged over chains = 72.2% | *
* +-------------------------------====--+ *
* | standard deviation = 9.3% | *
* +-------------------------------------+ *
* *
*..........................................................................*
* *
* Further measures of performance *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* Matthews correlation coefficient: *
* *
* +---------------------------------------------+ *
* | Chelix = 0.63, Cstrand = 0.53, Cloop = 0.52 | *
* +---------------------------------------------+ *
*..........................................................................*
* *
* Average length of predicted secondary structure segments: *
* *
* . +------------+----------+ *
* . | predicted | observed | *
* +-----------+------------+----------+ *
* | Lhelix = | 10.3 | 9.3 | *
* | Lstrand = | 5.0 | 5.3 | *
* | Lloop = | 7.2 | 5.9 | *
* +-----------+------------+----------+ *
*..........................................................................*
* *
* The accuracy matrix in detail: *
* *
* +---------------------------------------+ *
* | number of residues with H, E, L | *
* +---------+------+------+------+--------+ *
* | |net H |net E |net L |sum obs | *
* +---------+------+------+------+--------+ *
* | obs H |12447 | 1255 | 3990 | 17692 | *
* | obs E | 949 | 7493 | 3750 | 12192 | *
* | obs L | 2604 | 2875 |19962 | 25441 | *
* +---------+------+------+------+--------+ *
* | sum Net |16000 |11623 |27702 | 55325 | *
* +---------+------+------+------+--------+ *
* *
* Note: This table is to be read in the following manner: *
* 12447 of all residues predicted to be in helix, were observed to *
* be in helix, 949 however belong to observed strands, 2604 to *
* observed loop regions. The term "observed" refers to the DSSP *
* assignment of secondary structure calculated from 3D coordinates *
* of experimentally determined structures (Dictionary of Secondary *
* Structure of Proteins: Kabsch & Sander (1983) Biopolymers, 22, *
* 2577-2637). *
* *
* *
****************************************************************************
* *
* *
* Position-specific reliability index *
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *
* *
* The network predicts the three secondary structure types using real *
* numbers from the output units. The prediction is assigned by choosing *
* the maximal unit ("winner takes all"). However, the real numbers *
* contain additional information. *
* E.g. the difference between the maximal and the second largest output *
* unit can be used to derive a "reliability index". This index is given *
* for each residue along with the prediction. The index is scaled to *
* have values between 0 (lowest reliability), and 9 (highest). *
* The accuracies (Qtot) to be expected for residues with values above a *
* particular value of the index are given below as well as the fraction *
* of such residues (%res).: *
* *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* | index| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | *
* | %res |100.0| 99.2| 90.4| 80.9| 71.6| 62.5| 52.8| 42.3| 29.8| 14.1| *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* | | | | | | | | | | | | *
* | Qtot | 72.1| 72.3| 74.8| 77.7| 80.3| 82.9| 85.7| 88.5| 91.1| 94.2| *
* | | | | | | | | | | | | *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* | H%obs| 70.4| 70.6| 73.7| 77.1| 80.1| 83.1| 86.0| 89.3| 92.5| 96.4| *
* | E%obs| 61.5| 61.7| 63.7| 66.6| 69.1| 71.7| 74.6| 77.0| 77.8| 68.1| *
* | | | | | | | | | | | | *
* | H%prd| 77.8| 78.0| 80.0| 82.6| 84.7| 86.9| 89.2| 91.3| 93.1| 95.4| *
* | E%prd| 64.5| 64.7| 67.8| 71.0| 74.2| 77.6| 81.4| 85.1| 89.8| 93.5| *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* *
* The above table gives the cumulative results, e.g. 62.5% of all *
* residues have a reliability of at least 5. The overall three-state *
* accuracy for this subset of almost two thirds of all residues is 82.9%. *
* For this subset, e.g., 83.1% of the observed helices are correctly *
* predicted, and 86.9% of all residues predicted to be in helix are *
* correct. *
* *
*..........................................................................*
* *
* The following table gives the non-cumulative quantities, i.e. the *
* values per reliability index range. These numbers answer the question: *
* how reliable is the prediction for all residues labeled with the *
* particular index i. *
* *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* | index| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | *
* | %res | 8.8| 9.5| 9.3| 9.1| 9.7| 10.5| 12.5| 15.7| 14.1| *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* | | | | | | | | | | | *
* | Qtot | 46.6| 50.6| 57.7| 62.6| 67.9| 74.2| 82.2| 88.3| 94.2| *
* | | | | | | | | | | | *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* | H%obs| 36.8| 42.3| 49.5| 55.2| 61.7| 69.9| 78.8| 87.4| 96.4| *
* | E%obs| 44.7| 44.5| 52.1| 55.4| 60.9| 68.0| 75.9| 81.0| 68.1| *
* | | | | | | | | | | | *
* | H%prd| 49.9| 52.5| 60.3| 64.2| 69.2| 77.5| 85.4| 89.9| 95.4| *
* | E%prd| 41.7| 47.1| 53.6| 57.0| 64.0| 71.6| 78.8| 88.8| 93.5| *
* +------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ *
* *
* For example, for residues with Relindex = 5 64% of all predicted betha- *
* strand residues are correctly identified. *
* *
* *
****************************************************************************