Merge pull request #33 from andreskrey/v1.0

v1.0

.travis.yml

@@ -3,10 +3,9 @@ language: php
 install: composer install
 
 php:
-  - "5.4"
-  - "5.5"
   - "5.6"
   - "7.0"
   - "7.1"
+  - "7.2"
 
 sudo: false

AUTHORS.md

@@ -1,6 +1,9 @@
 # Authors
 
-Readability.php developed by **Andres Rey**. Copyright (c) 2010 Arc90 Inc
+Readability.php developed by **Andres Rey**. 
+
+Based on Arc90's readability.js (1.7.1) script available at: http://code.google.com/p/arc90labs-readability.
+Copyright (c) 2010 Arc90 Inc
 
 The AUTHORS/Contributors are (and/or have been):
 

CHANGELOG.md

@@ -3,6 +3,15 @@ All notable changes to this project will be documented in this file.
 
 ## Unreleased
 
+## [v1.0.0](https://github.com/andreskrey/readability.php/releases/tag/v1.0.0)
+
+- Node encapsulation is gone. Pre v1 all nodes where encapsulated in a Readability class, which created lots of trouble with dependencies, responsibilities, and properties. Now all the encapsulation is gone: all the DOMNodes inside the Readability class are extensions of the original DOM classes, which allows the system to take advantage of the functions and properties of DOMDocument.
+- HTMLParser is gone, Readability is the new main class. Switched things a bit for this release. Pre v1 you had to create an HTMLParser class to parse the HTML. Now you have to create a Readability class, feed it the text, and check the result.
+- No more dumb arrays as a result. If you want to get the title, content, images, or anything else you'll have to use the getters of the Readability class.
+- Environment class is gone. Now you have to create a configuration class and use setters to set your configuration options.
+- Exceptions. Make sure you wrap your Readability class in a try catch block, because if it fails to parse your HTML, it will throw a `ParseException`.
+- Minimum PHP version bumped to 5.6.
+
 ## [v0.3.1](https://github.com/andreskrey/readability.php/releases/tag/v0.3.1)
 
 - Trim titles when detecting hierarchical separators to avoid false negatives on strings with spaces.

README.md

@@ -1,66 +1,93 @@
 # Readability.php
 [![Latest Stable Version](https://poser.pugx.org/andreskrey/readability.php/v/stable)](https://packagist.org/packages/andreskrey/readability.php) [![StyleCI](https://styleci.io/repos/71042668/shield?branch=master)](https://styleci.io/repos/71042668) [![Build Status](https://travis-ci.org/andreskrey/readability.php.svg?branch=master)](https://travis-ci.org/andreskrey/readability.php) [![Total Downloads](https://poser.pugx.org/andreskrey/readability.php/downloads)](https://packagist.org/packages/andreskrey/readability.php) [![Monthly Downloads](https://poser.pugx.org/andreskrey/readability.php/d/monthly)](https://packagist.org/packages/andreskrey/readability.php)
 
-PHP port of *Mozilla's* **[Readability.js](https://github.com/mozilla/readability)**. Parses html text (usually news and other articles) and returns title, byline and text content without nav bars, ads, footers, or anything that isn't the main body of the text. Analizes each text node, gives an score and orders them based on this calculation.
+PHP port of *Mozilla's* **[Readability.js](https://github.com/mozilla/readability)**. Parses html text (usually news and other articles) and returns **title**, **author**, **main image** and **text content** without nav bars, ads, footers, or anything that isn't the main body of the text. Analyzes each node, gives them a score, and determines what's relevant and what can be discarded.
 
-**Requires**: PHP 5.4+ & DOMDocument (libxml)
+![Screenshot](https://raw.githubusercontent.com/andreskrey/readability.php/assets/screenshot.png)
 
-**Lead Developer**: Andres Rey
+The project aim is to be a 1 to 1 port of Mozilla's version and to follow closely all changes introduced there, but there are some major differences on the structure. Most of the code is a 1:1 copy –even the comments were imported– but some functions and structures were adapted to suit better the PHP language.
+
+## Requirements
 
-## Status
+PHP 5.6+, ext-dom, ext-xml, and ext-mbstring. To install all this dependencies (in the rare case your system does not have them already), you could try something like this in *nix like environments:
 
-Current status is stable. Version 1.0 is around the corner.
+`$ sudo apt-get install php7.1-xml php7.1-mbstring`
+
+**Lead Developer**: Andres Rey
 
 ## How to use it
 
 First you have to require the library using composer:
 
 `composer require andreskrey/readability.php`
 
-Then, create and HTMLParser object with your preferences, feed the `parse()` function with your HTML and check the resulting array:
+Then, create a Readability class and pass a Configuration class, feed the `parse()` function with your HTML and echo the variable:
 
 ```php 
 use andreskrey\Readability\HTMLParser;
+use andreskrey\Readability\Configuration;
 
-$readability = new HTMLParser();
+$readability = new Readability(new Configuration());
 
 $html = file_get_contents('http://your.favorite.newspaper/article.html');
 
-$result = $readability->parse($html);
+try {
+    $readability->parse($html);
+    echo $readability;
+} catch (ParseException $e) {
+    echo sprintf('Error processing text: %s', $e->getMessage);
+}
 ```
 
-The `$result` variable now will hold the following information:
+Your script will output the parsed text or inform about any errors. You should always wrap the `->parse` call in a try/catch block because if the HTML cannot be parsed correctly, a `ParseException` will be thrown.
+
+If you want to have a finer control on the output, just call the properties one by one, wrapping it with your own HTML.
+
+```php
+<h1><?= $readability->getTitle(); ?></h1>
+<h2>By <?= $readability->getAuthor(); ?></h2>
+<div class="content"><?= $readability->getContent(); ?></div>
 
-```
-$result = [
-    'title' => 'Title of the article',
-    'author' => 'Name of the author of the article',
-    'image' => 'Main image of the article',
-    'images' => 'All images of the article',
-    'article' => 'DOMDocument with the full article text, scored and parsed'
-]
 ```
 
-If the parsing process was unsuccessful the HTMLParser will return `false`
+Here's a list of the available properties:
+
+- Article title: `->getTitle();`
+- Article content: `->getContent();`
+- Excerpt: `->getExcerpt();`
+- Main image: `->getImage();`
+- All images: `->getImages();`
+- Author: `->getAuthor();`
+- Text direction (ltr or rtl): `->getDirection();`
 
 ## Options
 
-- **maxTopCandidates**: default value `5`, max amount of top level candidates.
-- **wordThreshold**: default value `500`, minimum amount of characters to consider that the article was parsed successful.
-- **articleByLine**: default value `false`, search for the article byline and remove it from the text. It will be moved to the article metadata. 
-- **stripUnlikelyCandidates**: default value `true`, remove nodes that are unlikely to have relevant information. Useful for debugging or parsing complex or non-standard articles. 
-- **cleanConditionally**: default value `true`, remove certain nodes after parsing to return a cleaner result. 
-- **weightClasses**: default value `true`, weight classes during the rating phase. 
-- **removeReadabilityTags**: default value `true`, remove the data-readability tags inside the nodes that are added during the rating phase. 
-- **fixRelativeURLs**: default value `false`, convert relative URLs to absolute. Like `/test` to `http://host/test`. 
-- **substituteEntities**: default value `false`, disables the `substituteEntities` flag of libxml. Will avoid substituting HTML entities. Like `&aacute;` to á.
-- **normalizeEntities**: default value `false`, converts UTF-8 characters to its HTML Entity equivalent. Useful to parse HTML with mixed encoding.
-- **originalURL**: default value `http://fakehost`, original URL from the article used to fix relative URLs.
-- **summonCthulhu**: default value `false`, remove all `<script>` nodes via regex. This is not ideal as it might break things, but might be the only solution to [libxml problems with unescaped javascript](https://github.com/andreskrey/readability.php#known-issues).
+You can change the behaviour of Readability via the Configuration object. For example, if you want to fix relative URLs and declare the original URL, you could set up the configuration like this:
+
+```php
+$configuration = new Configuration();
+$configuration->setFixRelativeURLs(true)
+    ->setOriginalURL('http://my.newspaper.url/article/something-interesting-to-read.html');
+```
+
+Then you pass this Configuration object to Readability. The following options are available. Remember to prepend `set` when calling them.
+
+- **MaxTopCandidates**: default value `5`, max amount of top level candidates.
+- **WordThreshold**: default value `500`, minimum amount of characters to consider that the article was parsed successful.
+- **ArticleByLine**: default value `false`, search for the article byline and remove it from the text. It will be moved to the article metadata. 
+- **StripUnlikelyCandidates**: default value `true`, remove nodes that are unlikely to have relevant information. Useful for debugging or parsing complex or non-standard articles. 
+- **CleanConditionally**: default value `true`, remove certain nodes after parsing to return a cleaner result. 
+- **WeightClasses**: default value `true`, weight classes during the rating phase. 
+- **RemoveReadabilityTags**: default value `true`, remove the data-readability tags inside the nodes that are added during the rating phase. 
+- **FixRelativeURLs**: default value `false`, convert relative URLs to absolute. Like `/test` to `http://host/test`. 
+- **SubstituteEntities**: default value `false`, disables the `substituteEntities` flag of libxml. Will avoid substituting HTML entities. Like `&aacute;` to á.
+- **NormalizeEntities**: default value `false`, converts UTF-8 characters to its HTML Entity equivalent. Useful to parse HTML with mixed encoding.
+- **OriginalURL**: default value `http://fakehost`, original URL from the article used to fix relative URLs.
+- **SummonCthulhu**: default value `false`, remove all `<script>` nodes via regex. This is not ideal as it might break things, but might be the only solution to [libxml problems with unescaped javascript](https://github.com/andreskrey/readability.php#known-issues). If you're not parsing Javascript tutorials, it's recommended to always set this option as `true`.
 
 ## Limitations
 
-Of course the main limitation is PHP. Websites that load the content through lazy loading, AJAX, or any type of javascript fueled call will be ignored (actually, *not ran*) and the resulting text will be incorrect, compared to the readability.js results. All the articles you want to parse with readability.php will need to be complete and all the content should be in the HTML already.  
+Of course the main limitation is PHP. Websites that load the content through lazy loading, AJAX, or any type of javascript fueled call will be ignored (actually, *not ran*) and the resulting text will be incorrect, compared to the readability.js results. All the articles you want to parse with readability.php need to be complete and all the content should be in the HTML already.  
 
 ## Known Issues
 
@@ -76,7 +103,7 @@ DOMDocument has some issues while parsing javascript with unescaped HTML on stri
 </script>
 ```
 
-If you would like to remove the scripts of the HTML (like readability does), you would expect ending up with just one div and one comment on the final HTML. The problem is that libxml takes that closing div tag inside the javascript string as a HTML tag, effectively closing the unclosed tag and leaving the rest of the javascript as a string withing a P tag. If you save that node, the final HTML will end up like this:
+If you would like to remove the scripts of the HTML (like readability does), you would expect ending up with just one div and one comment on the final HTML. The problem is that libxml takes that closing div tag inside the javascript string as a HTML tag, effectively closing the unclosed tag and leaving the rest of the javascript as a string within a P tag. If you save that node, the final HTML will end up like this:
 
 ```html
 <div> <!-- Offending div without closing tag -->
@@ -99,12 +126,12 @@ Self closing tags like `<br />` get automatically expanded to `<br></br`. No way
 
 ## Dependencies
 
-Readability uses the Element interface and class from *The PHP League's* **[html-to-markdown](https://github.com/thephpleague/html-to-markdown/)**. The Readability object is an extension of the Element class. It overrides some methods but relies on it for basic DOMElement parsing.
+Readability.php has no dependencies to other libraries.
 
 ## To-do
 
-- Right now the Readability object is an extension of the Element object of html-to-markdown. This is a problem because you lose context. The scoring when creating a new Readability object must be reloaded manually. The DOMDocument object is consistent across the same document. You change one value here and that will update all other nodes in other variables. By using the element interface you lose that reference and the score must be restored manually. Ideally, the Readability object should be an extension of the DOMDocument or DOMElement objects, the score should be saved within that object and no restoration or recalculation would be needed.
-- There are a lot of problems with responsabilities. Right now there are two classes: HTMLParser and Readability. HTMLParser does a lot of things that should be a responsibility of Readability. It also does a lot of things that should be part of another class, specially when building the final article DOMDocument.
+- Keep up with Readability.js changes
+- Add a small template engine for the __toString() method, instead of using a hardcoded one.
 
 ## How it works
 
@@ -114,12 +141,10 @@ Readability parses all the text with DOMDocument, scans the text nodes and gives
 
 Up to date with readability.js as of [16 Oct 2017](https://github.com/mozilla/readability/commit/c3ff1a2d2c94c1db257b2c9aa88a4b8fbeb221c5).
 
-## TO-DOs of the current port:
-
- - Port `_cleanStyles` to avoid style attributes inside other tags (like `<p style="hello   ">`) 
-
 ## License
 
+Based on Arc90's readability.js (1.7.1) script available at: http://code.google.com/p/arc90labs-readability
+
     Copyright (c) 2010 Arc90 Inc
 
     Licensed under the Apache License, Version 2.0 (the "License");

composer.json

@@ -18,12 +18,12 @@
         }
     },
     "require": {
-        "php": ">=5.4.0",
+        "php": ">=5.6.0",
         "ext-dom": "*",
         "ext-xml": "*",
-        "league/html-to-markdown": "^4.2"
+        "ext-mbstring": "*"
     },
     "require-dev": {
-        "phpunit/phpunit": "4.*"
+        "phpunit/phpunit": "^5.7"
     }
 }

phpunit.xml

@@ -8,4 +8,9 @@
             <directory>./test/</directory>
         </testsuite>
     </testsuites>
-</phpunit>
+    <filter>
+        <whitelist>
+            <directory suffix=".php">src/</directory>
+        </whitelist>
+    </filter>
+</phpunit>