diff --git a/.travis.yml b/.travis.yml
index a80e0947..aac35712 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,13 +1,16 @@
language: php
+before_install:
+ - apt-cache policy libxml2
+
install:
- composer install
php:
- - "5.6"
- "7.0"
- "7.1"
- "7.2"
+ - "7.3"
script:
- ./vendor/bin/phpunit --coverage-clover ./test/clover.xml
@@ -15,5 +18,3 @@ script:
after_script:
- composer require php-coveralls/php-coveralls:^2.0
- php ./vendor/php-coveralls/php-coveralls/bin/php-coveralls -v
-
-sudo: false
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7c636fe6..7879d1a3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,26 @@ All notable changes to this project will be documented in this file.
## Unreleased
+## [v2.0.0](https://github.com/andreskrey/readability.php/releases/tag/v2.0.0)
+
+- [BREAKING CHANGE] Bumped the minimum supported version of PHP to 7.0
+- Clean `
-
var res = Response.error();
+
var res = Response.error();try{
res.headers.set("Origin","http://mybank.com");}catch(e){
console.log("Cannot pretend to be a bank!");}
-
-
+
Request
@@ -153,36 +147,33 @@
Request
The simplest Request is of course, just a URL, as you may do to GET a
resource.
-
var req =new Request("/index.html");
+
var req =new Request("/index.html");
console.log(req.method);// "GET"
console.log(req.url);// "http://example.com/index.html"
-
-
+
You may also pass a Request to the Request() constructor to
create a copy.
(This is not the same as calling the clone() method, which
is covered in
the “Reading bodies” section.).
-
var copy =new Request(req);
+
var copy =new Request(req);
console.log(copy.method);// "GET"
console.log(copy.url);// "http://example.com/index.html"
-
-
+
Again, this form is probably only useful in ServiceWorkers.
The non-URL attributes of the Request can only be set by passing
initial
values as a second argument to the constructor. This argument is a dictionary.
-
var uploadReq =new Request("/uploadImage",{
+
var uploadReq =new Request("/uploadImage",{
method:"POST",
headers:{"Content-Type":"image/png",},
body:"image data"});
-
-
+
The Request’s mode is used to determine if cross-origin requests lead
to valid responses, and which properties on the response are readable.
Legal mode values are "same-origin", "no-cors" (default)
@@ -192,14 +183,13 @@
Request
this to ensure that
a request is always being made to your origin.
-
var arbitraryUrl = document.getElementById("url-input").value;
+
var arbitraryUrl = document.getElementById("url-input").value;
fetch(arbitraryUrl,{ mode:"same-origin"}).then(function(res){
console.log("Response succeeded?", res.ok);},function(e){
console.log("Please enter a same-origin URL!");});
-
-
+
The "no-cors" mode captures what the web platform does by default
for scripts you import from CDNs, images hosted on other domains, and so
on. First, it prevents the method from being anything other than “HEAD”,
@@ -218,7 +208,7 @@
Request
you could get a list of Flickr’s most interesting photos
today like this:
-
var u =new URLSearchParams();
+
var u =new URLSearchParams();
u.append('method','flickr.interestingness.getList');
u.append('api_key','<insert api key here>');
u.append('format','json');
@@ -236,15 +226,13 @@
Request
console.log(photo.title);});});
-
-
+
You may not read out the “Date” header since Flickr does not allow it
via
Access-Control-Expose-Headers.
-
response.headers.get("Date");// null
-
-
+
response.headers.get("Date");// null
+
The credentials enumeration determines if cookies for the other
domain are
sent to cross-origin requests. This is similar to XHR’s withCredentials
@@ -297,13 +285,12 @@
Response
idiomatic way to return a Response to an intercepted request in ServiceWorkers
is:
As you can see, Response has a two argument constructor, where both arguments
are optional. The first argument is a body initializer, and the second
is a dictionary to set the status, statusText and headers.
@@ -348,19 +335,17 @@
Dealing with bodies
non-text data!
Request bodies can be set by passing body parameters:
-
var form =new FormData(document.getElementById('login-form'));
+
var form =new FormData(document.getElementById('login-form'));
fetch("/login",{
method:"POST",
body: form
})
-
-
+
Responses take the first argument as the body.
-
var res =new Response(new File(["chunk","chunk"],"archive.zip",
+
var res =new Response(new File(["chunk","chunk"],"archive.zip",{ type:"application/zip"}));
-
-
+
Both Request and Response (and by extension the fetch() function),
will try to intelligently determine the content type.
Request will also automatically set a “Content-Type” header if none is
@@ -372,7 +357,7 @@
Streams and cloning
read once! Both interfaces have a boolean attribute bodyUsed to
determine if it is safe to read or not.
-
var res =new Response("one time use");
+
var res =new Response("one time use");
console.log(res.bodyUsed);// false
res.text().then(function(v){
console.log(res.bodyUsed);// true
@@ -382,8 +367,7 @@
Streams and cloning
res.text().catch(function(e){
console.log("Tried to read already consumed Response");});
-
-
+
This decision allows easing the transition to an eventual stream-based Fetch
API. The intention is to let applications consume data as it arrives, allowing
for JavaScript to deal with larger files like videos, and perform things
@@ -398,7 +382,7 @@
Streams and cloning
be called before the body of the corresponding object has been used. That
is, clone() first, read later.
diff --git a/test/test-pages/003-metadata-preferred/expected-images.json b/test/test-pages/003-metadata-preferred/expected-images.json
new file mode 100644
index 00000000..0637a088
--- /dev/null
+++ b/test/test-pages/003-metadata-preferred/expected-images.json
@@ -0,0 +1 @@
+[]
\ No newline at end of file
diff --git a/test/test-pages/003-metadata-preferred/expected-metadata.json b/test/test-pages/003-metadata-preferred/expected-metadata.json
new file mode 100644
index 00000000..a702e4a0
--- /dev/null
+++ b/test/test-pages/003-metadata-preferred/expected-metadata.json
@@ -0,0 +1,5 @@
+{
+ "Title": "Dublin Core property title",
+ "Author": "Dublin Core property author",
+ "Excerpt": "Dublin Core property description"
+}
diff --git a/test/test-pages/003-metadata-preferred/expected.html b/test/test-pages/003-metadata-preferred/expected.html
new file mode 100644
index 00000000..6b03dd38
--- /dev/null
+++ b/test/test-pages/003-metadata-preferred/expected.html
@@ -0,0 +1,18 @@
+
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
\ No newline at end of file
diff --git a/test/test-pages/003-metadata-preferred/source.html b/test/test-pages/003-metadata-preferred/source.html
new file mode 100644
index 00000000..c7275b3b
--- /dev/null
+++ b/test/test-pages/003-metadata-preferred/source.html
@@ -0,0 +1,45 @@
+
+
+
+
+ Title Element
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Test document title
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
+
+
\ No newline at end of file
diff --git a/test/test-pages/004-metadata-space-separated-properties/expected-images.json b/test/test-pages/004-metadata-space-separated-properties/expected-images.json
new file mode 100644
index 00000000..0637a088
--- /dev/null
+++ b/test/test-pages/004-metadata-space-separated-properties/expected-images.json
@@ -0,0 +1 @@
+[]
\ No newline at end of file
diff --git a/test/test-pages/004-metadata-space-separated-properties/expected-metadata.json b/test/test-pages/004-metadata-space-separated-properties/expected-metadata.json
new file mode 100644
index 00000000..a41ac6b0
--- /dev/null
+++ b/test/test-pages/004-metadata-space-separated-properties/expected-metadata.json
@@ -0,0 +1,5 @@
+{
+ "Title": "Preferred title",
+ "Author": "Creator Name",
+ "Excerpt": "Preferred description"
+}
diff --git a/test/test-pages/004-metadata-space-separated-properties/expected.html b/test/test-pages/004-metadata-space-separated-properties/expected.html
new file mode 100644
index 00000000..6b03dd38
--- /dev/null
+++ b/test/test-pages/004-metadata-space-separated-properties/expected.html
@@ -0,0 +1,18 @@
+
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
\ No newline at end of file
diff --git a/test/test-pages/004-metadata-space-separated-properties/source.html b/test/test-pages/004-metadata-space-separated-properties/source.html
new file mode 100644
index 00000000..9b951008
--- /dev/null
+++ b/test/test-pages/004-metadata-space-separated-properties/source.html
@@ -0,0 +1,35 @@
+
+
+
+
+ Title Element
+
+
+
+
+
+
+
+
+
+
+
Test document title
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+
+
+
\ No newline at end of file
diff --git a/test/test-pages/aclu/config.json b/test/test-pages/aclu/config.json
new file mode 100644
index 00000000..0ac6816e
--- /dev/null
+++ b/test/test-pages/aclu/config.json
@@ -0,0 +1,3 @@
+{
+ "ArticleByLine": true
+}
\ No newline at end of file
diff --git a/test/test-pages/aclu/expected-images.json b/test/test-pages/aclu/expected-images.json
new file mode 100644
index 00000000..0637a088
--- /dev/null
+++ b/test/test-pages/aclu/expected-images.json
@@ -0,0 +1 @@
+[]
\ No newline at end of file
diff --git a/test/test-pages/aclu/expected-metadata.json b/test/test-pages/aclu/expected-metadata.json
new file mode 100644
index 00000000..f954b98d
--- /dev/null
+++ b/test/test-pages/aclu/expected-metadata.json
@@ -0,0 +1,5 @@
+{
+ "Title": "Facebook Is Tracking Me Even Though I’m Not on Facebook",
+ "Author": "By Daniel Kahn Gillmor, Senior Staff Technologist, ACLU Speech, Privacy, and Technology Project",
+ "Excerpt": "I don't use Facebook. I'm not technophobic — I'm a geek. I've been using email since the early 1990s, I have accounts on hundreds of services around the net, and I do software development and internet protocol design both for work and for fun. I believe that a globe-spanning communications network like the internet can be a positive social force, and I publish much of my own work on the open web."
+}
diff --git a/test/test-pages/aclu/expected.html b/test/test-pages/aclu/expected.html
new file mode 100644
index 00000000..15801438
--- /dev/null
+++ b/test/test-pages/aclu/expected.html
@@ -0,0 +1,124 @@
+
+
+ I don't use Facebook. I'm not technophobic — I'm a geek. I've been using email since the early 1990s, I have accounts on hundreds of services around the net, and I do software development and internet protocol design both for work and for fun. I believe that a globe-spanning communications network like the internet can be a positive social force, and I publish much of my own work on the open web.
+
+
+ But Facebook and other massive web companies represent a strong push toward unaccountable centralized social control, which I think makes our society more unequal and more unjust. The Cambridge Analytica scandal is one instance of this long-running problem with what I call the "surveillance economy." I don't want to submit to these power structures, and I don’t want my presence on such platforms to serve as bait that lures other people into the digital panopticon.
+
+
+ But while I've never "opted in" to Facebook or any of the other big social networks, Facebook still has a detailed profile that can be used to target me. I've never consented to having Facebook collect my data, which can be used to draw very detailed inferences about my life, my habits, and my relationships. As we aim to take Facebook to task for its breach of user trust, we need to think about what its capabilities imply for society overall. After all, if you do #deleteFacebook, you'll find yourself in my shoes: non-consenting, but still subject to Facebook’s globe-spanning surveillance and targeting network.
+
+
+ There are at least two major categories of information available to Facebook about non-participants like me: information from other Facebook users, and information from sites on the open web.
+
+
+ Information from other Facebook users
+
+
+ When you sign up for Facebook, it encourages you to upload your list of contacts so that the site can "find your friends." Facebook uses this contact information to learn about people, even if those people don't agree to participate. It also links people together based on who they know, even if the shared contact hasn't agreed to this use.
+
+
+ For example, I received an email from Facebook that lists the people who have all invited me to join Facebook: my aunt, an old co-worker, a friend from elementary school, etc. This email includes names and email addresses — including my own name — and at least one web bug designed to identify me to Facebook’s web servers when I open the email. Facebook records this group of people as my contacts, even though I've never agreed to this kind of data collection.
+
+
+ Similarly, I'm sure that I'm in some photographs that someone has uploaded to Facebook — and I'm probably tagged in some of them. I've never agreed to this, but Facebook could still be keeping track.
+
+
+ So even if you decide you need to join Facebook, remember that you might be giving the company information about someone else who didn't agree to be part of its surveillance platform.
+
+
+ Information from sites on the open Web
+
+
+ Nearly every website that you visit that has a "Like" button is actually encouraging your browser to tell Facebook about your browsing habits. Even if you don't click on the "Like" button, displaying it requires your browser to send a request to Facebook's servers for the "Like" button itself. That request includes information mentioning the name of the page you are visiting and any Facebook-specific cookies your browser might have collected. (See Facebook's own description of this process.) This is called a "third-party request."
+
+
+ This makes it possible for Facebook to create a detailed picture of your browsing history — even if you've never even visited Facebook directly, let alone signed up for a Facebook account.
+
+
+ Think about most of the web pages you've visited — how many of them don't have a "Like" button? If you administer a website and you include a "Like" button on every page, you're helping Facebook to build profiles of your visitors, even those who have opted out of the social network. Facebook’s “Share” buttons on other sites — along with other tools — work a bit differently from the “Like” button, but do effectively the same thing.
+
+
+ The profiles that Facebook builds on non-users don't necessarily include so-called "personally identifiable information" (PII) like names or email addresses. But they do include fairly unique patterns. Using Chromium's NetLog dumping, I performed a simple five-minute browsing test last week that included visits to various sites — but not Facebook. In that test, the PII-free data that was sent to Facebook included information about which news articles I was reading, my dietary preferences, and my hobbies.
+
+
+ Given the precision of this kind of mapping and targeting, "PII" isn’t necessary to reveal my identity. How many vegans examine specifications for computer hardware from the ACLU's offices while reading about Cambridge Analytica? Anyway, if Facebook combined that information with the "web bug" from the email mentioned above — which is clearly linked to my name and e-mail address — no guesswork would be required.
+
+ We use the information we have to improve our advertising and measurement systems so we can show you relevant ads on and off our Services and measure the effectiveness and reach of ads and services.
+
+
+
+ This is, in essence, exactly what Cambridge Analytica did.
+
+
+ Consent
+
+
+ Facebook and other tech companies often deflect accusations against excessive data collection by arguing "consent" — that they harvest and use data with the consent of the users involved.
+
+
+ But even if we accept that clicking through a "Terms of Service" that no one reads can actually constitute true consent, even if we ignore the fact that these terms are overwhelmingly one-sided and non-negotiable, and even if we accept that it's meaningful for people to give consent when sharing data about other people who may have also opted in — what is the recourse for someone who has not opted into these systems at all?
+
+
+ Are those of us who have explicitly avoided agreeing to the Facebook terms of service simply fair game for an industry-wide surveillance and targeting network?
+
+
+ Privilege
+
+
+ I don’t mean to critique people who have created a Facebook profile or suggest they deserve whatever they get.
+
+
+ My ability to avoid Facebook comes from privilege — I have existing social contacts with whom I know how to stay in touch without using Facebook's network. My job does not require that I use Facebook. I can afford the time and expense to communicate with my electoral representatives and political allies via other channels.
+
+
+ Many people do not have these privileges and are compelled to "opt in" on Facebook's non-negotiable terms.
+
+
+ Many journalists, organizers, schools, politicians, and others who have good reasons to oppose Facebook's centralized social control feel compelled by Facebook's reach and scale to participate in their practices, even those we know to be harmful. That includes the ACLU.
+
+
+ Privacy should not be a luxury good, and while I'm happy to encourage people to opt out of these subtle and socially fraught arrangements, I do not argue that anyone who has signed up has somehow relinquished concerns about their privacy. We need to evaluate privacy concerns in their full social contexts. These are not problems that can be resolved on an individual level, because of the interpersonal nature of much of this data and the complexities of the tradeoffs involved.
+
+
+ Technical countermeasures
+
+
+ While they may not solve the problem, there are some technical steps people can take to limit the scope of these surveillance practices. For example, some web browsers do not send "third-party cookies" by default, or they scope cookies so that centralized surveillance doesn't get a single view of one user. The most privacy-preserving modern browser is the Tor Browser, which everyone should have installed and available, even if it's not the browser they choose to use every day. It limits the surveillance ability of systems that you have not signed up for to track you as you move around the web.
+
+ It can’t be denied, though, that these tools are harder to use than the web browsers most people are accustomed to, and they create barriers to some online activities. (For example, logging in to some sites and accessing some web applications is impossible without third-party cookies.)
+
+
+ Some website operators take their visitors' privacy more seriously than others, by reducing the amount of third-party requests. For example, it's possible to display "share on Facebook" or "Like" buttons without sending user requests to Facebook in the first place. The ACLU's own website does this because we believe that the right to read with privacy is a fundamental protection for civic discourse.
+
+
+ If you are responsible for running a website, try browsing it with a third-party-blocking extension turned on. Think about how much information you're requiring your users to send to third parties as a condition for using your site. If you care about being a good steward of your visitors' data, you can re-design your website to reduce this kind of leakage.
+
+
+ Opting out?
+
+
+ Some advertisers claim that you can "opt out" of their targeted advertising, and even offer a centralized place meant to help you do so. However, my experience with these tools isn't a positive one. They don't appear to work all of the time. (In a recent experiment I conducted, two advertisers’ opt-out mechanisms failed to take effect.) And while advertisers claim to allow the user to opt out of "interest-based ads," it's not clear that the opt-outs govern data collection itself, rather than just the use of the collected data for displaying ads. Moreover, opting out on their terms requires the use of third-party cookies, thereby enabling another mechanism that other advertisers can then exploit.
+
+
+ It's also not clear how they function over time: How frequently do I need to take these steps? Do they expire? How often should I check back to make sure I’m still opted out? I'd much prefer an approach requiring me to opt in to surveillance and targeting.
+
+
+ Fix the surveillance economy, not just Facebook
+
+
+ These are just a few of the mechanisms that enable online tracking. Facebook is just one culprit in this online "surveillance economy," albeit a massive one — the company owns Instagram, Atlas, WhatsApp, and dozens of other internet and technology companies and services. But it’s not the only player in this space. Google’s business model also relies on this kind of surveillance, and there are dozens of smaller players as well.
+
+
+ As we work to address the fallout from the current storm around Facebook and Cambridge Analytica, we can't afford to lose sight of these larger mechanisms at play. Cambridge Analytica's failures and mistakes are inherent to Facebook's business model. We need to seriously challenge the social structures that encourage people to opt in to this kind of surveillance. At the same time, we also need to protect those of us who manage to opt out.
+
+
\ No newline at end of file
diff --git a/test/test-pages/aclu/source.html b/test/test-pages/aclu/source.html
new file mode 100644
index 00000000..d8e99bbb
--- /dev/null
+++ b/test/test-pages/aclu/source.html
@@ -0,0 +1,1264 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Facebook Is Tracking Me Even Though I’m Not on Facebook | American Civil Liberties Union
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ The ACLU works tirelessly in courts, legislatures, and communities to defend and preserve the Constitution’s promise of liberty for everyone in our country.
+
+ For almost 100 years, the ACLU has worked to defend and preserve the individual rights and liberties guaranteed by the Constitution and laws of the United States.
+
+ I don't use Facebook. I'm not technophobic — I'm a geek. I've been using email since the early 1990s, I have accounts on hundreds of services around the net, and I do software development and internet protocol design both for work and for fun. I believe that a globe-spanning communications network like the internet can be a positive social force, and I publish much of my own work on the open web.
+
+
+ But Facebook and other massive web companies represent a strong push toward unaccountable centralized social control, which I think makes our society more unequal and more unjust. The Cambridge Analytica scandal is one instance of this long-running problem with what I call the "surveillance economy." I don't want to submit to these power structures, and I don’t want my presence on such platforms to serve as bait that lures other people into the digital panopticon.
+
+
+ But while I've never "opted in" to Facebook or any of the other big social networks, Facebook still has a detailed profile that can be used to target me. I've never consented to having Facebook collect my data, which can be used to draw very detailed inferences about my life, my habits, and my relationships. As we aim to take Facebook to task for its breach of user trust, we need to think about what its capabilities imply for society overall. After all, if you do #deleteFacebook, you'll find yourself in my shoes: non-consenting, but still subject to Facebook’s globe-spanning surveillance and targeting network.
+
+
+ There are at least two major categories of information available to Facebook about non-participants like me: information from other Facebook users, and information from sites on the open web.
+
+
+ Information from other Facebook users
+
+
+ When you sign up for Facebook, it encourages you to upload your list of contacts so that the site can "find your friends." Facebook uses this contact information to learn about people, even if those people don't agree to participate. It also links people together based on who they know, even if the shared contact hasn't agreed to this use.
+
+
+ For example, I received an email from Facebook that lists the people who have all invited me to join Facebook: my aunt, an old co-worker, a friend from elementary school, etc. This email includes names and email addresses — including my own name — and at least one web bug designed to identify me to Facebook’s web servers when I open the email. Facebook records this group of people as my contacts, even though I've never agreed to this kind of data collection.
+
+
+ Similarly, I'm sure that I'm in some photographs that someone has uploaded to Facebook — and I'm probably tagged in some of them. I've never agreed to this, but Facebook could still be keeping track.
+
+
+ So even if you decide you need to join Facebook, remember that you might be giving the company information about someone else who didn't agree to be part of its surveillance platform.
+
+
+ Information from sites on the open Web
+
+
+ Nearly every website that you visit that has a "Like" button is actually encouraging your browser to tell Facebook about your browsing habits. Even if you don't click on the "Like" button, displaying it requires your browser to send a request to Facebook's servers for the "Like" button itself. That request includes information mentioning the name of the page you are visiting and any Facebook-specific cookies your browser might have collected. (See Facebook's own description of this process.) This is called a "third-party request."
+
+
+ This makes it possible for Facebook to create a detailed picture of your browsing history — even if you've never even visited Facebook directly, let alone signed up for a Facebook account.
+
+
+ Think about most of the web pages you've visited — how many of them don't have a "Like" button? If you administer a website and you include a "Like" button on every page, you're helping Facebook to build profiles of your visitors, even those who have opted out of the social network. Facebook’s “Share” buttons on other sites — along with other tools — work a bit differently from the “Like” button, but do effectively the same thing.
+
+
+ The profiles that Facebook builds on non-users don't necessarily include so-called "personally identifiable information" (PII) like names or email addresses. But they do include fairly unique patterns. Using Chromium's NetLog dumping, I performed a simple five-minute browsing test last week that included visits to various sites — but not Facebook. In that test, the PII-free data that was sent to Facebook included information about which news articles I was reading, my dietary preferences, and my hobbies.
+
+
+ Given the precision of this kind of mapping and targeting, "PII" isn’t necessary to reveal my identity. How many vegans examine specifications for computer hardware from the ACLU's offices while reading about Cambridge Analytica? Anyway, if Facebook combined that information with the "web bug" from the email mentioned above — which is clearly linked to my name and e-mail address — no guesswork would be required.
+
+ We use the information we have to improve our advertising and measurement systems so we can show you relevant ads on and off our Services and measure the effectiveness and reach of ads and services.
+
+
+
+ This is, in essence, exactly what Cambridge Analytica did.
+
+
+ Consent
+
+
+ Facebook and other tech companies often deflect accusations against excessive data collection by arguing "consent" — that they harvest and use data with the consent of the users involved.
+
+
+ But even if we accept that clicking through a "Terms of Service" that no one reads can actually constitute true consent, even if we ignore the fact that these terms are overwhelmingly one-sided and non-negotiable, and even if we accept that it's meaningful for people to give consent when sharing data about other people who may have also opted in — what is the recourse for someone who has not opted into these systems at all?
+
+
+ Are those of us who have explicitly avoided agreeing to the Facebook terms of service simply fair game for an industry-wide surveillance and targeting network?
+
+
+ Privilege
+
+
+ I don’t mean to critique people who have created a Facebook profile or suggest they deserve whatever they get.
+
+
+ My ability to avoid Facebook comes from privilege — I have existing social contacts with whom I know how to stay in touch without using Facebook's network. My job does not require that I use Facebook. I can afford the time and expense to communicate with my electoral representatives and political allies via other channels.
+
+
+ Many people do not have these privileges and are compelled to "opt in" on Facebook's non-negotiable terms.
+
+
+ Many journalists, organizers, schools, politicians, and others who have good reasons to oppose Facebook's centralized social control feel compelled by Facebook's reach and scale to participate in their practices, even those we know to be harmful. That includes the ACLU.
+
+
+ Privacy should not be a luxury good, and while I'm happy to encourage people to opt out of these subtle and socially fraught arrangements, I do not argue that anyone who has signed up has somehow relinquished concerns about their privacy. We need to evaluate privacy concerns in their full social contexts. These are not problems that can be resolved on an individual level, because of the interpersonal nature of much of this data and the complexities of the tradeoffs involved.
+
+
+ Technical countermeasures
+
+
+ While they may not solve the problem, there are some technical steps people can take to limit the scope of these surveillance practices. For example, some web browsers do not send "third-party cookies" by default, or they scope cookies so that centralized surveillance doesn't get a single view of one user. The most privacy-preserving modern browser is the Tor Browser, which everyone should have installed and available, even if it's not the browser they choose to use every day. It limits the surveillance ability of systems that you have not signed up for to track you as you move around the web.
+
+ It can’t be denied, though, that these tools are harder to use than the web browsers most people are accustomed to, and they create barriers to some online activities. (For example, logging in to some sites and accessing some web applications is impossible without third-party cookies.)
+
+
+ Some website operators take their visitors' privacy more seriously than others, by reducing the amount of third-party requests. For example, it's possible to display "share on Facebook" or "Like" buttons without sending user requests to Facebook in the first place. The ACLU's own website does this because we believe that the right to read with privacy is a fundamental protection for civic discourse.
+
+
+ If you are responsible for running a website, try browsing it with a third-party-blocking extension turned on. Think about how much information you're requiring your users to send to third parties as a condition for using your site. If you care about being a good steward of your visitors' data, you can re-design your website to reduce this kind of leakage.
+
+
+ Opting out?
+
+
+ Some advertisers claim that you can "opt out" of their targeted advertising, and even offer a centralized place meant to help you do so. However, my experience with these tools isn't a positive one. They don't appear to work all of the time. (In a recent experiment I conducted, two advertisers’ opt-out mechanisms failed to take effect.) And while advertisers claim to allow the user to opt out of "interest-based ads," it's not clear that the opt-outs govern data collection itself, rather than just the use of the collected data for displaying ads. Moreover, opting out on their terms requires the use of third-party cookies, thereby enabling another mechanism that other advertisers can then exploit.
+
+
+ It's also not clear how they function over time: How frequently do I need to take these steps? Do they expire? How often should I check back to make sure I’m still opted out? I'd much prefer an approach requiring me to opt in to surveillance and targeting.
+
+
+ Fix the surveillance economy, not just Facebook
+
+
+ These are just a few of the mechanisms that enable online tracking. Facebook is just one culprit in this online "surveillance economy," albeit a massive one — the company owns Instagram, Atlas, WhatsApp, and dozens of other internet and technology companies and services. But it’s not the only player in this space. Google’s business model also relies on this kind of surveillance, and there are dozens of smaller players as well.
+
+
+ As we work to address the fallout from the current storm around Facebook and Cambridge Analytica, we can't afford to lose sight of these larger mechanisms at play. Cambridge Analytica's failures and mistakes are inherent to Facebook's business model. We need to seriously challenge the social structures that encourage people to opt in to this kind of surveillance. At the same time, we also need to protect those of us who manage to opt out.
+
+
+
+
+
+ View comments (27)
+
+
+
+
+ Read the Terms of Use
+
+
+
+
+
+
+
+
+
+ Anonymous
+
+
+
+ One valuable reform would be to mandate, under penalty of felony, that anyone that searches your information has to notify you - to include private investigotors screening potential employers to police officers trolling your online profile. It would be a great consumer protection.
+
+
+ For example: if you lost the job offer due to your Facebook post, the employer should be required to tell you that. If you got pulled over and ticketed because you "Liked" police body cameras on FB you should know that also.
+
+
+ The real danger is allowing punitive actions, in retaliation of your Free Speech to be hidden from us. If someone is spying on us, bypassing privacy filters, without a judicial warrant - we should know that.
+
+ The U.S. Justice Department and U.S. Congress - more than a decade behind schedule - still haven't reconciled the "Joseph Nacchio" case. Nacchio's telecommunications company, Qwest Communications, was apparently the one and only telecommunications company that refused to participate in the felony crime of Warrantless Wire-tapping around February of 2001 (roughly 6 months BEFORE 9/11 when no wartime emergency existed)., being pressured by the Bush Administration to commit a felony under federal law.
+
+
+ Congress then passed an ex post facto law - illegal under Article I of the U.S. Constitution - making those past felonies legal. Congress has no such authority under Article VI.
+
+
+ Joseph Nacchio claims he was then indicted by the U.S. Department of Justice - on trumped up charges - and served a few years in federal prison. A CEO of a large American corporation was sent to prison for COMPLYING with federal wiretaping laws. The Washington Post ran a story after Nacchio was released from prison, he still claims to this day that he went to prison for not participating in Bush felony crimes.
+
+
+ What is to prevent a future lawless Executive Branch from pressuring Facebook or other social media networks to also bypass Judicial Warrants? If an honest U.S. Attorney or state prosecutor started indicting the former Bush folks, it would create a powerful "deterrent effect" to lawless officials in the future.
+
+ There is strong circumstantial evidence that "Post 9/11 Blacklisting" of U.S. citizens is still alive and well. Maybe the most dangerous blacklisting tactic is "Employment Tampering". It basically means if you were blacklisted after 9/11, you may not be allowed to work in certain occupations. The Government essentially decides where you are allowed work, without an official hearing by judge - blacklistees are never informed of their 2nd Class status as Americans - not entitled to the same rights as other Americans.
+
+
+ "Employment Tampering" by government officials/contractors - which violates Title 18 US Code 242 - is one of the hardest blacklisting tactics for federal prosecutors to police. We may even need a new federal watchdog agency to properly police this felony crime by officials.
+
+
+ Maybe the most famous case, was exposed by a federal appeals court that severely reprimanded former Attorney General John Ashcroft (although he was never criminally indicted). Ashcroft grossly and fraudulently exploited the federal "Material Witness Statute" to be used as a quasi form of imprisonment and involntary servitude. The 13th Amendment makes this practice totally illegal without an official and overt constitutional due process.
+
+
+ This tactic has a long history in authoritarian regimes. It was originally called "Zersetzung" and was first used in 1940's Germany during World War Two to silence dissent. During the Cold War, this tactic was adopted by then Communist East Germany. The FBI started using it in the 1950's commonly called Cointelpro tactics. Ashcroft then exploited the Material Witness Statute to be used as a Zersetzung tactic. Employment Tampering was probably the most potent tactic.
+
+ Didn't you read the article? You might not use it, but they still know all about you, even including name, address, phone number, etc. if you are in someone's contacts.
+
+
+ Ironically, by refusing to register with them as a user, you have no control at all over the collection and usage of your data and information. At least users have some little control over it.
+
+
+
+
+
+
+
+ April 10, 2018
+
+
+ 10:14 PM
+
+
+
+
+
+
+
+
+
+ Ed B. -Netherlands
+
+
+
+ Well written, well-thought article. I agree with the content. Although I hate regulation in general, it seems about time that website should be required to offer a cookie free, traceless version at all times.
+
+ HTTP is stateless. That means that when you click on a link to another page, the second page doesn't know that you were the person on page 1. Cookies, or some form of tracking, are needed for things like eCommerce and shopping carts. You need a cookie to associate *you* with your cart. Anything where you log in needs a cookie in order to know you are you, such as subscriptions.
+
+
+ However, those cookies can easily be "session cookies". That is, cookies that don't persist after you log out or close your web browser. They exists solely for the duration of your session, and do not persist.
+
+
+ Cookies are only an association mechanism. The heart of the problem is the harvesting of every click and everything on a page -- the over-collection of data to begin with.
+
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat.
-
Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
diff --git a/test/test-pages/bbc-1/expected-images.json b/test/test-pages/bbc-1/expected-images.json
index 58269b18..137b4225 100644
--- a/test/test-pages/bbc-1/expected-images.json
+++ b/test/test-pages/bbc-1/expected-images.json
@@ -1 +1 @@
-{"0":"http:\/\/ichef.bbci.co.uk\/news\/1024\/cpsprodpb\/3D8B\/production\/_84455751_84455749.jpg","1":"http:\/\/ichef.bbci.co.uk\/news\/555\/cpsprodpb\/462D\/production\/_84456971_gettyimages-167501087.jpg","2":"http:\/\/ichef.bbci.co.uk\/news\/555\/media\/images\/76020000\/jpg\/_76020974_line976.jpg","3":"http:\/\/ichef-1.bbci.co.uk\/news\/555\/cpsprodpb\/6D3D\/production\/_84456972_p072315al-0500.jpg","5":"http:\/\/ichef-1.bbci.co.uk\/news\/555\/cpsprodpb\/142FD\/production\/_84458628_shirtreuters.jpg"}
\ No newline at end of file
+{"0":"http:\/\/ichef.bbci.co.uk\/news\/555\/cpsprodpb\/462D\/production\/_84456971_gettyimages-167501087.jpg","1":"http:\/\/ichef.bbci.co.uk\/news\/555\/media\/images\/76020000\/jpg\/_76020974_line976.jpg","2":"http:\/\/ichef-1.bbci.co.uk\/news\/555\/cpsprodpb\/6D3D\/production\/_84456972_p072315al-0500.jpg","4":"http:\/\/ichef-1.bbci.co.uk\/news\/555\/cpsprodpb\/142FD\/production\/_84458628_shirtreuters.jpg"}
\ No newline at end of file
diff --git a/test/test-pages/bbc-1/expected-metadata.json b/test/test-pages/bbc-1/expected-metadata.json
index 7569b8f6..545e3171 100644
--- a/test/test-pages/bbc-1/expected-metadata.json
+++ b/test/test-pages/bbc-1/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "Obama admits US gun laws are his 'biggest frustration'",
+ "Title": "Obama admits US gun laws are his 'biggest frustration' - BBC News",
"Author": null,
- "Excerpt": "President Barack Obama tells the BBC his failure to pass"
+ "Excerpt": "President Barack Obama tells the BBC his failure to pass \"common sense gun safety laws\" is the greatest frustration of his presidency."
}
diff --git a/test/test-pages/blogger/expected-images.json b/test/test-pages/blogger/expected-images.json
index fc0d8bfa..aa14e4f8 100644
--- a/test/test-pages/blogger/expected-images.json
+++ b/test/test-pages/blogger/expected-images.json
@@ -1 +1 @@
-["https:\/\/1.bp.blogspot.com\/-YIPC5jkXkDE\/Vy7YPSqFKWI\/AAAAAAAAAxI\/a7D6Ji2GxoUvcrwUkI4RLZcr2LFQEJCTACLcB\/w1200-h630-p-nu\/block-diagram.png","https:\/\/1.bp.blogspot.com\/-YIPC5jkXkDE\/Vy7YPSqFKWI\/AAAAAAAAAxI\/a7D6Ji2GxoUvcrwUkI4RLZcr2LFQEJCTACLcB\/s640\/block-diagram.png","https:\/\/1.bp.blogspot.com\/-k3naUT3uXao\/Vy7WFac246I\/AAAAAAAAAw8\/mePy_ostO8QJra5ZJrbP2WGhTlJ0B_r8gCLcB\/s640\/schematic-from-hell.png","https:\/\/2.bp.blogspot.com\/-kIekczO693g\/Vy7dBqYifXI\/AAAAAAAAAxc\/hMNJBs5bedIQOrBzzkhq4gbmhR-n58EQwCLcB\/s400\/graph-labels.png"]
\ No newline at end of file
+{"0":"https:\/\/1.bp.blogspot.com\/-YIPC5jkXkDE\/Vy7YPSqFKWI\/AAAAAAAAAxI\/a7D6Ji2GxoUvcrwUkI4RLZcr2LFQEJCTACLcB\/s640\/block-diagram.png","2":"https:\/\/1.bp.blogspot.com\/-k3naUT3uXao\/Vy7WFac246I\/AAAAAAAAAw8\/mePy_ostO8QJra5ZJrbP2WGhTlJ0B_r8gCLcB\/s640\/schematic-from-hell.png","3":"https:\/\/2.bp.blogspot.com\/-kIekczO693g\/Vy7dBqYifXI\/AAAAAAAAAxc\/hMNJBs5bedIQOrBzzkhq4gbmhR-n58EQwCLcB\/s400\/graph-labels.png"}
\ No newline at end of file
diff --git a/test/test-pages/blogger/expected.html b/test/test-pages/blogger/expected.html
index 10b48746..ec3a02a4 100644
--- a/test/test-pages/blogger/expected.html
+++ b/test/test-pages/blogger/expected.html
@@ -1,27 +1,27 @@
- I've written a couple of posts in the past few months but they were all for
so I figured I'm long overdue for one on Silicon Exposed.
+ I've written a couple of posts in the past few months but they were all for the blog at work so I figured I'm long overdue for one on Silicon Exposed.
So what's a GreenPak?
-
Silego Technology is a fabless semiconductor company located in the SF Bay area, which makes (among other things) a line of programmable logic devices known as GreenPak. Their
GreenPak devices are kind of like itty bitty PSoCs - they have a mixed signal fabric with an ADC, DACs, comparators, voltage references, plus a digital LUT/FF fabric and some typical digital MCU peripherals like counters and oscillators (but no CPU).
It's actually an interesting architecture - FPGAs (including some devices marketed as CPLDs) are a 2D array of LUTs connected via wires to adjacent cells, and true (product term) CPLDs are a star topology of AND-OR arrays connected by a crossbar. GreenPak, on the other hand, is a star topology of LUTs, flipflops, and analog/digital hard IP connected to a crossbar.
Without further ado, here's a block diagram showing all the cool stuff you get in the SLG46620V:
+
Silego Technology is a fabless semiconductor company located in the SF Bay area, which makes (among other things) a line of programmable logic devices known as GreenPak. Their 5th generation parts were just announced, but I started this project before that happened so I'm still targeting the 4th generation.
GreenPak devices are kind of like itty bitty PSoCs - they have a mixed signal fabric with an ADC, DACs, comparators, voltage references, plus a digital LUT/FF fabric and some typical digital MCU peripherals like counters and oscillators (but no CPU).
It's actually an interesting architecture - FPGAs (including some devices marketed as CPLDs) are a 2D array of LUTs connected via wires to adjacent cells, and true (product term) CPLDs are a star topology of AND-OR arrays connected by a crossbar. GreenPak, on the other hand, is a star topology of LUTs, flipflops, and analog/digital hard IP connected to a crossbar.
Without further ado, here's a block diagram showing all the cool stuff you get in the SLG46620V:
SLG46620V block diagram (from device datasheet)
-
- They're also tiny (the SLG46620V is a 20-pin 0.4mm pitch STQFN measuring 2x3 mm, and the lower gate count SLG46140V is a mere 1.6x2 mm) and probably the cheapest programmable logic device on the market - $0.50 in low volume and less than $0.40 in larger quantities.
The Vdd range of GreenPak4 is huge, more like what you'd expect from an MCU than an FPGA! It can run on anything from 1.8 to 5V, although performance is only specified at 1.8, 3.3, and 5V nominal voltages. There's also a dual-rail version that trades one of the GPIO pins for a second power supply pin, allowing you to interface to logic at two different voltage levels.
To support low-cost/space-constrained applications, they even have the configuration memory on die. It's one-time programmable and needs external Vpp to program (presumably Silego didn't want to waste die area on charge pumps that would only be used once) but has a SRAM programming mode for prototyping.
The best part is that the development software (GreenPak Designer) is free of charge and provided for all major operating systems including Linux! Unfortunately, the only supported design entry method is schematic entry and there's no way to write your design in a HDL.
While schematics may be fine for quick tinkering on really simple designs, they quickly get unwieldy. The nightmare of a circuit shown below is just a bunch of counters hooked up to LEDs that blink at various rates.
+
+ They're also tiny (the SLG46620V is a 20-pin 0.4mm pitch STQFN measuring 2x3 mm, and the lower gate count SLG46140V is a mere 1.6x2 mm) and probably the cheapest programmable logic device on the market - $0.50 in low volume and less than $0.40 in larger quantities.
The Vdd range of GreenPak4 is huge, more like what you'd expect from an MCU than an FPGA! It can run on anything from 1.8 to 5V, although performance is only specified at 1.8, 3.3, and 5V nominal voltages. There's also a dual-rail version that trades one of the GPIO pins for a second power supply pin, allowing you to interface to logic at two different voltage levels.
To support low-cost/space-constrained applications, they even have the configuration memory on die. It's one-time programmable and needs external Vpp to program (presumably Silego didn't want to waste die area on charge pumps that would only be used once) but has a SRAM programming mode for prototyping.
The best part is that the development software (GreenPak Designer) is free of charge and provided for all major operating systems including Linux! Unfortunately, the only supported design entry method is schematic entry and there's no way to write your design in a HDL.
While schematics may be fine for quick tinkering on really simple designs, they quickly get unwieldy. The nightmare of a circuit shown below is just a bunch of counters hooked up to LEDs that blink at various rates.
Schematic from hell!
-
- As if this wasn't enough of a problem, the largest GreenPak4 device (the SLG46620V) is split into two halves with limited routing between them, and the GUI doesn't help the user manage this complexity at all - you have to draw your schematic in two halves and add "cross connections" between them.
The icing on the cake is that schematics are a pain to diff and collaborate on. Although GreenPak schematics are XML based, which is a touch better than binary, who wants to read a giant XML diff and try to figure out what's going on in the circuit?
This isn't going to be a post on the quirks of Silego's software, though - that would be boring. As it turns out, there's one more exciting feature of these chips that I didn't mention earlier: the configuration bitstream is 100% documented in the device datasheet! This is unheard of in the programmable logic world. As Nick of Arachnid Labs says, the chip is "just dying for someone to write a VHDL or Verilog compiler for it". As you can probably guess by from the title of this post, I've been busy doing exactly that.
+
+ As if this wasn't enough of a problem, the largest GreenPak4 device (the SLG46620V) is split into two halves with limited routing between them, and the GUI doesn't help the user manage this complexity at all - you have to draw your schematic in two halves and add "cross connections" between them.
The icing on the cake is that schematics are a pain to diff and collaborate on. Although GreenPak schematics are XML based, which is a touch better than binary, who wants to read a giant XML diff and try to figure out what's going on in the circuit?
This isn't going to be a post on the quirks of Silego's software, though - that would be boring. As it turns out, there's one more exciting feature of these chips that I didn't mention earlier: the configuration bitstream is 100% documented in the device datasheet! This is unheard of in the programmable logic world. As Nick of Arachnid Labs says, the chip is "just dying for someone to write a VHDL or Verilog compiler for it". As you can probably guess by from the title of this post, I've been busy doing exactly that.
Great! How does it work?
-
Rather than wasting time writing a synthesizer, I decided to write a GreenPak technology library for Clifford Wolf's excellent open source synthesis tool,
, and then make a place-and-route tool to turn that into a final netlist. The post-PAR netlist can then be loaded into GreenPak Designer in order to program the device.
The first step of the process is to run the "synth_greenpak4" Yosys flow on the Verilog source. This runs a generic RTL synthesis pass, then some coarse-grained extraction passes to infer shift register and counter cells from behavioral logic, and finally maps the remaining logic to LUT/FF cells and outputs a JSON-formatted netlist.
Once the design has been synthesized, my tool (named, surprisingly, gp4par) is then launched on the netlist. It begins by parsing the JSON and constructing a directed graph of cell objects in memory. A second graph, containing all of the primitives in the device and the legal connections between them, is then created based on the device specified on the command line. (As of now only the SLG46620V is supported; the SLG46621V can be added fairly easily but the SLG46140V has a slightly different microarchitecture which will require a bit more work to support.)
After the graphs are generated, each node in the netlist graph is assigned a numeric label identifying the type of cell and each node in the device graph is assigned a list of legal labels: for example, an I/O buffer site is legal for an input buffer, output buffer, or bidirectional buffer.
+
Rather than wasting time writing a synthesizer, I decided to write a GreenPak technology library for Clifford Wolf's excellent open source synthesis tool, Yosys, and then make a place-and-route tool to turn that into a final netlist. The post-PAR netlist can then be loaded into GreenPak Designer in order to program the device.
The first step of the process is to run the "synth_greenpak4" Yosys flow on the Verilog source. This runs a generic RTL synthesis pass, then some coarse-grained extraction passes to infer shift register and counter cells from behavioral logic, and finally maps the remaining logic to LUT/FF cells and outputs a JSON-formatted netlist.
Once the design has been synthesized, my tool (named, surprisingly, gp4par) is then launched on the netlist. It begins by parsing the JSON and constructing a directed graph of cell objects in memory. A second graph, containing all of the primitives in the device and the legal connections between them, is then created based on the device specified on the command line. (As of now only the SLG46620V is supported; the SLG46621V can be added fairly easily but the SLG46140V has a slightly different microarchitecture which will require a bit more work to support.)
After the graphs are generated, each node in the netlist graph is assigned a numeric label identifying the type of cell and each node in the device graph is assigned a list of legal labels: for example, an I/O buffer site is legal for an input buffer, output buffer, or bidirectional buffer.
Example labeling for a subset of the netlist and device graphs
-
- The labeled nodes now need to be placed. The initial placement uses a simple greedy algorithm to create a valid (although not necessarily optimal or even routable) placement:
Loop over the cells in the netlist. If any cell has a LOC constraint, which locks the cell to a specific physical site, attempt to assign the node to the specified site. If the specified node is the wrong type, doesn't exist, or is already used by another constrained node, the constraint is invalid so fail with an error.
+
+ The labeled nodes now need to be placed. The initial placement uses a simple greedy algorithm to create a valid (although not necessarily optimal or even routable) placement:
Loop over the cells in the netlist. If any cell has a LOC constraint, which locks the cell to a specific physical site, attempt to assign the node to the specified site. If the specified node is the wrong type, doesn't exist, or is already used by another constrained node, the constraint is invalid so fail with an error.
Loop over all of the unconstrained cells in the netlist and assign them to the first unused site with the right label. If none are available, the design is too big for the device so fail with an error.
- Once the design is placed, the placement optimizer then loops over the design and attempts to improve it. A simulated annealing algorithm is used, where changes to the design are accepted unconditionally if they make the placement better, and with a random, gradually decreasing probability if they make it worse. The optimizer terminates when the design receives a perfect score (indicating an optimal placement) or if it stops making progress for several iterations. Each iteration does the following:
Compute a score for the current design based on the number of unroutable nets, the amount of routing congestion (number of nets crossing between halves of the device), and static timing analysis (not yet implemented, always zero).
+ Once the design is placed, the placement optimizer then loops over the design and attempts to improve it. A simulated annealing algorithm is used, where changes to the design are accepted unconditionally if they make the placement better, and with a random, gradually decreasing probability if they make it worse. The optimizer terminates when the design receives a perfect score (indicating an optimal placement) or if it stops making progress for several iterations. Each iteration does the following:
Compute a score for the current design based on the number of unroutable nets, the amount of routing congestion (number of nets crossing between halves of the device), and static timing analysis (not yet implemented, always zero).
Make a list of nodes that contributed to this score in some way (having some attached nets unroutable, crossing to the other half of the device, or failing timing).
Remove nodes from the list that are LOC'd to a specific location since we're not allowed to move them.
Remove nodes from the list that have only one legal placement in the device (for example, oscillator hard IP) since there's nowhere else for them to go.
@@ -34,24 +34,24 @@
Re-compute the score for the design. If it's better, accept this change and start the next iteration.
If the score is worse, accept it with a random probability which decreases as the iteration number goes up. If the change is not accepted, restore the previous placement.
- After optimization, the design is checked for routability. If any edges in the netlist graph don't correspond to edges in the device graph, the user probably asked for something impossible (for example, trying to hook a flipflop's output to a comparator's reference voltage input) so fail with an error.
The design is then routed. This is quite simple due to the crossbar structure of the device. For each edge in the netlist:
If dedicated (non-fabric) routing is used for this path, configure the destination's input mux appropriately and stop.
+ After optimization, the design is checked for routability. If any edges in the netlist graph don't correspond to edges in the device graph, the user probably asked for something impossible (for example, trying to hook a flipflop's output to a comparator's reference voltage input) so fail with an error.
The design is then routed. This is quite simple due to the crossbar structure of the device. For each edge in the netlist:
If dedicated (non-fabric) routing is used for this path, configure the destination's input mux appropriately and stop.
If the source and destination are in the same half of the device, configure the destination's input mux appropriately and stop.
A cross-connection must be used. Check if we already used one to bring the source signal to the other half of the device. If found, configure the destination to route from that cross-connection and stop.
Check if we have any cross-connections left going in this direction. If they're all used, the design is unroutable due to congestion so fail with an error.
Pick the next unused cross-connection and configure it to route from the source. Configure the destination to route from the cross-connection and stop.
- Once routing is finished, run a series of post-PAR design rule checks. These currently include the following:
If any node has no loads, generate a warning
+ Once routing is finished, run a series of post-PAR design rule checks. These currently include the following:
If any node has no loads, generate a warning
If an I/O buffer is connected to analog hard IP, fail with an error if it's not configured in analog mode.
Some signals (such as comparator inputs and oscillator power-down controls) are generated by a shared mux and fed to many loads. If different loads require conflicting settings for the shared mux, fail with an error.
- If DRC passes with no errors, configure all of the individual cells in the netlist based on the HDL parameters. Fail with an error if an invalid configuration was requested.
Finally, generate the bitstream from all of the per-cell configuration and write it to a file.
- Great, let's get started!
- If you don't already have one, you'll need to buy a GreenPak4 development kit. The kit includes samples of the SLG46620V (among other devices) and a programmer/emulation board. While you're waiting for it to arrive, install GreenPak Designer.
Download and install Yosys. Although Clifford is pretty good at merging my pull requests, only my fork on Github is guaranteed to have the most up-to-date support for GreenPak devices so don't be surprised if you can't use a bleeding-edge feature with mainline Yosys.
Write your HDL, compile with Yosys, P&R with gp4par, and import the bitstream into GreenPak Designer to program the target device. The most current gp4par manual is included in LaTeX source form in the source tree and is automatically built as part of the compile process. If you're just browsing, there's a relatively recent PDF version on my web server.
If you'd like to see the Verilog that produced the nightmare of a schematic I showed above, here it is.
Be advised that this project is still very much a work in progress and there are still a number of SLG46620V features I don't support (see the manual for exact details).
- I love it / it segfaulted / there's a problem in the manual!
- Hop in our IRC channel (##openfpga on Freenode) and let me know. Feedback is great, pull requests are even better,
- You're competing with Silego's IDE. Have they found out and sued you yet?
- Nope. They're fully aware of what I'm doing and are rolling out the red carpet for me. They love the idea of a HDL flow as an alternative to schematic entry and are pretty amazed at how fast it's coming together.
After I reported a few bugs in their datasheets they decided to skip the middleman and give me direct access to the engineer who writes their documentation so that I can get faster responses. The last time I found a problem (two different parts of the datasheet contradicted each other) an updated datasheet was in my inbox and on their website by the next day. I only wish Xilinx gave me that kind of treatment!
They've even offered me free hardware to help me add support for their latest product family, although I plan to get GreenPak4 support to a more stable state before taking them up on the offer.
+ If DRC passes with no errors, configure all of the individual cells in the netlist based on the HDL parameters. Fail with an error if an invalid configuration was requested.
Finally, generate the bitstream from all of the per-cell configuration and write it to a file.
+ Great, let's get started!
+ If you don't already have one, you'll need to buy a GreenPak4 development kit. The kit includes samples of the SLG46620V (among other devices) and a programmer/emulation board. While you're waiting for it to arrive, install GreenPak Designer.
Download and install Yosys. Although Clifford is pretty good at merging my pull requests, only my fork on Github is guaranteed to have the most up-to-date support for GreenPak devices so don't be surprised if you can't use a bleeding-edge feature with mainline Yosys.
Write your HDL, compile with Yosys, P&R with gp4par, and import the bitstream into GreenPak Designer to program the target device. The most current gp4par manual is included in LaTeX source form in the source tree and is automatically built as part of the compile process. If you're just browsing, there's a relatively recent PDF version on my web server.
If you'd like to see the Verilog that produced the nightmare of a schematic I showed above, here it is.
Be advised that this project is still very much a work in progress and there are still a number of SLG46620V features I don't support (see the manual for exact details).
+ I love it / it segfaulted / there's a problem in the manual!
+ Hop in our IRC channel (##openfpga on Freenode) and let me know. Feedback is great, pull requests are even better,
+ You're competing with Silego's IDE. Have they found out and sued you yet?
+ Nope. They're fully aware of what I'm doing and are rolling out the red carpet for me. They love the idea of a HDL flow as an alternative to schematic entry and are pretty amazed at how fast it's coming together.
After I reported a few bugs in their datasheets they decided to skip the middleman and give me direct access to the engineer who writes their documentation so that I can get faster responses. The last time I found a problem (two different parts of the datasheet contradicted each other) an updated datasheet was in my inbox and on their website by the next day. I only wish Xilinx gave me that kind of treatment!
They've even offered me free hardware to help me add support for their latest product family, although I plan to get GreenPak4 support to a more stable state before taking them up on the offer.
So what's next?
-
Better testing, for starters. I have to verify functionality by hand with a DMM and oscilloscope, which is time consuming.
My contact at Silego says they're going to be giving me documentation on the SRAM emulation interface soon, so I'm going to make a hardware-in-loop test platform that connects to my desktop and the Silego ZIF socket, and lets me load new bitstreams via a scriptable interface. It'll have FPGA-based digital I/O as well as an ADC and DAC on every device pin, plus an adjustable voltage regulator for power, so I can feed in arbitrary mixed-signal test waveforms and write PC-based unit tests to verify correct behavior.
Other than that, I want to finish support for the SLG46620V in the next month or two. The SLG46621V will be an easy addition since only one pin and the relevant configuration bits have changed from the 46620 (I suspect they're the same die, just bonded out differently).
Once that's done I'll have to do some more extensive work to add the SLG46140V since the architecture is a bit different (a lot of the combinatorial logic is merged into multi-function blocks). Luckily, the 46140 has a lot in common architecturally with the GreenPak5 family, so once that's done GreenPak5 will probably be a lot easier to add support for.
My thanks go out to Clifford Wolf, whitequark, the IRC users in ##openfpga, and everyone at Silego I've worked with to help make this possible. I hope that one day this project will become mature enough that Silego will ship it as an officially supported extension to GreenPak Designer, making history by becoming the first modern programmable logic vendor to ship a fully open source synthesis and P&R suite.
-
-
\ No newline at end of file
+
Better testing, for starters. I have to verify functionality by hand with a DMM and oscilloscope, which is time consuming.
My contact at Silego says they're going to be giving me documentation on the SRAM emulation interface soon, so I'm going to make a hardware-in-loop test platform that connects to my desktop and the Silego ZIF socket, and lets me load new bitstreams via a scriptable interface. It'll have FPGA-based digital I/O as well as an ADC and DAC on every device pin, plus an adjustable voltage regulator for power, so I can feed in arbitrary mixed-signal test waveforms and write PC-based unit tests to verify correct behavior.
Other than that, I want to finish support for the SLG46620V in the next month or two. The SLG46621V will be an easy addition since only one pin and the relevant configuration bits have changed from the 46620 (I suspect they're the same die, just bonded out differently).
Once that's done I'll have to do some more extensive work to add the SLG46140V since the architecture is a bit different (a lot of the combinatorial logic is merged into multi-function blocks). Luckily, the 46140 has a lot in common architecturally with the GreenPak5 family, so once that's done GreenPak5 will probably be a lot easier to add support for.
My thanks go out to Clifford Wolf, whitequark, the IRC users in ##openfpga, and everyone at Silego I've worked with to help make this possible. I hope that one day this project will become mature enough that Silego will ship it as an officially supported extension to GreenPak Designer, making history by becoming the first modern programmable logic vendor to ship a fully open source synthesis and P&R suite.
+
+
\ No newline at end of file
diff --git a/test/test-pages/breitbart/expected-images.json b/test/test-pages/breitbart/expected-images.json
index 81ca6a4c..0c7f5707 100644
--- a/test/test-pages/breitbart/expected-images.json
+++ b/test/test-pages/breitbart/expected-images.json
@@ -1 +1 @@
-["http:\/\/media.breitbart.com\/media\/2016\/11\/GettyImages-621866810.jpg","http:\/\/media.breitbart.com\/media\/2016\/11\/GettyImages-621866810-640x480.jpg"]
\ No newline at end of file
+["http:\/\/media.breitbart.com\/media\/2016\/11\/GettyImages-621866810-640x480.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/breitbart/expected-metadata.json b/test/test-pages/breitbart/expected-metadata.json
index 2b968e75..8bbbd411 100644
--- a/test/test-pages/breitbart/expected-metadata.json
+++ b/test/test-pages/breitbart/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "'Neutral' Snopes Fact-Checker David Emery: 'Are There Any Un-Angry Trump Supporters?'",
+ "Title": "'Neutral' Snopes Fact-Checker David Emery: 'Are There Any Un-Angry Trump Supporters?' - Breitbart",
"Author": "by Lucas Nolan22 Dec 2016651",
"Direction": null,
"Excerpt": "Snopes fact checker and staff writer David Emery posted to Twitter asking if there were “any un-angry Trump supporters?”"
diff --git a/test/test-pages/breitbart/expected.html b/test/test-pages/breitbart/expected.html
index eabc20da..e2f04c74 100644
--- a/test/test-pages/breitbart/expected.html
+++ b/test/test-pages/breitbart/expected.html
@@ -1,7 +1,7 @@
-
JIM WATSON/AFP/Getty Images
+
JIM WATSON/AFP/Getty Images
diff --git a/test/test-pages/bug-1255978/expected-images.json b/test/test-pages/bug-1255978/expected-images.json
index f6ff5d9f..c4dc5637 100644
--- a/test/test-pages/bug-1255978/expected-images.json
+++ b/test/test-pages/bug-1255978/expected-images.json
@@ -1 +1 @@
-["https:\/\/static.independent.co.uk\/s3fs-public\/thumbnails\/image\/2015\/12\/06\/10\/bed-hotel-room.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2014\/03\/18\/10\/bandb2.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2015\/05\/26\/11\/hotel-door-getty.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2013\/07\/31\/15\/luggage-3.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2015\/04\/13\/11\/Lifestyle-hotels.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2014\/03\/13\/16\/agenda7.jpg","http:\/\/fakehost\/sites\/all\/themes\/ines_themes\/independent_theme\/img\/reuse.png"]
\ No newline at end of file
+["https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2014\/03\/18\/10\/bandb2.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2015\/05\/26\/11\/hotel-door-getty.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2013\/07\/31\/15\/luggage-3.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2015\/04\/13\/11\/Lifestyle-hotels.jpg","https:\/\/static.independent.co.uk\/s3fs-public\/styles\/story_medium\/public\/thumbnails\/image\/2014\/03\/13\/16\/agenda7.jpg","http:\/\/fakehost\/sites\/all\/themes\/ines_themes\/independent_theme\/img\/reuse.png"]
\ No newline at end of file
diff --git a/test/test-pages/bug-1255978/expected-metadata.json b/test/test-pages/bug-1255978/expected-metadata.json
index 0acbbdf0..db7c6524 100644
--- a/test/test-pages/bug-1255978/expected-metadata.json
+++ b/test/test-pages/bug-1255978/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "The seven secrets that hotel owners don't want you to know",
+ "Title": "Seven secrets that hotel owners don't want you to know",
"Author": "Hazel Sheffield",
"Direction": null,
"Excerpt": "Most people go to hotels for the pleasure of sleeping in a giant bed with clean white sheets and waking up to fresh towels in the morning. But those towels and sheets might not be as clean as they look, according to the hotel bosses that responded to an online thread about the things hotel owners don’t want you to know."
diff --git a/test/test-pages/bug-1255978/expected.html b/test/test-pages/bug-1255978/expected.html
index 78f2f890..c281eef9 100644
--- a/test/test-pages/bug-1255978/expected.html
+++ b/test/test-pages/bug-1255978/expected.html
@@ -14,13 +14,13 @@
-
+
-
- Even posh hotels might not wash a blanket in between stays
-
-
+
+
Even posh hotels might not wash a blanket in between stays
+
+
1. Take any blankets or duvets off the bed
@@ -37,13 +37,13 @@
-
+
-
- Forrest Jones advised stuffing the peep hole with a strip of rolled up notepaper when not in use.
-
-
+
+
Forrest Jones advised stuffing the peep hole with a strip of rolled up notepaper when not in use.
+
+
2. Check the peep hole has not been tampered with
@@ -55,13 +55,10 @@
-
+
-
- Put luggage on the floor
-
-
+
3. Don’t use a wooden luggage rack
@@ -73,13 +70,13 @@
-
+
-
- The old rule of thumb is that for every 00 invested in a room, the hotel should charge in average daily rate
-
-
+
+
The old rule of thumb is that for every 00 invested in a room, the hotel should charge in average daily rate
+
+
4. Hotel rooms are priced according to how expensive they were to build
@@ -117,13 +114,13 @@
6. Mini bars almost always lose money
-
+
-
- Towels should always be cleaned between stays
-
-
+
+
Towels should always be cleaned between stays
+
+
7. Always made sure the hand towels are clean when you arrive
\ No newline at end of file
diff --git a/test/test-pages/buzzfeed-1/expected-images.json b/test/test-pages/buzzfeed-1/expected-images.json
index 1ee3a588..3064b985 100644
--- a/test/test-pages/buzzfeed-1/expected-images.json
+++ b/test/test-pages/buzzfeed-1/expected-images.json
@@ -1 +1 @@
-["http:\/\/s3-static-ak.buzzfed.com\/static\/2015-04\/22\/5\/campaign_images\/webdr03\/student-dies-after-diet-pills-she-bought-online-b-2-28712-1429696299-24_dblbig.jpg","http:\/\/ak-hdl.buzzfed.com\/static\/2015-04\/21\/5\/enhanced\/webdr12\/grid-cell-2501-1429608056-15.jpg","http:\/\/ak-hdl.buzzfed.com\/static\/2015-04\/21\/5\/enhanced\/webdr12\/grid-cell-2501-1429608057-18.jpg"]
\ No newline at end of file
+["http:\/\/s3-static-ak.buzzfed.com\/static\/2015-04\/22\/5\/campaign_images\/webdr03\/student-dies-after-diet-pills-she-bought-online-b-2-28712-1429696299-24_big.jpg","http:\/\/ak-hdl.buzzfed.com\/static\/2015-04\/21\/5\/enhanced\/webdr12\/grid-cell-2501-1429608056-15.jpg","http:\/\/ak-hdl.buzzfed.com\/static\/2015-04\/21\/5\/enhanced\/webdr12\/grid-cell-2501-1429608057-18.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/buzzfeed-1/expected-metadata.json b/test/test-pages/buzzfeed-1/expected-metadata.json
index d5eca27e..5b04b831 100644
--- a/test/test-pages/buzzfeed-1/expected-metadata.json
+++ b/test/test-pages/buzzfeed-1/expected-metadata.json
@@ -1,5 +1,4 @@
{
"Title": "Student Dies After Diet Pills She Bought Online \"Burned Her Up From Within\"",
- "Author": "Mark Di Stefano",
- "Excerpt": "An inquest into Eloise Parry's death has been adjourned until July..."
+ "Excerpt": "An inquest into Eloise Parry's death has been adjourned until July."
}
diff --git a/test/test-pages/buzzfeed-1/expected.html b/test/test-pages/buzzfeed-1/expected.html
index 82dc3a10..46a3f321 100644
--- a/test/test-pages/buzzfeed-1/expected.html
+++ b/test/test-pages/buzzfeed-1/expected.html
@@ -15,13 +15,13 @@
The mother of a woman who took suspected diet pills bought online has descri
-
+
Facebook
-
+
Facebook
diff --git a/test/test-pages/challenges/expected-images.json b/test/test-pages/challenges/expected-images.json
index 66b23712..0637a088 100644
--- a/test/test-pages/challenges/expected-images.json
+++ b/test/test-pages/challenges/expected-images.json
@@ -1 +1 @@
-["http:\/\/fakehost\/img\/cha\/placeholders\/placeholder_1000x750.jpg"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/clean-links/expected.html b/test/test-pages/clean-links/expected.html
index 41539618..8c1b8b12 100644
--- a/test/test-pages/clean-links/expected.html
+++ b/test/test-pages/clean-links/expected.html
@@ -1,5 +1,5 @@
-
+
Study Webtext
"Bartleby the Scrivener: A Story of Wall-Street " (1853)
Herman Melville
@@ -1383,5 +1383,4 @@
Click on text in red for hypertext notes and questions
Ah
Bartleby! Ah humanity!
-
-
\ No newline at end of file
+
\ No newline at end of file
diff --git a/test/test-pages/cnet-svg-classes/expected-images.json b/test/test-pages/cnet-svg-classes/expected-images.json
index 3402f790..3807681e 100644
--- a/test/test-pages/cnet-svg-classes/expected-images.json
+++ b/test/test-pages/cnet-svg-classes/expected-images.json
@@ -1 +1 @@
-["https:\/\/cdn1.cnet.com\/img\/JumVcu1ZSLtPP8ui0UWaSlgi5RU=\/670x503\/2017\/12\/01\/b36ce794-e0b8-495c-a198-184923a8f4e9\/twitter-lite.jpg","https:\/\/cdn2.cnet.com\/img\/LI8y19stcvIQUdzbYdH4-DAigtc=\/fit-in\/570x0\/2017\/12\/01\/b36ce794-e0b8-495c-a198-184923a8f4e9\/twitter-lite.jpg","https:\/\/cdn1.cnet.com\/img\/mWcZaiA8Ngv61OQcpdoh6Ra9nY0=\/170x96\/2017\/11\/13\/70540d7a-cbc5-4563-ab86-b5549ef68168\/oneplus-5t-product-21.jpg"]
\ No newline at end of file
+["https:\/\/cdn2.cnet.com\/img\/LI8y19stcvIQUdzbYdH4-DAigtc=\/fit-in\/570x0\/2017\/12\/01\/b36ce794-e0b8-495c-a198-184923a8f4e9\/twitter-lite.jpg","https:\/\/cdn1.cnet.com\/img\/mWcZaiA8Ngv61OQcpdoh6Ra9nY0=\/170x96\/2017\/11\/13\/70540d7a-cbc5-4563-ab86-b5549ef68168\/oneplus-5t-product-21.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/cnet-svg-classes/expected.html b/test/test-pages/cnet-svg-classes/expected.html
index 26b7440a..1b635124 100644
--- a/test/test-pages/cnet-svg-classes/expected.html
+++ b/test/test-pages/cnet-svg-classes/expected.html
@@ -13,9 +13,9 @@
Twitter dio a conocer Twitter Lite en abril en India, y desde entonces ha estado trabajando para llevarlo a más países. La empresa en los últimos meses también se ha involucrado de forma definitiva en la eliminación de los abusos en la red social, tomando medidas incluso en la verificación de cuentas.
-
+
Reproduciendo:Mira esto: Google Assistant mejora, hay más cambios en Twitter y...
-
+
8:09
diff --git a/test/test-pages/cnet/expected-images.json b/test/test-pages/cnet/expected-images.json
index 4ea0ebbb..c7a3000a 100644
--- a/test/test-pages/cnet/expected-images.json
+++ b/test/test-pages/cnet/expected-images.json
@@ -1 +1 @@
-["https:\/\/cnet3.cbsistatic.com\/img\/1JaRRjqhoGxDVkFxTRRWkZgyK2Q=\/670x503\/2014\/03\/21\/863df5d9-e8b8-4b38-851b-5e3f77f2cf0e\/mark-zuckerberg-facebook-home-10671610x407.jpg","https:\/\/cnet1.cbsistatic.com\/img\/nAMdBzIE1ogVw5bOBZBaiJCt3Ro=\/570x0\/2014\/03\/21\/863df5d9-e8b8-4b38-851b-5e3f77f2cf0e\/mark-zuckerberg-facebook-home-10671610x407.jpg"]
\ No newline at end of file
+["https:\/\/cnet1.cbsistatic.com\/img\/nAMdBzIE1ogVw5bOBZBaiJCt3Ro=\/570x0\/2014\/03\/21\/863df5d9-e8b8-4b38-851b-5e3f77f2cf0e\/mark-zuckerberg-facebook-home-10671610x407.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/cnn/expected-images.json b/test/test-pages/cnn/expected-images.json
index 87a39a35..b3a79917 100644
--- a/test/test-pages/cnn/expected-images.json
+++ b/test/test-pages/cnn/expected-images.json
@@ -1 +1 @@
-["http:\/\/i2.cdn.turner.com\/money\/dam\/assets\/141103182938-income-inequality-780x439.png","https:\/\/smrt.as\/ck"]
\ No newline at end of file
+["http:\/\/i2.cdn.turner.com\/money\/dam\/assets\/141103182938-income-inequality-780x439.png"]
\ No newline at end of file
diff --git a/test/test-pages/cnn/expected.html b/test/test-pages/cnn/expected.html
index 66c778db..552455fe 100644
--- a/test/test-pages/cnn/expected.html
+++ b/test/test-pages/cnn/expected.html
@@ -1,22 +1,5 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
The U.S. has long been heralded as a land of opportunity -- a place where anyone can succeed regardless of the economic class they were born into.
The U.S. has long been heralded as a land of opportunity -- a place where an
-
+
@@ -64,5 +47,5 @@
The U.S. has long been heralded as a land of opportunity -- a place where an
The report concluded that the American safety net was ineffective because it provides only half the financial help people need. Additionally, the levels of assistance in the U.S. are generally lower than in other countries.
-
CNNMoney (New York) First published February 1, 2016: 1:28 AM ET
+
CNNMoney (New York) First published February 1, 2016: 1:28 AM ET
\ No newline at end of file
diff --git a/test/test-pages/ehow-1/expected-images.json b/test/test-pages/ehow-1/expected-images.json
index 395e1b03..877f5f06 100644
--- a/test/test-pages/ehow-1/expected-images.json
+++ b/test/test-pages/ehow-1/expected-images.json
@@ -1 +1 @@
-["http:\/\/img-aws.ehowcdn.com\/200x200\/cme\/photography.prod.demandstudios.com\/16149374-814f-40bc-baf3-ca20f149f0ba.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/16149374-814f-40bc-baf3-ca20f149f0ba.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/fc249ef6-4d27-41b4-8c21-15f7a8512b50.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/aae11d4d-a4aa-4251-a4d9-41023ebf6d84.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/7afdfa1e-da74-44b5-b89c-ca8123516272.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/4f6612c0-316a-4c74-bb03-cb4e778f6d72.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/eeb1e0b4-e573-40a3-8db1-2c76f0b13b84.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/812d4649-4152-4363-97c0-f181d02e709a.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/0cb3988c-9318-47d6-bc9c-c798da1ede72.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/e3e18f0b-ab2c-4ffb-9988-a1ea63faef8b.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/2cd79f8d-0d16-4573-8861-e47fb74b0638.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/78670312-8636-4c42-a75c-3029f7aa6c73.jpg"]
\ No newline at end of file
+["http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/16149374-814f-40bc-baf3-ca20f149f0ba.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/fc249ef6-4d27-41b4-8c21-15f7a8512b50.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/aae11d4d-a4aa-4251-a4d9-41023ebf6d84.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/7afdfa1e-da74-44b5-b89c-ca8123516272.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/4f6612c0-316a-4c74-bb03-cb4e778f6d72.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/eeb1e0b4-e573-40a3-8db1-2c76f0b13b84.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/812d4649-4152-4363-97c0-f181d02e709a.jpg","http:\/\/img-aws.ehowcdn.com\/default\/cme\/photography.prod.demandstudios.com\/0cb3988c-9318-47d6-bc9c-c798da1ede72.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/e3e18f0b-ab2c-4ffb-9988-a1ea63faef8b.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/2cd79f8d-0d16-4573-8861-e47fb74b0638.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/photography.prod.demandstudios.com\/78670312-8636-4c42-a75c-3029f7aa6c73.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/ehow-1/expected-metadata.json b/test/test-pages/ehow-1/expected-metadata.json
index 6cfa7b0d..9e2782b5 100644
--- a/test/test-pages/ehow-1/expected-metadata.json
+++ b/test/test-pages/ehow-1/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "How to Build a Terrarium (with Pictures)",
+ "Title": "How to Build a Terrarium | eHow",
"Author": "Lucy Akins",
- "Excerpt": "How to Build a Terrarium. Glass cloche terrariums are not only appealing to the eye, but they also preserve a bit of nature in your home and serve as a simple, yet beautiful, piece of art. Closed terrariums are easy to care for, as they retain much of their own moisture and provide a warm environment with a consistent level of humidity. You..."
+ "Excerpt": "Glass cloche terrariums are not only appealing to the eye, but they also preserve a bit of nature in your home and serve as a simple, yet beautiful, piece of art. Closed terrariums are easy to care for, as they retain much of their own moisture and provide a warm environment with a consistent level of humidity. You won’t have to water the..."
}
diff --git a/test/test-pages/ehow-1/expected.html b/test/test-pages/ehow-1/expected.html
index 06e78ccb..397322e2 100644
--- a/test/test-pages/ehow-1/expected.html
+++ b/test/test-pages/ehow-1/expected.html
@@ -10,8 +10,7 @@
-
What You'll Need:
-
Cloche
+
What You'll Need:
Cloche
Planter saucer, small shallow dish or desired platform
Floral foam oasis
Ruler
@@ -26,8 +25,7 @@
-
Step 1
-
Measure the circumference of your cloche and cut the foam oasis about 3/4 inch (2 cm) smaller. Place the foam oasis into a container full of water and allow to soak until it sinks to the bottom. Dig out a hole on the oasis large enough to fit your plant, being careful not to pierce all the way through to the bottom.
+
Step 1
Measure the circumference of your cloche and cut the foam oasis about 3/4 inch (2 cm) smaller. Place the foam oasis into a container full of water and allow to soak until it sinks to the bottom. Dig out a hole on the oasis large enough to fit your plant, being careful not to pierce all the way through to the bottom.
Dig a hole in the oasis. (Lucy Akins)
@@ -35,64 +33,55 @@
-
Step 2
-
Insert your plant into the hole.
+
Step 2
Insert your plant into the hole.
Orchid in foam oasis (Lucy Akins)
-
Step 3
-
You can add various plants if you wish.
+
Step 3
You can add various plants if you wish.
Various foliage (Lucy Akins)
-
Step 4
-
Using floral pins, attach enough moss around the oasis to cover it.
+
Step 4
Using floral pins, attach enough moss around the oasis to cover it.
Attach moss. (Lucy Akins)
-
Step 5
-
Gently place the cloche over the oasis. The glass may push some of the moss upward, exposing some of the foam.
+
Step 5
Gently place the cloche over the oasis. The glass may push some of the moss upward, exposing some of the foam.
Place cloche over oasis. (Lucy Akins)
-
Step 6
-
Simply pull down the moss with tweezers or insert more moss to fill in the empty spaces.
+
Step 6
Simply pull down the moss with tweezers or insert more moss to fill in the empty spaces.
Rearrange moss. (Lucy Akins)
-
Step 7
-
You can use any platform you wish. In this case, a small saucer was used.
+
Step 7
You can use any platform you wish. In this case, a small saucer was used.
Place cloche on a platform to sit on. (Lucy Akins)
-
Step 8
-
This particular terrarium rests on a planter saucer and features a small white pumpkin.
+
Step 8
This particular terrarium rests on a planter saucer and features a small white pumpkin.
Cloche placed on a terracotta saucer (Lucy Akins)
-
Step 9
-
This particular terrarium was placed on a wood slice and a little toy squirrel was placed inside to add a little whimsy.
+
Step 9
This particular terrarium was placed on a wood slice and a little toy squirrel was placed inside to add a little whimsy.
Placed on a wooden slice (Lucy Akins)
-
Finished Terrarium
-
Displayed alone or in a group, these pretty arrangements allow you to add a little nature to your decor or tablescape.
+
Finished Terrarium
Displayed alone or in a group, these pretty arrangements allow you to add a little nature to your decor or tablescape.
Cloche terrarium (Lucy Akins)
diff --git a/test/test-pages/ehow-2/expected-images.json b/test/test-pages/ehow-2/expected-images.json
index 6a4dc341..fab0b158 100644
--- a/test/test-pages/ehow-2/expected-images.json
+++ b/test/test-pages/ehow-2/expected-images.json
@@ -1 +1 @@
-{"0":"http:\/\/img-aws.ehowcdn.com\/200x200\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/2F\/86\/5547EF62-EAF5-4256-945D-0496F61C862F\/5547EF62-EAF5-4256-945D-0496F61C862F.jpg","1":"http:\/\/img-aws.ehowcdn.com\/60x60\/cme\/cme_public_images\/www_demandstudios_com\/sitelife.studiod.com\/ver1.0\/Content\/images\/store\/9\/2\/d9dd6f61-b183-4893-927f-5b540e45be91.Small.jpg","3":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/2F\/86\/5547EF62-EAF5-4256-945D-0496F61C862F\/5547EF62-EAF5-4256-945D-0496F61C862F.jpg","4":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/FE\/CB\/121569D2-6984-4B2F-83C4-9D2D9A27CBFE\/121569D2-6984-4B2F-83C4-9D2D9A27CBFE.jpg","5":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/DF\/FC\/A05B0252-BD73-4BC7-A09A-96F0A504FCDF\/A05B0252-BD73-4BC7-A09A-96F0A504FCDF.jpg","6":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/EB\/DB\/8A04CCA7-3255-4225-B59A-C41441F8DBEB\/8A04CCA7-3255-4225-B59A-C41441F8DBEB.jpg","7":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/94\/10\/08035476-0167-4A03-AADC-13A7E7AA1094\/08035476-0167-4A03-AADC-13A7E7AA1094.jpg","8":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/06\/49\/4AD62696-FC95-4DA2-8351-42740C7B4906\/4AD62696-FC95-4DA2-8351-42740C7B4906.jpg","9":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/D0\/51\/B6AED06C-5E19-4A26-9AAD-0E175F6251D0\/B6AED06C-5E19-4A26-9AAD-0E175F6251D0.jpg","10":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/35\/B4\/DD5FD05A-B631-4AFE-BC8F-FDACAD1EB435\/DD5FD05A-B631-4AFE-BC8F-FDACAD1EB435.jpg","11":"http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/A1\/FA\/2C368B34-8F6A-45F6-9DFC-0B0C4E33FAA1\/2C368B34-8F6A-45F6-9DFC-0B0C4E33FAA1.jpg"}
\ No newline at end of file
+["http:\/\/img-aws.ehowcdn.com\/60x60\/cme\/cme_public_images\/www_demandstudios_com\/sitelife.studiod.com\/ver1.0\/Content\/images\/store\/9\/2\/d9dd6f61-b183-4893-927f-5b540e45be91.Small.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/2F\/86\/5547EF62-EAF5-4256-945D-0496F61C862F\/5547EF62-EAF5-4256-945D-0496F61C862F.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/FE\/CB\/121569D2-6984-4B2F-83C4-9D2D9A27CBFE\/121569D2-6984-4B2F-83C4-9D2D9A27CBFE.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/DF\/FC\/A05B0252-BD73-4BC7-A09A-96F0A504FCDF\/A05B0252-BD73-4BC7-A09A-96F0A504FCDF.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/EB\/DB\/8A04CCA7-3255-4225-B59A-C41441F8DBEB\/8A04CCA7-3255-4225-B59A-C41441F8DBEB.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/94\/10\/08035476-0167-4A03-AADC-13A7E7AA1094\/08035476-0167-4A03-AADC-13A7E7AA1094.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/06\/49\/4AD62696-FC95-4DA2-8351-42740C7B4906\/4AD62696-FC95-4DA2-8351-42740C7B4906.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/D0\/51\/B6AED06C-5E19-4A26-9AAD-0E175F6251D0\/B6AED06C-5E19-4A26-9AAD-0E175F6251D0.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/35\/B4\/DD5FD05A-B631-4AFE-BC8F-FDACAD1EB435\/DD5FD05A-B631-4AFE-BC8F-FDACAD1EB435.jpg","http:\/\/img-aws.ehowcdn.com\/640\/cme\/cme_public_images\/www_ehow_com\/cdn-write.demandstudios.com\/upload\/image\/A1\/FA\/2C368B34-8F6A-45F6-9DFC-0B0C4E33FAA1\/2C368B34-8F6A-45F6-9DFC-0B0C4E33FAA1.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/ehow-2/expected-metadata.json b/test/test-pages/ehow-2/expected-metadata.json
index 653912ee..2967b612 100644
--- a/test/test-pages/ehow-2/expected-metadata.json
+++ b/test/test-pages/ehow-2/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "How to Throw a Graduation Party on a Budget (with Pictures)",
+ "Title": "How to Throw a Graduation Party on a Budget | eHow",
"Author": "Gina Roberts-Grey",
- "Excerpt": "How to Throw a Graduation Party on a Budget. Graduation parties are a great way to commemorate the years of hard work teens and college co-eds devote to education. They’re also costly for mom and dad.The average cost of a graduation party in 2013 was a whopping $1,200, according to Graduationparty.com; $700 of that was allocated for food...."
+ "Excerpt": "Graduation parties are a great way to commemorate the years of hard work teens and college co-eds devote to education. They’re also costly for mom and dad.The average cost of a graduation party in 2013 was a whopping $1,200, according to Graduationparty.com; $700 of that was allocated for food. However that budget was based on Midwestern..."
}
diff --git a/test/test-pages/ehow-2/expected.html b/test/test-pages/ehow-2/expected.html
index e7d33e85..ea7d138e 100644
--- a/test/test-pages/ehow-2/expected.html
+++ b/test/test-pages/ehow-2/expected.html
@@ -3,18 +3,11 @@
Parties hosted at restaurants, clubhouses and country clubs eliminate the need to spend hours cleaning up once party guests have gone home. But that convenience comes with a price tag. A country club may charge as much as $2,000 for room rental and restaurant food and beverage will almost always cost more than food prepped and served at home.
+
Parties hosted at restaurants, clubhouses and country clubs eliminate the need to spend hours cleaning up once party guests have gone home. But that convenience comes with a price tag. A country club may charge as much as $2,000 for room rental and restaurant food and beverage will almost always cost more than food prepped and served at home.
Thomas Jackson/Digital Vision/Getty Images
@@ -47,9 +39,8 @@
-
-
Instead of hiring a DJ, use your iPod or Smartphone to spin the tunes. Both easily hook up to most speakers or mp3 compatible docks to play music from your music library. Or download Pandora, the free online radio app, and play hours of music for free.
-
Personalize the music with a playlist of the grad’s favorite songs or songs that were big hits during his or her years in school.
+
Instead of hiring a DJ, use your iPod or Smartphone to spin the tunes. Both easily hook up to most speakers or mp3 compatible docks to play music from your music library. Or download Pandora, the free online radio app, and play hours of music for free.
+
Personalize the music with a playlist of the grad’s favorite songs or songs that were big hits during his or her years in school.
Spencer Platt/Getty Images News/Getty Images
@@ -59,8 +50,7 @@
-
-
Avoid canned drinks, which guests often open, but don't finish. Serve pitchers of tap water with lemon and cucumber slices or sliced strawberries for an interesting and refreshing flavor. Opt for punches and non-alcoholic drinks for high school graduates that allow guests to dole out the exact amount they want to drink.
+
Avoid canned drinks, which guests often open, but don't finish. Serve pitchers of tap water with lemon and cucumber slices or sliced strawberries for an interesting and refreshing flavor. Opt for punches and non-alcoholic drinks for high school graduates that allow guests to dole out the exact amount they want to drink.
evgenyb/iStock/Getty Images
@@ -71,8 +61,7 @@
-
-
Instead of inviting everyone you – and the graduate – know or ever knew, scale back the guest list. Forgo inviting guests that you or your grad haven't seen for eons. There is no reason to provide provisions for people who are essentially out of your lives. Sticking to a small, but personal, guest list allows more time to mingle with loved ones during the party, too.
+
Instead of inviting everyone you – and the graduate – know or ever knew, scale back the guest list. Forgo inviting guests that you or your grad haven't seen for eons. There is no reason to provide provisions for people who are essentially out of your lives. Sticking to a small, but personal, guest list allows more time to mingle with loved ones during the party, too.
Kane Skennar/Photodisc/Getty Images
@@ -82,8 +71,7 @@
-
-
See if your grad and his best friend, girlfriend or close family member would consider hosting a joint party. You can split some of the expenses, especially when the two graduates share mutual friends. You'll also have another parent to bounce ideas off of and to help you stick to your budget when you're tempted to splurge.
+
See if your grad and his best friend, girlfriend or close family member would consider hosting a joint party. You can split some of the expenses, especially when the two graduates share mutual friends. You'll also have another parent to bounce ideas off of and to help you stick to your budget when you're tempted to splurge.
Mike Watson Images/Moodboard/Getty
@@ -93,9 +81,8 @@
-
-
Skip carving stations of prime rib and jumbo shrimp as appetizers, especially for high school graduation parties. Instead, serve some of the graduate's favorite side dishes that are cost effective, like a big pot of spaghetti with breadsticks. Opt for easy and simple food such as pizza, finger food and mini appetizers.
-
Avoid pre-packaged foods and pre-made deli platters. These can be quite costly. Instead, make your own cheese and deli platters for less than half the cost of pre-made.
+
Skip carving stations of prime rib and jumbo shrimp as appetizers, especially for high school graduation parties. Instead, serve some of the graduate's favorite side dishes that are cost effective, like a big pot of spaghetti with breadsticks. Opt for easy and simple food such as pizza, finger food and mini appetizers.
+
Avoid pre-packaged foods and pre-made deli platters. These can be quite costly. Instead, make your own cheese and deli platters for less than half the cost of pre-made.
Mark Stout/iStock/Getty Images
@@ -105,8 +92,7 @@
-
-
Instead of an evening dinner party, host a grad lunch or all appetizers party. Brunch and lunch fare or finger food costs less than dinner. Guests also tend to consume less alcohol in the middle of the day, which keeps cost down.
+
Instead of an evening dinner party, host a grad lunch or all appetizers party. Brunch and lunch fare or finger food costs less than dinner. Guests also tend to consume less alcohol in the middle of the day, which keeps cost down.
Mark Stout/iStock/Getty Images
@@ -118,8 +104,7 @@
-
-
Decorate your party in the graduate's current school colors or the colors of the school he or she will be headed to next. Décor that is not specifically graduation-themed may cost a bit less, and any leftovers can be re-used for future parties, picnics and events.
+
Decorate your party in the graduate's current school colors or the colors of the school he or she will be headed to next. Décor that is not specifically graduation-themed may cost a bit less, and any leftovers can be re-used for future parties, picnics and events.
jethuynh/iStock/Getty Images
diff --git a/test/test-pages/engadget/expected-images.json b/test/test-pages/engadget/expected-images.json
new file mode 100644
index 00000000..4c77ca8a
--- /dev/null
+++ b/test/test-pages/engadget/expected-images.json
@@ -0,0 +1 @@
+["https:\/\/o.aolcdn.com\/images\/dims?thumbnail=45%2C45&quality=80&image_uri=http%3A%2F%2Fwww.blogcdn.com%2Fwww.engadget.com%2Fmedia%2F2016%2F03%2Fdevindra-engadget-headshot-small.jpg&client=cbc79c14efcebee57402&signature=e6ffba7468c380581b6589a70ce5d7c1ec40cd1d","https:\/\/o.aolcdn.com\/images\/dims?thumbnail=980%2C653&quality=80&image_uri=https%3A%2F%2Fs.blogcdn.com%2Fslideshows%2Fimages%2Fslides%2F714%2F208%2F8%2FS7142088%2Fslug%2Fl%2Fxbox-one-x-review-gallery-1-1.jpg&client=cbc79c14efcebee57402&signature=9bb08b52e12de8e4060f863a52c613489529818d","https:\/\/o.aolcdn.com\/images\/dims?crop=1600%2C1067%2C0%2C0&quality=85&format=jpg&resize=1600%2C1067&image_uri=http%3A%2F%2Fo.aolcdn.com%2Fhss%2Fstorage%2Fmidas%2F93beb86758ae1cf95721699e1e006e35%2F205826074%2FXbox%2BOne%2BX%2Breview%2Bgallery%2B7.jpg&client=a1acac3e1b3290917d92&signature=c0f2d36259c2c1decfb60aae364527cda2560d4a","https:\/\/o.aolcdn.com\/images\/dims?crop=1600%2C949%2C0%2C0&quality=85&format=jpg&resize=1600%2C949&image_uri=http%3A%2F%2Fo.aolcdn.com%2Fhss%2Fstorage%2Fmidas%2F9ece7fdad1e7025dec06ac9bf98688d0%2F205826075%2FXbox%2BOne%2BX%2Breview%2Bgallery%2B5.jpg&client=a1acac3e1b3290917d92&signature=9913883753141e7df322616bfe0bc41c6ecd80c8","https:\/\/o.aolcdn.com\/images\/dims?resize=980%2C640&quality=100&image_uri=https%3A%2F%2Fo.aolcdn.com%2Fimages%2Fdims%3Fcrop%3D1599%252C1043%252C0%252C0%26quality%3D85%26format%3Djpg%26resize%3D1600%252C1043%26image_uri%3Dhttp%253A%252F%252Fo.aolcdn.com%252Fhss%252Fstorage%252Fmidas%252F8b98ec8f6649158fe7448ac2f2695ac5%252F205826072%252FXbox%252BOne%252BX%252Breview%252Bgallery%252B6.jpg%26client%3Da1acac3e1b3290917d92%26signature%3D353dad1308f98c2c9dfc82c58a540a8b2f1fe63c&client=cbc79c14efcebee57402&signature=60b7c061460d0d45f5d367b8a9c62978af6b76ce","https:\/\/o.aolcdn.com\/images\/dims?resize=980%2C640&quality=100&image_uri=https%3A%2F%2Fo.aolcdn.com%2Fimages%2Fdims%3Fcrop%3D1600%252C900%252C0%252C0%26quality%3D85%26format%3Djpg%26resize%3D1600%252C900%26image_uri%3Dhttp%253A%252F%252Fo.aolcdn.com%252Fhss%252Fstorage%252Fmidas%252F1885534bd201fc37481b806645c1fc8b%252F205828119%252FXbox%252Bone%252BX%252Bscreenshot%252Bgallery%252B8.jpg%26client%3Da1acac3e1b3290917d92%26signature%3Df63cf67c88b37fd9424855984e45f6b950c8c11a&client=cbc79c14efcebee57402&signature=0adca80fc8ee26a7353be639082881450a5ad49f","https:\/\/o.aolcdn.com\/images\/dims?crop=1600%2C900%2C0%2C0&quality=85&format=jpg&resize=1600%2C900&image_uri=http%3A%2F%2Fo.aolcdn.com%2Fhss%2Fstorage%2Fmidas%2F8352a8a14e88e2ca2ba5be4d8381a055%2F205828115%2FXbox%2Bone%2BX%2Bscreenshot%2Bgallery%2B1.jpg&client=a1acac3e1b3290917d92&signature=d2ccb22e0eaabeb05bfe46e83dbe26fd07f01da8","https:\/\/o.aolcdn.com\/images\/dims?thumbnail=980%2C653&quality=80&image_uri=https%3A%2F%2Fs.blogcdn.com%2Fslideshows%2Fimages%2Fslides%2F714%2F292%2F4%2FS7142924%2Fslug%2Fl%2Fxbox-one-x-screenshot-gallery-2-1.jpg&client=cbc79c14efcebee57402&signature=38c95635c7aad58a8a48038e05589f5cf35b1e28","https:\/\/o.aolcdn.com\/images\/dims?crop=1600%2C900%2C0%2C0&quality=85&format=jpg&resize=1600%2C900&image_uri=http%3A%2F%2Fo.aolcdn.com%2Fhss%2Fstorage%2Fmidas%2Facb08903fbe26ad77b80db8c8e7e8fb1%2F205828118%2FXbox%2Bone%2BX%2Bscreenshot%2Bgallery%2B7.jpg&client=a1acac3e1b3290917d92&signature=21630fa5ec6d8fdce2c35f7e1f652636a2d8efe7","https:\/\/o.aolcdn.com\/images\/dims?resize=980%2C640&quality=100&image_uri=https%3A%2F%2Fo.aolcdn.com%2Fimages%2Fdims%3Fcrop%3D1600%252C1027%252C0%252C0%26quality%3D85%26format%3Djpg%26resize%3D1600%252C1027%26image_uri%3Dhttp%253A%252F%252Fo.aolcdn.com%252Fhss%252Fstorage%252Fmidas%252Fa2c8ba1caccdbb9e0559797e5141eafd%252F205826078%252FXbox%252BOne%252BX%252Breview%252Bgallery%252B11.jpg%26client%3Da1acac3e1b3290917d92%26signature%3Da11bcddced805c6e3698f8ce0494102aef057265&client=cbc79c14efcebee57402&signature=1e9bd192add2772bc842a34e67b7572cfd1b265a","https:\/\/o.aolcdn.com\/images\/dims?crop=1600%2C1028%2C0%2C0&quality=85&format=jpg&resize=1600%2C1028&image_uri=http%3A%2F%2Fo.aolcdn.com%2Fhss%2Fstorage%2Fmidas%2F5396460ef8b6bde7fb7272d9e66a7701%2F205826076%2FXbox%2BOne%2BX%2Breview%2Bgallery%2B9.jpg&client=a1acac3e1b3290917d92&signature=f5b5b4b986c2f8b5031a4469ae0ecec82aff65b0"]
\ No newline at end of file
diff --git a/test/test-pages/engadget/expected-metadata.json b/test/test-pages/engadget/expected-metadata.json
new file mode 100644
index 00000000..eb1ed25f
--- /dev/null
+++ b/test/test-pages/engadget/expected-metadata.json
@@ -0,0 +1,4 @@
+{
+ "Title": "Xbox One X review: A console that keeps up with gaming PCs",
+ "Excerpt": "The Xbox One X is the most powerful gaming console ever, but it's not for everyone yet."
+}
diff --git a/test/test-pages/engadget/expected.html b/test/test-pages/engadget/expected.html
new file mode 100644
index 00000000..44a2afe0
--- /dev/null
+++ b/test/test-pages/engadget/expected.html
@@ -0,0 +1,479 @@
+
The Xbox
+ One X is the ultimate video game system. It sports
+ more horsepower than any system ever. And it plays more
+ titles in native 4K than Sony's
+ PlayStation 4 Pro. It's just about everything
+ you could want without investing in a gaming PC. The
+ only problem? It's now been a year since the PS4 Pro
+ launched, and the One X costs $500, while Sony's console
+ launched at $400. That high price limits the Xbox One X
+ to diehard Microsoft fans who don't mind paying a bit
+ more to play the console's exclusive titles in 4K.
+ Everyone else might be better off waiting, or opting for
+ the $279 Xbox
+ One S.
As promised, the Xbox One X is the
+ most powerful game console ever. In practice, though, it
+ really just puts Microsoft on equal footing with Sony’s
+ PlayStation 4 Pro. 4K/HDR enhanced games look great, but
+ it’s lack of VR is disappointing in 2017.
+
+
+
+
+
+
+
+
+
+
+
Hardware
+
+
Despite all the power inside, the One X is
+ Microsoft's smallest console to date. It looks
+ similar to the Xbox One S, except it has an entirely
+ matte black case and is slightly slimmer. It's also
+ surprisingly dense -- the console weighs 8.4 pounds,
+ but it feels far heavier than you'd expect for its
+ size, thanks to all of its new hardware. The One S,
+ in comparison, weighs two pounds less.
+
The Xbox One X's real upgrades are under the hood. It
+ features an 8-core CPU running at 2.3Ghz, 12GB of
+ GDDR5 RAM, a 1 terabyte hard drive and an upgraded
+ AMD Polaris GPU with 6 teraflops of computing power.
+ The PS4 Pro has only 8GB of RAM and tops out at 4.2
+ teraflops. Microsoft's console is clearly faster.
+ That additional horsepower means the Xbox One X can
+ run more games in full native 4K than the Sony's
+ console.
+
+
Along the front, there's the slot-loading 4K Blu-ray
+ drive, a physical power button, a single USB port
+ and a controller pairing button. And around back,
+ there are HDMI out and in ports, the latter of which
+ lets you plug in your cable box. Additionally, there
+ are two USB ports, connections for optical audio, IR
+ out, and gigabit Ethernet. If you've still got a
+ Kinect around, you'll need to use a USB adapter to
+ plug it in.
+
+
+
+
+
+
+
+
+
+ Devindra Hardawar/AOL
+
+
+
+
+
+
+
+
The console's controller hasn't changed since its
+ last mini-upgrade with the Xbox One S. That revision
+ rounded out its seams, improved bumper performance
+ and added a 3.5mm headphone jack. It's still a great
+ controller, though I'm annoyed Microsoft is sticking
+ with AA batteries as their default power source.
+ Sure, you could just pick up some renewable
+ batteries, or the Play and Charge kit, but that's an
+ extra expense. And manually swapping batteries feels
+ like a bad user experience when every other console
+ has rechargeable controllers.
+
In use
+
+
+
+
+
+
+
+
+
+ Devindra Hardawar/AOL
+
+
+
+
+
+
+
+
You won't find any major differences between the One
+ X and the last Xbox at first — aside from a more
+ dramatic startup sequence. Navigating the Xbox
+ interface is fast and zippy, but mostly that's due
+ to a recent OS upgrade. If you're moving over from
+ an older Xbox One, you can use the backup tool to
+ transfer your games and settings to an external hard
+ drive. Just plug that into the new console during
+ setup and it'll make it feel just like your old
+ machine. It's also a lot faster than waiting for
+ everything to download from Xbox Live.
+
You'll still have to set aside some time if you want
+ to play an Xbox One X-enhanced title, though. Those
+ 4K textures will make games significantly larger,
+ but Microsoft says it's come up with a few ways to
+ help developers make downloading them more
+ efficient. For example, language packs and other
+ optional content won't get installed by default.
+
We only had a few enhanced titles to test out during
+ our review: Gears of War 4, Killer
+ Instinct and Super Lucky's Tale.
+ They each took advantage of the console in different
+ ways. Gears of War 4 runs natively in 4K at
+ 30 FPS with Dolby Atmos and HDR (high dynamic range
+ lighting) support. It looked great -- especially
+ with HDR, which highlighted bright elements like
+ lightning strikes -- but I noticed the frame rate
+ dip occasionally. I was also surprised that load
+ times were on-par with what I've seen with the game
+ on the Xbox One S.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
You can also play in Performance mode, which bumps
+ the frame rate up to 60FPS and uses higher quality
+ graphical effects, while rendering it lower in
+ 1080p. Personally, I preferred this, since it makes
+ the game much smoother -- as if you're playing it on
+ a high-end gaming PC, not a console. Some
+ PlayStation 4 Pro games also let you choose how you
+ wanted to distribute its power, so in some ways
+ Microsoft is just following in its footsteps.
+
I've been playing Gears of War 4 on my
+ gaming PC (which is connected to my home theater)
+ over the past year, and I was impressed that the
+ Xbox One X is able to deliver a similar experience.
+ It didn't quite match my rig though, which is
+ powered by Intel Core i7 4790k CPU running at 4GHz,
+ 16GB DDR3 RAM and an NVIDIA GTX 1080 GPU. Typically,
+ I play at 1,440p (2,560 by 1,440 pixels) with HDR
+ and all of the graphical settings set to their
+ highest level, and I can easily maintain a 60FPS
+ frame rate. The One X felt just as solid at 1080p,
+ but there were clearly plenty of graphics settings
+ it couldn't take advantage of, in particular higher
+ levels of bloom lighting and shadow detail.
Killer Instinct and Super Lucky's
+ Tale run in 4K at a smooth 60FPS. They both
+ looked and played better than their standard
+ versions, though I was surprised they didn't take
+ advantage of HDR. As usual, I noticed the
+ improvement in frame rates more than the higher
+ resolution. Unless you're sitting very close to a TV
+ above 50-inches, you'd likely have a hard time
+ telling between 4K and 1080p.
+
That poses a problem for Microsoft: It's betting that
+ gamers will actually want true 4K rendering. In
+ practice, though, PlayStation 4 Pro titles running
+ in HDR and resolutions between 1080p and 4K often
+ look just as good to the naked eye. The Xbox One X's
+ big advantage is that its hardware could let more
+ games reach 60FPS compared to Sony's console.
+
Microsoft says over 130 Xbox One X-enhanced titles
+ are in the works. That includes already-released
+ games like Forza Motorsport 7 and Assassin's
+ Creed Origins, as well as upcoming titles
+ like Call of Duty: WW2. You'll be able to
+ find them easily in a special section in the Xbox
+ store. There is also a handful of Xbox 360 games
+ that'll get enhanced eventually, including Halo
+ 3 and Fallout 3. Some of those
+ titles will get bumped up to a higher resolution,
+ while others will get HDR support. Microsoft
+ describes these upgrades as a bonus for developers
+ who were prescient about how they built their games.
+ Basically, don't expect your entire 360 library to
+ get enhanced.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Even if a game isn't specifically tuned for the new
+ console, Microsoft says you might still see some
+ performance improvements. The PlayStation 4 Pro,
+ meanwhile, has over one hundred games built for its
+ hardware, and its boost mode can speed up some older
+ games.
+
Microsoft is still pushing the Xbox as more than just
+ a game console, though. 4K Blu-rays loaded up
+ quickly, and I didn't notice many delays as I
+ skipped around films. Planet Earth II, in
+ particular, looked fantastic thanks to its brilliant
+ use of HDR. Unfortunately, the One X doesn't support
+ Dolby Vision, so you're stuck with the slightly less
+ capable HDR 10 standard. That makes sense since it's
+ more widely supported, but it would have been nice
+ to see Dolby's, too.
+
+
+
+
And speaking of Dolby technology, Microsoft is also
+ highlighting Atmos support on the One X, just like
+ it did with the One S. The company's app lets you
+ configure the console to pass audio Atmos signals to
+ your audio receiver. You can also shell out $15 to
+ get Atmos support for headphones, which simulates
+ immersive surround sound. It's strange to pay money
+ to unlock Dolby features, but it's worth it since
+ it's significantly better than Microsoft's audio
+ virtualization technology. The Netflix app also
+ supports Atmos for a handful of films (something
+ that the Xbox One S and PlayStation 4 offer, as
+ well).
+
One thing you won't find in the new Xbox is VR
+ support. Microsoft has mentioned that the console
+ will offer some sort of mixed reality, but it hasn't
+ offered up any details yet. It's technically
+ powerful enough to work with any of the Windows
+ Mixed Reality headsets launching this fall. It's a
+ shame that Microsoft is being so wishy-washy because
+ Sony has had a very successful head start with the
+ PlayStation VR.
+
Pricing and the competition
+
+
+
+
+
+
+
+
+
+ Devindra Hardawar/AOL
+
+
+
+
+
+
+
+
The biggest knock against the Xbox One X is its $500
+ price. The PS4 Pro launched at $400 last year, and
+ there's a good chance we'll see plenty of deals
+ around the holidays. If your friends are on Xbox
+ Live, or you're a devotee of Microsoft's first party
+ franchises, then the X makes more sense. If you just
+ want to play third-party titles that come to both
+ platforms, though, the PS4 Pro is clearly the better
+ deal.
+
If you're looking to upgrade from an original Xbox
+ One, and you have a new TV, the One X might be more
+ compelling. It's faster and offers more features
+ than the One S, and more importantly, it'll last you
+ much longer without needing an upgrade. There's also
+ plenty of wisdom in simply waiting a while before
+ you buy the One X, especially if you haven't moved
+ to a 4K TV yet. The new console can make games look
+ better on 1080p sets, since it'll supersample
+ high-res textures and have more graphical effects,
+ but it's simply not worth the upgrade since those
+ TVs don't support HDR.
+
If price isn't a huge concern for you, it's worth
+ considering investing in a gaming PC. A decent one
+ costs between $600 and $800, plus the price of a
+ monitor, but it'll easily be more powerful than the
+ One X. And you have the added benefit of upgrading
+ components down the line. Now that Microsoft and
+ game publishers are offering most major titles on
+ PC, you won't be missing out on much by ditching
+ consoles.
+
Wrap-up
+
+
Ultimately, the Xbox One X offers some major
+ performance upgrades that gamers will notice --
+ especially if you're coming from an original Xbox
+ One. But it's also a bit disappointing since it's
+ coming a year after the PS4 Pro, and it doesn't
+ offer VR yet. For Microsoft fans, though, none of
+ that will matter. It's exactly what the company
+ promised: the fastest game console ever made.
The Xbox
+ One X is the ultimate video game system. It sports
+ more horsepower than any system ever. And it plays more
+ titles in native 4K than Sony's
+ PlayStation 4 Pro. It's just about everything
+ you could want without investing in a gaming PC. The
+ only problem? It's now been a year since the PS4 Pro
+ launched, and the One X costs $500, while Sony's console
+ launched at $400. That high price limits the Xbox One X
+ to diehard Microsoft fans who don't mind paying a bit
+ more to play the console's exclusive titles in 4K.
+ Everyone else might be better off waiting, or opting for
+ the $279 Xbox
+ One S.
As promised, the Xbox One X is the
+ most powerful game console ever. In practice, though, it
+ really just puts Microsoft on equal footing with Sony’s
+ PlayStation 4 Pro. 4K/HDR enhanced games look great, but
+ it’s lack of VR is disappointing in 2017.
+
+
+
+
+
+
+
+
+
+
+
Hardware
+
+
Despite all the power inside, the One X is
+ Microsoft's smallest console to date. It looks
+ similar to the Xbox One S, except it has an entirely
+ matte black case and is slightly slimmer. It's also
+ surprisingly dense -- the console weighs 8.4 pounds,
+ but it feels far heavier than you'd expect for its
+ size, thanks to all of its new hardware. The One S,
+ in comparison, weighs two pounds less.
+
The Xbox One X's real upgrades are under the hood. It
+ features an 8-core CPU running at 2.3Ghz, 12GB of
+ GDDR5 RAM, a 1 terabyte hard drive and an upgraded
+ AMD Polaris GPU with 6 teraflops of computing power.
+ The PS4 Pro has only 8GB of RAM and tops out at 4.2
+ teraflops. Microsoft's console is clearly faster.
+ That additional horsepower means the Xbox One X can
+ run more games in full native 4K than the Sony's
+ console.
+
+
Along the front, there's the slot-loading 4K Blu-ray
+ drive, a physical power button, a single USB port
+ and a controller pairing button. And around back,
+ there are HDMI out and in ports, the latter of which
+ lets you plug in your cable box. Additionally, there
+ are two USB ports, connections for optical audio, IR
+ out, and gigabit Ethernet. If you've still got a
+ Kinect around, you'll need to use a USB adapter to
+ plug it in.
+
+
+
+
+
+
+
+
+
+
+ Devindra Hardawar/AOL
+
+
+
+
+
+
+
+
+
+
The console's controller hasn't changed since its
+ last mini-upgrade with the Xbox One S. That revision
+ rounded out its seams, improved bumper performance
+ and added a 3.5mm headphone jack. It's still a great
+ controller, though I'm annoyed Microsoft is sticking
+ with AA batteries as their default power source.
+ Sure, you could just pick up some renewable
+ batteries, or the Play and Charge kit, but that's an
+ extra expense. And manually swapping batteries feels
+ like a bad user experience when every other console
+ has rechargeable controllers.
+
In use
+
+
+
+
+
+
+
+
+
+
+ Devindra Hardawar/AOL
+
+
+
+
+
+
+
+
+
+
You won't find any major differences between the One
+ X and the last Xbox at first — aside from a more
+ dramatic startup sequence. Navigating the Xbox
+ interface is fast and zippy, but mostly that's due
+ to a recent OS upgrade. If you're moving over from
+ an older Xbox One, you can use the backup tool to
+ transfer your games and settings to an external hard
+ drive. Just plug that into the new console during
+ setup and it'll make it feel just like your old
+ machine. It's also a lot faster than waiting for
+ everything to download from Xbox Live.
+
You'll still have to set aside some time if you want
+ to play an Xbox One X-enhanced title, though. Those
+ 4K textures will make games significantly larger,
+ but Microsoft says it's come up with a few ways to
+ help developers make downloading them more
+ efficient. For example, language packs and other
+ optional content won't get installed by default.
+
We only had a few enhanced titles to test out during
+ our review: Gears of War 4, Killer
+ Instinct and Super Lucky's Tale.
+ They each took advantage of the console in different
+ ways. Gears of War 4 runs natively in 4K at
+ 30 FPS with Dolby Atmos and HDR (high dynamic range
+ lighting) support. It looked great -- especially
+ with HDR, which highlighted bright elements like
+ lightning strikes -- but I noticed the frame rate
+ dip occasionally. I was also surprised that load
+ times were on-par with what I've seen with the game
+ on the Xbox One S.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
You can also play in Performance mode, which bumps
+ the frame rate up to 60FPS and uses higher quality
+ graphical effects, while rendering it lower in
+ 1080p. Personally, I preferred this, since it makes
+ the game much smoother -- as if you're playing it on
+ a high-end gaming PC, not a console. Some
+ PlayStation 4 Pro games also let you choose how you
+ wanted to distribute its power, so in some ways
+ Microsoft is just following in its footsteps.
+
I've been playing Gears of War 4 on my
+ gaming PC (which is connected to my home theater)
+ over the past year, and I was impressed that the
+ Xbox One X is able to deliver a similar experience.
+ It didn't quite match my rig though, which is
+ powered by Intel Core i7 4790k CPU running at 4GHz,
+ 16GB DDR3 RAM and an NVIDIA GTX 1080 GPU. Typically,
+ I play at 1,440p (2,560 by 1,440 pixels) with HDR
+ and all of the graphical settings set to their
+ highest level, and I can easily maintain a 60FPS
+ frame rate. The One X felt just as solid at 1080p,
+ but there were clearly plenty of graphics settings
+ it couldn't take advantage of, in particular higher
+ levels of bloom lighting and shadow detail.
Killer Instinct and Super Lucky's
+ Tale run in 4K at a smooth 60FPS. They both
+ looked and played better than their standard
+ versions, though I was surprised they didn't take
+ advantage of HDR. As usual, I noticed the
+ improvement in frame rates more than the higher
+ resolution. Unless you're sitting very close to a TV
+ above 50-inches, you'd likely have a hard time
+ telling between 4K and 1080p.
+
That poses a problem for Microsoft: It's betting that
+ gamers will actually want true 4K rendering. In
+ practice, though, PlayStation 4 Pro titles running
+ in HDR and resolutions between 1080p and 4K often
+ look just as good to the naked eye. The Xbox One X's
+ big advantage is that its hardware could let more
+ games reach 60FPS compared to Sony's console.
+
Microsoft says over 130 Xbox One X-enhanced titles
+ are in the works. That includes already-released
+ games like Forza Motorsport 7 and Assassin's
+ Creed Origins, as well as upcoming titles
+ like Call of Duty: WW2. You'll be able to
+ find them easily in a special section in the Xbox
+ store. There is also a handful of Xbox 360 games
+ that'll get enhanced eventually, including Halo
+ 3 and Fallout 3. Some of those
+ titles will get bumped up to a higher resolution,
+ while others will get HDR support. Microsoft
+ describes these upgrades as a bonus for developers
+ who were prescient about how they built their games.
+ Basically, don't expect your entire 360 library to
+ get enhanced.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Even if a game isn't specifically tuned for the new
+ console, Microsoft says you might still see some
+ performance improvements. The PlayStation 4 Pro,
+ meanwhile, has over one hundred games built for its
+ hardware, and its boost mode can speed up some older
+ games.
+
Microsoft is still pushing the Xbox as more than just
+ a game console, though. 4K Blu-rays loaded up
+ quickly, and I didn't notice many delays as I
+ skipped around films. Planet Earth II, in
+ particular, looked fantastic thanks to its brilliant
+ use of HDR. Unfortunately, the One X doesn't support
+ Dolby Vision, so you're stuck with the slightly less
+ capable HDR 10 standard. That makes sense since it's
+ more widely supported, but it would have been nice
+ to see Dolby's, too.
+
+
+
+
And speaking of Dolby technology, Microsoft is also
+ highlighting Atmos support on the One X, just like
+ it did with the One S. The company's app lets you
+ configure the console to pass audio Atmos signals to
+ your audio receiver. You can also shell out $15 to
+ get Atmos support for headphones, which simulates
+ immersive surround sound. It's strange to pay money
+ to unlock Dolby features, but it's worth it since
+ it's significantly better than Microsoft's audio
+ virtualization technology. The Netflix app also
+ supports Atmos for a handful of films (something
+ that the Xbox One S and PlayStation 4 offer, as
+ well).
+
One thing you won't find in the new Xbox is VR
+ support. Microsoft has mentioned that the console
+ will offer some sort of mixed reality, but it hasn't
+ offered up any details yet. It's technically
+ powerful enough to work with any of the Windows
+ Mixed Reality headsets launching this fall. It's a
+ shame that Microsoft is being so wishy-washy because
+ Sony has had a very successful head start with the
+ PlayStation VR.
+
Pricing and the competition
+
+
+
+
+
+
+
+
+
+
+ Devindra Hardawar/AOL
+
+
+
+
+
+
+
+
+
+
The biggest knock against the Xbox One X is its $500
+ price. The PS4 Pro launched at $400 last year, and
+ there's a good chance we'll see plenty of deals
+ around the holidays. If your friends are on Xbox
+ Live, or you're a devotee of Microsoft's first party
+ franchises, then the X makes more sense. If you just
+ want to play third-party titles that come to both
+ platforms, though, the PS4 Pro is clearly the better
+ deal.
+
If you're looking to upgrade from an original Xbox
+ One, and you have a new TV, the One X might be more
+ compelling. It's faster and offers more features
+ than the One S, and more importantly, it'll last you
+ much longer without needing an upgrade. There's also
+ plenty of wisdom in simply waiting a while before
+ you buy the One X, especially if you haven't moved
+ to a 4K TV yet. The new console can make games look
+ better on 1080p sets, since it'll supersample
+ high-res textures and have more graphical effects,
+ but it's simply not worth the upgrade since those
+ TVs don't support HDR.
+
If price isn't a huge concern for you, it's worth
+ considering investing in a gaming PC. A decent one
+ costs between $600 and $800, plus the price of a
+ monitor, but it'll easily be more powerful than the
+ One X. And you have the added benefit of upgrading
+ components down the line. Now that Microsoft and
+ game publishers are offering most major titles on
+ PC, you won't be missing out on much by ditching
+ consoles.
+
Wrap-up
+
+
Ultimately, the Xbox One X offers some major
+ performance upgrades that gamers will notice --
+ especially if you're coming from an original Xbox
+ One. But it's also a bit disappointing since it's
+ coming a year after the PS4 Pro, and it doesn't
+ offer VR yet. For Microsoft fans, though, none of
+ that will matter. It's exactly what the company
+ promised: the fastest game console ever made.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
From around the web
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/test/test-pages/fortressofsolitude/config.json b/test/test-pages/fortressofsolitude/config.json
new file mode 100644
index 00000000..0ac6816e
--- /dev/null
+++ b/test/test-pages/fortressofsolitude/config.json
@@ -0,0 +1,3 @@
+{
+ "ArticleByLine": true
+}
\ No newline at end of file
diff --git a/test/test-pages/fortressofsolitude/expected-images.json b/test/test-pages/fortressofsolitude/expected-images.json
new file mode 100644
index 00000000..0c484103
--- /dev/null
+++ b/test/test-pages/fortressofsolitude/expected-images.json
@@ -0,0 +1 @@
+["http:\/\/www.fortressofsolitude.co.za\/wp-content\/uploads\/2018\/05\/C\u00e9line-Dion-Belches-Her-Way-Onto-Deadpool-2s-Soundtrack-750x400.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/fortressofsolitude/expected-metadata.json b/test/test-pages/fortressofsolitude/expected-metadata.json
new file mode 100644
index 00000000..2efe2a72
--- /dev/null
+++ b/test/test-pages/fortressofsolitude/expected-metadata.json
@@ -0,0 +1,5 @@
+{
+ "Title": "Céline Dion Belches Her Way Onto Deadpool 2's Soundtrack",
+ "Author": "Written by Edward Wallace",
+ "Excerpt": "Just when you think the marketing behind the film couldn't get any more amazing, they go and release a music video from the Deadpool 2 soundtrack starring Canadian singer Céline Dion."
+}
diff --git a/test/test-pages/fortressofsolitude/expected.html b/test/test-pages/fortressofsolitude/expected.html
new file mode 100644
index 00000000..5dc67914
--- /dev/null
+++ b/test/test-pages/fortressofsolitude/expected.html
@@ -0,0 +1 @@
+
Ryan Reynolds never stops surprising fans with his Deadpool, a.k.a. the Merc with the Mouth, persona. Just when you think the marketing behind the film couldn’t get any more amazing, they go and release a music video from the Deadpool 2 soundtrack starring Canadian singer Céline Dion. Yes, the one who sang the Titanic theme song. Your eyes do not deceive you.
“What’s left to say? These prayers ain’t working anymore. Every word shot down in flames. What’s left to do with these broken pieces on the floor? I’m losing my voice calling on you,” Dion croons in the video for “Ashes”. Near the end of it, the filthy, foul-mouthed Deadpool shows up in a surprising twist mentioning that he should have gotten N’sync instead.
Deadpool 2 opens in theatres on May 18.
\ No newline at end of file
diff --git a/test/test-pages/fortressofsolitude/source.html b/test/test-pages/fortressofsolitude/source.html
new file mode 100644
index 00000000..af8f61a2
--- /dev/null
+++ b/test/test-pages/fortressofsolitude/source.html
@@ -0,0 +1,141 @@
+ Céline Dion Belches Her Way Onto Deadpool 2's Soundtrack
Ryan Reynolds never stops surprising fans with his Deadpool, a.k.a. the Merc with the Mouth, persona. Just when you think the marketing behind the film couldn’t get any more amazing, they go and release a music video from the Deadpool 2 soundtrack starring Canadian singer Céline Dion. Yes, the one who sang the Titanic theme song. Your eyes do not deceive you.
“What’s left to say? These prayers ain’t working anymore. Every word shot down in flames. What’s left to do with these broken pieces on the floor? I’m losing my voice calling on you,” Dion croons in the video for “Ashes”. Near the end of it, the filthy, foul-mouthed Deadpool shows up in a surprising twist mentioning that he should have gotten N’sync instead.
Hollywood has run out of ideas again. With that comes another Robin Hood film just eight years after Ridley Scott’s film starring Russell Crowe. It seems audiences can’t get enough
Todd McFarlane’s long-discussed gritty R-rated Spawn movie reboot is moving forward. There are now rumours that Jamie Foxx, who has been lobbying for the role since 2013, is in early
diff --git a/test/test-pages/heise/expected-images.json b/test/test-pages/heise/expected-images.json
index 61370afe..9cc49ac8 100644
--- a/test/test-pages/heise/expected-images.json
+++ b/test/test-pages/heise/expected-images.json
@@ -1 +1 @@
-["http:\/\/www.heise.de\/imgs\/18\/1\/4\/6\/2\/3\/5\/1\/Barcode-Scanner-With-Border-f0c62350bd8d9d96.jpeg","http:\/\/3.f.ix.de\/scale\/geometry\/600\/q75\/imgs\/18\/1\/4\/6\/2\/3\/5\/1\/Barcode-Scanner-With-Border-fc08c913da5cea5d.jpeg"]
\ No newline at end of file
+["http:\/\/3.f.ix.de\/scale\/geometry\/600\/q75\/imgs\/18\/1\/4\/6\/2\/3\/5\/1\/Barcode-Scanner-With-Border-fc08c913da5cea5d.jpeg"]
\ No newline at end of file
diff --git a/test/test-pages/heise/expected-metadata.json b/test/test-pages/heise/expected-metadata.json
index c723ccd0..ae180bf9 100644
--- a/test/test-pages/heise/expected-metadata.json
+++ b/test/test-pages/heise/expected-metadata.json
@@ -1,5 +1,5 @@
{
"Title": "1Password für Mac generiert Einmal-Passwörter",
- "Author": null,
+ "Author": "Mac & i",
"Excerpt": "Das in der iOS-Version bereits enthaltene TOTP-Feature ist nun auch für OS X 10.10 verfügbar. Zudem gibt es neue Zusatzfelder in der Datenbank und weitere Verbesserungen."
}
diff --git a/test/test-pages/herald-sun-1/expected-metadata.json b/test/test-pages/herald-sun-1/expected-metadata.json
index 42194645..b8ad6c26 100644
--- a/test/test-pages/herald-sun-1/expected-metadata.json
+++ b/test/test-pages/herald-sun-1/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "Angry media won’t buckle over new surveillance laws\n\t\t\t\t\t\t| Herald Sun",
+ "Title": "Angry media won’t buckle over new surveillance laws",
"Author": "JOE HILDEBRAND",
"Excerpt": "A HIGH-powered federal government team has been doing the rounds of media organisations in the past few days in an attempt to allay concerns about the impact of new surveillance legislation on press freedom. It failed."
}
diff --git a/test/test-pages/herald-sun-1/expected.html b/test/test-pages/herald-sun-1/expected.html
index 6b0cbe6e..6896eeda 100644
--- a/test/test-pages/herald-sun-1/expected.html
+++ b/test/test-pages/herald-sun-1/expected.html
@@ -1,8 +1,7 @@
-
-
+
A new Bill would require telecommunications service providers to store so-called ‘metadata’ for two years.Source:
@@ -14,9 +13,9 @@
A HIGH-powered federal government team has been doing the rounds of media organisations in the past few days in an attempt to allay concerns about the impact of new surveillance legislation on press freedom. It failed.
The roadshow featured the Prime Minister’s national security adviser, Andrew Shearer, Justin Bassi, who advises Attorney-General George Brandis on crime and security matters, and Australian Federal Police Commissioner Andrew Colvin. Staffers from the office of Communications Minister Malcolm Turnbull also took part.
They held meetings with executives from News Corporation and Fairfax, representatives of the TV networks, the ABC top brass and a group from the media union and the Walkley journalism foundation. I was involved as a member of the Walkley board.
The initiative, from Tony Abbott’s office, is evidence that the Government has been alarmed by the strength of criticism from media of the Data Retention Bill it wants passed before Parliament rises in a fortnight. Bosses, journalists, even the Press Council, are up in arms, not only over this measure, but also over aspects of two earlier pieces of national security legislation that interfere with the ability of the media to hold government to account.
The roadshow featured the Prime Minister’s national security adviser, Andrew Shearer, Justin Bassi, who advises Attorney-General George Brandis on crime and security matters, and Australian Federal Police Commissioner Andrew Colvin. Staffers from the office of Communications Minister Malcolm Turnbull also took part.
They held meetings with executives from News Corporation and Fairfax, representatives of the TV networks, the ABC top brass and a group from the media union and the Walkley journalism foundation. I was involved as a member of the Walkley board.
The initiative, from Tony Abbott’s office, is evidence that the Government has been alarmed by the strength of criticism from media of the Data Retention Bill it wants passed before Parliament rises in a fortnight. Bosses, journalists, even the Press Council, are up in arms, not only over this measure, but also over aspects of two earlier pieces of national security legislation that interfere with the ability of the media to hold government to account.
So it’s fair to make a tentative judgement on what people are paying for
@@ -230,12 +230,12 @@
Welcome to DoctorX’s Barcelona l
-
+
Photo by Joan Bardeletti
-
+
Despite the prevalence of people using the service to gauge the quality
of what goes up their nose, many users send samples to Energy Control in
the spirit of its original mission: keeping themselves alive and healthy.
@@ -274,7 +274,7 @@
Welcome to DoctorX’s Barcelona l
-
+
Photo by Joan Bardeletti
The particular case of cocaine cut with Levamisole is one of the reasons
@@ -299,7 +299,7 @@
Welcome to DoctorX’s Barcelona l
-
+
Photo by Joan Bardeletti
Other members of the community have been helping out, too. PlutoPete,
@@ -356,7 +356,7 @@
Welcome to DoctorX’s Barcelona l
-
+
Photo by Joan Bardeletti
“In the white market, they have quality control. In the dark market, it
@@ -369,7 +369,7 @@
Welcome to DoctorX’s Barcelona l
good, and relatively safe drugs. And now they know it.
-
+
Top photo by Joan Bardeletti
Follow Backchannel: Twitter
diff --git a/test/test-pages/la-nacion/expected-images.json b/test/test-pages/la-nacion/expected-images.json
index e9074fac..25ef5985 100644
--- a/test/test-pages/la-nacion/expected-images.json
+++ b/test/test-pages/la-nacion/expected-images.json
@@ -1 +1 @@
-["http:\/\/bucket.glanacion.com\/anexos\/fotos\/77\/2585177.jpg","http:\/\/bucket2.glanacion.com\/anexos\/fotos\/77\/conflicto-mapuche-2585177w280.jpg"]
\ No newline at end of file
+["http:\/\/bucket2.glanacion.com\/anexos\/fotos\/77\/conflicto-mapuche-2585177w280.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/la-nacion/expected-metadata.json b/test/test-pages/la-nacion/expected-metadata.json
index 512e3e0d..ddba3d8f 100644
--- a/test/test-pages/la-nacion/expected-metadata.json
+++ b/test/test-pages/la-nacion/expected-metadata.json
@@ -1,4 +1,4 @@
{
- "Title": "Una solución no violenta para la cuestión mapuche - 07.12.2017",
- "Excerpt": "Una solución no violenta para la cuestión mapuche | Los pueblos indígenas reclaman por derechos que permanecen incumplidos, por eso es más eficiente canalizar la protesta que reprimirla - LA NACION"
+ "Title": "Una solución no violenta para la cuestión mapuche",
+ "Excerpt": "Los pueblos indígenas reclaman por derechos que permanecen incumplidos, por eso es más eficiente canalizar la protesta que reprimirla"
}
diff --git a/test/test-pages/la-nacion/expected.html b/test/test-pages/la-nacion/expected.html
index 0ebb7e97..5640699c 100644
--- a/test/test-pages/la-nacion/expected.html
+++ b/test/test-pages/la-nacion/expected.html
@@ -21,7 +21,7 @@
supuestos contactos internacionales de organizaciones mapuches. Entre ellos aparecía Ocalan, a quien el
informe ubicó "con domicilios en Palermo y en el centro porteño", y aseguraba incluso que había sido
visto "en Neuquén, Río Negro y Chubut durante el juicio a Jones Huala".
-
+
Foto: LA NACION
Esta falsa noticia fue la más rocambolesca de una larga cadena. Dos hechos quedaron en evidencia:
primero, que hay periodistas que no chequean la información; segundo, que los servicios de inteligencia
los utilizan para manipular la agenda pública. Y sobre los servicios hay dos posibilidades: o son burros
@@ -82,5 +82,4 @@
del Estado argentino no fue tanto quiénes lo desafiaron como quiénes lo gobernaron. Cambiemos.
Andrés Malamud es politólogo e investigador en la Universidad de Lisboa. Martín Schapiro es abogado
administrativista y analista internacional
-
\ No newline at end of file
+
\ No newline at end of file
diff --git a/test/test-pages/lemonde-1/expected-images.json b/test/test-pages/lemonde-1/expected-images.json
index 5aad499c..0637a088 100644
--- a/test/test-pages/lemonde-1/expected-images.json
+++ b/test/test-pages/lemonde-1/expected-images.json
@@ -1 +1 @@
-["http:\/\/s1.lemde.fr\/image\/2015\/05\/05\/600x315\/4628128_3_47fc_projet-de-loi-renseignement_aeba800424730d672d1bd08faf203438.jpg"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/lemonde-2/expected-images.json b/test/test-pages/lemonde-2/expected-images.json
index 8417ec80..0637a088 100644
--- a/test/test-pages/lemonde-2/expected-images.json
+++ b/test/test-pages/lemonde-2/expected-images.json
@@ -1 +1 @@
-["http:\/\/s1.lemde.fr\/medias\/web\/1.2.705\/img\/placeholder\/default.png"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/lemonde-2/expected.html b/test/test-pages/lemonde-2/expected.html
index 8be48485..af69efe3 100644
--- a/test/test-pages/lemonde-2/expected.html
+++ b/test/test-pages/lemonde-2/expected.html
@@ -10,4 +10,4 @@
L'union monétaire s'enlise dans la crise, l'euro reste soumis au feu des cr
En tant qu'industriels allemands et français, nous voulons souligner les immenses avantages que l'espace monétaire commun a apportés. Nous envoyons un message de soutien aux responsables politiques, afin qu'ils mettent en place des conditions nécessaires à un euro durablement stable et compétitif. C'est la base de la future prospérité en Europe. L'union monétaire a durablement besoin de finances publiques solides, de règles de responsabilité claires, de structures transparentes et de conditions de concurrence équitables.
Ce n'est qu'à ces conditions que l'euro sortira renforcé de la crise de la dette. Il n'y a pas d'alternative sérieuse à l'euro commun. L'euro symbolise l'Europe d'aujourd'hui. Un échec de l'euro serait un revers fatal pour l'Europe.
-Appel Frank (Deutsche Post) ; Aschenbroich Jacques (Valeo) ; Berger Roland (Roland Berger Strategy Consultants) ; Bock Kurt (BASF) ; Bories Christel (Constellium) ; Borsig Clemens (Deutsche Bank) ; Brunck Robert (CGG Veritas) ; Buffet Patrick (Eramet) ; Burda Hubert (Hubert Burda Media Holding) ; Caparros Alain (Rewe group) ; Clamadieu Jean-Pierre (Rhodia) ; Cromme Gerhard (ThyssenKrupp) ; Crouzet Philippe (Vallourec) ; de Chalendar Pierre-André (Saint-Gobain) ; de Margerie Christophe (Total) ; de Romanet Augustin (Caisse des dépôts et consignations) ; Degenhart Elmar (Continental) ; Diekmann Michael (Allianz) ; Engel Klaus (Evonik Industries) ; Fehrenbach Franz (Robert Bosch) ; Frérot Antoine (Veolia
KLM) ; Teyssen Johannes (E.ON) ; Thumann Jürgen (BusinessEurope) ; Verwaayen Ben (Alcatel-Lucent) ; Viehbacher Chris (Sanofi) ; Zetsche Dieter (Daimler).
\ No newline at end of file
+Appel Frank (Deutsche Post) ; Aschenbroich Jacques (Valeo) ; Berger Roland (Roland Berger Strategy Consultants) ; Bock Kurt (BASF) ; Bories Christel (Constellium) ; Borsig Clemens (Deutsche Bank) ; Brunck Robert (CGG Veritas) ; Buffet Patrick (Eramet) ; Burda Hubert (Hubert Burda Media Holding) ; Caparros Alain (Rewe group) ; Clamadieu Jean-Pierre (Rhodia) ; Cromme Gerhard (ThyssenKrupp) ; Crouzet Philippe (Vallourec) ; de Chalendar Pierre-André (Saint-Gobain) ; de Margerie Christophe (Total) ; de Romanet Augustin (Caisse des dépôts et consignations) ; Degenhart Elmar (Continental) ; Diekmann Michael (Allianz) ; Engel Klaus (Evonik Industries) ; Fehrenbach Franz (Robert Bosch) ; Frérot Antoine (Veolia Environnement) ; Hagemann Snabe Jim (SAP) ; Henrot François (Groupe Rothschild Cie Banque) ; Hermelin Paul (Cap Gemini) ; Kormann Hermut (Lazard) ; Kron Patrick (Alstom) ; Lachmann Henri (Schneider Electric) ; Lafont Bruno (Lafarge) ; Lauvergeon Anne (Areva) ; Lévy Maurice (Publicis Groupe SA) ; Lévy Jean-Bernard (Vivendi) ; Löscher Peter (Siemens) ; Mestrallet Gérard (GDF Suez) ; Obermann René (Deutsche Telekom) ;Oetker Arend (Dr Arend Oetker Holding) ; Oudéa Frédéric (Société Générale) ; Potier Benoit (Air Liquide) ; Ranque Denis (Cercle de l'industrie/Technicolor) ; Reithofer Norbert (BMW) ; Reitzle Wolfgang (Linde) ; Richard Stéphane (France Telecom) ; Rollier Michel (Michelin) ; Rose Frédéric (Technicolor) ; Schnepp Gilles (Legrand) ; Spinetta Jean-Cyril (Air-France KLM) ; Teyssen Johannes (E.ON) ; Thumann Jürgen (BusinessEurope) ; Verwaayen Ben (Alcatel-Lucent) ; Viehbacher Chris (Sanofi) ; Zetsche Dieter (Daimler).
\ No newline at end of file
diff --git a/test/test-pages/liberation-1/expected-images.json b/test/test-pages/liberation-1/expected-images.json
index cd3c9547..0637a088 100644
--- a/test/test-pages/liberation-1/expected-images.json
+++ b/test/test-pages/liberation-1/expected-images.json
@@ -1 +1 @@
-["http:\/\/md1.libe.com\/photo\/755923-000_hkg10175905.jpg?modified_at=1430371146&width=750"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/lifehacker-post-comment-load/expected-images.json b/test/test-pages/lifehacker-post-comment-load/expected-images.json
index 2dc09553..a8b6052e 100644
--- a/test/test-pages/lifehacker-post-comment-load/expected-images.json
+++ b/test/test-pages/lifehacker-post-comment-load/expected-images.json
@@ -1 +1 @@
-{"0":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--hqqO9fze--\/n1s6c2m6kc07iqdyllj6.jpg","2":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--QeUTCiuW--\/o4dpyrcbiqyfrc3bxx6p.jpg","3":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mlR3Ku0_--\/xrhkwleyurcizy4akiae.jpg","4":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--Tacb0tyW--\/qodag11euf2npkawkn9v.jpg","5":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--x9hLbIKJ--\/imfc9ybqfw0jmztbhfrh.jpg","6":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--6NwBgQLy--\/afy7n45jfvsjdmmhonct.jpg","7":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--ciqk42G0--\/s3pq8vjrvyjgne4lfsod.jpg","8":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mtob1sjR--\/y2ldv5eufb3jcrtfouye.jpg","9":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--4Ajak63w--\/atb9qm07fvvg7hqkumkw.jpg"}
\ No newline at end of file
+["http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--hqqO9fze--\/n1s6c2m6kc07iqdyllj6.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--QeUTCiuW--\/o4dpyrcbiqyfrc3bxx6p.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mlR3Ku0_--\/xrhkwleyurcizy4akiae.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--Tacb0tyW--\/qodag11euf2npkawkn9v.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--x9hLbIKJ--\/imfc9ybqfw0jmztbhfrh.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--6NwBgQLy--\/afy7n45jfvsjdmmhonct.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--ciqk42G0--\/s3pq8vjrvyjgne4lfsod.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mtob1sjR--\/y2ldv5eufb3jcrtfouye.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--4Ajak63w--\/atb9qm07fvvg7hqkumkw.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/lifehacker-post-comment-load/expected.html b/test/test-pages/lifehacker-post-comment-load/expected.html
index d5e6c7c9..2f862a91 100644
--- a/test/test-pages/lifehacker-post-comment-load/expected.html
+++ b/test/test-pages/lifehacker-post-comment-load/expected.html
@@ -56,8 +56,7 @@
Understand
-
For example, confirmation bias makes you only believe the information that conforms to your prior beliefs, while you discount everything else. Advertisers appeal to this bias directly by convincing you one item is better than another with imagery and other tricks, regardless of what hard facts might say. Keep your mind open, do your own research, and accept when you're wrong about a product. The Decoy effect is also a commonly used tactic. You think one product is a deal because it's next to a similar product that's priced way higher. Even if it's a product you need, it's probably not as good of a deal as it looks right then and there. Again, always research beforehand and be on the lookout for this common trick to avoid impulse buys.
+
For example, confirmation bias makes you only believe the information that conforms to your prior beliefs, while you discount everything else. Advertisers appeal to this bias directly by convincing you one item is better than another with imagery and other tricks, regardless of what hard facts might say. Keep your mind open, do your own research, and accept when you're wrong about a product. The Decoy effect is also a commonly used tactic. You think one product is a deal because it's next to a similar product that's priced way higher. Even if it's a product you need, it's probably not as good of a deal as it looks right then and there. Again, always research beforehand and be on the lookout for this common trick to avoid impulse buys.
@@ -103,8 +102,7 @@
Make a Lis
-
The purpose of this exercise is twofold: you see what you already have and don't need to ever buy again, and you get to see what you shouldn't have bought in the first place. As you list everything out, separate items into categories. It's extremely important that you are as honest with yourself as possible while you do this. It's also important you actually write this all down or type it all out. Here is the first set of categories to separate everything into:
+
The purpose of this exercise is twofold: you see what you already have and don't need to ever buy again, and you get to see what you shouldn't have bought in the first place. As you list everything out, separate items into categories. It's extremely important that you are as honest with yourself as possible while you do this. It's also important you actually write this all down or type it all out. Here is the first set of categories to separate everything into:
@@ -381,8 +379,7 @@
Learn to D
-
The last thing you should consider when it comes to impulse buys is "artificial replacement." As Trent Hamm at The Simple Dollar explains, artificial replacement can happen when you start to reduce the time you get with your main interests:
+
The last thing you should consider when it comes to impulse buys is "artificial replacement." As Trent Hamm at The Simple Dollar explains, artificial replacement can happen when you start to reduce the time you get with your main interests:
\ No newline at end of file
diff --git a/test/test-pages/lifehacker-working/expected-images.json b/test/test-pages/lifehacker-working/expected-images.json
index 2dc09553..a8b6052e 100644
--- a/test/test-pages/lifehacker-working/expected-images.json
+++ b/test/test-pages/lifehacker-working/expected-images.json
@@ -1 +1 @@
-{"0":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--hqqO9fze--\/n1s6c2m6kc07iqdyllj6.jpg","2":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--QeUTCiuW--\/o4dpyrcbiqyfrc3bxx6p.jpg","3":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mlR3Ku0_--\/xrhkwleyurcizy4akiae.jpg","4":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--Tacb0tyW--\/qodag11euf2npkawkn9v.jpg","5":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--x9hLbIKJ--\/imfc9ybqfw0jmztbhfrh.jpg","6":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--6NwBgQLy--\/afy7n45jfvsjdmmhonct.jpg","7":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--ciqk42G0--\/s3pq8vjrvyjgne4lfsod.jpg","8":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mtob1sjR--\/y2ldv5eufb3jcrtfouye.jpg","9":"http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--4Ajak63w--\/atb9qm07fvvg7hqkumkw.jpg"}
\ No newline at end of file
+["http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--hqqO9fze--\/n1s6c2m6kc07iqdyllj6.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--QeUTCiuW--\/o4dpyrcbiqyfrc3bxx6p.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mlR3Ku0_--\/xrhkwleyurcizy4akiae.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--Tacb0tyW--\/qodag11euf2npkawkn9v.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--x9hLbIKJ--\/imfc9ybqfw0jmztbhfrh.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--6NwBgQLy--\/afy7n45jfvsjdmmhonct.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--ciqk42G0--\/s3pq8vjrvyjgne4lfsod.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--mtob1sjR--\/y2ldv5eufb3jcrtfouye.jpg","http:\/\/i.kinja-img.com\/gawker-media\/image\/upload\/s--4Ajak63w--\/atb9qm07fvvg7hqkumkw.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/lifehacker-working/expected.html b/test/test-pages/lifehacker-working/expected.html
index d5e6c7c9..2f862a91 100644
--- a/test/test-pages/lifehacker-working/expected.html
+++ b/test/test-pages/lifehacker-working/expected.html
@@ -56,8 +56,7 @@
Understand
-
For example, confirmation bias makes you only believe the information that conforms to your prior beliefs, while you discount everything else. Advertisers appeal to this bias directly by convincing you one item is better than another with imagery and other tricks, regardless of what hard facts might say. Keep your mind open, do your own research, and accept when you're wrong about a product. The Decoy effect is also a commonly used tactic. You think one product is a deal because it's next to a similar product that's priced way higher. Even if it's a product you need, it's probably not as good of a deal as it looks right then and there. Again, always research beforehand and be on the lookout for this common trick to avoid impulse buys.
+
For example, confirmation bias makes you only believe the information that conforms to your prior beliefs, while you discount everything else. Advertisers appeal to this bias directly by convincing you one item is better than another with imagery and other tricks, regardless of what hard facts might say. Keep your mind open, do your own research, and accept when you're wrong about a product. The Decoy effect is also a commonly used tactic. You think one product is a deal because it's next to a similar product that's priced way higher. Even if it's a product you need, it's probably not as good of a deal as it looks right then and there. Again, always research beforehand and be on the lookout for this common trick to avoid impulse buys.
@@ -103,8 +102,7 @@
Make a Lis
-
The purpose of this exercise is twofold: you see what you already have and don't need to ever buy again, and you get to see what you shouldn't have bought in the first place. As you list everything out, separate items into categories. It's extremely important that you are as honest with yourself as possible while you do this. It's also important you actually write this all down or type it all out. Here is the first set of categories to separate everything into:
+
The purpose of this exercise is twofold: you see what you already have and don't need to ever buy again, and you get to see what you shouldn't have bought in the first place. As you list everything out, separate items into categories. It's extremely important that you are as honest with yourself as possible while you do this. It's also important you actually write this all down or type it all out. Here is the first set of categories to separate everything into:
@@ -381,8 +379,7 @@
Learn to D
-
The last thing you should consider when it comes to impulse buys is "artificial replacement." As Trent Hamm at The Simple Dollar explains, artificial replacement can happen when you start to reduce the time you get with your main interests:
+
The last thing you should consider when it comes to impulse buys is "artificial replacement." As Trent Hamm at The Simple Dollar explains, artificial replacement can happen when you start to reduce the time you get with your main interests:
\ No newline at end of file
diff --git a/test/test-pages/links-in-tables/expected-images.json b/test/test-pages/links-in-tables/expected-images.json
index 1d81db65..04d70fb6 100644
--- a/test/test-pages/links-in-tables/expected-images.json
+++ b/test/test-pages/links-in-tables/expected-images.json
@@ -1 +1 @@
-["https:\/\/2.bp.blogspot.com\/-chCZZinlUTg\/WEcxvJo9gdI\/AAAAAAAADnk\/3ND_BspqN6Y2j5xxkLFW3RyS2Ig0NHZpQCLcB\/w1200-h630-p-k-nu\/ipsum-opsum.gif","https:\/\/2.bp.blogspot.com\/-chCZZinlUTg\/WEcxvJo9gdI\/AAAAAAAADnk\/3ND_BspqN6Y2j5xxkLFW3RyS2Ig0NHZpQCLcB\/s640\/ipsum-opsum.gif","https:\/\/2.bp.blogspot.com\/-5aRh1dM6Unc\/WEcNs55RGhI\/AAAAAAAADnI\/tzr_oOJjZwgWd9Vu25ydY0UwB3eXKupXwCLcB\/s200\/image01.png"]
\ No newline at end of file
+{"0":"https:\/\/2.bp.blogspot.com\/-chCZZinlUTg\/WEcxvJo9gdI\/AAAAAAAADnk\/3ND_BspqN6Y2j5xxkLFW3RyS2Ig0NHZpQCLcB\/s640\/ipsum-opsum.gif","2":"https:\/\/2.bp.blogspot.com\/-5aRh1dM6Unc\/WEcNs55RGhI\/AAAAAAAADnI\/tzr_oOJjZwgWd9Vu25ydY0UwB3eXKupXwCLcB\/s200\/image01.png"}
\ No newline at end of file
diff --git a/test/test-pages/links-in-tables/expected.html b/test/test-pages/links-in-tables/expected.html
index 73b554d4..05e07233 100644
--- a/test/test-pages/links-in-tables/expected.html
+++ b/test/test-pages/links-in-tables/expected.html
@@ -54,7 +54,7 @@
Watch how much the compressed text on the right side changes from a one-letter
change in the uncompressed text on the left:
-
+
File-by-File therefore is based on detecting changes in the uncompressed data.
To generate a patch, we first decompress both old and new files before computing
@@ -107,65 +107,47 @@
-
+
-Disclaimer: if you see different patch sizes when you press "update"
+
Disclaimer: if you see different patch sizes when you press "update"
manually, that is because we are not currently using File-by-file for
-interactive updates, only those done in the background.
-
+interactive updates, only those done in the background.
Saving data and making our
users (& developers!) happy
\ No newline at end of file
diff --git a/test/test-pages/lwn-1/expected.html b/test/test-pages/lwn-1/expected.html
index b2bef415..358f9565 100644
--- a/test/test-pages/lwn-1/expected.html
+++ b/test/test-pages/lwn-1/expected.html
@@ -30,27 +30,23 @@
As the name might suggest, QGIS is a Qt application; the latest release will, in fact, build on both Qt4 and Qt5, although the binaries released by the project come only in Qt4 form at present. 2.8 has been labeled a long-term release (LTR)—which, in this case, means that the project has committed to providing backported bug fixes for one full calendar year, and that the 2.8.x series is in permanent feature freeze. The goal, according to the change log, is to provide a stable version suitable for businesses and deployments in other large organizations. The change log itself points out that the development of quite a few new features was underwritten by various GIS companies or university groups, which suggests that taking care of these organizations' needs is reaping dividends for the project.
+
As the name might suggest, QGIS is a Qt application; the latest release will, in fact, build on both Qt4 and Qt5, although the binaries released by the project come only in Qt4 form at present. 2.8 has been labeled a long-term release (LTR)—which, in this case, means that the project has committed to providing backported bug fixes for one full calendar year, and that the 2.8.x series is in permanent feature freeze. The goal, according to the change log, is to provide a stable version suitable for businesses and deployments in other large organizations. The change log itself points out that the development of quite a few new features was underwritten by various GIS companies or university groups, which suggests that taking care of these organizations' needs is reaping dividends for the project.
For those new to QGIS (or GIS in general), there is a detailed new-user tutorial that provides a thorough walk-through of the data-manipulation, mapping, and analysis functions. Being a new user, I went through the tutorial; although there are a handful of minor differences between QGIS 2.8 and the version used in the text (primarily whether specific features were accessed through a toolbar or right-click menu), on the whole it is well worth the time.
QGIS is designed to make short work of importing spatially oriented data sets, mining information from them, and turning the results into a meaningful visualization. Technically speaking, the visualization output is optional: one could simply extract the needed statistics and results and use them to answer some question or, perhaps, publish the massaged data set as a database for others to use.
But well-made maps are often the easiest way to illuminate facts about populations, political regions, geography, and many other topics when human comprehension is the goal. QGIS makes importing data from databases, web-mapping services (WMS), and even unwieldy flat-file data dumps a painless experience. It handles converting between a variety of map-referencing systems more or less automatically, and allows the user to focus on finding the useful attributes of the data sets and rendering them on screen.
Here be data
The significant changes in QGIS 2.8 fall into several categories. There are updates to how QGIS handles the mathematical expressions and queries users can use to filter information out of a data set, improvements to the tools used to explore the on-screen map canvas, and enhancements to the "map composer" used to produce visual output. This is on top of plenty of other under-the-hood improvements, naturally.
-
-
In the first category are several updates to the filtering tools used to mine a data set. Generally speaking, each independent data set is added to a QGIS project as its own layer, then transformed with filters to focus in on a specific portion of the original data. For instance, the land-usage statistics for a region might be one layer, while roads and buildings for the same region from OpenStreetMap might be two additional layers. Such filters can be created in several ways: there is a "query builder" that lets the user construct and test expressions on a data layer, then save the results, an SQL console for performing similar queries on a database, and spreadsheet-like editing tools for working directly on data tables.
+
In the first category are several updates to the filtering tools used to mine a data set. Generally speaking, each independent data set is added to a QGIS project as its own layer, then transformed with filters to focus in on a specific portion of the original data. For instance, the land-usage statistics for a region might be one layer, while roads and buildings for the same region from OpenStreetMap might be two additional layers. Such filters can be created in several ways: there is a "query builder" that lets the user construct and test expressions on a data layer, then save the results, an SQL console for performing similar queries on a database, and spreadsheet-like editing tools for working directly on data tables.
All three have been improved in this release. New are support for if(condition, true, false) conditional statements, a set of operations for geometry primitives (e.g., to test whether regions overlap or lines intersect), and an "integer divide" operation. Users can also add comments to their queries to annotate their code, and there is a new custom
function editor for writing Python functions that can be called in mathematical expressions within the query builder.
It is also now possible to select only some rows in a table, then perform calculations just on the selection—previously, users would have to extract the rows of interest into a new table first. Similarly, in the SQL editor, the user can highlight a subset of the SQL query and execute it separately, which is no doubt helpful for debugging.
There have also been several improvements to the Python and Processing plugins. Users can now drag-and-drop Python scripts onto QGIS and they will be run automatically. Several new analysis algorithms are now available through the Processing interface that were previously Python-only; they include algorithms for generating grids of points or vectors within a region, splitting layers and lines, generating hypsometric
curves, refactoring data sets, and more.
Maps in, maps out
-
-
The process of working with on-screen map data picked up some improvements in the new release as well. Perhaps the most fundamental is that each map layer added to the canvas is now handled in its own thread, so fewer hangs in the user interface are experienced when re-rendering a layer (as happens whenever the user changes the look of points or shapes in a layer). Since remote databases can also be layers, this multi-threaded approach is more resilient against connectivity problems, too. The interface also now supports temporary "scratch" layers that can be used to merge, filter, or simply experiment with a data set, but are not saved when the current project is saved.
+
The process of working with on-screen map data picked up some improvements in the new release as well. Perhaps the most fundamental is that each map layer added to the canvas is now handled in its own thread, so fewer hangs in the user interface are experienced when re-rendering a layer (as happens whenever the user changes the look of points or shapes in a layer). Since remote databases can also be layers, this multi-threaded approach is more resilient against connectivity problems, too. The interface also now supports temporary "scratch" layers that can be used to merge, filter, or simply experiment with a data set, but are not saved when the current project is saved.
For working on the canvas itself, polygonal regions can now use raster images (tiled, if necessary) as fill colors, the map itself can be rotated arbitrarily, and objects can be "snapped" to align with items on any layer (not just the current layer). For working with raster image layers (e.g., aerial photographs) or simply creating new geometric shapes by hand, there is a new digitizing tool that can offer assistance by locking lines to specific angles, automatically keeping borders parallel, and other niceties.
There is a completely overhauled "simplify" tool that is used to reduce the number of extraneous vertices of a vector layer (thus reducing its size). The old simplify tool provided only a relative "tolerance" setting that did not correspond directly to any units. With the new tool, users can set a simplification threshold in terms of the underlying map units, layer-specific units, pixels, and more—and, in addition, the tool reports how much the simplify operation has reduced the size of the data.
-
-
There has also been an effort to present a uniform interface to one of the most important features of the map canvas: the ability to change the symbology used for an item based on some data attribute. The simplest example might be to change the line color of a road based on whether its road-type attribute is "highway," "service road," "residential," or so on. But the same feature is used to automatically highlight layer information based on the filtering and querying functionality discussed above. The new release allows many more map attributes to be controlled by these "data definition" settings, and provides a hard-to-miss button next to each attribute, through which a custom data definition can be set.
+
There has also been an effort to present a uniform interface to one of the most important features of the map canvas: the ability to change the symbology used for an item based on some data attribute. The simplest example might be to change the line color of a road based on whether its road-type attribute is "highway," "service road," "residential," or so on. But the same feature is used to automatically highlight layer information based on the filtering and querying functionality discussed above. The new release allows many more map attributes to be controlled by these "data definition" settings, and provides a hard-to-miss button next to each attribute, through which a custom data definition can be set.
QGIS's composer module is the tool used to take project data and generate a map that can be used outside of the application (in print, as a static image, or as a layer for MapServer or some other software tool, for example). Consequently, it is not a simple select-and-click-export tool; composing the output can involve a lot of choices about which data to make visible, how (and where) to label it, and how to make it generally accessible.
The updated composer in 2.8 now has a full-screen mode and sports several new options for configuring output. For instance, the user now has full control over how map axes are labeled. In previous releases, the grid coordinates of the map could be turned on or off, but the only options were all or nothing. Now, the user can individually choose whether coordinates are displayed on all four sides, and can even choose in which direction vertical text labels will run (so that they can be correctly justified to the edge of the map, for example).
There are, as usual, many more changes than there is room to discuss. Some particularly noteworthy improvements include the ability to save and load bookmarks for frequently used data sources (perhaps most useful for databases, web services, and other non-local data) and improvements to QGIS's server module. This module allows one QGIS instance to serve up data accessible to other QGIS applications (for example, to simply team projects). The server can now be extended with Python plugins and the data layers that it serves can be styled with style rules like those used in the desktop interface.
and found a new home as an Apache project. It is fair to say that the rivalry between the two projects in the time since then has been strong. Predictions that one project or the other would fail have not been borne out, but that does not mean that the two projects are equally successful. A look at the two projects' development communities reveals some interesting differences.
+ March 25, 2015
The LibreOffice project was announced with great fanfare in September 2010. Nearly one year later, the OpenOffice.org project (from which LibreOffice was forked) was
+cut loose from Oracle and found a new home as an Apache project. It is fair to say that the rivalry between the two projects in the time since then has been strong. Predictions that one project or the other would fail have not been borne out, but that does not mean that the two projects are equally successful. A look at the two projects' development communities reveals some interesting differences.
Release histories
Apache OpenOffice has made two releases in the past year: 4.1 in April 2014 and 4.1.1 (described as "a micro update" in the release announcement) in August. The main feature added during that time would appear to be significantly improved accessibility support.
@@ -74,9 +70,9 @@
Development statistics
In the one-year period since late March 2014, there have been 381 changesets committed to the OpenOffice Subversion repository. The most active committers are:
-
Most active OpenOffice developers
+
Most active OpenOffice developers
-
By changesets
+
By changesets
Herbert Dürr
63
16.6%
@@ -127,7 +123,7 @@
Development statistics
0.5%
-
By changed lines
+
By changed lines
Jürgen Schmidt
455499
88.1%
@@ -183,9 +179,9 @@
Development statistics
The picture for LibreOffice is just a little bit different; in the same one-year period, the project has committed 22,134 changesets from 268 developers. The most active of these developers were:
-
Most active LibreOffice developers
+
Most active LibreOffice developers
-
By changesets
+
By changesets
Caolán McNamara
4307
19.5%
@@ -248,7 +244,7 @@
Development statistics
1.0%
-
By changed lines
+
By changed lines
Lionel Elie Mamane
244062
12.5%
@@ -314,7 +310,7 @@
Development statistics
To a first approximation, the top ten companies supporting LibreOffice in the last year are:
-
Companies supporting LibreOffice development
+
Companies supporting LibreOffice development
(by changesets)
Red Hat
8417
@@ -366,8 +362,8 @@
Inside this week's LWN.net Weekly Edition
Distributions: A look at Debian's 2015 DPL candidates; Debian, Fedora, ...
Development: A look at GlusterFS; LibreOffice Online; Open sourcing existing code; Secure Boot in Windows 10; ...
Announcements: A Turing award for Michael Stonebraker, Sébastien Jodogne, ReGlue are Free Software Award winners, Kat Walsh joins FSF board of directors, Cyanogen, ...
diff --git a/test/test-pages/medium-1/expected-images.json b/test/test-pages/medium-1/expected-images.json
index 507d7d07..2aaba2e2 100644
--- a/test/test-pages/medium-1/expected-images.json
+++ b/test/test-pages/medium-1/expected-images.json
@@ -1 +1 @@
-["https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*oBWUXtszDsiv_-Qq2bFLTQ.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*AzYWbe4cZkMMEUbfRjysLQ.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*d0Hp6KlzyIcGHcL6to1sYQ.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*_9KYIFrk_PqWFgptsMDeww.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*Vh2MpQjqjPkzYJaaWExoVg.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*2Ln_DmC95Xpz6LzgywkcFQ.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*bXaR_NBJdoHpRc8lUWSsow.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*lulfisQxgSQ209vPHMAifg.png"]
\ No newline at end of file
+["https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*AzYWbe4cZkMMEUbfRjysLQ.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*d0Hp6KlzyIcGHcL6to1sYQ.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*_9KYIFrk_PqWFgptsMDeww.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*Vh2MpQjqjPkzYJaaWExoVg.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*2Ln_DmC95Xpz6LzgywkcFQ.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*bXaR_NBJdoHpRc8lUWSsow.png","https:\/\/d262ilb51hltx0.cloudfront.net\/max\/800\/1*lulfisQxgSQ209vPHMAifg.png"]
\ No newline at end of file
diff --git a/test/test-pages/medium-1/expected-metadata.json b/test/test-pages/medium-1/expected-metadata.json
index b27963a4..e3e098c1 100644
--- a/test/test-pages/medium-1/expected-metadata.json
+++ b/test/test-pages/medium-1/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "Better Student Journalism — Medium",
+ "Title": "The Open Journalism Project: Better Student Journalism",
"Author": "Pippin Lee",
"Excerpt": "We pushed out the first version of the Open Journalism site in January. Here’s what we’ve learned about student journali…"
}
diff --git a/test/test-pages/medium-1/expected.html b/test/test-pages/medium-1/expected.html
index 0ab250cc..679d8b75 100644
--- a/test/test-pages/medium-1/expected.html
+++ b/test/test-pages/medium-1/expected.html
@@ -27,7 +27,7 @@
Circa 2011
considered.
-
+
topleftpixel.com
I started discovering beautiful things the web could do with images:
things not possible with print. Just as every generation revolts against
walking in the previous generations shoes, I found myself questioning the
@@ -46,7 +46,7 @@
Circa 2011
was, we were told, all part of “becoming a journalist.”
-
+
We don’t know what we don’t know
We spent much of the rest of the school year asking “what should we be
doing in the newsroom?”, which mainly led us to ask “how do we use the
@@ -90,7 +90,7 @@
Common problems in student newsrooms (2011)
Not enough discussion between the business side and web efforts
-
+
From our 2011 research
Common problems in student newsrooms (2013)
Two years later, we went back and looked at what had changed. We talked
to a dozen more newsrooms and weren’t surprised by our findings.
@@ -138,7 +138,7 @@
A train or light at the end of the tunnel: are student
-
+
We designed many of these slides to help explain to ourselves what we were doing
When a newsroom decides to create a position for the web, it’s often with
the intent of having content flow steadily from writers onto the web. This
@@ -180,7 +180,7 @@
A train or light at the end of the tunnel: are student
web.
-
+
The current Open Journalism site was a few years in the making. This was
an original launch page we use in 2012
What we know
New process
@@ -251,7 +251,7 @@
A note to professional news orgs
between student journalism and professional newsrooms.
-
+
2012
This is a start
We going to continue slowly growing the content on Open Journalism. We still consider this the beta version,
but expect to polish it, and beef up the content for a real launch at the
@@ -269,7 +269,7 @@
A note to professional news orgs
a lot.
-
+
Let’s talk. Let’s listen.
diff --git a/test/test-pages/medium-2/expected-metadata.json b/test/test-pages/medium-2/expected-metadata.json
index d451476a..395fb031 100644
--- a/test/test-pages/medium-2/expected-metadata.json
+++ b/test/test-pages/medium-2/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "On Behalf of “Literally” — Medium",
+ "Title": "On Behalf of “Literally”",
"Author": "Courtney Kirchoff",
"Excerpt": "In defense of the word “literally” and why you or someone you know should stop misusing the word, lest they drive us fig…"
}
diff --git a/test/test-pages/medium-2/expected.html b/test/test-pages/medium-2/expected.html
index b3711807..222b09db 100644
--- a/test/test-pages/medium-2/expected.html
+++ b/test/test-pages/medium-2/expected.html
@@ -1 +1 @@
-
Words need defenders.
On Behalf of “Literally”
You either are a “literally” abuser or know of one. If you’re anything like me, hearing the word “literally” used incorrectly causes a little piece of your soul to whither and die. Of course I do not mean that literally, I mean that figuratively. An abuser would have said: “Every time a person uses that word, a piece of my soul literally withers and dies.” Which is terribly, horribly wrong.
For whatever bizarre reason, people feel the need to use literally as a sort of verbal crutch. They use it to emphasize a point, which is silly because they’re already using an analogy or a metaphor to illustrate said point. For example: “Ugh, I literally tore the house apart looking for my remote control!” No, you literally did not tear apart your house, because it’s still standing. If you’d just told me you “tore your house apart” searching for your remote, I would’ve understood what you meant. No need to add “literally” to the sentence.
Maybe I should define literally.
Literally means actually. When you say something literally happened, you’re describing the scene or situation as it actually happened.
So you should only use literally when you mean it. It should not be used in hyperbole. Example: “That was so funny I literally cried.” Which is possible. Some things are funny enough to elicit tears. Note the example stops with “literally cried.” You cannot literally cry your eyes out. The joke wasn’t so funny your eyes popped out of their sockets.
When in Doubt, Leave it Out
“I’m so hungry I could eat a horse,” means you’re hungry. You don’t need to say “I’m so hungry I could literally eat a horse.” Because you can’t do that in one sitting, I don’t care how big your stomach is.
“That play was so funny I laughed my head off,” illustrates the play was amusing. You don’t need to say you literally laughed your head off, because then your head would be on the ground and you wouldn’t be able to speak, much less laugh.
“I drove so fast my car was flying,” we get your point: you were speeding. But your car is never going fast enough to fly, so don’t say your car was literally flying.
Insecurities?
Maybe no one believed a story you told as a child, and you felt the need to prove that it actually happened. No really, mom, I literally climbed the tree. In efforts to prove truth, you used literally to describe something real, however outlandish it seemed. Whatever the reason, now your overuse of literally has become a habit.
Hard Habit to Break?
Abusing literally isn’t as bad a smoking, but it’s still an unhealthy habit (I mean that figuratively). Help is required in order to break it.
This is my version of an intervention for literally abusers. I’m not sure how else to do it other than in writing. I know this makes me sound like a know-it-all, and I accept that. But there’s no excuse other than blatant ignorance to misuse the word “literally.” So just stop it.
Don’t say “Courtney, this post is so snobbish it literally burned up my computer.” Because nothing is that snobbish that it causes computers to combust. Or: “Courtney, your head is so big it literally cannot get through the door.” Because it can, unless it’s one of those tiny doors from Alice in Wonderland and I need to eat a mushroom to make my whole body smaller.
No One’s Perfect
And I’m not saying I am. I’m trying to restore meaning to a word that’s lost meaning. I’m standing up for literally. It’s a good word when used correctly. People are butchering it and destroying it every day (figuratively speaking) and the massacre needs to stop. Just as there’s a coalition of people against the use of certain fonts (like Comic Sans and Papyrus), so should there be a coalition of people against the abuse of literally.
Saying it to Irritate?
Do you misuse the word “literally” just to annoy your know-it-all or grammar police friends/acquaintances/total strangers? If so, why? Doing so would be like me going outside when it’s freezing, wearing nothing but a pair of shorts and t-shirt in hopes of making you cold by just looking at me. Who suffers more?
Graphical Representation
Matthew Inman of “The Oatmeal” wrote a comic about literally. Abusers and defenders alike should check it out. It’s clear this whole craze about literally is driving a lot of us nuts. You literally abusers are killing off pieces of our souls. You must be stopped, or the world will be lost to meaninglessness forever. Figuratively speaking.
\ No newline at end of file
+
Words need defenders.
On Behalf of “Literally”
You either are a “literally” abuser or know of one. If you’re anything like me, hearing the word “literally” used incorrectly causes a little piece of your soul to whither and die. Of course I do not mean that literally, I mean that figuratively. An abuser would have said: “Every time a person uses that word, a piece of my soul literally withers and dies.” Which is terribly, horribly wrong.
For whatever bizarre reason, people feel the need to use literally as a sort of verbal crutch. They use it to emphasize a point, which is silly because they’re already using an analogy or a metaphor to illustrate said point. For example: “Ugh, I literally tore the house apart looking for my remote control!” No, you literally did not tear apart your house, because it’s still standing. If you’d just told me you “tore your house apart” searching for your remote, I would’ve understood what you meant. No need to add “literally” to the sentence.
Maybe I should define literally.
Literally means actually. When you say something literally happened, you’re describing the scene or situation as it actually happened.
So you should only use literally when you mean it. It should not be used in hyperbole. Example: “That was so funny I literally cried.” Which is possible. Some things are funny enough to elicit tears. Note the example stops with “literally cried.” You cannot literally cry your eyes out. The joke wasn’t so funny your eyes popped out of their sockets.
When in Doubt, Leave it Out
“I’m so hungry I could eat a horse,” means you’re hungry. You don’t need to say “I’m so hungry I could literally eat a horse.” Because you can’t do that in one sitting, I don’t care how big your stomach is.
“That play was so funny I laughed my head off,” illustrates the play was amusing. You don’t need to say you literally laughed your head off, because then your head would be on the ground and you wouldn’t be able to speak, much less laugh.
“I drove so fast my car was flying,” we get your point: you were speeding. But your car is never going fast enough to fly, so don’t say your car was literally flying.
Insecurities?
Maybe no one believed a story you told as a child, and you felt the need to prove that it actually happened. No really, mom, I literally climbed the tree. In efforts to prove truth, you used literally to describe something real, however outlandish it seemed. Whatever the reason, now your overuse of literally has become a habit.
Hard Habit to Break?
Abusing literally isn’t as bad a smoking, but it’s still an unhealthy habit (I mean that figuratively). Help is required in order to break it.
This is my version of an intervention for literally abusers. I’m not sure how else to do it other than in writing. I know this makes me sound like a know-it-all, and I accept that. But there’s no excuse other than blatant ignorance to misuse the word “literally.” So just stop it.
Don’t say “Courtney, this post is so snobbish it literally burned up my computer.” Because nothing is that snobbish that it causes computers to combust. Or: “Courtney, your head is so big it literally cannot get through the door.” Because it can, unless it’s one of those tiny doors from Alice in Wonderland and I need to eat a mushroom to make my whole body smaller.
No One’s Perfect
And I’m not saying I am. I’m trying to restore meaning to a word that’s lost meaning. I’m standing up for literally. It’s a good word when used correctly. People are butchering it and destroying it every day (figuratively speaking) and the massacre needs to stop. Just as there’s a coalition of people against the use of certain fonts (like Comic Sans and Papyrus), so should there be a coalition of people against the abuse of literally.
Saying it to Irritate?
Do you misuse the word “literally” just to annoy your know-it-all or grammar police friends/acquaintances/total strangers? If so, why? Doing so would be like me going outside when it’s freezing, wearing nothing but a pair of shorts and t-shirt in hopes of making you cold by just looking at me. Who suffers more?
Graphical Representation
Matthew Inman of “The Oatmeal” wrote a comic about literally. Abusers and defenders alike should check it out. It’s clear this whole craze about literally is driving a lot of us nuts. You literally abusers are killing off pieces of our souls. You must be stopped, or the world will be lost to meaninglessness forever. Figuratively speaking.
\ No newline at end of file
diff --git a/test/test-pages/medium-3/expected-images.json b/test/test-pages/medium-3/expected-images.json
index ad3992f2..0637a088 100644
--- a/test/test-pages/medium-3/expected-images.json
+++ b/test/test-pages/medium-3/expected-images.json
@@ -1 +1 @@
-["https:\/\/cdn-images-1.medium.com\/max\/1200\/1*kbPh7V97eyRodSOw2-ALDw.png"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/medium-3/expected-metadata.json b/test/test-pages/medium-3/expected-metadata.json
index c4eaa563..b2e66b9f 100644
--- a/test/test-pages/medium-3/expected-metadata.json
+++ b/test/test-pages/medium-3/expected-metadata.json
@@ -1,6 +1,6 @@
{
- "Title": "Samantha and The Great Big Lie – John C. Welch – Medium",
+ "Title": "Samantha and The Great Big Lie",
"Author": "John C. Welch",
"Direction": null,
- "Excerpt": "(EDIT: removed the link to Samantha’s post, because the arments and the grubers and the rest of The Deck Clique got what they wanted: a non-proper person driven off the internet lightly capped with a…"
+ "Excerpt": "How to get shanked doing what people say they want"
}
diff --git a/test/test-pages/medium-3/source.html b/test/test-pages/medium-3/source.html
index 6e3d0856..37c24641 100644
--- a/test/test-pages/medium-3/source.html
+++ b/test/test-pages/medium-3/source.html
@@ -2216,10 +2216,10 @@
Samantha and The Great Big Lie – Medium
-
+
-
+
diff --git a/test/test-pages/mozilla-1/expected-images.json b/test/test-pages/mozilla-1/expected-images.json
index 7d3792eb..d9a80d0c 100644
--- a/test/test-pages/mozilla-1/expected-images.json
+++ b/test/test-pages/mozilla-1/expected-images.json
@@ -1 +1 @@
-["https:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/template\/page-image.af8027a425de.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/animations\/flexible-top-fallback.c960365ba781.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/designed-redesigned.fbd3ee9402e6.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/animations\/flexible-bottom-fallback.cafd48a3d0a4.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/theme-red.61611c5734ab.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/add-ons.63a4b761f822.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/awesome-bar.437df162126c.png"]
\ No newline at end of file
+["http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/animations\/flexible-top-fallback.c960365ba781.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/designed-redesigned.fbd3ee9402e6.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/animations\/flexible-bottom-fallback.cafd48a3d0a4.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/theme-red.61611c5734ab.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/add-ons.63a4b761f822.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/desktop\/customize\/awesome-bar.437df162126c.png"]
\ No newline at end of file
diff --git a/test/test-pages/mozilla-1/expected-metadata.json b/test/test-pages/mozilla-1/expected-metadata.json
index b6bac46a..3a93f8f0 100644
--- a/test/test-pages/mozilla-1/expected-metadata.json
+++ b/test/test-pages/mozilla-1/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "Firefox — Customize and make it your own — The most flexible browser on\n the Web — Mozilla",
+ "Title": "Firefox — Customize and make it your own — The most flexible browser on the Web",
"Author": null,
"Direction": "ltr",
"Excerpt": "It’s easier than ever to personalize Firefox and make it work the way\n you do.\n No other browser gives you so much choice and flexibility."
diff --git a/test/test-pages/mozilla-1/expected.html b/test/test-pages/mozilla-1/expected.html
index 495b2307..ba1896ea 100644
--- a/test/test-pages/mozilla-1/expected.html
+++ b/test/test-pages/mozilla-1/expected.html
@@ -1,11 +1,11 @@
-
+
It’s easier than ever to personalize Firefox and make it work the way
you do.
No other browser gives you so much choice and flexibility.
-
+
@@ -16,9 +16,9 @@
Designed to be redesigned
Open the “Customize” panel to add, move or remove any button you want.
Keep your favorite features — add-ons, private browsing, Sync and more
— one quick click away.
-
+
-
+
@@ -29,46 +29,37 @@
Themes
Make Firefox match your style. Choose from thousands of themes and dress
up your browser with a single click.
Add-ons are like apps that you install to add features to Firefox. They
let you compare prices, check the weather, listen to music, send a tweet
and more.
diff --git a/test/test-pages/mozilla-2/expected-images.json b/test/test-pages/mozilla-2/expected-images.json
index dd00d590..7322a3a0 100644
--- a/test/test-pages/mozilla-2/expected-images.json
+++ b/test/test-pages/mozilla-2/expected-images.json
@@ -1 +1 @@
-["https:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/developer\/page-image.03bbe7da3199.png","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-webide.16763db341cb.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-valence.251f9def4d8d.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-inspector.c791bf1f1a59.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-console.42666aaf6d03.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-debugger.02ed86fb0c9f.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-network.740d6082b3f6.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-webaudio.a10ebc48d017.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-style-editor.87c5d2017506.jpg"]
\ No newline at end of file
+["http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-webide.16763db341cb.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-valence.251f9def4d8d.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-inspector.c791bf1f1a59.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-console.42666aaf6d03.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-debugger.02ed86fb0c9f.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-network.740d6082b3f6.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-webaudio.a10ebc48d017.jpg","http:\/\/mozorg.cdn.mozilla.net\/media\/img\/firefox\/firstrun\/dev\/feature-style-editor.87c5d2017506.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/mozilla-2/expected.html b/test/test-pages/mozilla-2/expected.html
index 8b858948..8ce4c070 100644
--- a/test/test-pages/mozilla-2/expected.html
+++ b/test/test-pages/mozilla-2/expected.html
@@ -1,4 +1,4 @@
-
+
Get to know the features that make it the most complete browser for building the Web.
-Además, las asambleas afectan el normal funcionamiento en las escuelas municipales y jardines maternales.
-Ayer, Damián Bizzi, vocero del Sindicato Unión Obreros y Empleados Municipales (Suoem) manifestó a Cadena 3 que las asambleas “son la única forma de hacernos escuchar”: “Lo esencial sería que no hubiéramos tenido este problema. No hay otra forma que las autoridades municipales nos escuchen".
-"Estamos en una situación de conflicto y ésta es la única forma de hacernos escuchar. Es una gestión que no escucha mucho”, dijo.
\ No newline at end of file
diff --git a/test/test-pages/needs-entity-normalization/source.html b/test/test-pages/needs-entity-normalization/source.html
deleted file mode 100755
index c8375fd4..00000000
--- a/test/test-pages/needs-entity-normalization/source.html
+++ /dev/null
@@ -1,1433 +0,0 @@
-Daniele, a Mestre: "En unos meses se va a borrar la sonrisa"
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Daniele, a Mestre: "En unos meses se va a borrar la sonrisa"
30/05/2017 | 11:57 | El titular del Suoem le dijo al intendente que "va a pagar caro, esta infamia y canallada"–por la publiación de sueldos–. Los municipales realizaron una ruidosa protesta en el Palacio 6 de Julio
Los municipales de la ciudad de Córdoba realizan este martes una ruidosa asamblea general en el Palacio 6 de Julio y todas las reparticiones como medida de presión en contra de la difusión de las lista de los empleados y los sueldos que perciben.
-
-"Esta infamia, esta canallada, este intento por pisotear nuestra dignidad este ataque a nuestra seguridad y la de nuestras familias le prometo que lo va a pagar caro; vamos a responder con toda la fuerza", disparó al hablar ante la multitud el titular del Suoem, Rubén Daniele.
-
-"En estos días señor intendente lo he visto suelto de cuerpo, diría que casi con una sonrisa burlona. Le prometo y lo juro acá: en unos meses se va a borrar la sonrisa y se va a cagar como se cagó otras veces, pidiendo que vayamos a solucionar los problemas de la ciudad", agregó.
-
-Si bien la medida de fuerza arrancó a las 11 en la explanada de la Municipalidad, la atención en el municipio, los CPC y otras dependencias se vio resentida durante toda la jornada.
-
-
-
-
-Además, las asambleas afectan el normal funcionamiento en las escuelas municipales y jardines maternales.
-
-Ayer, Damián Bizzi, vocero del Sindicato Unión Obreros y Empleados Municipales (Suoem) manifestó a Cadena 3 que las asambleas “son la única forma de hacernos escuchar”: “Lo esencial sería que no hubiéramos tenido este problema. No hay otra forma que las autoridades municipales nos escuchen".
-
-"Estamos en una situación de conflicto y ésta es la única forma de hacernos escuchar. Es una gestión que no escucha mucho”, dijo.
11:57 | El titular del Suoem le dijo al intendente que "va a pagar caro, esta infamia y canallada"–por la publiación de sueldos–. Los municipales realizaron una ruidosa protesta en el Palacio 6 de Julio
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
\ No newline at end of file
diff --git a/test/test-pages/nytimes-1/expected-images.json b/test/test-pages/nytimes-1/expected-images.json
index 7b773c56..342fd8ae 100644
--- a/test/test-pages/nytimes-1/expected-images.json
+++ b/test/test-pages/nytimes-1/expected-images.json
@@ -1 +1 @@
-["https:\/\/static01.nyt.com\/images\/2017\/01\/14\/world\/13SUDAN-1\/13SUDAN-1-facebookJumbo.jpg","https:\/\/static01.nyt.com\/images\/2017\/01\/14\/world\/13SUDAN-1\/13SUDAN-1-master768.jpg"]
\ No newline at end of file
+["https:\/\/static01.nyt.com\/images\/2017\/01\/14\/world\/13SUDAN-1\/13SUDAN-1-master768.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/nytimes-1/expected.html b/test/test-pages/nytimes-1/expected.html
index 13933733..60f7148b 100644
--- a/test/test-pages/nytimes-1/expected.html
+++ b/test/test-pages/nytimes-1/expected.html
@@ -2,7 +2,7 @@
Photo
-
+
United Nations peacekeepers at a refugee camp in Sudan on Monday. In exchange for the lifting of United States trade sanctions, Sudan has said it will improve access for aid groups, stop supporting rebels in neighboring South Sudan and cooperate with American intelligence agents.
@@ -29,8 +29,8 @@
Obama administration officials said that they had briefed President-elect Donald J. Trump’s transition team, but that they did not know if Mr. Trump would stick with a policy of warmer relations with Sudan.
They said that Sudan had a long way to go in terms of respecting human rights, but that better relations could help increase American leverage.
Mr. Reeves said he thought that the American government was being manipulated and that the Obama administration had made a “deal with the devil.”
diff --git a/test/test-pages/nytimes-2/expected-images.json b/test/test-pages/nytimes-2/expected-images.json
index 4d533ca4..7966a507 100644
--- a/test/test-pages/nytimes-2/expected-images.json
+++ b/test/test-pages/nytimes-2/expected-images.json
@@ -1 +1 @@
-["https:\/\/static01.nyt.com\/images\/2016\/07\/30\/business\/db-dealprof\/db-dealprof-facebookJumbo.jpg","https:\/\/static01.nyt.com\/images\/2016\/07\/30\/business\/db-dealprof\/db-dealprof-master315.jpg"]
\ No newline at end of file
+["https:\/\/static01.nyt.com\/images\/2016\/07\/30\/business\/db-dealprof\/db-dealprof-master315.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/nytimes-2/expected.html b/test/test-pages/nytimes-2/expected.html
index e6d2662e..1584ca43 100644
--- a/test/test-pages/nytimes-2/expected.html
+++ b/test/test-pages/nytimes-2/expected.html
@@ -2,15 +2,15 @@
Photo
-
+
Credit Harry Campbell
Yahoo’s $4.8 billion sale to Verizon is a complicated beast, showing how different acquisition structures can affect how shareholders are treated.
First, let’s say what the Yahoo sale is not. It is not a sale of the publicly traded company. Instead, it is a sale of the Yahoo subsidiary and some related assets to Verizon.
The sale is being done in two steps. The first step will be the transfer of any assets related to Yahoo business to a singular subsidiary. This includes the stock in the business subsidiaries that make up Yahoo that are not already in the single subsidiary, as well as the odd assets like benefit plan rights. This is what is being sold to Verizon. A license of Yahoo’s oldest patents is being held back in the so-called Excalibur portfolio. This will stay with Yahoo, as will Yahoo’s stakes in Alibaba Group and Yahoo Japan.
-
It is hard to overestimate how complex an asset sale like this is. Some of the assets are self-contained, but they must be gathered up and transferred. Employees need to be shuffled around and compensation arrangements redone. Many contracts, like the now-infamous one struck with the search engine Mozilla, which may result in a payment of up to a $1 billion, will contain change-of-control provisions that will be set off and have to be addressed. Tax issues always loom large.
It is hard to overestimate how complex an asset sale like this is. Some of the assets are self-contained, but they must be gathered up and transferred. Employees need to be shuffled around and compensation arrangements redone. Many contracts, like the now-infamous one struck with the search engine Mozilla, which may result in a payment of up to a $1 billion, will contain change-of-control provisions that will be set off and have to be addressed. Tax issues always loom large.
In Yahoo’s case, this is not deliberate, though. It is simply the most expedient way to get rid of the assets.
Whether this is the most tax-efficient way is unclear to me as a nontax lawyer (email me if you know). Yahoo is likely to have a tax bill on the sale, possibly a substantial one. And I presume there were legal reasons for not using a Morris Trust structure, in which Yahoo would have been spun off and immediately sold to Verizon so that only Yahoo’s shareholders paid tax on the deal. In truth, the Yahoo assets being sold are only about 10 percent of the value of the company, so the time and logistics for such a sale when Yahoo is a melting ice cube may not have been worth it.
Finally, if another bidder still wants to acquire Yahoo, it has time. The agreement with Verizon allows Yahoo to terminate the deal and accept a superior offer by paying a $144 million breakup fee to Verizon. And if Yahoo shareholders change their minds and want to stick with Yahoo’s chief executive, Marissa Mayer, and vote down the deal, there is a so-called naked no-vote termination fee of $15 million payable to Verizon to reimburse expenses.
-
All in all, this was as hairy a deal as they come. There was the procedural and logistical complications of selling a company when the chief executive wanted to stay. Then there was the fact that this was an asset sale, including all of the challenges that go with it. Throw in all of the tax issues and the fact that this is a public company, and it is likely that the lawyers involved will have nightmares for years to come.
All in all, this was as hairy a deal as they come. There was the procedural and logistical complications of selling a company when the chief executive wanted to stay. Then there was the fact that this was an asset sale, including all of the challenges that go with it. Throw in all of the tax issues and the fact that this is a public company, and it is likely that the lawyers involved will have nightmares for years to come.
diff --git a/test/test-pages/pixnet/expected-images.json b/test/test-pages/pixnet/expected-images.json
index 5d68a0df..f79702ef 100644
--- a/test/test-pages/pixnet/expected-images.json
+++ b/test/test-pages/pixnet/expected-images.json
@@ -1 +1 @@
-["http:\/\/pic.pimg.tw\/stevenhgm\/1387895093-631461272.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894842-1217674167.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894971-1486345289.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894979-1252095111.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387971416-4261675924.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387971406-2480195851.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894752-3567294980.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894771-2897027724.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894778-2035483089.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387897405-3236217457.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894848-3695967443.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894863-3269042540.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894868-3997219746.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894873-1524806724.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894788-105924953.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894798-1063855065.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894807-309560703.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894882-1881930036.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894887-407829597.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894857-470378275.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895099-4119123008.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387971426-4277312474.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387971436-2828193592.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894823-4061326865.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894911-3706194096.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894894-1173705525.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894901-1058040075.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894925-1582979930.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894989-1689510758.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894933-2886337976.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895113-4041265313.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894940-3359449338.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895106-1387217970.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894947-2636431527.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894956-618198074.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894961-2201609427.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387971396-2999285851.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387894999-1588465034.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895007-4184988815.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895016-2193615729.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895047-92554161.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387971446-966387512.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895036-848978834.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895067-717977929.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895083-1227791497.jpg","http:\/\/pic.pimg.tw\/stevenhgm\/1387895075-2647157523.jpg"]
\ No newline at end of file
+{"0":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894842-1217674167.jpg","2":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894971-1486345289.jpg","3":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894979-1252095111.jpg","4":"http:\/\/pic.pimg.tw\/stevenhgm\/1387971416-4261675924.jpg","5":"http:\/\/pic.pimg.tw\/stevenhgm\/1387971406-2480195851.jpg","6":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894752-3567294980.jpg","7":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894771-2897027724.jpg","8":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894778-2035483089.jpg","9":"http:\/\/pic.pimg.tw\/stevenhgm\/1387897405-3236217457.jpg","10":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894848-3695967443.jpg","11":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894863-3269042540.jpg","12":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894868-3997219746.jpg","13":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894873-1524806724.jpg","14":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894788-105924953.jpg","15":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894798-1063855065.jpg","16":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894807-309560703.jpg","17":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894882-1881930036.jpg","18":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894887-407829597.jpg","19":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894857-470378275.jpg","20":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895099-4119123008.jpg","21":"http:\/\/pic.pimg.tw\/stevenhgm\/1387971426-4277312474.jpg","22":"http:\/\/pic.pimg.tw\/stevenhgm\/1387971436-2828193592.jpg","23":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894823-4061326865.jpg","24":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894911-3706194096.jpg","25":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894894-1173705525.jpg","26":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894901-1058040075.jpg","27":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894925-1582979930.jpg","28":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894989-1689510758.jpg","29":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894933-2886337976.jpg","30":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895113-4041265313.jpg","31":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894940-3359449338.jpg","32":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895106-1387217970.jpg","33":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894947-2636431527.jpg","34":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894956-618198074.jpg","35":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894961-2201609427.jpg","36":"http:\/\/pic.pimg.tw\/stevenhgm\/1387971396-2999285851.jpg","37":"http:\/\/pic.pimg.tw\/stevenhgm\/1387894999-1588465034.jpg","38":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895007-4184988815.jpg","39":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895016-2193615729.jpg","40":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895047-92554161.jpg","41":"http:\/\/pic.pimg.tw\/stevenhgm\/1387971446-966387512.jpg","42":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895036-848978834.jpg","43":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895067-717977929.jpg","44":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895083-1227791497.jpg","45":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895075-2647157523.jpg","46":"http:\/\/pic.pimg.tw\/stevenhgm\/1387895093-631461272.jpg"}
\ No newline at end of file
diff --git a/test/test-pages/remove-extra-brs/expected.html b/test/test-pages/remove-extra-brs/expected.html
index f8f8a6c1..e064a49e 100644
--- a/test/test-pages/remove-extra-brs/expected.html
+++ b/test/test-pages/remove-extra-brs/expected.html
@@ -1,12 +1,12 @@
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.
-
Ut enim ad minim veniam,
+
Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
-
+
Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat.
amet, consectetur adipisicing elit, sed do eiusmod
- tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
- quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
- consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
- cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
- Tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
- quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
- consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
- cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ Tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
\ No newline at end of file
diff --git a/test/test-pages/replace-brs/source.html b/test/test-pages/replace-brs/source.html
index e1d2e78f..2283d654 100644
--- a/test/test-pages/replace-brs/source.html
+++ b/test/test-pages/replace-brs/source.html
@@ -8,19 +8,19 @@
Lorem
- Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod
- tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
- quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
- consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
- cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ Lorem ipsum dolor sit
amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Foo
- Tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
- quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
- consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
- cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+ Tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
- Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
diff --git a/test/test-pages/rtl-3/expected.html b/test/test-pages/rtl-3/expected.html
index eba1686e..75448882 100644
--- a/test/test-pages/rtl-3/expected.html
+++ b/test/test-pages/rtl-3/expected.html
@@ -1,4 +1,4 @@
-
+
Lorem ipsum dolor sit amet.
diff --git a/test/test-pages/salon-1/expected-images.json b/test/test-pages/salon-1/expected-images.json
index e43ea0c0..0637a088 100644
--- a/test/test-pages/salon-1/expected-images.json
+++ b/test/test-pages/salon-1/expected-images.json
@@ -1 +1 @@
-["http:\/\/media.salon.com\/2014\/12\/uber_rand_paul.jpg"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/salon-1/expected-metadata.json b/test/test-pages/salon-1/expected-metadata.json
index 9c39de6c..71fc109d 100644
--- a/test/test-pages/salon-1/expected-metadata.json
+++ b/test/test-pages/salon-1/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "The sharing economy is a lie: Uber, Ayn Rand and the truth about tech\n and libertarians",
+ "Title": "The sharing economy is a lie: Uber, Ayn Rand and the truth about tech and libertarians",
"Author": "Joanna Rothkopf",
"Excerpt": "Disruptive companies talk a good game about sharing. Uber's really just an under-regulated company making riches"
}
diff --git a/test/test-pages/simplyfound-1/expected-images.json b/test/test-pages/simplyfound-1/expected-images.json
index f4321b98..0637a088 100644
--- a/test/test-pages/simplyfound-1/expected-images.json
+++ b/test/test-pages/simplyfound-1/expected-images.json
@@ -1 +1 @@
-["https:\/\/d34hb2g9mvfppu.cloudfront.net\/m\/images\/cache\/images\/2016\/02\/29\/apcnews2012raspberry_pi_logo_mainimage8_jpg8_322_27630a8388eb_lg.jpg"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/simplyfound-1/expected-metadata.json b/test/test-pages/simplyfound-1/expected-metadata.json
index 9d1a0bad..3772bbd9 100644
--- a/test/test-pages/simplyfound-1/expected-metadata.json
+++ b/test/test-pages/simplyfound-1/expected-metadata.json
@@ -1,5 +1,5 @@
{
"Title": "Raspberry Pi 3 - The credit card sized PC that cost only $35 - All-time bestselling computer in UK",
"Author": null,
- "Excerpt": "The Raspberry Pi Foundation started by a handful of volunteers in 2012 when they released the original Raspberry Pi 256MB Model B without knowing what to expect. In a short four-year period they have grown to over sixty full-time employees and ha..."
+ "Excerpt": "The Raspberry Pi Foundation started by a handful of volunteers in 2012 when they released the original Raspberry Pi 256MB Model B without knowing what to expect. In a short four-year period they have grown to over sixty full-time employees and ha..."
}
diff --git a/test/test-pages/table-style-attributes/expected.html b/test/test-pages/table-style-attributes/expected.html
index c1f0a1a1..77c083b2 100644
--- a/test/test-pages/table-style-attributes/expected.html
+++ b/test/test-pages/table-style-attributes/expected.html
@@ -6,7 +6,7 @@
-
+
In December 2002, I tried to install some software on my computer. The experience was, shall we say, less than pleasant. On many levels. I wrote about my experience, as I so often do.
Then in January, the jackasses over at Slashdot posted a link to it, calling it a "review" of Linux video software. I guess you could consider it a review, if you were to squint at it just right. But really what it is is a rant about how I had an evening stolen from me by crap software design. It is a flame about the pathetic state of Linux usability in general, and the handful of video players I tried out in particular. It makes no attempt to be balanced or objective or exhaustive. It is a description of my experience. Perhaps your experience was different. Good for you.
So of course that day I got hundreds of emails about it. Every Linux apologist in the world wanted to make sure I was fully informed of their opinion. The replies were roughly in the following groups:
@@ -33,7 +33,9 @@
"Red Hat 7.2 is totally obsolete! It's almost 14 months old! What were you expecting!"
-
While I am flattered that so many logorrheic Linux fanboys are sufficiently interested in my opinions and experiences to share their deeply heartfelt views with me, you can all rest assured that:
+
+
+
While I am flattered that so many logorrheic Linux fanboys are sufficiently interested in my opinions and experiences to share their deeply heartfelt views with me, you can all rest assured that:
"Red Hat 7.2 is totally obsolete! It's almost 14 months old! What were you expecting!"
+
+
While I am flattered that so many logorrheic Linux fanboys are sufficiently interested in my opinions and experiences to share their deeply heartfelt views with me, you can all rest assured that:
diff --git a/test/test-pages/telegraph/expected-images.json b/test/test-pages/telegraph/expected-images.json
index 44c6f049..0637a088 100644
--- a/test/test-pages/telegraph/expected-images.json
+++ b/test/test-pages/telegraph/expected-images.json
@@ -1 +1 @@
-["http:\/\/www.telegraph.co.uk\/content\/dam\/news\/2017\/11\/16\/TELEMMGLPICT000146889449-xlarge_trans_NvBQzQNjv4BqySoB6nTCgtc7U4LQ_FPO4hKi2sT3vi7ux2-RDZwC4QA.jpeg"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/telegraph/expected-metadata.json b/test/test-pages/telegraph/expected-metadata.json
index 0664ae6f..196d045e 100644
--- a/test/test-pages/telegraph/expected-metadata.json
+++ b/test/test-pages/telegraph/expected-metadata.json
@@ -1,6 +1,4 @@
{
"Title": "Zimbabwe coup: Robert Mugabe and wife Grace 'insisting he finishes his term', as priest steps in to mediate",
- "Author": null,
- "Direction": null,
"Excerpt": "Zimbabwe President Robert Mugabe, his wife Grace and two key figures from her G40 political faction are under house arrest at Mugabe's "Blue House" compound in Harare and are insisting the 93 year-old finishes his presidential term, a source said."
}
diff --git a/test/test-pages/tmz-1/expected.html b/test/test-pages/tmz-1/expected.html
index 2c237889..6eb48b97 100644
--- a/test/test-pages/tmz-1/expected.html
+++ b/test/test-pages/tmz-1/expected.html
@@ -28,9 +28,7 @@
morning. We know they were in the manager's office and we're told
they have looked at security footage to determine if they can ID the culprit.
-
-
-
+
\ No newline at end of file
diff --git a/test/test-pages/tumblr/expected-images.json b/test/test-pages/tumblr/expected-images.json
index 1b0151f3..0637a088 100644
--- a/test/test-pages/tumblr/expected-images.json
+++ b/test/test-pages/tumblr/expected-images.json
@@ -1 +1 @@
-["http:\/\/assets.tumblr.com\/images\/og\/fb_landscape_share.png"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/tumblr/expected-metadata.json b/test/test-pages/tumblr/expected-metadata.json
index 09c83ac8..4589af02 100644
--- a/test/test-pages/tumblr/expected-metadata.json
+++ b/test/test-pages/tumblr/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "Minecraft 1.8 - The Bountiful Update - Minecraft 1.8 - The Bountiful Update",
+ "Title": "Minecraft 1.8 - The Bountiful Update",
"Author": null,
"Direction": null,
"Excerpt": "+ Added Granite, Andesite, and Diorite stone blocks, with smooth versions\n+ Added Slime Block\n+ Added Iron Trapdoor\n+ Added Prismarine and Sea Lantern blocks\n+ Added the Ocean Monument\n+ Added Red..."
diff --git a/test/test-pages/tumblr/expected.html b/test/test-pages/tumblr/expected.html
index ad7a7d88..56df5965 100644
--- a/test/test-pages/tumblr/expected.html
+++ b/test/test-pages/tumblr/expected.html
@@ -4,7 +4,7 @@
+ Added Granite, Andesite, and Diorite stone blocks, with smooth versions + Added Slime Block + Added Iron Trapdoor + Added Prismarine and Sea Lantern blocks + Added the Ocean Monument + Added Red Sandstone + Added Banners + Added Armor Stands + Added Coarse Dirt (dirt where grass won’t grow) + Added Guardian mobs, with item drops + Added Endermite mob + Added Rabbits, with item drops + Added Mutton and Cooked Mutton + Villagers will harvest crops and plant new ones + Mossy Cobblestone and Mossy Stone Bricks are now craftable + Chiseled Stone Bricks are now craftable + Doors and fences now come in all wood type variants + Sponge block has regained its water-absorbing ability and becomes wet + Added a spectator game mode (game mode 3) + Added one new achievement + Added “Customized” world type + Added hidden “Debug Mode” world type + Worlds can now have a world barrier + Added @e target selector for Command Blocks + Added /blockdata command + Added /clone command + Added /execute command + Added /fill command + Added /particle command + Added /testforblocks command + Added /title command + Added /trigger command + Added /worldborder command + Added /stats command + Containers can be locked in custom maps by using the “Lock” data tag + Added logAdminCommands, showDeathMessages, reducedDebugInfo, sendCommandFeedback, and randomTickSpeed game rules + Added three new statistics + Player skins can now have double layers across the whole model, and left/right arms/legs can be edited independently + Added a new player model with smaller arms, and a new player skin called Alex? + Added options for configuring what pieces of the skin that are visible + Blocks can now have custom visual variations in the resource packs + Minecraft Realms now has an activity chart, so you can see who has been online + Minecraft Realms now lets you upload your maps * Difficulty setting is saved per world, and can be locked if wanted * Enchanting has been redone, now costs lapis lazuli in addition to enchantment levels * Villager trading has been rebalanced * Anvil repairing has been rebalanced * Considerable faster client-side performance * Max render distance has been increased to 32 chunks (512 blocks) * Adventure mode now prevents you from destroying blocks, unless your items have the CanDestroy data tag * Resource packs can now also define the shape of blocks and items, and not just their textures * Scoreboards have been given a lot of new features * Tweaked the F3 debug screen * Block ID numbers (such as 1 for stone), are being replaced by ID names (such as minecraft:stone) * Server list has been improved * A few minor changes to village and temple generation * Mob heads for players now show both skin layers * Buttons can now be placed on the ceiling * Lots and lots of other changes * LOTS AND LOTS of other changes - Removed Herobrine
+
+ Added Granite, Andesite, and Diorite stone blocks, with smooth versions + Added Slime Block + Added Iron Trapdoor + Added Prismarine and Sea Lantern blocks + Added the Ocean Monument + Added Red Sandstone + Added Banners + Added Armor Stands + Added Coarse Dirt (dirt where grass won’t grow) + Added Guardian mobs, with item drops + Added Endermite mob + Added Rabbits, with item drops + Added Mutton and Cooked Mutton + Villagers will harvest crops and plant new ones + Mossy Cobblestone and Mossy Stone Bricks are now craftable + Chiseled Stone Bricks are now craftable + Doors and fences now come in all wood type variants + Sponge block has regained its water-absorbing ability and becomes wet + Added a spectator game mode (game mode 3) + Added one new achievement + Added “Customized” world type + Added hidden “Debug Mode” world type + Worlds can now have a world barrier + Added @e target selector for Command Blocks + Added /blockdata command + Added /clone command + Added /execute command + Added /fill command + Added /particle command + Added /testforblocks command + Added /title command + Added /trigger command + Added /worldborder command + Added /stats command + Containers can be locked in custom maps by using the “Lock” data tag + Added logAdminCommands, showDeathMessages, reducedDebugInfo, sendCommandFeedback, and randomTickSpeed game rules + Added three new statistics + Player skins can now have double layers across the whole model, and left/right arms/legs can be edited independently + Added a new player model with smaller arms, and a new player skin called Alex? + Added options for configuring what pieces of the skin that are visible + Blocks can now have custom visual variations in the resource packs + Minecraft Realms now has an activity chart, so you can see who has been online + Minecraft Realms now lets you upload your maps * Difficulty setting is saved per world, and can be locked if wanted * Enchanting has been redone, now costs lapis lazuli in addition to enchantment levels * Villager trading has been rebalanced * Anvil repairing has been rebalanced * Considerable faster client-side performance * Max render distance has been increased to 32 chunks (512 blocks) * Adventure mode now prevents you from destroying blocks, unless your items have the CanDestroy data tag * Resource packs can now also define the shape of blocks and items, and not just their textures * Scoreboards have been given a lot of new features * Tweaked the F3 debug screen * Block ID numbers (such as 1 for stone), are being replaced by ID names (such as minecraft:stone) * Server list has been improved * A few minor changes to village and temple generation * Mob heads for players now show both skin layers * Buttons can now be placed on the ceiling * Lots and lots of other changes * LOTS AND LOTS of other changes - Removed Herobrine
diff --git a/test/test-pages/wapo-1/expected-images.json b/test/test-pages/wapo-1/expected-images.json
index 5a8b6ba9..95ae4640 100644
--- a/test/test-pages/wapo-1/expected-images.json
+++ b/test/test-pages/wapo-1/expected-images.json
@@ -1 +1 @@
-["http:\/\/img.washingtonpost.com\/rw\/2010-2019\/WashingtonPost\/2015\/03\/18\/Foreign\/Images\/Nic6429927.jpg","https:\/\/img.washingtonpost.com\/rf\/image_480w\/2010-2019\/WashingtonPost\/2015\/03\/18\/Foreign\/Graphics\/tunisia600.jpg?uuid=1_yuLs2LEeSHME9HNBbnWQ"]
\ No newline at end of file
+["https:\/\/img.washingtonpost.com\/rf\/image_480w\/2010-2019\/WashingtonPost\/2015\/03\/18\/Foreign\/Graphics\/tunisia600.jpg?uuid=1_yuLs2LEeSHME9HNBbnWQ"]
\ No newline at end of file
diff --git a/test/test-pages/wapo-1/expected.html b/test/test-pages/wapo-1/expected.html
index 1853a354..ed051e88 100644
--- a/test/test-pages/wapo-1/expected.html
+++ b/test/test-pages/wapo-1/expected.html
@@ -48,7 +48,7 @@
the investigation.
-
Gunmen in military uniforms stormed Tunisia's national museum, killing at least 19 people, most of them foreign tourists. (Reuters)
+
Gunmen in military uniforms stormed Tunisia's national museum, killing at least 19 people, most of them foreign tourists. (Reuters)
“This attack today is meant to threaten authorities, to frighten tourists
@@ -85,8 +85,8 @@
fire for what many Tunisians saw as a failure to crack down on Islamist
extremists.
- Map: Flow of foreign fighters to Syria
-
+
Map: Flow of foreign fighters to Syria
+
After the collapse of the authoritarian system in 2011, hard-line Muslims
known as Salafists attacked bars and art galleries. Then, in 2012, hundreds
of Islamists assaulted the U.S. Embassy in
@@ -112,8 +112,7 @@
In January, Libyan militants loyal to the Islamic State beheaded 21 Christians —
20 of them Egyptian Copts — along the country’s coast. They later seized
the Libyan city of Sirte.
-
-
+
Officials are worried about the number of Tunisian militants who may have
joined the jihadists in Libya — with the goal of returning home to fight
the Tunis government.
diff --git a/test/test-pages/wapo-2/expected-images.json b/test/test-pages/wapo-2/expected-images.json
index 7783ad73..c5ed2629 100644
--- a/test/test-pages/wapo-2/expected-images.json
+++ b/test/test-pages/wapo-2/expected-images.json
@@ -1 +1 @@
-["http:\/\/img.washingtonpost.com\/rw\/2010-2019\/WashingtonPost\/2015\/03\/18\/National-Economy\/Images\/Nic6429750-1138.jpg","http:\/\/img.washingtonpost.com\/wp-apps\/imrs.php?src=http:\/\/www.washingtonpost.com\/blogs\/wonkblog\/files\/2014\/07\/mufson_steve.jpg&h=180&w=180"]
\ No newline at end of file
+["https:\/\/img.washingtonpost.com\/rf\/image_400w\/2010-2019\/WashingtonPost\/2015\/03\/18\/National-Economy\/Images\/Nic6429750-1140.jpg?uuid=zLIZQs2KEeSip5UXo6cFBg","http:\/\/img.washingtonpost.com\/wp-apps\/imrs.php?src=http:\/\/www.washingtonpost.com\/blogs\/wonkblog\/files\/2014\/07\/mufson_steve.jpg&h=180&w=180"]
\ No newline at end of file
diff --git a/test/test-pages/wapo-2/expected.html b/test/test-pages/wapo-2/expected.html
index 2f540e2d..cb2a8ced 100644
--- a/test/test-pages/wapo-2/expected.html
+++ b/test/test-pages/wapo-2/expected.html
@@ -1,4 +1,6 @@
-
President Obama told the U.N. General Assembly 18 months ago that he would
+
+ Israeli Prime Minister Benjamin Netanyahu reacts as he visits the Western Wall in Jerusalem on March 18 following his party's victory in Israel's general election. (Thomas Coex/AFP/Getty Images)
+
President Obama told the U.N. General Assembly 18 months ago that he would
seek “real breakthroughs on these two issues — Iran’s nuclear program and
Israeli-Palestinian peace.”
But Benjamin Netanyahu’s triumph in Tuesday’s
@@ -18,7 +20,7 @@
stand up to Obama and even fewer among longtime allies.
-
Israeli Prime Minister Benjamin Netanyahu pledged to form a new governing coalition quickly after an upset election victory that was built on a shift to the right. (Reuters)
+
Israeli Prime Minister Benjamin Netanyahu pledged to form a new governing coalition quickly after an upset election victory that was built on a shift to the right. (Reuters)
In the past, Israeli leaders who risked damaging the country’s most important
@@ -93,8 +95,7 @@
“That could be an issue forced onto the agenda about the same time as
a potential nuclear deal.”
-
-
Steven Mufson covers the White House. Since joining The Post, he has covered
+
Steven Mufson covers the White House. Since joining The Post, he has covered
economics, China, foreign policy and energy.
\ No newline at end of file
diff --git a/test/test-pages/webmd-1/expected.html b/test/test-pages/webmd-1/expected.html
index 6f773fe3..ddbe3d84 100644
--- a/test/test-pages/webmd-1/expected.html
+++ b/test/test-pages/webmd-1/expected.html
@@ -39,8 +39,6 @@
another study presented at the same meeting held out hope of a treatment.
A new skin patch called Viaskin allowed people with peanut allergies to
eat tiny amounts of peanuts after they wore it for a year.
-
-
A Change in Guidelines?
Allergies to peanuts and other foods are on the rise. In the U.S., more
diff --git a/test/test-pages/webmd-2/expected.html b/test/test-pages/webmd-2/expected.html
index ca22ee7f..d9e18169 100644
--- a/test/test-pages/webmd-2/expected.html
+++ b/test/test-pages/webmd-2/expected.html
@@ -12,7 +12,6 @@
-
But First: What's a Superbug?
It's a term coined by the media to describe bacteria that cannot be killed using multiple antibiotics. "It resonates because it's scary," says Stephen Calderwood, MD, president of the Infectious Diseases Society of America. "But in fairness, there is no real definition."
Instead, doctors often use phrases like "multidrug-resistant bacteria." That's because a superbug isn't necessarily resistant to all antibiotics. It refers to bacteria that can't be treated using two or more, says Brian K. Coombes, PhD, of McMaster University in Ontario.
diff --git a/test/test-pages/wikia/expected-images.json b/test/test-pages/wikia/expected-images.json
index 36da355a..1a756264 100644
--- a/test/test-pages/wikia/expected-images.json
+++ b/test/test-pages/wikia/expected-images.json
@@ -1 +1 @@
-["https:\/\/vignette.wikia.nocookie.net\/1fb5ee36-d9ae-4125-96d9-f52eb403f1c9\/thumbnail-down\/width\/1280\/height\/720","https:\/\/vignette.wikia.nocookie.net\/e80dae8a-b955-43f6-8ada-f023385e622b\/scale-to-width-down\/627","https:\/\/vignette.wikia.nocookie.net\/375e0e5a-170d-4560-8f20-240c9f0624e9\/scale-to-width-down\/627","https:\/\/vignette.wikia.nocookie.net\/1fb5ee36-d9ae-4125-96d9-f52eb403f1c9\/scale-to-width-down\/627"]
\ No newline at end of file
+["https:\/\/vignette.wikia.nocookie.net\/e80dae8a-b955-43f6-8ada-f023385e622b\/scale-to-width-down\/627","https:\/\/vignette.wikia.nocookie.net\/375e0e5a-170d-4560-8f20-240c9f0624e9\/scale-to-width-down\/627","https:\/\/vignette.wikia.nocookie.net\/1fb5ee36-d9ae-4125-96d9-f52eb403f1c9\/scale-to-width-down\/627"]
\ No newline at end of file
diff --git a/test/test-pages/wikipedia/expected.html b/test/test-pages/wikipedia/expected.html
index 3ae3f69a..8de3ffab 100644
--- a/test/test-pages/wikipedia/expected.html
+++ b/test/test-pages/wikipedia/expected.html
@@ -38,14 +38,13 @@
Firefox began as an experimental branch of the Mozilla codebase by Dave Hyatt, Joe Hewitt and Blake Ross. They believed the commercial requirements of Netscape's sponsorship and developer-driven feature creep compromised the utility of the Mozilla browser.[46] To combat what they saw as the Mozilla Suite'ssoftware bloat, they created a stand-alone browser, with which they intended to replace the Mozilla Suite.
Firefox was originally named Phoenix but the name was changed so as to avoid trademark conflicts with Phoenix Technologies. The initially-announced replacement, Firebird, provoked objections from the Firebird project community.[47][48] The current name, Firefox, was chosen on February 9, 2004.[49]
SeaMonkey (formerly the Mozilla Application Suite) is a free and open source cross platform suite of Internet software components including a web browser component, a client for sending and receiving email and USENET newsgroup messages, an HTML editor (Mozilla Composer) and the ChatZilla IRC client.
There are a number of sub-communities that exist based on their geographical locations, where contributors near each other work together on particular activities, such as localization, marketing, PR and user support.
@@ -160,8 +156,7 @@
Mozilla Reps[
-
+
The Mozilla Reps program aims to empower and support volunteer Mozillians who want to become official representatives of Mozilla in their region/locale.
discuss the future of news at the 2011 Mozilla Festival in London.
+ Speakers from the Knight Foundation discuss the future of news at the 2011 Mozilla Festival in London.
The Mozilla Festival is an annual event where hundreds of passionate people explore the Web, learn together and make things that can change the world. With the emphasis on making—the mantra of the Festival is "less yack, more hack." Journalists, coders, filmmakers, designers, educators, gamers, makers, youth and anyone else, from all over the world, are encouraged to attend, with attendees from more than 40 countries, working together at the intersection between freedom, the Web, and that years theme.
@@ -204,221 +198,221 @@
See also[^For exceptions, see "Values" section below
-
^"Introduction to Mozilla Source Code". Mozilla. Retrieved 2012-08-18. However, mozilla.org wants to emphasize that these milestones are being produced for testing purposes only.
+
^"Introduction to Mozilla Source Code". Mozilla. Retrieved 2012-08-18. However, mozilla.org wants to emphasize that these milestones are being produced for testing purposes only.
^"Comments on Cisco, Mozilla, and H.264". By endorsing Cisco's plan, there's no getting around the fact that we've caved on our principles. That said, principles can't replace being in a practical position to make a difference in the future. - Christopher Montgomery wrote in a personal capacity but works for Mozilla in their codecs team
+
^"Comments on Cisco, Mozilla, and H.264". By endorsing Cisco's plan, there's no getting around the fact that we've caved on our principles. That said, principles can't replace being in a practical position to make a difference in the future. - Christopher Montgomery wrote in a personal capacity but works for Mozilla in their codecs team
^"Mozilla browser becomes Firebird". IBPhoenix. Archived from the original on 2007-09-14. Retrieved 2013-06-10. We at IBPhoenix think that having a browser and a database with the same name in the same space will confuse the market, especially as browsers and databases are often used in the same applications
+
^"Mozilla browser becomes Firebird". IBPhoenix. Archived from the original on 2007-09-14. Retrieved 2013-06-10. We at IBPhoenix think that having a browser and a database with the same name in the same space will confuse the market, especially as browsers and databases are often used in the same applications
diff --git a/test/test-pages/wordpress/expected-images.json b/test/test-pages/wordpress/expected-images.json
index 0ec604ad..18cf3678 100644
--- a/test/test-pages/wordpress/expected-images.json
+++ b/test/test-pages/wordpress/expected-images.json
@@ -1 +1 @@
-["https:\/\/i0.wp.com\/wptavern.com\/wp-content\/uploads\/2016\/07\/stack-overflow.png?fit=1200%2C470&ssl=1","https:\/\/i0.wp.com\/wptavern.com\/wp-content\/uploads\/2016\/07\/stack-overflow.png?resize=1025%2C401&ssl=1","https:\/\/i2.wp.com\/wptavern.com\/wp-content\/uploads\/2017\/03\/ChangesinDemand.png?resize=975%2C1115&ssl=1","https:\/\/i1.wp.com\/wptavern.com\/wp-content\/uploads\/2017\/03\/HighDemand.png?resize=975%2C854&ssl=1"]
\ No newline at end of file
+["https:\/\/i0.wp.com\/wptavern.com\/wp-content\/uploads\/2016\/07\/stack-overflow.png?resize=1025%2C401&ssl=1","https:\/\/i2.wp.com\/wptavern.com\/wp-content\/uploads\/2017\/03\/ChangesinDemand.png?resize=975%2C1115&ssl=1","https:\/\/i1.wp.com\/wptavern.com\/wp-content\/uploads\/2017\/03\/HighDemand.png?resize=975%2C854&ssl=1"]
\ No newline at end of file
diff --git a/test/test-pages/wordpress/expected-metadata.json b/test/test-pages/wordpress/expected-metadata.json
index 3d860dcc..c9d6abdf 100644
--- a/test/test-pages/wordpress/expected-metadata.json
+++ b/test/test-pages/wordpress/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "Stack Overflow Jobs Data Shows ReactJS Skills in High Demand, WordPress Market Oversaturated with Developers – WordPress Tavern",
+ "Title": "Stack Overflow Jobs Data Shows ReactJS Skills in High Demand, WordPress Market Oversaturated with Developers",
"Author": null,
"Direction": null,
"Excerpt": "Stack Overflow published its analysis of 2017 hiring trends based on the targeting options employers selected when posting to Stack Overflow Jobs. The report, which compares data from 200 companies…"
diff --git a/test/test-pages/yahoo-1/expected-images.json b/test/test-pages/yahoo-1/expected-images.json
index 494126bb..4747f024 100644
--- a/test/test-pages/yahoo-1/expected-images.json
+++ b/test/test-pages/yahoo-1/expected-images.json
@@ -1 +1 @@
-["http:\/\/l3.yimg.com\/uu\/api\/res\/1.2\/4eRCPf9lJt_3q29.outekQ--\/aD02Njk7dz03NDQ7c209MTthcHBpZD15dGFjaHlvbg--\/http:\/\/media.zenfs.com\/en\/homerun\/feed_manager_auto_publish_494\/4406ef57dcb40376c513903b03bef048","http:\/\/l1.yimg.com\/ny\/api\/res\/1.2\/589noY9BZNdmsUUQf6L1AQ--\/YXBwaWQ9aGlnaGxhbmRlcjtzbT0xO3c9NzQ0O2g9NjY5\/http:\/\/media.zenfs.com\/en\/homerun\/feed_manager_auto_publish_494\/4406ef57dcb40376c513903b03bef048"]
\ No newline at end of file
+["http:\/\/l1.yimg.com\/ny\/api\/res\/1.2\/589noY9BZNdmsUUQf6L1AQ--\/YXBwaWQ9aGlnaGxhbmRlcjtzbT0xO3c9NzQ0O2g9NjY5\/http:\/\/media.zenfs.com\/en\/homerun\/feed_manager_auto_publish_494\/4406ef57dcb40376c513903b03bef048"]
\ No newline at end of file
diff --git a/test/test-pages/yahoo-1/expected.html b/test/test-pages/yahoo-1/expected.html
index 1736b046..deea6c82 100644
--- a/test/test-pages/yahoo-1/expected.html
+++ b/test/test-pages/yahoo-1/expected.html
@@ -1,6 +1,6 @@
-
+
Sony’s PlayStation VR.
@@ -8,28 +8,28 @@
Virtual reality has officially reached the consoles. And it’s pretty good! Sony’s PlayStation VR is extremely comfortable and reasonably priced, and while it’s lacking killer apps, it’s loaded with lots of interesting ones.
But which ones should you buy? I’ve played just about every launch game, and while some are worth your time, others you might want to skip. To help you decide what’s what, I’ve put together this list of the eight PSVR games worth considering.
Beloved cult hit “Rez” gets the VR treatment to help launch the PSVR, and the results are terrific. It includes a fully remastered take on the original “Rez” – you zoom through a Matrix-like computer system, shooting down enemies to the steady beat of thumping electronica – but the VR setting makes it incredibly immersive. It gets better the more you play it, too; unlock the amazing Area X mode and you’ll find yourself flying, shooting and bobbing your head to some of the trippiest visuals yet seen in VR.
What would happen if Tron, the board game Simon, a Clown beetle, Cthulhu and a noise band met in VR? Chaos, for sure, and also “Thumper.” Called a “violent rhythm game” by its creators, “Thumper” is, well, a violent rhythm game that’s also a gorgeous, unsettling and totally captivating assault on the senses. With simple controls and a straightforward premise – click the X button and the analog stick in time with the music as you barrel down a neon highway — it’s one of the rare games that works equally well both in and out of VR. But since you have PSVR, play it there. It’s marvelous.
Cheeky horror game “Until Dawn” was a breakout hit for the PS4 last year, channeling the classic “dumb teens in the woods” horror trope into an effective interactive drama. Well, forget all that if you fire up “Rush of Blood,” because this one sticks you front and center on a rollercoaster ride from Hell. Literally. You ride through a dimly-lit carnival of terror, dual-wielding pistols as you take down targets, hideous pig monsters and, naturally, maniac clowns. Be warned: If the bad guys don’t get you, the jump scares will.
Soccer meets “Portal” in the weird (and weirdly fun) “Headmaster,” a game about heading soccer balls into nets, targets and a variety of other things while stuck in some diabolical training facility. While at first it seems a little basic, increasingly challenging shots and a consistently entertaining narrative keep it from running off the pitch. Funny, ridiculous and as easy as literally moving your head back and forth, it’s a pleasant PSVR surprise.
Giant mechs + sports? That’s the gist of this robotic blast-a-thon, which pits two teams of three against one another in gorgeous, explosive and downright fun VR combat. At its best, “RIGS” marries the thrill of fast-paced competitive shooters with the insanity of piloting a giant mech in VR. It can, however, be one of the barfier PSVR games. So pack your Dramamine, you’re going to have to ease yourself into this one.
“I’m Batman,” you will say. And you’ll actually be right this time, because you are Batman in this detective yarn, and you know this because you actually grab the famous cowl and mask, stick it on your head, and stare into the mirrored reflection of Rocksteady Games’ impressive Dark Knight character model. It lacks the action of its fellow “Arkham” games and runs disappointingly short, but it’s a high-quality experience that really shows off how powerfully immersive VR can be.
There are a number of good VR ports in the PSVR launch lineup, but the HTC Vive launch game “Job Simulator” might be the best. Your task? Lots of tasks, actually, from cooking food to fixing cars to working in an office, all for robots, because did I mention you were in the future? Infinitely charming and surprisingly challenging, it’s a great showpiece for VR.
Already a hit on the Oculus Rift, this space dogfighting game was one of the first to really show off how VR can turn a traditional game experience into something special. It’s pricey and not quite as hi-res as the Rift version, but “Eve Valkyrie” does an admirable job filling the void left since “Battlestar Galactica” ended. Too bad there aren’t any Cylons in it (or are there?)
+
\ No newline at end of file
diff --git a/test/test-pages/yahoo-2/expected-images.json b/test/test-pages/yahoo-2/expected-images.json
index 1045158a..0637a088 100644
--- a/test/test-pages/yahoo-2/expected-images.json
+++ b/test/test-pages/yahoo-2/expected-images.json
@@ -1 +1 @@
-["https:\/\/s.yimg.com\/os\/mit\/media\/m\/social\/images\/social_default_logo-1481777.png"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/yahoo-2/expected-metadata.json b/test/test-pages/yahoo-2/expected-metadata.json
index 562a72b1..fe406c35 100644
--- a/test/test-pages/yahoo-2/expected-metadata.json
+++ b/test/test-pages/yahoo-2/expected-metadata.json
@@ -1,5 +1,5 @@
{
- "Title": "Russia: Space ship malfunctions, breaks up over Siberia",
+ "Title": "Yahoo News - Latest News & Headlines",
"Author": "NATALIYA VASILYEVA",
"Excerpt": "The latest news and headlines from Yahoo! News. Get breaking news stories and in-depth coverage with videos and photos."
}
diff --git a/test/test-pages/yahoo-2/expected.html b/test/test-pages/yahoo-2/expected.html
index f603205e..9bc5f6f5 100644
--- a/test/test-pages/yahoo-2/expected.html
+++ b/test/test-pages/yahoo-2/expected.html
@@ -30,5 +30,5 @@
Aerospace Writer Marcia Dunn in Cape Canaveral, Florida, and Vladimir Isachenkov in Moscow contributed to this report.
-
+
\ No newline at end of file
diff --git a/test/test-pages/yahoo-3/expected-images.json b/test/test-pages/yahoo-3/expected-images.json
index 33eb9b3f..f9727c23 100644
--- a/test/test-pages/yahoo-3/expected-images.json
+++ b/test/test-pages/yahoo-3/expected-images.json
@@ -1 +1 @@
-["https:\/\/s.yimg.com\/bt\/api\/res\/1.2\/qZaM9MLUOrxLg4IfXt_Niw--\/YXBwaWQ9eW5ld3NfbGVnbztxPTc1O3c9NjAw\/http:\/\/media.zenfs.com\/en-US\/video\/video.abcnewsplus.com\/559ecdbafdb839129816b5c79a996975.cf.png","https:\/\/s3.yimg.com\/bt\/api\/res\/1.2\/GNtA09EDJWzWfpBzGYJS0Q--\/YXBwaWQ9eW5ld3NfbGVnbztxPTg1O3c9NjMw\/http:\/\/media.zenfs.com\/en_us\/gma\/us.abcnews.gma.com\/HT_flag_baby_jtm_150311_16x9_992.jpg"]
\ No newline at end of file
+["https:\/\/s3.yimg.com\/bt\/api\/res\/1.2\/GNtA09EDJWzWfpBzGYJS0Q--\/YXBwaWQ9eW5ld3NfbGVnbztxPTg1O3c9NjMw\/http:\/\/media.zenfs.com\/en_us\/gma\/us.abcnews.gma.com\/HT_flag_baby_jtm_150311_16x9_992.jpg"]
\ No newline at end of file
diff --git a/test/test-pages/yahoo-3/expected-metadata.json b/test/test-pages/yahoo-3/expected-metadata.json
index 75d461da..1ffa6e7a 100644
--- a/test/test-pages/yahoo-3/expected-metadata.json
+++ b/test/test-pages/yahoo-3/expected-metadata.json
@@ -1,4 +1,5 @@
{
"Title": "Veteran Wraps Baby in American Flag, Photo Sparks Controversy",
- "Excerpt": "From Yahoo: A photographer and Navy veteran is fighting back after a photo she posted to Facebook started an online backlash. Vanessa Hicks said she had no idea her photo would be considered controversial. The photo, from a military family’s newborn photo shoot, showed a newborn infant wrapped in an American flag held by his father, who was in his military uniform. Hicks, a Navy veteran herself and the wife of an active-duty Navy member, said her intention was to honor the flag as well as her clients, who wanted to incorporate their military service in the photo shoot."
+ "Excerpt": "A photographer and Navy veteran is fighting back after a photo she posted to Facebook started an online backlash. Vanessa Hicks said she had no idea her photo would be considered controversial. The photo, from a military family’s newborn photo shoot, showed a newborn infant wrapped in an American flag held by his father, who was in his military uniform. Hicks, a Navy veteran herself and the wife of an active-duty Navy member, said her intention was to honor the flag as well as her clients, who wanted to incorporate their military service in the photo shoot.",
+ "Author": "By GILLIAN MOHNEY\n March 11, 2015 3:46 PM"
}
diff --git a/test/test-pages/yahoo-3/expected.html b/test/test-pages/yahoo-3/expected.html
index f3113b6b..c5d7e33d 100644
--- a/test/test-pages/yahoo-3/expected.html
+++ b/test/test-pages/yahoo-3/expected.html
@@ -1,6 +1,6 @@
-
'GMA' Cookie Search:
+
'GMA' Cookie Search:
@@ -19,11 +19,10 @@
“This is what he was fighting for, his son wrapped in an American flag,” Hicks told ABC News. However, when she posted the image on her page, she started to get comments accusing her of desecrating the flag.
On one Facebook page an unidentified poster put up her picture writing and wrote they found it was “disrespectful, rude, tacky, disgusting, and against the U.S. Flag Code.”
-
-
View photo
.
Vanessa Hicks
+
View photo
.
Vanessa Hicks
-
The Federal Flag Code has guidelines for the proper treatment of the U.S. Flag but there are no rules for punishment related to violations. In the past, the
has found that people are protected from punishment under the First Amendment for manipulating or even burning the flag.
+
The Federal Flag Code has guidelines for the proper treatment of the U.S. Flag but there are no rules for punishment related to violations. In the past, the Supreme Court has found that people are protected from punishment under the First Amendment for manipulating or even burning the flag.
Hicks said she was surprised when messages suddenly started to pop up on her Facebook page and even her own website criticizing her photos.
She said she stayed up until 4 a.m. recently to take down comments from her business and company page, even on shoots that had nothing to do with the flag.
diff --git a/test/test-pages/yahoo-4/expected-images.json b/test/test-pages/yahoo-4/expected-images.json
index 605166a6..0637a088 100644
--- a/test/test-pages/yahoo-4/expected-images.json
+++ b/test/test-pages/yahoo-4/expected-images.json
@@ -1 +1 @@
-["http:\/\/i.yimg.jp\/images\/jpnews\/cre\/common\/all\/images\/fbico_ogp_600x600.png"]
\ No newline at end of file
+[]
\ No newline at end of file
diff --git a/test/test-pages/yahoo-4/expected-metadata.json b/test/test-pages/yahoo-4/expected-metadata.json
index 0dac267f..8048fe7e 100644
--- a/test/test-pages/yahoo-4/expected-metadata.json
+++ b/test/test-pages/yahoo-4/expected-metadata.json
@@ -1,6 +1,6 @@
{
- "Title": "トレンドマイクロ、公衆無線LANを安全に使うためのアプリ「フリーWi-Fiプロテクション」 (CNET Japan) - Yahoo!ニュース",
+ "Title": "トレンドマイクロ、公衆無線LANを安全に使うためのアプリ「フリーWi-Fiプロテクション」(CNET Japan) - Yahoo!ニュース",
"Author": null,
"Direction": null,
- "Excerpt": " トレンドマイクロは3月9日、Wi-Fi利用時の通信を暗号化し保護するスマホ・タブレッ"
+ "Excerpt": "トレンドマイクロは3月9日、Wi-Fi利用時の通信を暗号化し保護するスマホ・タブレット - Yahoo!ニュース(CNET Japan)"
}
diff --git a/test/test-pages/yahoo-4/expected.html b/test/test-pages/yahoo-4/expected.html
index 4a50c77c..db7694ab 100644
--- a/test/test-pages/yahoo-4/expected.html
+++ b/test/test-pages/yahoo-4/expected.html
@@ -1,7 +1,9 @@
-
+ Anonymous +
++ One valuable reform would be to mandate, under penalty of felony, that anyone that searches your information has to notify you - to include private investigotors screening potential employers to police officers trolling your online profile. It would be a great consumer protection. +
++ For example: if you lost the job offer due to your Facebook post, the employer should be required to tell you that. If you got pulled over and ticketed because you "Liked" police body cameras on FB you should know that also. +
++ The real danger is allowing punitive actions, in retaliation of your Free Speech to be hidden from us. If someone is spying on us, bypassing privacy filters, without a judicial warrant - we should know that. +
++-
+ reply
+
+ -
+ April 5, 2018
+
+ -
+ 6:32 PM
+
+
++ Lars +
++ Couldn’t I just bloc facebook in my private filer (squid filter)? +
++-
+ reply
+
+ -
+ April 6, 2018
+
+ -
+ 2:20 AM
+
+
++ Anonymous +
++ privacy badger is also a good plugin to mention +
++-
+ reply
+
+ -
+ April 6, 2018
+
+ -
+ 12:13 PM
+
+
++ Anonymous +
++ The U.S. Justice Department and U.S. Congress - more than a decade behind schedule - still haven't reconciled the "Joseph Nacchio" case. Nacchio's telecommunications company, Qwest Communications, was apparently the one and only telecommunications company that refused to participate in the felony crime of Warrantless Wire-tapping around February of 2001 (roughly 6 months BEFORE 9/11 when no wartime emergency existed)., being pressured by the Bush Administration to commit a felony under federal law. +
++ Congress then passed an ex post facto law - illegal under Article I of the U.S. Constitution - making those past felonies legal. Congress has no such authority under Article VI. +
++ Joseph Nacchio claims he was then indicted by the U.S. Department of Justice - on trumped up charges - and served a few years in federal prison. A CEO of a large American corporation was sent to prison for COMPLYING with federal wiretaping laws. The Washington Post ran a story after Nacchio was released from prison, he still claims to this day that he went to prison for not participating in Bush felony crimes. +
++ What is to prevent a future lawless Executive Branch from pressuring Facebook or other social media networks to also bypass Judicial Warrants? If an honest U.S. Attorney or state prosecutor started indicting the former Bush folks, it would create a powerful "deterrent effect" to lawless officials in the future. +
++-
+ reply
+
+ -
+ April 6, 2018
+
+ -
+ 12:23 PM
+
+
++ Anonymous +
++ There is strong circumstantial evidence that "Post 9/11 Blacklisting" of U.S. citizens is still alive and well. Maybe the most dangerous blacklisting tactic is "Employment Tampering". It basically means if you were blacklisted after 9/11, you may not be allowed to work in certain occupations. The Government essentially decides where you are allowed work, without an official hearing by judge - blacklistees are never informed of their 2nd Class status as Americans - not entitled to the same rights as other Americans. +
++ "Employment Tampering" by government officials/contractors - which violates Title 18 US Code 242 - is one of the hardest blacklisting tactics for federal prosecutors to police. We may even need a new federal watchdog agency to properly police this felony crime by officials. +
++ Maybe the most famous case, was exposed by a federal appeals court that severely reprimanded former Attorney General John Ashcroft (although he was never criminally indicted). Ashcroft grossly and fraudulently exploited the federal "Material Witness Statute" to be used as a quasi form of imprisonment and involntary servitude. The 13th Amendment makes this practice totally illegal without an official and overt constitutional due process. +
++ This tactic has a long history in authoritarian regimes. It was originally called "Zersetzung" and was first used in 1940's Germany during World War Two to silence dissent. During the Cold War, this tactic was adopted by then Communist East Germany. The FBI started using it in the 1950's commonly called Cointelpro tactics. Ashcroft then exploited the Material Witness Statute to be used as a Zersetzung tactic. Employment Tampering was probably the most potent tactic. +
++-
+ reply
+
+ -
+ April 6, 2018
+
+ -
+ 2:35 PM
+
+
++ Anonymous +
++ This was the most elightening piece on Facebook Ive seen of late. +
++ Hate their format, so don't use it. +
++-
+ reply
+
+ -
+ April 6, 2018
+
+ -
+ 3:12 PM
+
+
++ Anonymous +
++ Didn't you read the article? You might not use it, but they still know all about you, even including name, address, phone number, etc. if you are in someone's contacts. +
++ Ironically, by refusing to register with them as a user, you have no control at all over the collection and usage of your data and information. At least users have some little control over it. +
++-
+ April 10, 2018
+
+ -
+ 10:14 PM
+
+
++ Ed B. -Netherlands +
++ Well written, well-thought article. I agree with the content. Although I hate regulation in general, it seems about time that website should be required to offer a cookie free, traceless version at all times. +
++-
+ reply
+
+ -
+ April 6, 2018
+
+ -
+ 5:29 PM
+
+
++ Anonymous +
++ HTTP is stateless. That means that when you click on a link to another page, the second page doesn't know that you were the person on page 1. Cookies, or some form of tracking, are needed for things like eCommerce and shopping carts. You need a cookie to associate *you* with your cart. Anything where you log in needs a cookie in order to know you are you, such as subscriptions. +
++ However, those cookies can easily be "session cookies". That is, cookies that don't persist after you log out or close your web browser. They exists solely for the duration of your session, and do not persist. +
++ Cookies are only an association mechanism. The heart of the problem is the harvesting of every click and everything on a page -- the over-collection of data to begin with. +
++-
+ April 11, 2018
+
+ -
+ 2:51 PM
+
+
++ Anonymous +
++ Great article. +
++-
+ reply
+
+ -
+ April 6, 2018
+
+ -
+ 5:42 PM
+
+
++ Pages +
++- 1
+
+ -
+ 2
+
+ -
+ 3
+
+ -
+ next
+
+ -
+ last »
+
+
+