changes

- Button now reflects whitelist status instantly
- Fixed bug where only last site whitelisted displayed properly
- No longer breaks when spammed
- Removed test code

.gitlab-ci.yml

@@ -24,26 +24,24 @@ hash rules:
       - data.minify.json
       - rules.minify.hash
 
-bundle addon firefox:
+bundle addon:
   stage: build
   script:
-    - zip ClearURLs-firefox -r -FS clearurls.js browser-polyfill.js manifest.json img/* external_js/* html/* core_js/* css/* fonts/* _locales/*
-  only:
-    - master
+    - zip ClearURLs -r -FS clearurls.js browser-polyfill.js manifest.json img/* external_js/* html/* core_js/* css/* fonts/* _locales/*
   artifacts:
     paths:
-      - ClearURLs-firefox.zip
+      - ClearURLs.zip
 
-bundle addon chrome:
-  stage: build
-  script:
-    - jq 'del(.applications)' manifest.json > manifest.json.tmp && mv manifest.json.tmp manifest.json
-    - zip ClearURLs-chrome -r -FS clearurls.js browser-polyfill.js manifest.json img/* external_js/* html/* core_js/* css/* fonts/* _locales/*
-  only:
-    - master
-  artifacts:
-    paths:
-      - ClearURLs-chrome.zip
+#bundle addon chrome:
+  #stage: build
+  #script:
+    #- jq 'del(.applications)' manifest.json > manifest.json.tmp && mv manifest.json.tmp manifest.json
+    #- zip ClearURLs-chrome -r -FS clearurls.js browser-polyfill.js manifest.json img/* external_js/* html/* core_js/* css/* fonts/* _locales/*
+  #only:
+  #  - master
+  #artifacts:
+  #  paths:
+  #    - ClearURLs-chrome.zip
 
 pages:
   stage: deploy
@@ -55,6 +53,8 @@ pages:
     - sha256sum public/data/data.min.json | awk '{print $1}' > public/data/rules.min.hash
     - node build_tools/minifyDataJSON.js "public/data/data.min.json" "public/data/data.minify.json"
     - sha256sum public/data/data.minify.json | awk '{print $1}' > public/data/rules.minify.hash
+  only:
+    - master
   artifacts:
     paths:
       - public

CHANGELOG.md

@@ -15,6 +15,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Require Firefox >= 55
 - Require Chrome >= 37
 
+## [1.27.0] - 2023-XX-XX
+
+### Fixed
+- [#276](https://github.com/ClearURLs/Addon/issues/276)
+- [#196](https://github.com/ClearURLs/Addon/issues/196)
+
+# Removed
+- ETag filtering for Firefox. Since Firefox 85, ETags can no longer be used for tracking users over multiple sites.
+
+### Compatibility note
+- Require Firefox >= 55
+- Require Chrome >= 37
+
+## [1.26.0] - 2022-11-18
+
+### Compatibility note
+- Require Firefox >= 55
+- Require Chrome >= 37
+
+### Changed
+- The popup window is now responsive 
 
 ## [1.25.0] - 2022-07-27
 

README.md

@@ -4,7 +4,7 @@
 
 [<img src="https://blog.mozilla.org/addons/files/2020/04/get-the-addon-fx-apr-2020.svg" alt="for Firefox" height="60px">](https://addons.mozilla.org/firefox/addon/clearurls/) [<img src="https://docs.clearurls.xyz/1.22.0/assets/img/MEA-button.png" alt="for Edge" height="60px">](https://microsoftedge.microsoft.com/addons/detail/mdkdmaickkfdekbjdoojfalpbkgaddei) [<img src="https://storage.googleapis.com/chrome-gcs-uploader.appspot.com/image/WlD8wC6g8khYWPJUsQceQkhXSlv1/HRs9MPufa1J1h5glNhut.png" alt="for Chrome" height="60px">](https://chrome.google.com/webstore/detail/clearurls/lckanjgmijmafbedllaakclkaicjfmnk)
 
-# <sub><img src="https://gitlab.com/KevinRoebert/ClearUrls/raw/master/img/clearurls.svg" width="64px" height="64px"></sub> ClearURLs [![Gitter](https://badges.gitter.im/ClearURLs/ClearURLs.svg)](https://gitter.im/ClearURLs/ClearURLs)
+# <sub><img src="https://gitlab.com/ClearURLs/ClearUrls/raw/master/img/clearurls.svg" width="64px" height="64px"></sub> ClearURLs [![Gitter](https://badges.gitter.im/ClearURLs/ClearURLs.svg)](https://gitter.im/ClearURLs/ClearURLs)
 
 **ClearURLs** is an add-on based on the new WebExtensions technology and is optimized for *Firefox* and *Chrome* based browsers.
 
@@ -37,7 +37,7 @@ Indeed most of the above URL is tracking code. Once ClearURLs has cleaned the ad
 * Prevents Yandex from rewriting the search results (to include tracking elements)
 
 ## Permissons
-Reasoning for needed permissions can be found under [here](https://gitlab.com/KevinRoebert/ClearUrls/issues/159).
+Reasoning for needed permissions can be found under [here](https://gitlab.com/ClearURLs/ClearUrls/issues/159).
 
 ## Screenshot
 ![Interface (version 1.14.0)](https://docs.clearurls.xyz/1.22.0/assets/img/interface.png)
@@ -45,14 +45,13 @@ Reasoning for needed permissions can be found under [here](https://gitlab.com/Ke
 ## CI/CD Artifacts Download (for Firefox- and Chrome-Dev only)
 Here you can download the packed files for the Firefox- and Chrome-Dev:
 
- - [ClearURLs-firefox.zip](https://gitlab.com/KevinRoebert/ClearUrls/-/jobs/artifacts/master/raw/ClearURLs-firefox.zip?job=bundle%20addon%20firefox)
- - [ClearURLs-chrome.zip](https://gitlab.com/KevinRoebert/ClearUrls/-/jobs/artifacts/master/raw/ClearURLs-chrome.zip?job=bundle%20addon%20chrome)
+ - [ClearURLs.zip](https://gitlab.com/ClearURLs/ClearUrls/-/jobs/artifacts/master/raw/ClearURLs.zip?job=bundle%20addon)
 
 ## Test
 If you want to test whether ClearURLs works correctly on your system, you can go to this test page: [https://test.clearurls.xyz/](https://test.clearurls.xyz/)
 
 ## Contribute
-If you have any suggestions or complaints, please [create an issue.](https://gitlab.com/KevinRoebert/ClearUrls/issues/new)
+If you have any suggestions or complaints, please [create an issue.](https://gitlab.com/ClearURLs/ClearUrls/issues/new)
 
 **Note: If you have any suggestions or complaints regarding the rules, please [create an issue in this repo](https://gitlab.com/anti-tracking/ClearURLs/rules/-/issues/new) or email us rules.support (at) clearurls.xyz (this mail will automatically create a new issue in this repo).**
 
@@ -65,7 +64,7 @@ You can choose between two options to contribute. You can create a merge request
 It is not necessary to translate the description field; in the most cases it is empty.*
 
 #### Merge request
-If you want to create a merge request, you must open the path [`_locales/en/messages.json`](https://github.com/KevinRoebert/ClearUrls/blob/master/_locales/en/messages.json) in the ClearURLs repo 
+If you want to create a merge request, you must open the path [`_locales/en/messages.json`](https://gitlab.com/ClearURLs/ClearUrls/-/blob/master/_locales/en/messages.json) in the ClearURLs repo 
 and translate the english terms into terms of your language. Once you have translated all the terms, you make a pull request of your translation. 
 Please push your translation into the folder `_locales/{language code}/messages.json`.
 

_locales/en/messages.json

@@ -87,6 +87,14 @@
         "message": "Show numbers of cleaned urls",
         "description": "This string is used as title for the badges switch button on the popup page."
     },
+    "popup_html_configs_whitelist_button_add": {
+        "message": "Whitelist Site",
+        "description": "This string is used as name for the whitelist button on the popup page."
+    },
+    "popup_html_configs_whitelist_button_remove": {
+        "message": "Remove from Whitelist",
+        "description": "This string is used as name for the whitelist button on the popup page."
+    },
     "popup_html_statistics_head": {
         "message": "Statistics",
         "description": "This string is used as title for the statistics on the popup page."
@@ -179,6 +187,10 @@
         "message": "The url to the rules.hash file (hash)",
         "description": "This string is used as name for the rule.hash url label."
     },
+    "setting_whitelist_list_label": {
+        "message": "Whitelisted sites",
+        "description": "This string is used as name for the whitelisted sites list label."
+    },
     "setting_types_label": {
         "message": "<a href='https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/ResourceType' target='_blank'>Request types</a> (expert level)",
         "description": "This string is used as name for the types label."

clearurls.js

@@ -47,6 +47,19 @@ function removeFieldsFormURL(provider, pureUrl, quiet = false, request = null) {
     let rawRules = provider.getRawRules();
     let urlObject = new URL(url);
 
+    /*
+    * Skip whitelisted sites
+    */
+    for (const site of storage.whitelist) {
+        if (url.indexOf(site) !== -1) {
+            return {
+                "changes": false,
+                "url": url,
+                "cancel": false
+            }
+        }
+    }
+
     if (storage.localHostsSkipping && checkLocalURL(urlObject)) {
         return {
             "changes": false,
@@ -152,7 +165,7 @@ function removeFieldsFormURL(provider, pureUrl, quiet = false, request = null) {
 
         let finalURL = domain;
 
-        if (fields.toString() !== "") finalURL += "?" + fields.toString();
+        if (fields.toString() !== "") finalURL += "?" + urlSearchParamsToString(fields);
         if (fragments.toString() !== "") finalURL += "#" + fragments.toString();
 
         url = finalURL.replace(new RegExp("\\?&"), "?").replace(new RegExp("#&"), "#");

core_js/eTagFilter.js

@@ -31,8 +31,16 @@ function eTagFilter(requestDetails) {
             continue;
         }
 
+        const etag = header.value.toLowerCase();
+        const w = etag.startsWith('w');
+        const quotes = etag.endsWith('"');
+
+        let len = etag.length;
+        if (w) len -= 2;
+        if (quotes) len -= 2;
+
         // insert dummy etag
-        requestDetails.responseHeaders[i].value = generateDummyEtag();
+        requestDetails.responseHeaders[i].value = generateDummyEtag(len, quotes, w);
 
         pushToLog(requestDetails.url, requestDetails.url, translate("eTag_filtering_log"));
 
@@ -42,12 +50,29 @@ function eTagFilter(requestDetails) {
     return {responseHeaders: requestDetails.responseHeaders};
 }
 
-function generateDummyEtag() {
-    return Math.random().toString();
+/**
+ * Generates a random ETag.
+ * 
+ * Must be ASCII characters placed between double quotes.
+ * See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag
+ */
+function generateDummyEtag(len, quotes = true, w = false) {
+    let rtn = randomASCII(len);
+
+    if (quotes) rtn = '"' + rtn + '"';
+    if (w) rtn = 'W/' + rtn;
+
+    return rtn;
 }
 
-browser.webRequest.onHeadersReceived.addListener(
-    eTagFilter,
-    {urls: ["<all_urls>"]},
-    ["blocking", "responseHeaders"]
-);
+/**
+ * Since Firefox 85, eTags can no longer be 
+ * used for tracking users over multiple sites.
+ */
+if(getBrowser() !== "Firefox") {
+    browser.webRequest.onHeadersReceived.addListener(
+        eTagFilter,
+        {urls: ["<all_urls>"]},
+        ["blocking", "responseHeaders"]
+    );
+}

core_js/google_link_fix.js

@@ -27,6 +27,9 @@
     function injectFunction() {
         let ele = document.createElement('script');
         let s = document.getElementsByTagName('script')[0];
+        if (s === undefined) {
+            return;
+        }
 
         ele.type = 'text/javascript';
         ele.textContent = "Object.defineProperty(window, 'rwt', {" +

core_js/historyListener.js

@@ -44,7 +44,7 @@ function historyCleaner(details) {
         if(urlBefore !== urlAfter) {
             browser.tabs.executeScript(details.tabId, {
                 frameId: details.frameId,
-                   code: 'history.replaceState({state: "cleaned_history"},"",'+JSON.stringify(urlAfter)+');'
+                   code: 'history.replaceState(null,"",'+JSON.stringify(urlAfter)+');'
             }).then(() => {}, onError);
         }
     }

core_js/popup.js

@@ -62,6 +62,34 @@ function changeStatistics()
     elTotal.textContent = totalCounter.toLocaleString();
 }
 
+/**
+* Set the whitelist button text
+*/
+function setWhitelistText()
+{
+    let element = document.getElementById('whitelist_btn');
+    let currentSite;
+    browser.tabs.query({active: true, currentWindow: true}, function(tabs) {
+        currentSite = tabs[0].url;
+    });
+
+    browser.runtime.sendMessage({
+        function: "getData",
+        params: ['whitelist']
+    }).then((data) => {
+        let siteFound = data.response.some(site => currentSite.indexOf(site) !== -1);
+        if (siteFound) {
+            element.classList.replace('btn-primary', 'btn-danger')
+            element.textContent = translate('popup_html_configs_whitelist_button_remove')
+            document.getElementById('whitelist_btn').onclick = () => {changeWhitelist(true)};
+        } else {
+            element.classList.replace('btn-danger', 'btn-primary')
+            element.textContent = translate('popup_html_configs_whitelist_button_add')
+            document.getElementById('whitelist_btn').onclick = () => {changeWhitelist(false)};
+        }
+    }).catch(handleError);
+}
+
 /**
 * Set the value for the hashStatus on startUp.
 */
@@ -155,6 +183,36 @@ function setSwitchButton(id, varname)
     element.checked = this[varname];
 }
 
+/**
+* Adds (or removes) the site the user is on to the whitelist
+* Whitelisted sites do not get filtered
+* @param {boolean} removeWl If true remove current site instead of adding
+*/
+function changeWhitelist(removeWl) {
+    let site;
+    browser.tabs.query({active: true, currentWindow: true}, function(tabs) { // Couldn't figure out how to access currentUrl var
+        site = tabs[0].url;                                                  // So this is used instead
+    });
+    browser.runtime.sendMessage({
+        function: "getData",
+        params: ['whitelist']
+    }).then((data) => {
+        let siteUrl = new URL(site)
+        let domain = siteUrl.hostname
+        if (removeWl) {
+            data.response = data.response.filter(wlSite => wlSite !== domain)
+        } else {
+            data.response.push(domain)
+        }
+        browser.runtime.sendMessage({
+            function: "setData",
+            params: ['whitelist', data.response]
+        }).then(() => {
+            setWhitelistText();
+        }).catch(handleError);
+    }).catch(handleError);
+}
+
 /**
 * Reset the global statistic
 */
@@ -220,6 +278,7 @@ function setText()
     injectText('configs_switch_filter','popup_html_configs_switch_filter');
     injectText('configs_head','popup_html_configs_head');
     injectText('configs_switch_statistics','configs_switch_statistics');
+    setWhitelistText();
     document.getElementById('donate').title = translate('donate_button');
 }
 

core_js/settings.js

@@ -82,6 +82,7 @@ function save() {
     saveData("badged_color", pickr.getColor().toHEXA().toString())
         .then(() => saveData("ruleURL", document.querySelector('input[name=ruleURL]').value))
         .then(() => saveData("hashURL", document.querySelector('input[name=hashURL]').value))
+        .then(() => saveData("whitelist", document.querySelector('input[name=whitelist]').value.split(',')))
         .then(() => saveData("types", document.querySelector('input[name=types]').value))
         .then(() => saveData("logLimit", Math.max(0, Math.min(5000, document.querySelector('input[name=logLimit]').value))))
         .then(() => browser.runtime.sendMessage({
@@ -122,6 +123,7 @@ function getData() {
 
     loadData("ruleURL")
         .then(() => loadData("hashURL"))
+        .then(() => loadData("whitelist"))
         .then(() => loadData("types"))
         .then(() => loadData("logLimit"))
         .then(logData => {
@@ -147,6 +149,20 @@ function getData() {
             changeSwitchButton("domainBlocking", "domainBlocking");
             changeSwitchButton("pingBlocking", "pingBlocking");
             changeSwitchButton("eTagFiltering", "eTagFiltering");
+        })
+        .then(() => {
+            /**
+             * Since Firefox 85, eTags can no longer be 
+             * used for tracking users over multiple sites.
+             */
+            browser.runtime.sendMessage({
+                function: "getBrowser",
+                params: []
+            }).then(resp => {
+                if(resp.response === "Firefox") {
+                    document.getElementById('etag_p').remove();
+                }
+            }, null);
         }).catch(handleError);
 }
 
@@ -202,6 +218,7 @@ function setText() {
     document.getElementById('reset_settings_btn').setAttribute('title', translate('setting_html_reset_button_title'));
     document.getElementById('rule_url_label').textContent = translate('setting_rule_url_label');
     document.getElementById('hash_url_label').textContent = translate('setting_hash_url_label');
+    document.getElementById('whitelist_list_label').textContent = translate('setting_whitelist_list_label');
     document.getElementById('types_label').innerHTML = translate('setting_types_label');
     document.getElementById('save_settings_btn').textContent = translate('settings_html_save_button');
     document.getElementById('save_settings_btn').setAttribute('title', translate('settings_html_save_button_title'));

core_js/storage.js

@@ -216,6 +216,7 @@ function initSettings() {
     storage.badged_color = "#ffa500";
     storage.hashURL = "https://rules2.clearurls.xyz/rules.minify.hash";
     storage.ruleURL = "https://rules2.clearurls.xyz/data.minify.json";
+    storage.whitelist = []; // TODO: If we do whitelist per rule, this needs to be obj
     storage.contextMenuEnabled = true;
     storage.historyListenerEnabled = true;
     storage.localHostsSkipping = true;

core_js/tools.js

@@ -328,3 +328,30 @@ async function sha256(message) {
 
     return hashArray.map(b => b.toString(16).padStart(2, '0')).join('');
 }
+
+/**
+ * Generates a non-secure random ASCII string of length {@code len}.
+ * 
+ * @returns non-secure random ASCII
+ */
+function randomASCII(len) {
+    return [...Array(len)].map(() => (~~(Math.random() * 36)).toString(36)).join('');
+}
+
+/**
+ * Returns an URLSearchParams as string.
+ * Does handle spaces correctly.
+ */
+function urlSearchParamsToString(searchParams) {
+    const rtn = []
+
+    searchParams.forEach((value, key) => {
+        if (value) {
+            rtn.push(key + '=' + value)
+        } else {
+            rtn.push(key)
+        }
+    })
+
+    return rtn.join('&')
+}

html/popup.html

@@ -32,9 +32,9 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 </head>
 <body>
         <div class="container-fluid"
-        style="min-width: 200px;">
+        style="min-width: 200px">
         <div class="row">
-            <nav class="col-sm-1 navbar-margin navbar navbar-dark bg-dark">
+            <nav class="col-sm-12 navbar-margin navbar navbar-dark bg-dark">
                 <div class="container">
                         <span class="navbar-brand">
                             <span class="float-left"><img src="../img/clearurls.svg"
@@ -53,14 +53,14 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
             </div>
 
             <div class="row" id="references_section">
-                <div class="col-sm-1 text-center">
+                <div class="col-sm-4 offset-sm-4 text-center">
                     <a id="cleaning_tools" target="_blank"><span class="fas fa-tools" style="font-size: 1.5em; margin-right: 1em;"></span></a>
                     <a id="settings" target="_blank"><span class="fas fa-cog" style="font-size: 1.5em"></span></a>
                 </div>
             </div>
 
             <div class="row" id="config_section">
-                <div class="col-sm-1">
+                <div class="col-sm-4 offset-sm-4">
                     <h5><b id="configs_head"></b></h5>
                     <label class="switch">
                         <input type="checkbox" id="globalStatus">
@@ -86,12 +86,15 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
                         <label id="configs_switch_statistics"></label>
                     </label>
                     <div class="clearfix"></div>
+                    <div class="text-center">
+                        <button type="button" id="whitelist_btn" class="btn btn-primary btn-sm text-wrap"></button>
+                    </div>
                     <br />
                 </div>
             </div>
 
             <div class="row" id="statistic_section">
-                <div class="col-sm-1">
+                <div class="col-sm-4 offset-sm-4">
                     <h5><b id="statistics_head"></b></h5>
 
                     <div class="progress">
@@ -133,7 +136,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
         </div>
 
         <div class="row" id="status_section">
-            <div class="col-sm-1">
+            <div class="col-sm-4 offset-sm-4">
                 <h5><b id="rules_status_head"></b></h5>
                 <div class="text-center">
                     <a href="https://gitlab.com/ClearURLs/rules/-/commits/master/data.min.json"
@@ -145,7 +148,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
         </div>
         <br />
         <div class="row" id="log_section">
-            <div class="col-sm-1">
+            <div class="col-sm-4 offset-sm-4">
                 <div class="text-center">
                     <a id="loggingPage" target="_blank"
                     class="btn btn-default btn-sm btn-block text-wrap"></a>

html/settings.html

@@ -105,6 +105,11 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
                     <input type="url" id="hashURL" value="" name="hashURL" class="form-control" />
                 </p>
                 <br />
+                <p>
+                    <label id="whitelist_list_label"></label><br />
+                    <input type="text" id="whitelist" value="" name="whitelist" class="form-control" />
+                </p>
+                <br />
                 <p>
                     <label id="types_label"></label><br />
                     <input type="text" id="types" value="" name="types" class="form-control" />
@@ -157,7 +162,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
                         <span class="slider round"></span>
                     </label>
                 </p>
-                <p>
+                <p id="etag_p">
                     <label id="eTag_filtering_enabled" style="font-weight: bold;"></label><br />
                     <label class="switch">
                         <input type="checkbox" id="eTagFiltering">

manifest.json

@@ -1,13 +1,13 @@
 {
   "manifest_version": 2,
   "name": "ClearURLs",
-  "version": "1.25.0",
+  "version": "1.27.0",
   "author": "Kevin Roebert",
   "description": "__MSG_extension_description__",
   "homepage_url": "https://docs.clearurls.xyz",
   "default_locale": "en",
   "content_security_policy": "script-src 'self'; object-src 'none'",
-  "applications": {
+  "browser_specific_settings": {
     "gecko": {
       "id": "{74145f27-f039-47ce-a470-a662b129930a}"
     }