| BlackWeb brings together public domain blocklists and adapts them for use as access control lists (ACLs) in Squid-Cache. It includes domains related to adult content, downloads, drugs, malware, spyware, trackers, bots, social networks, warez, weapons, and other categories. | BlackWeb reúne listas públicas de dominios y las adapta para usarlas como listas de control de acceso (ACL) en Squid-Cache. Incluye dominios relacionados con contenido para adultos, descargas, drogas, malware, spyware, rastreadores, bots, redes sociales, warez y armas, entre otras categorías. |
squid(orsquid-openssl) to apply the BlackWeb rules
apt install -y squid
The archive includes blackweb.txt, prepared for use as a domain ACL in Squid. To use it, download the archive and save the file in a path Squid can access; then add the block rule shown below to the configuration.
|
El archivo comprimido incluye blackweb.txt, preparado para usarse como ACL de dominios en Squid. Para aplicarlo, debe descargarse y guardarse en una ruta accesible para Squid; después, se añade a la configuración la regla de bloqueo que aparece abajo.
|
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -#!/bin/bash
# Variables
url="https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz"
wgetd="wget -q -c --timestamping --no-check-certificate --retry-connrefused --timeout=10 --tries=4 --show-progress"
# TMP folder
output_dir="bwtmp"
mkdir -p "$output_dir"
# Download
if $wgetd "$url"; then
echo "File downloaded: $(basename $url)"
else
echo "Main file not found. Searching for multiparts..."
# Multiparts from a to z
all_parts_downloaded=true
for part in {a..z}{a..z}; do
part_url="${url%.*}.$part"
if $wgetd "$part_url"; then
echo "Part downloaded: $(basename $part_url)"
else
echo "Part not found: $part"
all_parts_downloaded=false
break
fi
done
if $all_parts_downloaded; then
# Rebuild the original file in the current directory
cat blackweb.tar.gz.* > blackweb.tar.gz
echo "Multipart file rebuilt"
else
echo "Multipart process cannot be completed"
exit 1
fi
fi
# Unzip the file to the output folder
tar -xzf blackweb.tar.gz -C "$output_dir"
echo "Done"wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.txt.sha256
LOCAL=$(sha256sum blackweb.txt | awk '{print $1}'); REMOTE=$(awk '{print $1}' blackweb.txt.sha256); echo "$LOCAL" && echo "$REMOTE" && [ "$LOCAL" = "$REMOTE" ] && echo OK || echo FAILBlackWeb Rule for Squid-Cache
| Open: | Abra: |
/etc/squid/squid.conf| Add the following lines at the indicated location: | Añada las siguientes líneas en el punto indicado: |
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
# Block Rule for Blackweb
acl blackweb dstdomain "/path_to/blackweb.txt"
http_access deny blackweb| BlackWeb contains millions of domains, therefore it is recommended: | BlackWeb contiene millones de dominios, por tanto se recomienda: |
Add domains that should be exempt from blocking to allowdomains.txt, such as .accounts.google.com, .yahoo.com or .github.com. Place http_access allow allowdomains before the rules that deny access; otherwise, the exception may not take effect. According to Squid's documentation, Google may use the subdomains accounts.google.com and accounts.youtube.com for authentication within its ecosystem. Blocking them could disrupt access to services such as Gmail, Drive and Docs.
|
En allowdomains.txt se incluyen los dominios que deben excluirse del bloqueo, como .accounts.google.com, .yahoo.com o .github.com. La regla http_access allow allowdomains debe colocarse antes de las reglas que deniegan el acceso; de lo contrario, la excepción puede no surtir efecto. Según la documentación de Squid, Google puede utilizar los subdominios accounts.google.com y accounts.youtube.com para la autenticación dentro de su ecosistema. Bloquearlos podría interrumpir el acceso a servicios como Gmail, Drive y Docs.
|
acl allowdomains dstdomain "/path_to/allowdomains.txt"
http_access allow allowdomains
Use blockdomains.txt to list additional domains to block. This is a separate ACL; it does not automatically block every domain absent from blackweb.txt.
|
blockdomains.txt contiene los dominios adicionales que se bloquearán mediante esa ACL. Es una lista independiente: no bloquea automáticamente todos los dominios que no aparezcan en blackweb.txt.
|
acl blockdomains dstdomain "/path_to/blockdomains.txt"
http_access deny blockdomains
Use blocktlds.txt to block entire top-level domains, such as gTLDs, sTLDs or ccTLDs.
|
blocktlds.txt contiene los TLD completos que se bloquearán, como gTLD, sTLD o ccTLD.
|
acl blocktlds dstdomain "/path_to/blocktlds.txt"
http_access deny blocktldsInput:
.bardomain.xxx
.subdomain.bardomain.xxx
.bardomain.ru
.bardomain.adult
.foodomain.com
.foodomain.pornOutput:
.foodomain.com
This optional rule blocks domain names containing a Punycode label (xn--). It may reduce some homograph risks, but it can also block legitimate internationalized domains. Review and test the rule before enabling it. For background, see Welivesecurity: Homograph attacks.
|
Esta regla opcional bloquea dominios cuyo nombre contiene una etiqueta Punycode (xn--). Puede ayudar a reducir ciertos riesgos de ataques homográficos, pero también puede bloquear dominios internacionalizados legítimos. Conviene revisar y probar la regla antes de activarla. Para más información, puede consultarse Welivesecurity: Ataques homográficos.
|
acl punycode dstdom_regex -i \.xn--.*
http_access deny punycodeInput:
.bücher.com
.mañana.com
.google.com
.auth.wikimedia.org
.xn--fiqz9s
.xn--p1aiASCII Output:
.google.com
.auth.wikimedia.org| This optional rule blocks URLs that match patterns in the file. It may produce false positives, so review the list before enabling the rule. | Esta regla opcional bloquea las URL que coincidan con los patrones del archivo. Puede generar falsos positivos; conviene revisar la lista antes de activarla. |
# Example: Download ACL:
sudo wget -P /etc/acl/squid https://raw.githubusercontent.com/maravento/vault/refs/heads/master/gateproxy/acl/squid/blockpatterns.txt
# Squid Rule to Block Patterns (change path):
acl blockpatterns url_regex -i "/etc/acl/squid/blockpatterns.txt"
http_access deny blockpatternsInput:
.bittorrent.com
https://www.google.com/search?q=torrent
https://www.google.com/search?q=mydomain
https://www.google.com/search?q=porn
.mydomain.comOutput:
https://www.google.com/search?q=mydomain
.mydomain.com
Use streaming.txt to block streaming domains not included in blackweb.txt, such as .youtube.com, .googlevideo.com or .ytimg.com. To allow them instead, use an allow rule in place of the example deny rule.
|
streaming.txt permite bloquear dominios de servicios de streaming que no estén en blackweb.txt, por ejemplo .youtube.com, .googlevideo.com o .ytimg.com. Para permitirlos, se puede configurar una regla allow en lugar de la regla deny del ejemplo.
|
acl streaming dstdomain "/path_to/streaming.txt"
http_access deny streaming
Note: This list may contain overlapping domains. It is important to manually clean it according to the proposed objective. Example:
|
Nota: Esta lista puede contener dominios superpuestos. Conviene depurarla manualmente según el objetivo. Por ejemplo:
|
# Block Facebook
.fbcdn.net
.facebook.com
# Block some Facebook streaming content
.z-p3-video.flpb1-1.fna.fbcdn.net
Use ai.txt to block domains related to artificial intelligence. To allow them instead, use an allow rule in place of the example deny rule.
|
ai.txt permite bloquear dominios relacionados con servicios de inteligencia artificial. Para permitirlos, se puede configurar una regla allow en lugar de la regla deny del ejemplo.
|
acl ai dstdomain "/path_to/ai.txt"
http_access deny ai# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
# Allow Rule for Domains
# https://raw.githubusercontent.com/maravento/blackweb/master/bwupdate/lst/allowdomains.txt
acl allowdomains dstdomain "/path_to/allowdomains.txt"
http_access allow allowdomains
# Block Rule for Punycode
acl punycode dstdom_regex -i \.xn--.*
http_access deny punycode
# Block Rule for gTLD, sTLD, ccTLD
# https://raw.githubusercontent.com/maravento/blackweb/master/bwupdate/lst/blocktlds.txt
acl blocktlds dstdomain "/path_to/blocktlds.txt"
http_access deny blocktlds
# Block Rule for Domains
# https://raw.githubusercontent.com/maravento/blackweb/master/bwupdate/lst/blockdomains.txt
acl blockdomains dstdomain "/path_to/blockdomains.txt"
http_access deny blockdomains
# Block Rule for Patterns (Optional)
# https://raw.githubusercontent.com/maravento/vault/refs/heads/master/gateproxy/acl/squid/blockpatterns.txt
acl blockpatterns url_regex -i "/path_to/blockpatterns.txt"
http_access deny blockpatterns
# Block Rule for Blackweb
# https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz
acl blackweb dstdomain "/path_to/blackweb.txt"
http_access deny blackweb| ACL | Blocked Domains | File Size |
|---|---|---|
| blackweb.txt | 7963236 | 194,5 MB |
bwupdate/
├── bwupdate.sh # Main script: downloads lists, builds blackweb.txt and reloads Squid
├── lst/ # ACLs and helper lists used by bwupdate.sh
│ ├── ai.txt # Separate Squid ACL for AI-related domains; not modified by bwupdate.sh
│ ├── allowdomains.txt # Separate Squid ACL for exempt domains; not modified by bwupdate.sh
│ ├── allowtlds.txt # Allowed-TLD pattern used during debugging
│ ├── blockdomains.txt # Separate Squid ACL for additional blocked domains; not modified by bwupdate.sh
│ ├── blocktlds.txt # Separate Squid ACL for blocking whole TLDs; not modified by bwupdate.sh
│ ├── debugbl.txt # Added to the candidate blocklist before debugging
│ ├── debugwl.txt # Excludes false positives during debugging (Google, Hotmail, Yahoo, etc.)
│ ├── invalid.txt # Added to urls.txt during debugging, along with debugwl.txt
│ ├── pendingtld.txt # Not read by scripts; maintainer reference
│ ├── streaming.txt # Separate Squid ACL for streaming domains; not modified by bwupdate.sh
│ ├── tldsappx.txt # TLD source used by dofi/domfilter.py
│ └── tldsbk.txt # Not read by scripts; maintainer reference
└── tools/
├── checksources.sh # Downloads the source lists and searches them for a domain
└── debugerror.py # Removes domains flagged in Squid's cache.log when their parent domain is already blocked
This section describes how bwupdate.sh updates and processes the list. Running the script is not required to use the published blackweb.txt. Updates can take time and use substantial CPU, memory, network and bandwidth resources; run the process on a test system.
|
Esta sección describe el proceso que sigue bwupdate.sh para actualizar y depurar la lista. No es necesario ejecutar este script para usar el archivo blackweb.txt publicado. La actualización puede tardar y consumir muchos recursos de CPU, memoria, red y ancho de banda; se recomienda ejecutarla en un equipo de pruebas.
|
Note: bwupdate.sh tested on Ubuntu 24.04/26.04 LTS. Use on other versions or distributions is at your own risk.
|
Nota: bwupdate.sh ha sido probado en Ubuntu 24.04/26.04 LTS. Su uso en otras versiones o distribuciones queda bajo su propio riesgo.
|
- Python 3.x, Bash 5.x
wget,curl,tar,gzip,idn2,squid(orsquid-openssl),python3,bind9-host,findutils,grep,sed,coreutils,util-linux,file,libc-bin,sudo- Python module:
requests(required bydomfilter.py)
apt install -y wget curl tar gzip idn2 squid python3 bind9-host findutils grep sed coreutils python3-requests util-linux file libc-bin sudo
bwupdate.sh runs the steps needed to generate and update blackweb.txt in sequence. The script requests elevated privileges when an operation requires them.
|
bwupdate.sh ejecuta en secuencia los pasos necesarios para generar y actualizar blackweb.txt. Cuando una operación requiere privilegios, el script los solicita.
|
git clone --depth=1 https://github.com/maravento/blackweb.git && cd blackweb/bwupdate && ./bwupdate.sh| Before continuing, check that Squid is installed and its service is running. | Antes de continuar, conviene comprobar que Squid esté instalado y que el servicio funcione. |
| Downloads the public blocklists listed under SOURCES, extracts their domains and combines them for further processing. | Descarga las listas públicas indicadas en SOURCES, extrae sus dominios y los reúne para continuar con la depuración. |
Removes redundant domains when a broader entry already covers them, normalizes entries for Squid ACLs, and applies debugwl.txt to retain domains that should not be blocked.
|
Elimina dominios redundantes cuando uno ya queda cubierto por otro más general, normaliza las entradas al formato de ACL de Squid y aplica la lista debugwl.txt para conservar dominios que deben excluirse del bloqueo.
|
Input:
com
.com
.domain.com
domain.com
0.0.0.0 domain.com
127.0.0.1 domain.com
::1 domain.com
domain.com.co
foo.bar.subdomain.domain.com
.subdomain.domain.com.co
www.domain.com
www.foo.bar.subdomain.domain.com
domain.co.uk
xxx.foo.bar.subdomain.domain.co.ukOutput:
.domain.com
.domain.com.co
.domain.co.uk| Removes domains whose suffix is not in the configured list of valid TLDs. The list covers different levels of public and private suffixes, including domains with up to four labels. | Elimina los dominios cuyo sufijo no figura entre los TLD válidos de las fuentes configuradas. La lista contempla distintos niveles de sufijos públicos y privados, incluidos dominios de hasta cuatro etiquetas. |
Input:
.domain.exe
.domain.com
.domain.edu.coOutput:
.domain.com
.domain.edu.co|
Removes domain names that exceed length limits or contain unsupported characters, as defined by RFC 1035 and the IDN tools. It also converts domains containing non-ASCII characters to Punycode/IDNA so they can be validated and processed as ASCII names. |
Elimina nombres de dominio que exceden los límites de longitud o contienen caracteres no admitidos. También convierte los dominios con caracteres no ASCII a formato Punycode/IDNA, para que puedan validarse y procesarse como nombres ASCII. Esta conversión no significa que todos esos dominios se usen para ataques homográficos. |
Input:
bücher.com
café.fr
españa.com
köln-düsseldorfer-rhein-main.de
mañana.com
mūsųlaikas.lt
sendesık.com
президент.рфOutput:
xn--bcher-kva.com
xn--caf-dma.fr
xn--d1abbgf6aiiy.xn--p1ai
xn--espaa-rta.com
xn--kln-dsseldorfer-rhein-main-cvc6o.de
xn--maana-pta.com
xn--mslaikas-qzb5f.lt
xn--sendesk-wfb.com| Removes entries with invalid encoding, non-printable characters, whitespace or unsupported symbols. It then converts the output to ASCII text for further domain validation and processing: | Elimina entradas con codificación inválida, caracteres no imprimibles, espacios o símbolos no admitidos. Después convierte la salida a texto ASCII para continuar con la validación y el procesamiento de los dominios: |
Input:
M-C$
-$
.$
0$
1$
23andmê.com
.òutlook.com
.ălibăbă.com
.ămăzon.com
.ăvăst.com
.amùazon.com
.aməzon.com
.avalón.com
.bĺnance.com
.bitdẹfender.com
.blóckchain.site
.blockchaiǹ.com
.cashpluÈ™.com
.dẹll.com
.diócesisdebarinas.org
.disnẹylandparis.com
.ebăy.com
.əməzon.com
.evo-bancó.com
.goglÄ™.com
.gooÄŸle.com
.googļę.com
.googlÉ™.com
.google.com
.ibẹria.com
.imgúr.com
.lloydÅŸbank.com
.mýetherwallet.com
.mrgreęn.com
.myẹthẹrwallet.com
.myẹthernwallet.com
.myethẹrnwallet.com
.myetheá¹™wallet.com
.myethernwallẹt.com
.nętflix.com
.paxfùll.com
.türkiyeisbankasi.com
.třezor.com
.westernúnion.com
.yòutube.com
.yăhoo.com
.yoütübe.co
.yoütübe.com
.yoütu.beOutput:
.google.com
Many SOURCES contain invalid or nonexistent domains, so DNS queries run in two stages to discard domains that do not resolve. The queries run in parallel and can use substantial CPU and bandwidth. The parallel_procs variable controls concurrency:
|
El proceso comprueba los dominios mediante consultas DNS en dos etapas para descartar los que no resuelven. Las consultas se ejecutan en paralelo y pueden consumir bastante CPU y ancho de banda. parallel_procs permite ajustar la cantidad de consultas simultáneas según los recursos del equipo y la red:
|
parallel_procs=$(($(nproc))) # Conservative (network-friendly)
parallel_procs=$(($(nproc) * 2)) # Balanced
parallel_procs=$(($(nproc) * 4)) # Aggressive (default)
parallel_procs=$(($(nproc) * 8)) # Extreme (8 or higher, use with caution)| For example, on a system with a Core i5 CPU (4 physical cores and 8 threads with Hyper-Threading): | Por ejemplo, en un equipo con CPU Core i5, cuatro núcleos físicos y ocho subprocesos con Hyper-Threading: |
nproc → 8
parallel_procs=$((8 * 4)) → 32 parallel queries
parallel_procs values can speed up DNS resolution, but may saturate CPU or bandwidth, especially on constrained networks such as satellite links. The following shows an example of the live progress output:
|
parallel_procs pueden acelerar la resolución DNS, pero también saturar la CPU o el ancho de banda, especialmente en redes limitadas como los enlaces satelitales. El siguiente ejemplo muestra el progreso durante la ejecución:
|
Processed: 2463489 / 7244989 (34.00%)Output (manual run):
HIT google.com
google.com has address 142.251.35.238
google.com has IPv6 address 2607:f8b0:4008:80b::200e
google.com mail is handled by 10 smtp.google.com.
FAULT testfaultdomain.com
Host testfaultdomain.com not found: 3(NXDOMAIN)| Remove government domains (.gov) and other related TLDs from BlackWeb. | Elimina de BlackWeb los dominios de gobierno (.gov) y otros TLD relacionados. |
Input:
.argentina.gob.ar
.mydomain.com
.gob.mx
.gov.uk
.navy.milOutput:
.mydomain.com
When Squid runs with BlackWeb, errors are written to SquidErrors.txt.
|
Al ejecutar Squid-Cache con BlackWeb, los errores se escriben en SquidErrors.txt.
|
Both bwupdate.sh and checksources.sh write a log file (bwupdate.log or checksources.log) in the directory containing the script.
|
bwupdate.sh y checksources.sh escriben un archivo de registro (bwupdate.log o checksources.log) en el directorio donde se encuentra cada script.
|
| Tag | Shows | English | Español |
|---|---|---|---|
SAVED: |
File name | The transfer completed and the file was written | La descarga terminó y el archivo se guardó |
PARTIAL: |
Full URL | The download started and was cut off before finishing | La descarga comenzó y se interrumpió antes de terminar |
BUSY: |
Full URL | The server answered 5xx: it is up but not serving the list right now | El servidor respondió con un error 5xx: está disponible, pero no entrega la lista en ese momento |
TIMEOUT: |
Full URL | The server did not answer at all | El servidor no respondió |
BROKEN: |
Full URL | The server answered 404 or 410: broken or nonexistent URL | El servidor respondió 404 o 410: el recurso no existe o la URL ya no es válida |
|
|
- ABPindo - indonesianadblockrules
- abuse.ch - hostfile
- Adaway - host
- adblockplus - advblock Russian
- adblockplus - antiadblockfilters
- adblockplus - easylistchina
- adblockplus - easylistlithuania
- anudeepND - adservers
- anudeepND - coinminer
- AssoEchap - stalkerware-indicators
- azet12 - KADhosts
- BarbBlock - blacklists
- BBcan177 - minerchk
- BBcan177 - MS-2
- BBcan177 - referrer-spam-blacklist
- betterwebleon - slovenian-list
- bigdargon - hostsVN
- BlackJack8 - iOSAdblockList
- BlackJack8 - webannoyances
- blocklistproject - everything
- chadmayfield - porn top
- chadmayfield - porn_all
- chainapsis - phishing-block-list
- cjx82630 - Chinese CJX's Annoyance List
- cobaltdisco - Google-Chinese-Results-Blocklist
- CriticalPathSecurity - Public-Intelligence-Feeds
- DandelionSprout - adfilt
- Dawsey21 - adblock-list
- Dawsey21 - main-blacklist
- developerdan - ads-and-tracking-extended
- Disconnect.me - simple_ad
- Disconnect.me - simple_malvertising
- Disconnect.me - simple_tracking
- Eallion - uBlacklist
- EasyList - EasyListHebrew
- ethanr - dns-blacklists
- fabriziosalmi - blacklists
- firebog - AdguardDNS
- firebog - Admiral
- firebog - Easylist
- firebog - Easyprivacy
- firebog - Kowabit
- firebog - neohostsbasic
- firebog - Prigent-Ads
- firebog - Prigent-Crypto
- firebog - Prigent-Malware
- firebog - RPiList-Malware
- firebog - RPiList-Phishing
- firebog - WaLLy3K
- frogeye - firstparty-trackers-hosts
- gardar - Icelandic ABP List
- greatis - Anti-WebMiner
- hagezi - dns-blocklists
- hexxium - threat-list/
- hoshsadiq - adblock-nocoin-list
- jawz101 - potentialTrackers
- jdlingyu - ad-wars
- kaabir - AdBlock_Hosts
- kevle1 - Windows-Telemetry-Blocklist - xiaomiblock
- liamja - Prebake Filter Obtrusive Cookie Notices
- malware-filter - URLhaus Malicious URL Blocklist
- malware-filter.- phishing-filter-hosts
- Matomo-org - spammers
- MBThreatIntel - malspam
- mine.nu - hosts0
- mitchellkrogza - Badd-Boyz-Hosts
- mitchellkrogza - hacked-domains
- mitchellkrogza - nginx-ultimate-bad-bot-blocker
- mitchellkrogza - strip_domains
- molinero - hBlock
- NanoAdblocker - NanoFilters
- neodevpro - neodevhost
- notracking - hosts-blocklists
- Oleksiig - Squid-BlackList
- openphish - feed
- pengelana - domains blocklist
- phishing.army - phishing_army_blocklist_extended
- piperun - iploggerfilter
- quidsup - notrack-blocklists
- quidsup - notrack-malware
- reddestdream - MinimalHostsBlocker
- RooneyMcNibNug - pihole-stuff
- Rpsl - adblock-leadgenerator-list
- ruvelro - Halt-and-Block-Mining
- ryanbr - fanboy-adblock
- scamaNet - blocklist
- ShadowWhisperer - Adult
- ShadowWhisperer - Cryptocurrency
- ShadowWhisperer - Dating
- ShadowWhisperer - Gambling
- ShadowWhisperer - Malware
- ShadowWhisperer - Risk
- ShadowWhisperer - Scam
- ShadowWhisperer - Shock
- ShadowWhisperer - Tracking
- ShadowWhisperer - Typo
- ShadowWhisperer - UrlShortener
- simeononsecurity/System-Wide-Windows-Ad-Blocker
- Someonewhocares - hosts
- stanev.org - Bulgarian adblock list
- StevenBlack - add.2o7Net
- StevenBlack - add.Risk
- StevenBlack - fakenews-gambling-porn-social
- StevenBlack - hosts
- StevenBlack - spam
- StevenBlack - uncheckyAds
- Stopforumspam - Toxic Domains
- sumatipru - squid-blacklist
- tomasko126 - Easylist Czech and Slovak filter list
- txthinking - blackwhite
- txthinking - bypass china domains
- Ultimate Hosts Blacklist - hosts
- Université Toulouse 1 Capitole - Blacklists UT1 - Olbat
- Université Toulouse 1 Capitole - Blacklists UT1
- Winhelp2002 - hosts
- yourduskquibbles - Web Annoyances Ultralist
- yous - YousList
- yoyo - Peter Lowe's Ad and tracking server list
- zoso - Romanian Adblock List
- google supported domains
- iana
- ipv6-hosts (Partial)
- publicsuffix
- Ransomware Database
- University Domains and Names Data List
- whoisxmlapi
BlackWeb's core is the domain and TLD blocklist (blackweb.txt and bwupdate.sh). It also includes related projects, each with its own tools and update process.
|
El núcleo de BlackWeb es la lista de bloqueo de dominios y TLD (blackweb.txt y bwupdate.sh). También incluye proyectos relacionados, cada uno con sus propias herramientas y proceso de actualización.
|
fpack/ contains optional ACLs outside BlackWeb's core flow; bwupdate.sh does not process them. To generate the ACLs, download the lists and run fpack.sh, then apply the Squid rules shown below. They cover ransomware-related file extensions, malicious User-Agents and static web3 lists. fpack.sh generates the first two; the web3 lists are maintained manually and are not modified by the script.
|
fpack/ contiene ACL opcionales, ajenas al flujo principal de BlackWeb; bwupdate.sh no las procesa. Para generarlas, se descargan las listas y se ejecuta fpack.sh; después, se aplican las reglas de Squid que aparecen abajo. Incluyen extensiones de archivo relacionadas con ransomware, User-Agents maliciosos y listas web3 estáticas. fpack.sh genera las dos primeras; las listas web3 se mantienen manualmente y el script no las modifica.
|
fpack/
├── fpack.sh # generates rw/rwext.txt and ua/blockua.txt
├── rw/
│ ├── rw.txt # administrator ransomware blacklist (source)
│ ├── rwext.txt # Squid url_regex ACL (generated)
│ └── wl.txt # administrator ransomware whitelist (source)
├── ua/
│ └── blockua.txt # Squid browser ACL for bad User-Agents (generated)
└── web3/
├── web3domains.txt # Squid dstdomain ACL, web3/wallet domains (static)
└── web3tld.txt # Squid dstdomain ACL, web3/wallet TLDs (static)
wget,curl,grep,sed,coreutils,util-linux
apt install -y wget curl grep sed coreutils util-linux
Either clone the whole BlackWeb repo, or download just the fpack/ folder with gitfolder.py (same tool vault subprojects use):
|
Clone todo el repo de BlackWeb, o descargue solo la carpeta fpack/ con gitfolder.py (la misma herramienta que usan los subproyectos de vault):
|
# Option 1: full clone
git clone --depth=1 https://github.com/maravento/blackweb.git
cd blackweb/fpack
# Option 2: folder-only download
wget -qO gitfolder.py https://raw.githubusercontent.com/maravento/vault/master/scripts/python/gitfolder.py
chmod +x gitfolder.py
python3 gitfolder.py https://github.com/maravento/blackweb/fpack
cd fpack
fpack.sh only produces Squid ACLs. Run it manually, as a non-root user — it is not scheduled by any installer:
|
fpack.sh solo produce ACLs para Squid. Se ejecuta manualmente, como usuario no root — ningún instalador la programa:
|
bash fpack.sh
Log: fpack.log, generated in fpack/, emptied at the start of every run.
|
Log: fpack.log, generado en fpack/, vaciado al inicio de cada ejecución.
|
fpack.sh downloads and normalizes the User-Agent list into ua/blockua.txt. It then downloads several ransomware extension/pattern lists, merges them with the administrator list (rw/rw.txt), normalizes extension forms (ext, .ext, _ext, -ext, *ext all become *.ext), and keeps only simple *.ext patterns. It removes entries listed in the administrator whitelist (rw/wl.txt) and discards ransom-note-style or implausibly long extensions. The result is written as a Squid url_regex ACL, rw/rwext.txt, with regex metacharacters escaped so they match literally.
|
fpack.sh descarga y normaliza la lista de User-Agents en ua/blockua.txt. Luego descarga varias listas de extensiones/patrones de ransomware, las combina con la lista del administrador (rw/rw.txt), normaliza las formas de extensión (ext, .ext, _ext, -ext, *ext pasan todas a *.ext) y conserva solo patrones simples *.ext. Elimina las entradas de la lista blanca del administrador (rw/wl.txt) y descarta extensiones tipo nota-de-rescate o implausiblemente largas. El resultado se escribe como ACL url_regex de Squid, rw/rwext.txt, con los metacaracteres de regex escapados para que coincidan como texto literal.
|
rw/wl.txtexcludes entries by exact match, format*.ext, one per line. Do not add*.zipor*.rar: they are common legitimate ransomware suffixes (e.g.*.bart.zip) and would be discarded if whitelisted.
rw/wl.txtexcluye entradas por coincidencia exacta, formato*.ext, una por línea. No agregue*.zipni*.rar: son sufijos legítimos comunes de ransomware (ej.*.bart.zip) y se descartarían si se agregan a la lista blanca.
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
# Block Rule for Ransomware Extensions/Patterns (Optional)
# https://raw.githubusercontent.com/maravento/blackweb/master/fpack/rw/rwext.txt
acl block_ransomware url_regex -i "/path_to/rwext.txt"
http_access deny block_ransomware
# Block Rule for web3 domains (Optional)
# https://raw.githubusercontent.com/maravento/blackweb/master/fpack/web3/web3domains.txt
acl web3domains dstdomain "/path_to/web3domains.txt"
http_access deny web3domains
# Block Rule for web3 TLDs (Optional)
# https://raw.githubusercontent.com/maravento/blackweb/master/fpack/web3/web3tld.txt
acl web3tld dstdomain "/path_to/web3tld.txt"
http_access deny web3tld
# Block Rule for User-Agents (Optional)
# https://raw.githubusercontent.com/maravento/blackweb/master/fpack/ua/blockua.txt
acl bad_useragents browser -i "/path_to/blockua.txt"
http_access deny bad_useragents
Domain Filtering (DoFi) removes redundant domains, validates their suffixes, and can check DNS resolution. bwupdate.sh uses domfilter.py during updates and stops if that step fails. DoFi can also be used independently to process other domain lists:
|
DoFi (Domain Filtering) elimina dominios redundantes, valida sus sufijos y puede comprobar su resolución DNS. bwupdate.sh usa domfilter.py durante la actualización y se detiene si esa etapa falla. DoFi también puede emplearse por separado para procesar otras listas:
|
python3 domfilter.py --input mylst.txtdofi/
├── domcheck.sh # Checks domain existence with the host command
└── domfilter.py # Removes overlapping domains, validates TLDs
- Python 3.12.3, Bash 5.2.21
python3-requests(required bydomfilter.py),bind9-host,findutils,grep,sed,coreutils,util-linux(required bydomcheck.sh)
apt install -y python3 python3-requests bind9-host findutils grep sed coreutils util-linux# Option 1: full clone
git clone --depth=1 https://github.com/maravento/blackweb.git
cd blackweb/dofi
# Option 2: folder-only download
wget -qO gitfolder.py https://raw.githubusercontent.com/maravento/vault/master/scripts/python/gitfolder.py
chmod +x gitfolder.py
python3 gitfolder.py https://github.com/maravento/blackweb/dofi
cd dofi
Before running the filter, remove http://, https:// and www. prefixes from the input list. The script:- Downloads public suffix lists from multiple sources. - Removes invalid or duplicate TLDs. - Filters domains against the valid suffix list. - Removes overlapping domains. - Excludes duplicates from previously processed domains. - Writes the results to a file. |
Antes de ejecutar el filtro, deben quitarse de la lista los prefijos http://, https:// y www.. El script:- Descarga listas de sufijos públicos de varias fuentes. - Elimina TLD no válidos o duplicados. - Filtra dominios según la lista de sufijos válidos. - Elimina dominios superpuestos. - Excluye dominios duplicados ya procesados. - Guarda los resultados en un archivo. |
python3 domfilter.py --input mylst.txt
In the command, replace mylst.txt with the input filename. By default, results go to output.txt and removed lines to removed.txt; these paths can be changed with the following options:
|
En el comando, mylst.txt debe sustituirse por el nombre del archivo de entrada. De forma predeterminada, el resultado se guarda en output.txt y las líneas descartadas en removed.txt; esas rutas pueden cambiarse con las opciones siguientes:
|
python3 domfilter.py --input mylst.txt --output outlst.txt --removed removelst.txtThe
tlds.txtlist includes ccTLDs, gTLDs, sTLDs, eTLDs, and domains of up to four labels.La lista
tlds.txtincluye ccTLD, gTLD, sTLD, eTLD y dominios de hasta cuatro etiquetas.
The script queries each domain with the host command and separates results into two files:- hit.txt: domains that resolved.- fault.txt: domains that did not resolve.
|
El script consulta cada dominio con el comando host y separa los resultados en dos archivos:- hit.txt: dominios que resolvieron.- fault.txt: dominios que no pudieron resolverse.
|
Before running the script, remove
http://,https://, andwww.prefixes from the input list.Antes de ejecutar el script, quite de la lista los prefijos
http://,https://ywww..
chmod +x domcheck.sh
bash domcheck.sh my_domain_list.txt
In the command, replace my_domain_list.txt with the input filename. The optional second argument sets the number of parallel processes (default: nproc × 4, capped at 200):
|
En el comando, my_domain_list.txt debe sustituirse por el nombre del archivo de entrada. El segundo argumento, opcional, indica la cantidad de procesos paralelos (por defecto: nproc × 4, con tope de 200):
|
bash domcheck.sh my_domain_list.txt 502026-07-07 13:29:58 domcheck start...
2026-07-07 13:29:58 Step 1...
2026-07-07 13:30:17 OK
2026-07-07 13:30:17 Step 2...
2026-07-07 13:30:28 hit.txt: domains successfully resolved
2026-07-07 13:30:28 fault.txt: unresolved domains
2026-07-07 13:30:28 Summary:
2026-07-07 13:30:28 Input domains : 7896
2026-07-07 13:30:28 Resolved : 7349
2026-07-07 13:30:28 Unresolved : 547
2026-07-07 13:30:28 Elapsed time : 30s
2026-07-07 13:30:29 domcheck done at: mar 07 jul 2026 13:30:29 -05
Log: dofi.log, generated in dofi/, emptied at the start of every run.
|
Log: dofi.log, generado en dofi/, vaciado al inicio de cada ejecución.
|
- Awesome Open Source: Blackweb
- Community IPfire: url filter and self updating blacklists
- covert.io: Getting Started with DGA Domain Detection Research
- Crazymax: WindowsSpyBlocker
- egirna: Allowing/Blocking Websites Using Squid
- Jason Trost: Getting Started with DGA Domain Detection Research
- Kandi Openweaver: Domains Blocklist for Squid-Cache
- Kerry Cordero: Blocklists of Suspected Malicious IPs and URLs
- Keystone Solutions: blocklists
- Lifars: Sites with blocklist of malicious IPs and URLs
- Opensourcelibs: Blackweb
- OSINT Framework: Domain Name/Domain Blacklists/Blackweb
- Osintbay: Blackweb
- Reddit: Blackweb
- Secrepo: Samples of Security Related Data
- Segu-Info: Análisis de malware y sitios web en tiempo real
- Segu-Info: Dominios/TLD dañinos que pueden ser bloqueados para evitar spam y #phishing
- Soficas: CiberSeguridad - Protección Activa
- Stackoverflow: Blacklist IP database
- Wikipedia: Blacklist_(computing)
- Xploitlab: Projects using WindowsSpyBlocker
- Zeltser: Free Blocklists of Suspected Malicious IPs and URLs
- Zenarmor: How-to-enable-web-filtering-on-OPNsense-proxy?
|
|
wget https://raw.githubusercontent.com/maravento/blackweb/refs/heads/master/bwupdate/tools/checksources.sh
chmod +x checksources.sh
./checksources.she.g:
[?] Enter domain to search: kickass.to
[*] Searching for 'kickass.to'...
[+] Domain found in: https://github.com/fabriziosalmi/blacklists/releases/download/latest/blacklist.txt
[+] Domain found in: https://hostsfile.org/Downloads/hosts.txt
[+] Domain found in: https://raw.githubusercontent.com/blocklistproject/Lists/master/everything.txt
[+] Domain found in: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/ultimate-onlydomains.txt
[+] Domain found in: https://raw.githubusercontent.com/Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist/master/hosts/hosts0
[+] Domain found in: https://sysctl.org/cameleon/hosts
[+] Domain found in: https://v.firebog.net/hosts/Kowabit.txt
DoneSpecial thanks to: Jhonatan Sneider
| This project uses a dual-licensing model to balance software freedom with content protection: | Este proyecto utiliza un modelo de licencia dual para equilibrar la libertad del software con la protección del contenido: |
| Content | Licensed Under |
|---|---|
| Scripts, Binaries, Infrastructure |  |
| RAG, Workers, Specialized Modules, Docs |  |
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
Due to recent arbitrary changes in computer terminology, it is necessary to clarify the meaning and connotation of the term blacklist, associated with this project:
In computing, a blacklist, denylist or blocklist is a basic access control mechanism that allows through all elements (email addresses, users, passwords, URLs, IP addresses, domain names, file hashes, etc.), except those explicitly mentioned. Those items on the list are denied access. The opposite is a whitelist, which means only items on the list are let through whatever gate is being used. Source Wikipedia Therefore, blacklist, blocklist, blackweb, blackip, whitelist and similar, are terms that have nothing to do with racial discrimination. |
Debido a los recientes cambios arbitrarios en la terminología informática, es necesario aclarar el significado y connotación del término blacklist, asociado a este proyecto:
En informática, una lista negra, lista de denegación o lista de bloqueo es un mecanismo básico de control de acceso que permite a través de todos los elementos (direcciones de correo electrónico, usuarios, contraseñas, URL, direcciones IP, nombres de dominio, hashes de archivos, etc.), excepto los mencionados explícitamente. Esos elementos en la lista tienen acceso denegado. Lo opuesto es una lista blanca, lo que significa que solo los elementos de la lista pueden pasar por cualquier puerta que se esté utilizando. Fuente Wikipedia Por tanto, blacklist, blocklist, blackweb, blackip, whitelist y similares, son términos que no tienen ninguna relación con la discriminación racial. |