Documentation/DETAILS.md

@@ -28,7 +28,7 @@
 - lib/geoip-shell-lib-non-owrt.sh
 - lib/geoip-shell-lib-arrays.sh
 - lib/geoip-shell-lib-uninstall.sh
-- lib/geoip-shell-lib-detect-lan.sh
+- lib/geoip-shell-lib-ip-tools.sh
 
 
 The -lib-common script includes a large number of functions used throughout the suite, and assigns some essential variables.
@@ -45,7 +45,7 @@ The -lib-arrays script implements a minimal subset of functions emulating the fu
 
 The -lib-uninstall script has some functions which are used both for uninstallation and for reset if required.
 
-The lib-detect-lan script is only used under specific conditions:
+The lib-ip-tools script is only used under specific conditions:
 - During initial setup, with whitelist mode, and only if wan interfaces were set to 'all', and LAN subnets were not specified via command line args. geoip-shell then assumes that it is being configured on a host behind a router and firewall, uses this script to detect the LAN subnets and offers the user to add them to the whitelist, and to enable automatic detection of LAN subnets in the future.
 - At the time of creating/updating firewall rules, and only if LAN subnets automatic detection is enabled. geoip-shell then re-detects LAN subnets automatically.
 
@@ -122,7 +122,7 @@ These options apply to geoblocking in both directions.
 
 `-f <ipv4|ipv6|"ipv4 ipv6">`: Families (defaults to 'ipv4 ipv6'). Use double quotes for multiple families.
 
-`-u [ripe|ipdeny]`: Change ip lists source.
+`-u [ripe|ipdeny|maxmind]`: Change ip lists source.
 
 `-i <[ifaces]|auto|all>`: Change which network interfaces geoip firewall rules are applied to. `auto` will attempt to automatically detect WAN network interfaces. `auto` works correctly in **most** cases but not in **every** case. Don't use `auto` if the machine has no dedicated WAN network interfaces. The automatic detection occurs only when manually triggered by the user via this command.
 
@@ -131,7 +131,7 @@ These options apply to geoblocking in both directions.
 `-t <"[trusted_ips]|none">`: Specify trusted ip's or subnets (anywhere on the Internet) to exclude from geoip blocking (both ipv4 and ipv6).
 
 `-U <auto|pause|none|"[ip_addresses]">`: Policy for allowing automatic ip list updates when outbound geoblocking is enabled. Use `auto` to detect server ip addresses automatically once and always allow outbound connection to detected addresses. Or use `pause` to always temporarily pause outbound geoblocking before fetching ip list updates.
-Or specify ip addresses for ip lists source (ripe or ipdeny) to allow - for multiple addresses, use double quotes.
+Or specify ip addresses for ip lists source (ripe or ipdeny or maxmind) to allow - for multiple addresses, use double quotes.
 Or use `none` to remove previously assigned server ip addresses and disable this feature.
 
 `-r <[user_country_code]|none>` : Specify user's country code. Used to prevent accidental lockout of a remote machine. `none` disables this feature.
@@ -203,7 +203,7 @@ List id has the format of `<country_code>_<family>`. For example, **US_ipv4** an
 
 
 **geoip-shell-fetch.sh**
-- Fetches ip lists for given list id's from RIPE or from ipdeny.
+- Fetches ip lists for given list id's from RIPE or from ipdeny or from MaxMind.
 - Parses, validates, compiles the downloaded lists, and saves each one to a separate file.
 - Implements extensive sanity checks at each stage (fetching, parsing, validating and saving) and handles errors if they occur.
 
@@ -217,7 +217,7 @@ Options:
 
 `-s <path>`        : Path to a file to register fetch results in.
 
-`-u <ripe|ipdeny>` : Use this ip list source for download. Supported sources: ripe, ipdeny.
+`-u <ripe|ipdeny|maxmind>` : Use this ip list source for download. Supported sources: ripe, ipdeny, maxmind.
 
 Extra options:
 
@@ -252,6 +252,6 @@ Actions: `add`, `update`, `restore`, `on`, `off`
 
 `sh check-ip-in-source.sh -c <country_code> -i <"ip [ip] [ip] ... [ip]"> [-u <source>]`
 
-- Supported sources are 'ripe' and 'ipdeny'.
+- Supported sources are `ripe`, `ipdeny` and `maxmind`.
 - Any combination of ipv4 and ipv6 addresses is supported.
 - If passing multiple ip addresses, use double quotes around them.

Documentation/NOTES.md

@@ -41,7 +41,12 @@ Depending on the options you specified during interactive setup or via the comma
     - With nftables: `nft -t list table inet geoip-shell`. This will display all geoip-shell rules and sets.
     - With iptables: `iptables -vL -t mangle` and `ip6tables -vL -t mangle`. This will report all geoip-shell rules. To check ipsets created by geoip-shell, use `ipset list -n | grep geoip-shell`. For a more detailed view, use this command: `ipset list -t`.
 
-5) geoip-shell uses RIPE as the default source for ip lists, except on OpenWrt. RIPE is a regional registry, and as such, is expected to stay online and free for the foreseeable future. However, RIPE may be fairly slow in some regions. For that reason, I implemented support for fetching ip lists from ipdeny. ipdeny provides aggregated ip lists, meaning in short that there are less entries for same effective geoblocking. With iptables, the machine which these lists are loaded on has to do less work when processing incoming connection requests. With nftables, this does not affect the load on the CPU because nftables automatically optimizes loaded ip lists, so ip lists from RIPE perform identically to ip lists from ipdeny. The downloaded lists from ipdeny are still smaller, so even with nftables, there is still a benefit of lower transient memory consumption while loading the lists (which mostly matters for embedded hardware with very limited memory capacity).
+5) Supported ip list sources:
+	- geoip-shell currently supports the following sources: RIPE, ipdeny and MaxMind.
+	- RIPE and ipdeny sources are free-to-use and require no license. Maxmind requires a license and offers both a free (GeoLite2) and a paid (GeoIP2) databases.
+	- RIPE and ipdeny ip lists are typically effectively the same, except ipdeny provides aggregated ip lists, meaning in short that there are less entries for same effective geoblocking. With iptables, the machine which these lists are loaded on has to do less work when processing incoming connection requests. With nftables, this does not affect the load on the CPU because nftables automatically optimizes loaded ip lists, so ip lists from RIPE perform identically to ip lists from ipdeny. The downloaded lists from ipdeny are still smaller, so even with nftables, there is still a benefit of lower transient memory consumption while loading the lists (which mostly matters for embedded hardware with very limited memory capacity).
+	- MaxMind manages proprietary geoip database which typically provides more accurate data than RIPE and ipdeny, both in the paid and in the free version (the paid version may provide even higher accuracy). Note that in order to use the MaxMind source, you need to have a MaxMind account.
+	- geoip-shell uses RIPE as the default source for ip lists, except on OpenWrt. RIPE is a regional registry, and as such, is expected to stay online and free for the foreseeable future. However, RIPE may be fairly slow in some regions, and the downloads from RIPE are typically larger than from ipdeny. For this reason, I implemented support for fetching ip lists from ipdeny, which is also the default source on OpenWrt.
 
 6) Scripts intended as user interface are: **-install**, **-uninstall**, **-manage** (also called by running '**geoip-shell**' after installation) and **check-ip-in-source.sh**. The -manage script saves the config to a file and implements coherence checks between that file and the actual firewall state. While you can run the other scripts individually, if you make changes to firewall geoip rules, next time you run the -manage script it may insist on reverting those changes since they are not reflected in the config file.
 

OpenWrt/README.md

@@ -60,7 +60,7 @@ Generally the defaults are the same as for other systems, except:
 - the data directory which geoip-shell uses to store the status file and the backups is by default in `/tmp/geoip-shell-data`, rather than in `/var/lib/geoip-shell` as on other Linux systems. This is to avoid flash wear. You can change this by running the install script with the `-a <path>` option, or after installation via the command `geoip-shell configure -a <path>`.
 - the 'nobackup' option is set to 'true', which configures geoip-shell to not create backups of the ip lists. With this option, geoip-shell will work as usual, except after reboot (and for iptables-based systems, after firewall restart) it will re-fetch the ip lists, rather than loading them from backup. You can change this by running the -install script with the `-o false` option (`-o` stands for nobackup), or after installation via the command `geoip-shell configure -o false`. To have persistent ip list backups, you will also need to change the data directory path as explained above.
 - if using geoip-shell on a router with just a few MB of embedded flash storage, consider either leaving the nobackup and datadir path defaults as is, or connecting an external storage device to your router (preferably formatted to ext4) and configuring a directory on it as your geoip-shell data directory, then enabling automatic backups. For example, if your external storage device is mounted on _/mnt/somedevice_, you can do all this via this command: `geoip-shell configure -a /mnt/somedevice/geoip-shell-data -o false`.
-- the default ip lists source for OpenWrt is ipdeny (rather than ripe). While ipdeny is a 3rd party, they provide aggregated lists which consume less memory (on nftables-based systems the ip lists are automatically optimized after loading into memory, so there the source does not matter, but a smaller initial ip lists size will cause a smaller memory consumption spike while loading the ip list).
+- the default ip lists source for OpenWrt is ipdeny (rather than RIPE). While ipdeny is a 3rd party, they provide aggregated lists which consume less memory (on nftables-based systems the ip lists are automatically optimized after loading into memory, so there the source does not matter, but a smaller initial ip lists size will cause a smaller memory consumption spike while loading the ip list).
 
 ## Building an OpenWrt package
 The repository includes the _mk-owrt-package.sh_ and _prep-owrt-package_ scripts which automate creation of geoip-shell ipk packages. The _makefile.tpl_ file is used as the base template for the Makefile. If you want to build the ipk package by yourself, read the comments inside the _mk-_ script for instructions.

README.md

@@ -1,5 +1,5 @@
 # **geoip-shell**
-Powerful geoblocker for Linux. Supports both **nftables** and **iptables** firewall management utilities.
+User-friendly and powerful geoblocker for Linux. Supports both **nftables** and **iptables** firewall management utilities.
 
 The idea of this project is making geoblocking (i.e. restricting access from or to Internet addresses based on geolocation) easy on (almost) any Linux system, no matter which hardware, including desktop, server, container, VPS or router, while also being reliable and providing flexible configuration options for the advanced users.
 
@@ -24,7 +24,7 @@ If you find this project useful, please take a second to give it a star on Githu
 ## **Main Features**
 * Core functionality is creating either a whitelist or a blacklist in the firewall using automatically downloaded ip lists for user-specified countries.
 
-* ip lists are fetched either from **RIPE** (regional Internet registry for Europe, the Middle East and parts of Central Asia) or from **ipdeny**. Both sources provide updated ip lists for all regions.
+* ip lists are fetched either from **RIPE** (regional Internet registry for Europe, the Middle East and parts of Central Asia) or from **ipdeny**, or from **MaxMind**. All 3 sources provide updated ip lists for all regions.
 
 * All firewall rules and ip sets required for geoblocking to work are created automatically during the initial setup.
 
@@ -48,6 +48,7 @@ If you find this project useful, please take a second to give it a star on Githu
 <details> <summary>Read more:</summary>
 
 - Default source for ip lists is RIPE, which allows to avoid dependency on non-official 3rd parties.
+- Supports the 'MaxMind' commercial source which provides more accurate ip lists, both the free GeoLite2 database and the paid GeoIP2 database. Note that in order to use the MaxMind source, you need to have a MaxMind account.
 - With nftables, utilizes nftables atomic rules replacement to make the interaction with the system firewall fault-tolerant and to completely eliminate time when geoip is disabled during an automatic update.
 - All scripts perform extensive error detection and handling.
 - All user input is validated to reduce the chance of accidental mistakes.
@@ -80,7 +81,7 @@ If you find this project useful, please take a second to give it a star on Githu
 - Extensive and (usually) up-to-date documentation.
 - Comes with an uninstall script which completely removes the suite and the geoblocking firewall rules. No restart is required.
 - Sane settings are applied during installation by default, but also lots of command-line options for advanced users or for special corner cases are provided.
-- Pre-installation, provides a utility _(check-ip-in-source.sh)_ to check whether specific ip addresses you might want to blacklist or whitelist are indeed included in the ip lists fetched from the source (RIPE or ipdeny).
+- Pre-installation, provides a utility _(check-ip-in-source.sh)_ to check whether specific ip addresses you might want to blacklist or whitelist are indeed included in the ip lists fetched from the source (RIPE or ipdeny or MaxMind).
 - Post-installation, provides a utility (symlinked to _'geoip-shell'_) for the user to change geoblocking config (turn geoblocking on or off, configure outbound geoblocking, change country codes, change geoblocking mode, change ip lists source, change the cron schedule etc).
 - Post-installation, provides a command _('geoip-shell status')_ to check geoblocking status, which also reports if there are any issues.
 - In case of an error or invalid user input, provides useful error messages to help with troubleshooting.
@@ -173,7 +174,7 @@ _(for detailed description of this feature, read [NOTES.md](/Documentation/NOTES
 
 `geoip-shell <on|off>`
 
-**To change ip lists source:** `geoip-shell configure -u <ripe|ipdeny>`
+**To change ip lists source:** `geoip-shell configure -u <ripe|ipdeny|maxmind>`
 
 **To have certain trusted ip addresses or subnets, either in your LAN or anywhere on the Internet, bypass geoblocking:**
 
@@ -201,7 +202,7 @@ _<details><summary>Example</summary>_
 
 **To update or re-install geoip-shell:** run the -install script from the (updated) distribution directory.
 
-**To temporarily stop geoip-shell:** `geoip-shell stop`. This will kill any running geoip-shell processes, remove geoip-shell firewall rules and unload ip sets.
+**To temporarily stop geoip-shell:** `geoip-shell stop`. This will kill any running geoip-shell processes, remove geoip-shell firewall rules and unload ip sets. To reactivate geoblocking, run `geoip-shell configure`.
 
 **To uninstall:** `geoip-shell-uninstall.sh`
 
@@ -259,6 +260,7 @@ Examples:
 - standard Unix utilities including **tr**, **cut**, **sort**, **wc**, **awk**, **sed**, **grep**, **pgrep**, **pidof** and **logger** which are included with every server/desktop linux distribution (and with OpenWrt). Both GNU and non-GNU versions are supported, including BusyBox implementation.
 - **wget** or **curl** or **uclient-fetch** (OpenWRT-specific utility).
 - for the autoupdate functionality, requires the **cron** service to be enabled. Except on OpenWrt, persistence also requires the cron service.
+- for the MaxMind source, requires the utilities: `unzip`, `gzip`, `gunzip` (`apt install unzip gzip`)
 
 **Optional**: the _check-ip-in-source.sh_ optional script requires **grepcidr**. install it with `apt install grepcidr` on Debian and derivatives. For other distros, use their built-in package manager.
 
@@ -273,7 +275,7 @@ For information about OpenWrt support, read the [OpenWrt README](/OpenWrt/README
 
 ## **Privacy**
 geoip-shell does not share your data with anyone.
-If you are using the ipdeny source then note that they are a 3rd party which has its own data privacy policy.
+If you are using the ipdeny or the maxmind source then note that they are a 3rd party which has its own data privacy policy.
 
 ## **P.s.**
 

check-ip-in-source.sh

@@ -41,22 +41,22 @@ set_path cca2 cca2.list "$conf_dir"
 usage() {
 cat <<EOF
 
-Usage: $me -c <country_code> -i <"ip [ip ... ip]"> [-u <ripe|ipdeny>] [-d] [-h]
+Usage: $me -c <country_code> -i <"ip [ip ... ip]"> [-u <ripe|ipdeny|maxmind>] [-d] [-h]
 
 For each of the specified ip addresses, checks whether it belongs to one of the subnets
-    in the list fetched from a source (either RIPE or ipdeny) for a given country code.
+    in the list fetched from a source (RIPE or ipdeny or MaxMind) for a given country code.
 Accepts a mix of ipv4 and ipv6 addresses.
 
 Requires the 'grepcidr' utility
 
 Options:
-  -c <country_code>    : 2-letter country code
-  -i <"ip_addresses">  : ip addresses to check
-                         if specifying multiple addresses, use double quotes
-  -u <ripe|ipdeny>     : Source to check in. By default checks in RIPE.
+  -c <country_code>        : 2-letter country code
+  -i <"ip_addresses">      : ip addresses to check
+                             if specifying multiple addresses, use double quotes
+  -u <ripe|ipdeny|maxmind> : Source to check in. By default checks in RIPE.
 
-  -d                   : Debug
-  -h                   : This help
+  -d                       : Debug
+  -h                       : This help
 
 EOF
 }
@@ -103,7 +103,7 @@ validate_ip() {
 
 
 #### Constants
-valid_sources="ripe ipdeny"
+valid_sources="ripe ipdeny maxmind"
 default_source=ripe
 
 
@@ -150,6 +150,10 @@ done
 [ -z "$val_ipv4s$val_ipv6s" ] && die "All ip addresses failed validation."
 [ -z "$families" ] && die "\$families variable is empty."
 
+if [ "$dl_source" = maxmind ]; then
+	setup_maxmind || die
+fi
+
 ### Fetch the ip list file
 
 for family in $families; do