[REQ] - If online checksum check fails fallback to using offline checksum check

[Razor221]

According to merlin's recent post and i quote

"My hosting provider regularly gets flooded with bots, in which case they end up blocking multiple IP ranges (sometimes entire countries). Nothing I can do about.

I'm just surprised that despite the site being behind Cloudflare, the provider can still end up blocking visitors even tho these shouldn't be going through Cloudflare, not directly accessing the website. I suspect they might be looking at the IP of origin header that Cloudflare publishes."

Problem is Turkish ip addresses are usually first ones to get to the chopping block. I tried to update fw 3006.102.8_4 on my asus rt-88u-pro but merlin au script kept failing in online checksum checking because of 403 error on https://www.asuswrt-merlin.net/download.

So, I modified your script and changed the online checksum verification code to fallback to offline checksum verification if online verification fails.

Here is my modified online checksum check with offline fallback.

`CheckOnlineFirmwareSHA256()
{
# Fetch the latest SHA256 checksums from ASUSWRT-Merlin website #
checksums="$(curl -Ls --retry 4 --retry-delay 5 --retry-connrefused
https://www.asuswrt-merlin.net/download |
sed -n '/<.>SHA256 signatures:</.>/,/</pre>/p' |
sed -n '/<pre[^>].>/,/</pre>/p' |
sed -e 's/<[^>].>//g; s/^[[:space:]]//; s/[[:space:]]$//')"

if [ -z "$checksums" ]
then
    Say "${YELLOWct}**WARNING**${NOct}: Could not fetch signatures online (HTTP 403/Block)."
    Say "Falling back to offline SHA256 verification..."
    _CheckOfflineFirmwareSHA256_
    return $?
fi

if [ -f "$firmware_file" ]
then
    fw_sig="$(openssl sha256 "$firmware_file" | awk -F ' ' '{print $2}')"
    dl_sig="$(echo "$checksums" | grep "$(basename "$firmware_file")" | awk -F ' ' '{print $1}')"
    if [ "$fw_sig" != "$dl_sig" ]
    then
        Say "${REDct}**ERROR**${NOct}: SHA256 signature from extracted firmware file does not match the SHA256 signature from the website."
        _DoCleanUp_ 1
        _SendEMailNotification_ FAILED_FW_CHECKSUM_STATUS
        return 1
    else
        Say "SHA256 signature check for firmware image file passed successfully."
        return 0
    fi
else
    Say "${REDct}**ERROR**${NOct}: Firmware image file NOT found!"
    _DoCleanUp_ 1
    return 1
fi

}`

I also share my full MerlinAU.sh for troubleshooting as well.

MerlinAU.sh

[ExtremeFiretop]

My concern is security; we initially only consulted the .zip until version 1.1.4 when we went the online route:
https://www.snbforums.com/threads/merlinau-v1-2-6-the-ultimate-firmware-auto-updater-available-in-amtm-thread-closed-due-to-age.88577/post-908412

@Martinski4GitHub
Thoughts?

[Razor221]

that's why i didn't just bypassed the online verification and just make it fallback to zip. Some countries ip blocks have a bad rep for example turkey,brazil,russia...etc. This fallback would be quite useful for users in those countries while also not affecting the majority of users. It's better than merlin au doesn't for them in my opinion. 1st it tests https://www.asuswrt-merlin.net/download to get the sha256 and only then it fallsback to zip verfication. Or maybe host a sha256.txt file here in this repo scraped from https://www.asuswrt-merlin.net/download and then use that for online verification. I think that would be a even better solution because github rarely block ip ranges.

[Razor221]

I understand that offline method relies on unzipping that can sometimes corrupt data if it fails or improperly executes and for an automated firmware update script that's a really bad thing security wise. My way still scrapes the https://www.asuswrt-merlin.net/download for proper sha256 checksums but does it in a different way that more resilient to provider blocking ip addresses to access.

I fully adapted my idea on my own fork and i think it works now.
here is the link for my fork https://github.com/Razor221/MerlinAutoUpdate-Router

what i did was

1)I created a workflow that scraped https://www.asuswrt-merlin.net/download everyday at 02:00 UTC and got the sha256 sigs and full file names and copied it to merlin-sha256.txt in a clean format like this example file.

merlin-sha256.txt

2. Modified MerlinAU.sh like this

`CheckOnlineFirmwareSHA256()
{
# Fetch the latest SHA256 checksums from your custom GitHub repository #
local GITHUB_RAW_URL="https://github.com/Razor221/MerlinAutoUpdate-Router/raw/refs/heads/merlin-sha256.txt-verification/merlin-sha256.txt"

checksums="$(curl -Ls --retry 4 --retry-delay 5 --retry-connrefused "$GITHUB_RAW_URL")"

if [ -z "$checksums" ]
then
    Say "${YELLOWct}**WARNING**${NOct}: Could not fetch signatures from GitHub."
    Say "Falling back to offline SHA256 verification..."
    _CheckOfflineFirmwareSHA256_
    return $?
fi

if [ -f "$firmware_file" ]
then
    fw_sig="$(openssl sha256 "$firmware_file" | awk -F ' ' '{print $2}')"
    # The downloaded text file is already formatted, so we just grep for the filename
    dl_sig="$(echo "$checksums" | grep "$(basename "$firmware_file")" | awk -F ' ' '{print $1}')"
    
    if [ "$fw_sig" != "$dl_sig" ]
    then
        Say "${REDct}**ERROR**${NOct}: SHA256 signature from extracted firmware file does not match the SHA256 signature from GitHub."
        _DoCleanUp_ 1
        _SendEMailNotification_ FAILED_FW_CHECKSUM_STATUS
        return 1
    else
        Say "SHA256 signature check for firmware image file passed successfully."
        return 0
    fi
else
    Say "${REDct}**ERROR**${NOct}: Firmware image file NOT found!"
    _DoCleanUp_ 1
    return 1
fi

}`

cleaned sed filtering and kept a fail safe if github get down or curl fails.

I put forks link so you all can look it for themselves. I think this method is much reliable since not many providers block access to azure ip addresses.

[ExtremeFiretop]

Daily checks using Github workflows are too slow unfortunately.

1. Merlin Releases Firmware
2. Excited User Goes to MerlinAU right away to install
3. MerlinAU would detect an update available, but fail at the checksum stage every time until the Github workflow runs 20+ hours later?

Even if you made it run hourly instead of daily, that would still be too slow.

1. Merlin Releases Firmware
2. Excited User Goes to MerlinAU right away to install
3. MerlinAU would detect an update available, but fail at the checksum stage every time until the Github workflow runs an hour later.

While it is a fun idea. we are not looking to mirror anything to get around this behavior for a single person.
Your original idea of a fallback was much more convincing tbh.

[Razor221]

I think merlin's website provider blocked the entire turkish ip address range because his website gives 403 error on my home isp, mobile isp and also proton vpn's turkish servers.

You are right it's not fast for excited users without making a workflow that abuse merlin's site.

I'm fine with the fallback method to offline check if online fails.
I would use my custom script file but that breaks the update process for the script from amtm because of checksum mismatch.

Here is my modified CheckOnlineFirmwareSHA256() it checks https://www.asuswrt-merlin.net/download if that fails then it jumps to offline verification. I'm not very good at coding so feel free to improve.

fallbackback.sh

[ExtremeFiretop]

I think merlin's website provider blocked the entire turkish ip address range because his website gives 403 error on my home isp, mobile isp and also proton vpn's turkish servers.

You are right it's not fast for excited users without making a workflow that abuse merlin's site.

I'm fine with the fallback method to offline check if online fails. I would use my custom script file but that breaks the update process for the script from amtm because of checksum mismatch.

Here is my modified CheckOnlineFirmwareSHA256() it checks https://www.asuswrt-merlin.net/download if that fails then it jumps to offline verification. I'm not very good at coding so feel free to improve.

fallbackback.sh

I understand why you picked the workflow idea. It's a way to mitigate the security concerns while avoiding the block by controlling my own mirror / clone.

But the truth is it's not something I'm interested in, and as discussed the delays involved would likely cause for a bad user experience whenever a block was found.

The fallback to offline makes the most sense. But the only question is if we are okay with the security implications that instead of failing when unable to get the online SHA256 values that it will fallback to the values inside the zip offline.

The status quo for now has been fail. I'm interested in @Martinski4GitHub opinion on this one.
We did use this method up until 1.1.4... so at one point in time we thought it was okay... Is it still okay as a fallback?

[Razor221]

I also understand your concern as well zip file might be damaged or might get subjected to mitm attacks or update server attacks like it happened to notepad++. Given how popular merlin firmware is.

I also messaged RMerlin on snbforums about it. I told him about the situation and and also told him that i use a static ip for my home internet and asked him if he could whitelist my ip address but i didn't receive a reply yet.

If nothing works, then i would manually maintain my own fork that scrapes merlin's website with workflow and update MerlinAU manually whenever a new update released instead of using amtm to update it.

[Razor221]

RMerlin replied to me on snbforums and unfortunately the block is something he didn't do his website's provider block some regions due to botnet attacks on their network level in which he has no control over.

After consideration I have decided to use own fork for in my merlin firmware. I can't expect you people to change how you run your script just for me and also i think there isn't much Turkish users use this script anyway. I defer firmware update for 2 days so scraping merlin's website once everyday is good enough for me.

I'm asking for your permission to use your repo as a base for my fork. I pretty much kept all the info the same so it's clear that your project is properly acknowledged.

I changed how the original script's checks for version number and how it updates by pointing it to my github fork like this.

Set version for each Production Release

readonly SCRIPT_VERSION=1.6.6
readonly SCRIPT_VERSTAG="26081603"
readonly SCRIPT_NAME="MerlinAU"

Set to "master" for Production Releases

SCRIPT_BRANCH="merlin-sha256.txt-verification"

##----------------------------------------##

Modified by Martinski W. [2024-Jul-03]

##----------------------------------------##

Script URL Info

readonly SCRIPT_URL_BASE="https://raw.githubusercontent.com/Razor221/MerlinAutoUpdate-Router"
SCRIPT_URL_REPO="${SCRIPT_URL_BASE}/$SCRIPT_BRANCH"

I think this is the only check for updating the script for a new version. If there is any more please let me know so that i can modify them accordingly as well.

In case you people release a new version, I will just sync the new things from your repo and keep my modifications and i will use the "Force Update merlinAU script option in terminal.

That way I wouldn't also compromise my own security as well by having proper sha256 checksums.

Thanks for the help and insight. I learned quite new things in this.

[ExtremeFiretop]

@Razor221

Sorry for the delay, you can fork all you wish, i don't mind.
I should mention though that we still have not come to a decision on this matter and I rather keep the ticket open until we have.

Can you link me to whatever response RMerlin gave you about this block and the website provider?
At least with that documented it gives us some grounds to re-open this issue.

Once we have decided what we will do. I will let you know if you need your own fork or if we will incorporate some logic for you in a final release.

Thanks

[Razor221]

I dmed RMerlin with this message

"Hi RMerlin,

I live in Türkiye and your website giving me consistent 403 errors for almost 3-4 days. I suspect that your website provider is blocking access from all Turkish ip addresses because i tried with my home isp, mobile isp and even proton vpns turkish servers and they all give 403 error. This really broke a script that i for automatically update firmware called MerlinAU because they check the downloaded firmware file's integrity via scraping your website for sha256 checksums and since I get 403 error MerlinAU fails. I'm in the process of talking with the MerlinAU team about adding a fallback method where if online check fails their script uses the checksum file in the firmware's sourceforge zip file. In the meantime, i wanted to ask you if there is anything you can do?

I use a static ip on my home isp and maybe you can whitelist my ip."

His response was

"Nothing I can do about it, it's my hosting provider blocking certain regions."

I came to the conclusion that his hosting provider blocks certain regions on their network level he doesn't have control over that from his response.

here is the link for dm conversation but i don't think you can access it

https://www.snbforums.com/direct-messages/consistent-403-error-on-your-website.16971/

And also he talks about it in this forum thread as well

https://www.snbforums.com/threads/asus-merlin-site-down.97648/post-999566

[ExtremeFiretop]

Here's what I think I'm willing to do for you.

online independent verification for unattended updates always. Not optional.
That means in no way, even with the bundled checksum, that an unattended update would pass with a block in place.

However for interactive flashing, we can allow bundled checksum only as an explicitly acknowledged manual fallback.

Something like...
"WARNING: Merlin's independently published SHA256 signature could not be retrieved.
The checksum included inside the firmware archive can verify file consistency but does not provide independent verification of the downloaded firmware.
Continue using bundled checksum? [y/N]"

[Razor221]

thank you for your suggestion but the only reason i use merlinAU is to have an automatic unattended firmware update process and manually accepting a warning kind of kills the whole reason for using it for me.

I'm quite happy with my fork with github workflow scrape method. I always defer the automatic firmware update for 1 day and my github scraper runs once every day so by the time MerlinAU starts the update process my checksum mirror would be up to date.

I'm completely abandoned the idea of using checksum file in the downloaded zip as i genuinely agree with you about security.

I'm closing this ticket but if your team could come up with a better idea without compromising the security for other users you can reopen it.

Thanks for everything. I learned quite a lot of things in this journey though 😄.