Why doesn't WGET work with Cosmote or Wind GR'

[DaveRo]

A rather technical question. On my blog about using Greek phones for weather forecasts
Greek Internet Weather Yacht Vigdis
I explain how I try to reduce the data volume when using Poseidon's site.

I wrote a script that uses WGET to download 2 day's weather maps at 6 hour intervals. It works fine over a wifi connection but not over Cosmote or Wind WAP proxied connections.

So for example this doesn't work:
wget http://www.poseidon.hcmr.gr/images/m...db09082515.png

wheras if I type that address into the browser it does. But I then have to save each image - which is tedious.

On Cosmote myview'n'web the wget hangs.
On Wind Plus NS I get a temporary resolution errror.

Any ideas or suggestions? It must be the proxy. (Spoofing the UAS does nothing.)

Dave

[inquisitor]

I understand, that you run the wget command locally (i.e. on your linux machine, which is connected to the internet via the named providers).
Proceeding on this assumption the problem appears pretty simple. http-applications do always send the so-called USER_AGENT parameter revealing it's identity with each http-request.
When you use wget the http requests sent to the server (and passing the network operator's proxy) will contain the USER_AGENT "wget 1.9.2". The proxy will so identify the request as not coming from a handset and so will block it.
What you need to do now, is faking the USER_AGENT. Therefore simply set the user_agent parameter to some mobile phone when executing wget. In the following example you will send the (pretty long) USER-AGENT of a Nokia N95 with your request:

wget --user-agent:"Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaN95/11.0.026; Profile MIDP-2.0 Configuration/CLDC-1.1) AppleWebKit/413 (KHTML, like Gecko) Safari/413" http://www.poseidon.hcmr.gr/images/m...db09082515.png

That should do the job.

Depending on which user_agents are blocked by the corresponding operator you could also use the user agent of a desktop browser like Firefox, which would look like this:

wget --user-agent:"Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)" http://www.poseidon.hcmr.gr/images/m...db09082515.png

For details on wget parameters, check the documentation: http://www.gnu.org/software/wget/manual/wget.html

[DaveRo]

Thanks for that comprehensive reply, but I already tried that. I just tried again with Cosmote and your suggested User Agent Strings (UAS) and it definitely makes no difference.

Is it likely that these Wap services will sniff for browsers? I've never seen evidence of that.

There are dozens of params for WGET - I have the manpage; any other suggestions?

Edit: I'm using a Nokia N810 tablet tethered to a Nokia N70 phone.

Dave

[inquisitor]

Have you checked, if faking the user_agent does actually take effect? I just found another syntax for changing the user_agent on Google:

$ wget -U "user agent" http://example.com/

You should first make sure that the user_agent manipulation works for sure. You should download http://pgl.yoyo.org/http/browser-headers.php while connected through wifi and check if the user_agent has actually been changed.

If the user_agent is actually changed and it still doesn't work through the cell network, they seem to perform some deeper inspection of your requests. Maybe the proxy checks in the EIR (a network element of every GSM-network) what type of phone you use and then only accepts the user_agent of that certain phone. Then you should use only the user_agent of the phone, which is registered on the network (the Nokia N70 in your case):

NokiaN70-1/5.0609.2.0.1 Series60/2.8 Profile/MIDP-2.0 Configuration/CLDC-1.1

If that doesn't work either, you should check if you can download that image when you use the native browser of your N70 and/or of the N810. That could help isolating the cause.

[DaveRo]

Yes, I usually use -U; I tried the longer version yesterday.

I'll try checking if the UAS is actually changed next time I find a wifi - may be a week or two. Maybe this version of WGET, compiled for Maemo, is faulty.

I just tried -S: no server response returned (Cosmote).

[DaveRo]

I solved it. I need to set the http_proxy environment variable. I assumed that the connection manager would specify the proxy for WGET since the browser picks it up.