Minor features

[BashSer]

Good day @theyosh !

You did great job ! 👍 🥇

Is it possible to add SOCKS5 or MTP proxy to telegram options ?
In several countries government blocks telegram and peoples use different ways for bypass this.

In time where was "memory leak" (3.4.2 ?) i need reboot RPi every three-five days, may be add "watchdog/timer" where RPi reboot automatically once in week for example ?

Thank you !

P.S. In last setup on RPi zero W:

1. First setup(run install.sh) - several errors with python, error with rc.local, TerrariumPi not started, "screen -r" does not work.
2. Second setup(run install.sh in a few minutes) on same board and same OS - errors on screenshot below, errors with rc.locals disappeared, TerrariumPi started, "screen -r" works.
3. After second re-install - It seems like it works, but i did not testing.
   

[theyosh]

Yes, a socks5 proxy can be added. Will be done in a few days.

The installation is strange. So that will need some extra testing. And therefore I need to buy a Pi Zero W. Because on a Pi3, the installation is looking fine. But I will look into it probably next week.

Euh, the idea is that the Pi does not need to be rebooted at all. The memory leak is fixed and I can just run it for weeks without issues. So that should not be needed. Else I have to figure this out. But I am thinking that this should not be needed anymore.

[theyosh]

Proxy support added. To enable it follow the steps below:

1. On the pi run: sudo pip install requests[socks]
2. In the folder TerrariumPI: git pull
3. Restart software
4. Clear browser cache and there is an extra option for proxy server.

Valid proxy servers are:

• http://user:pass@server:port
• https://user:pass@server:port
• socks5://user:pass@server:port

Where user and pass is optional. More information: http://docs.python-requests.org/en/latest/user/advanced/#proxies

[BashSer]

2018-06-24 12:35:42,000 - INFO    - terrariumWrapper     - Starting TerrariumPI server at location: http://192.168.43.105:8090 ...

('Connection aborted.', BadStatusLine("''",))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-24 12:37:09,715 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-06-24 12:37:09,718 - INFO    - terrariumNotificatio - Stopping TelegramBot. This can take up to 76 seconds...
2018-06-24 12:37:10,223 - INFO    - terrariumNotificatio - TelegramBot is stopped
2018-06-24 12:37:25,000 - INFO    - terrariumWebserver   - Running webserver at :::8090

In web-interface field "Proxy" after save and refresh: [object Object] or empty.
I type in this field: socks5://telegram:[email protected]:1080

UPD:
After reboot software:

SOCKSHTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot521739490:AAG8Ivy5M5pDEXsJI3zSY0qXCGoHhZvdbDo/getUpdates?timeout=120 (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0xb5b560d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-24 12:47:40,802 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-06-24 12:47:40,806 - INFO    - terrariumNotificatio - Stopping TelegramBot. This can take up to 70 seconds...
2018-06-24 12:47:46,000 - INFO    - terrariumWebserver   - Running webserver at :::8090
2018-06-24 12:47:46,845 - INFO    - terrariumNotificatio - TelegramBot is stopped

UPD2:
Mail notifications not work and web-server loads after all "notifications" runs.
In this case i will wait 6 minutes before all notifications refused and web-server starts:

2018-06-24 13:31:26,000 - INFO    - terrariumWrapper     - Restarting in 10 seconds after running for 258 seconds. Press Ctrl+C now to terminate TerrariumPI.
10 9 8 7 6 5 4 3 2 1 restart!
--->2018-06-24 13:31:37,000 - INFO    - terrariumWrapper     - Starting TerrariumPI server at location: http://192.168.43.105:8090 ...
SOCKSHTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot521739490:AAG8Ivy5M5pDEXsJI3zSY0qXCGoHhZvdbDo/getUpdates?timeout=120 (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0xb5bbb050>: Failed to establish a new connection: [Errno 110] Connection timed out',))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-24 13:36:33,429 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-06-24 13:36:33,430 - INFO    - terrariumNotificatio - Stopping TelegramBot. This can take up to -157 seconds...
2018-06-24 13:36:33,934 - INFO    - terrariumNotificatio - TelegramBot is stopped
Connection unexpectedly closed: timed out
(503, '5.5.4 Error: send AUTH command first.', '[email protected]')
--->2018-06-24 13:37:02,000 - INFO    - terrariumWebserver   - Running webserver at :::8090
2018-06-24 13:37:19,535 - WARNING - terrariumTranslations - No translation available for 'notification_telegram_proxy'

[theyosh]

Hmm, for me socks5 does not work either. But I can also use https for it. So I changed socks5:// just to https:// and it does work for me. Used my own proxy. If that does not help you, I am not sure if it will work. Because I use simple HTTPS connections to Telegram servers.

Unfortunately the Telegram Bot does lock the system during startup. But that should not take much time. If it does, there is a connections problem.

Als the output of the telegram proxy bot socks on the notification page is also fixed.

Also changed the installer script so that it will keep the output clean. That it will not work in 1 run, could be caused by network problems. I have run this script 20+ times now, and always it does the job the first time.

Do a git pull, restart and clear browser cache.

[theyosh]

Mail notification do not work due to missing authentication (username and password). Also I guess [email protected] is not valid? That should be a valid mail address

[BashSer]

Ok, now i use socks5://telegram:[email protected]:1080(wonderful, if "blah-blah" is unavaliable then random change to asdf_random_letters.tgvpnproxy... :D)
https://... don't work.
As result in screen -r:

10 9 8 7 6 5 4 3 2 1 restart!
2018-06-29 01:34:16,000 - INFO    - terrariumWrapper     - Starting TerrariumPI server at location: http://192.168.43.105:8090 ...
SOCKSHTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot521739490:AAGgKNR2X-nZsWiajchQ4kFCvusOobkZwTM/sendMessage?text=Hi+UserName%2C+you+are+now+getting+messages+from+TerrariumPI.+I+do+not+accept+commands.&chat_id=000000000 (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0xabe0bb30>: Failed to establish a new connection: [Errno 110] Connection timed out',))
2018-06-29 01:39:10,294 - ERROR   - terrariumWeather     - Error getting online data from yr.no
Traceback (most recent call last):
  File "/home/pi/TerrariumPI/terrariumWeather.py", line 86, in load_data
    xmldata = untangle.parse(self.source_url.strip('/') + '/forecast_hour_by_hour.xml')
  File "/usr/local/lib/python2.7/dist-packages/untangle.py", line 177, in parse
    parser.parse(filename)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 105, in parse
    source = saxutils.prepare_input_source(source)
  File "/usr/lib/python2.7/xml/sax/saxutils.py", line 349, in prepare_input_source
    f = urllib.urlopen(source.getSystemId())
  File "/usr/lib/python2.7/urllib.py", line 87, in urlopen
    return opener.open(url)
  File "/usr/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.7/urllib.py", line 443, in open_https
    h.endheaders(data)
  File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 1263, in connect
    server_hostname=server_hostname)
  File "/usr/local/lib/python2.7/dist-packages/gevent/_sslgte279.py", line 68, in wrap_socket
    _context=self)
  File "/usr/local/lib/python2.7/dist-packages/gevent/_sslgte279.py", line 270, in __init__
    raise x
IOError: [Errno socket error] [Errno 0] Error
('Connection aborted.', BadStatusLine("''",))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-29 01:41:59,529 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-06-29 01:42:01,000 - INFO    - terrariumWebserver   - Running webserver at :::8090
('Connection aborted.', BadStatusLine("''",))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-29 01:42:36,289 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
('Connection aborted.', BadStatusLine("''",))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-29 01:43:13,013 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
('Connection aborted.', BadStatusLine("''",))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-29 01:43:49,691 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
('Connection aborted.', BadStatusLine("''",))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-29 01:44:26,496 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
('Connection aborted.', BadStatusLine("''",))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-29 01:45:03,366 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data

And in telegram:

But web-interface very long loading or unavaliable and i restart TerrariumPi:

    self.set_state(terrariumSwitch.OFF)
  File "/home/pi/TerrariumPI/terrariumSwitch.py", line 285, in set_state
    self.callback(data)
  File "/home/pi/TerrariumPI/terrariumEngine.py", line 618, in toggle_switch
    self.notification.message('switch_toggle_' + ('off' if data['state'] == 0 else 'on'),data)
  File "/home/pi/TerrariumPI/terrariumNotification.py", line 629, in message
    self.send_telegram(title,message)
  File "/home/pi/TerrariumPI/terrariumNotification.py", line 589, in send_telegram
    self.telegram.send_message(message)
  File "/home/pi/TerrariumPI/terrariumNotification.py", line 155, in send_message
    self.__get_url(url)
  File "/home/pi/TerrariumPI/terrariumNotification.py", line 101, in __get_url
    response = requests.get(url,proxies=self.__proxy)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/socks.py", line 88, in _new_conn
    **extra_kw
  File "/usr/local/lib/python2.7/dist-packages/socks.py", line 252, in create_connection
    sock.connect((remote_host, remote_port))
  File "/usr/local/lib/python2.7/dist-packages/socks.py", line 100, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/socks.py", line 832, in connect
    super(socksocket, self).connect(proxy_addr)
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
KeyboardInterrupt
2018-06-28T22:48:18Z {'REMOTE_PORT': '63887', 'HTTP_HOST': '192.168.43.105:8090', 'REMOTE_ADDR': '::ffff:192.168.43.145', (hidden keys: 32)} failed with KeyboardInterrupt

2018-06-29 01:48:18,174 - INFO    - terrariumNotificatio - Stopping TelegramBot. This can take up to -106 seconds...
2018-06-29 01:48:18,692 - INFO    - terrariumNotificatio - TelegramBot is stopped
2018-06-29 01:48:19,000 - INFO    - terrariumWrapper     - Restarting in 10 seconds after running for 843 seconds. Press Ctrl+C now to terminate TerrariumPI.
10 9 8 7 6 5 4 3 2 1 restart!
2018-06-29 01:48:30,000 - INFO    - terrariumWrapper     - Starting TerrariumPI server at location: http://192.168.43.105:8090 ...
SOCKSHTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot521739490:AAGgKNR2X-nZsWiajchQ4kFCvusOobkZwTM/getUpdates?timeout=120 (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0xb5bdd0d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
No JSON object could be decoded
{'description': 'Did not receive valid JSON data'}
2018-06-29 01:53:25,271 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-06-29 01:53:34,000 - INFO    - terrariumWebserver   - Running webserver at :::8090

And telegram again don't work and i dont know how i can help you =(
How about sent photo from camera on user request and "Emeregency power off" ?

Mail:
I solved problem with FAQ https://yandex.com/support/passport/authorization/app-passwords.html and all works :)

P.S. Valid only proxy and token, because it's "public" proxy from gigantic pool and uses only for telegram many countries, and token is test bot where i every day revoke them :)

[theyosh]

Hmm, this is getting somewhat complicated. I have changed some code in the development branch to tackle this.

I do get the feeling that when you use Telegram proxy, also the weather data is trying to use the same proxy, what does not work and not what should happen. So I have changed the code so that every request has explicit proxy settings. Either for Telegram the socks/https proxy, and all other requests none. This way I hope to make sure that the proxy settings is not affecting other parts of the software.

When the Telegram try to connect it blocks the rest of the software. This cannot be changed. That means when the proxy does not work, it will hold everything. This is now tried three times, and then it stops and continue with the other parts of the software. This is only thing I can do right now.

So, make sure your proxy is working, else it will lock up the software.

[BashSer]

Nice improvements !
I do:

git checkout development
git pull
shutdown -r now

Zero W was loaded faster, but telegram bot does not worked :(

10 9 8 7 6 5 4 3 2 1 restart!
2018-07-01 01:39:43,000 - INFO    - terrariumWrapper     - Starting TerrariumPI server at location: http://192.168.43.105:8090 ...
Connection unexpectedly closed: timed out
2018-07-01 01:40:33,000 - INFO    - terrariumWebserver   - Running webserver at :::8090
('Connection aborted.', BadStatusLine("''",))
2018-07-01 01:40:37,480 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
('Connection aborted.', BadStatusLine("''",))
2018-07-01 01:41:14,673 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-07-01 01:41:20,186 - INFO    - terrariumNotificatio - TelegramBot is stopped

Now i use socks5://telegram:[email protected]:1080 and this proxy work on my desktop and pda on different connections.

[theyosh]

Hmmm, I just did a test using socks5h://telegram:[email protected]:1080 and seems to work. So add a 'h' to it. This changes the DNS behavior and I think will solve it for you.

[theyosh]

Does the change to socks5h change the behavior?

[BashSer]

Sorry, i cant check this in 5 days :( You can close this issue and if they need i reopen issue :)

[theyosh]

Ah that is ok.... I will leave it open then...

[BashSer]

Nope :(

2018-07-07 03:33:00,000 - INFO    - terrariumWrapper     - Starting TerrariumPI server at location: http://192.168.43.105:8090 ...
2018-07-07 03:33:46,491 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-07-07 03:33:56,020 - ERROR  - terrariumNotificatio - TelegramBot has issues: Did not receive valid JSON data
2018-07-07 03:34:01,531 - INFO    - terrariumNotificatio - TelegramBot is stopped
Connection unexpectedly closed: timed out
2018-07-07 03:34:12,000 - INFO    - terrariumWebserver   - Running webserver at :::8090

Use socks5h://telegram:[email protected]:1080

[theyosh]

Then I am currently out of options. But in #161 (comment) it shows messages trough Telegram. How did you fix that?

Could it be that a max amount of connections per IP is used at the VPN?

[BashSer]

Then I am currently out of options. But in #161 (comment) it shows messages trough Telegram. How did you fix that?

I dont know 😞
This was done once and dont repeat.

Could it be that a max amount of connections per IP is used at the VPN?

Nope, this gigantic pool and last data they has over 15 millions connections from several countries.

[theyosh]

Could it be that a max amount of connections per IP is used at the VPN?

Nope, this gigantic pool and last data they has over 15 millions connections from several countries.

I meant max devices per home IP. If you use it on your pc, mobile, etc.. could it be that you are limited based on your home ip?

[BashSer]

No, its unlimited connections.
I use this through phone AP. On phone telegram works, but on RPi W zero dont work. And i try through wi-fi from another ISP.

[theyosh]

I have trying to get this to work for a couple of hours. But socks proxies does not want to work. I have no clue :(

[theyosh]

Can't leave it to rest... So I was looking at: https://www.sslproxies.org/

And used a server in CZ. And that seems to work for me: https://185.142.208.197:41258

Not sure how long those proxy will be running. But a generic HTTP/HTTPS proxy could also help in stead of a socks5. Your Pi and Mobile/Desktop does not need to use the same proxy server. As long all devices reach telegram....

[theyosh]

@BashSer I am closing this now, as the proxy option is hard to fix and test with socks proxies. And I think this is not needed for the mass.

I do understand your problem with Telegram and Russia. But then I would like to say try to find some HTTP(S) based proxies.