### Weblog Tomasza Przechlewskiego

random image
Archiwum
O stronie
wykorzystywany jest blosxom plus nastêpuj±ce wtyczki: tagging, flatarchives, rss10, lastbuilddatexhtmlmime. Niektóre musia³em dopasowaæ nieco do swoich potrzeb. Wiêcej o blosxom jest tutaj
Subskrypcja
Perl encoding problem

SW asked me to augment a Perl script that originally processes ISO-8859-2 encoded text (TeX) files only by adding UTF-8 and CP1250 (one byte MS Windows encoding for Central Europe) encodings as well.

I made up it as follows (not sure if correct):

use Getopt::Long;
my $coding = 'utf8'; my$showhelp= '' ;
GetOptions( "coding=s"  => \$coding, "help|\?" => \$showhelp,) ;
if ( $showhelp ) { print "***$0 [-coding=[cp1250|iso88592|utf-8]] file1 file2...\n" ;
exit 1; }

if ( $coding =~ /cp1250/ ) {$coding='cp1250'; use open ':encoding(cp1250)'; }
elsif ( $coding =~ /iso8859\-?2/ ) {$coding='iso-8859-2'; use open ':encoding(iso-8859-2)'; }
elsif ( $coding =~ /utf\-?8/ ) {$coding='UTF-8'; use open ':encoding(UTF-8)'; }
else { die "*** Unknown coding: $coding\n"; exit 1; } print STDERR "*** Coding:$coding\n";
## rest of the script omitted ....


I reencoded the script from original ISO-8859-2 to UTF-8 as well with iconv, so all strings are UTF-8 encoded now.

url | Thu, 20/12/2012 20:30 | tagi: , ,
GDP per capita in purchasing power standards of EU member states

While GDP per capita is mainly an indicator reflecting the level of economic activity, Actual Individual Consumption (AIC) per capita is an alternative indicator better adapted to describe the material welfare situation of households.

GDP and AIC per capita in PPS, EU27 = 100

+-------------------------------------------------+
GDP per capita    AIC per capita
2009 2010 2011    2009 2010 2011
+-------------------------------------------------+
EU27             100   100  100     100  100  100
Euro area (EA17) 109   108  108     107  107  107
+-------------------------------------------------+
Luxembourg       255   267  271     144  141  140
Netherlands      132   131  131     118  114  113
Ireland          130   129  129     103  103  101
Austria          125   127  129     116  118  119
Sweden           120   124  127     116  114  116
Denmark          123   128  125     116  116  113
Germany          115   119  121     115  117  120
Belgium          118   119  119     109  111  111
Finland          114   113  114     110  111  112
United Kingdom   111   111  109     121  120  118
France           109   108  108     113  113  113
Italy            104   101  100     103  102  101
Spain            103    99   98      96   95   94
Cyprus           100    97   94     102   99   98
Malta             83    85   85      85   83   84
Slovenia          87    84   84      81   80   81
Czech Republic    83    80   80      73   71   71
Greece            94    87   79     104   97   91
Portugal          80    80   77      83   84   81
Slovakia          73    73   73      72   71   70
Estonia           63    63   67      58   56   58
Lithuania*        55    57   66      63   61   70
Hungary           65    65   66      62   60   61
Poland            61    63   64      64   67   69
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Latvia            54    54   58      52   53   57
Romania           47    47   49      46   46   47
Bulgaria          44    44   46      43   43   45
+-------------------------------------------------+
Norway           177   181  186     134  136  135
Switzerland      150   154  157     128  129  130
Iceland          120   112  111     111  106  107
Croatia           62    59   61      58   57   59
Turkey            46    50   52      51   54   57
Montenegro        41    42   42      50   52   53
FYR Macedonia     36    36   35      41   40   40
Serbia            36    35   35      44   44   43
Albania**         28    27   30      32   30   34
BosniaHerzegovina 31    30   30      37   36   36
+-------------------------------------------------+


* 2011 population figures adjusted on the basis of the 2011 Census. Therefore the per capita indices for 2011 are not entirely comparable with previous years due to this break in time series.

** Figures for all years based on Eurostat estimate of GDP

The euro area (EA17) consists of Belgium, Germany, Estonia, Ireland, Greece, Spain, France, Italy, Cyprus, Luxembourg, Malta, the Netherlands, Austria, Portugal, Slovenia, Slovakia and Finland.

url | Mon, 17/12/2012 09:52 | tagi: , , , ,
Grim prospects of EU economy

Source: Eurostat
Click on image to enlarge

The collapse of production during April-2008--April-2009 period (left figure) corresponds to a significant rise in unemployment (right figure), however in April 2009, when production was at the lowest level, unemployment rate was approx 9% while currently is is approx 12%.

url | Fri, 14/12/2012 09:34 | tagi: , , , ,
Raspberry Pi/DHT-22 sensor: registering temperature and humidity

Fig. #1: DHT-22 sensor

Fig. #2: Testing DHT-22

Fig. #3: Wiring

To build the installation one has to buy:

DHT-22 temperature/humidity sensor (8 USD per sensor). Not cheap...

0,25W 10K OHM carbon resistor (very cheap).

female connectors (pol. kable po³±czeniowe ¿eñskie), telephone cables or similar four core cable (pol. kabel cztero¿y³owy), terminal block (pol. kostka po³±czeniowa) and heat shrink tubing (pol. rurka termokurczliwa) to insulate and strengthen connections. The recommended way is to use a breadboard (pol. p³ytka stykowa/prototypowa) as described in learn.adafruit.com. My interest in electronics is limited, I've never used breadboards etc... I had some spare cables and terminal blocks so I designed it that way (cf. pictures).

NOTE: The cheaper version of the DHT-22 is a DHT-11 (aka SHT-11). Tempted by the lower price I bought two DHT-11 sensors but I do not recommend it. First of all, the temperature is measured in the range of 0 °C to 50 °C (with poor accuracy of +/- 2 °C) so is not suitable for outdoor (at least in Europe). Second, the humidity seems to be understated. Third, it does not work when DQ line is connected to other GPIO pins than pin #24 (maybe it's a software problem). For comparison, DHT-22 measures the temperature in the range of-40C to +80 C with an accuracy of +/- 0.5 °C.

I follow the tutorial available at learn.adafruit.com but some details were modified.

#### Hardware

There are four pins in DHT-22 (see Figure # 1). I connected data line (DQ) of each sensor to pins P22, P24 and P25 respectively (each sensor must have a separate data line). Vdd pin of each sensor to P1 (3.3 V supply). GND (ground) pin of each sensor to P6. In addition, each DQ was connected via the resistor with the power line Vdd.

Pin Null is not used.

The sensors were connected to GPIO pins via terminal blocks, cables and some soldering.

#### Software

pi@raspberrystar $wget http://www.open.com.au/mikem/bcm2835/bcm2835-1.8.tar.gz pi@raspberrystar$ tar -zxvf bcm2835-1.8.tar.gz
pi@raspberrystar $cd bcm2835-1.8 pi@raspberrystar$ ./configure && make && sudo make install


then the application retrieving the data from the sensors has to be installed:

pi@raspberrystar $git clone https://github.com/adafruit/Adafruit-Raspberry-Pi-Python-Code.git pi@raspberrystar$ cd Adafruit-Raspberry-Pi-Python-Code
pi@raspberrystar $cd Adafruit_DHT_Driver  One has to modify Makefile file, namely add -l rt at the end of the line that starts with CFLAGS: CFLAGS = -std=c99 -I. -lbcm2835 -l rt  now: ## in Adafruit_DHT_Driver directory pi@raspberrystar$ make


If everyting works, then:

# Run ./Adafruit_DHT sensor-type DQ-pin-number
pi@raspberrystar $sudo ./Adafruit_DHT 22 25 Using pin #25 Data (40): 0x3 0xe7 0x0 0x17 0x1 Temp = 2.3 *C, Hum = 99.9 %  The directory Adafruit_DHT_Driver contains also Adafruit_DHT_googledocs.ex.py Python script which can upload sensor readings directly to google.docs spreadsheet. To run Adafruit_DHT_googledocs.ex.py one has to install gspread module first: pi@raspberrystar$ wget http://pypi.python.org/packages/source/g/gspread/gspread-0.0.13.tar.gz
pi@raspberrystar $tar -zxvf gspread-0.0.13.tar.gz pi@raspberrystar$ cd gspread
pi@raspberrystar $sudo python setup.py install  Adafruit_DHT_googledocs.ex.py script: 1) in an infinite loop runs every 30 seconds the program Adafruit_DHT, 2) retrieves temperature/humidity, 3) sends temperature/humidity readings to google.docs. A fragment of the script looks like: while(True): output = subprocess.check_output(["./Adafruit_DHT", "2302", "4"]); print output # search for humidity printout matches = re.search("Hum =\s+([0-9.]+)", output) if (not matches): time.sleep(3) continue humidity = float(matches.group(1)) ## omitted code ... time.sleep(30)  Because I want to process somehow the data (not only to retrieve and upload to google.docs) I modify Adafruit_DHT_googledocs.ex.py script. My version Adafruit_DHT_googledocs.ex.py is limited to sending to google.docs values passed as arguments to the call: temp = float(sys.argv[1]) humidity = float(sys.argv[2]) ## omitted code ...  The following bash script takes care of the rest: #!/bin/bash # LOG_DIR=/home/pi/Logs/DHT BIN_DIR=/home/pi/bin SENSTYPE=22 SLEEP_TIME=5 function ReadSensor() { local sensorType="$1"
local sensorId="$2" local WYNIK="" local SUCCESS="" ## 5 tries with 5s sleep between them for i in 1 2 3 4 5; do WYNIK=sudo$BIN_DIR/Adafruit_DHT $sensorType$sensorId | tr '\n' ' '
SUCCESS=echo $WYNIK | awk ' { if (NF > 10) {print "YES"} else { print "NO"}}' if [ "$SUCCESS" = "YES" ] ; then
echo "$sensorId=$i $WYNIK" >>$LOG_DIR/DHT22.log
DHT_CURR_TEMP=echo $WYNIK | awk '{print$13}'
DHT_CURR_HUM=echo $WYNIK | awk '{print$17}'
break
fi
sleep $SLEEP_TIME; done ## All attempts to read sensors were unsuccessful if [$SUCCESS = "NO" ] ; then
echo "$sensorId=?$WYNIK" >> $LOG_DIR/DHT22.log DHT_CURR_TEMP="999.9" DHT_CURR_HUM="999.9" fi } echo "@date "+%Y%m%d%H%M%S"" >>$LOG_DIR/DHT22.log

## A sensor in the room:
ReadSensor $SENSTYPE "24" READINGS="$DHT_CURR_TEMP $DHT_CURR_HUM" sleep 12 ## Outdoor sensor: ReadSensor$SENSTYPE "25"
READINGS="$READINGS$DHT_CURR_TEMP $DHT_CURR_HUM" sleep 12 ## A sensor in the porch: ReadSensor$SENSTYPE "22"
READINGS="$READINGS$DHT_CURR_TEMP $DHT_CURR_HUM" ## HTML + chart /usr/bin/perl /home/pi/bin/dht2ht.pl > /var/www/stats/DHT22.html # Upload to google /home/pi/bin/DHT_googledocs.ex.py$READINGS


As in the case of 1-Wire bus there are problems with the reading of the sensor. That's why the function ReadSensor is trying to read the sensor several times. Maximum number of failed attempts, we have observed during several days of operation is 3.

The script runs every 30 minutes from cron:

1,31 * * * * /home/pi/bin/dht2ht.sh


LOG file looks something like this:

@20121113230101
24=1 Using pin #24 Data (40): 0x2 0x22 0x0 0xc9 0xed Temp =  20.1 *C, Hum = 54.6 %
25=1 Using pin #25 Data (40): 0x3 0xe7 0x0 0x1c 0x6 Temp =  2.8 *C, Hum = 99.9 %
22=4 Using pin #22 Data (40): 0x2 0x73 0x0 0xb0 0x25 Temp =  17.6 *C, Hum = 62.7 %


Row starting with the @ contains the date and time (@ is added for subsequnt easy parsing). Lines that begin with nn = m contain the data retrived from the sensor (nn is the sensor number, m denotes the number of successful attempt or ? in case when all attempts were unsuccessful)

Note: I noticed that higher system load (including intensive I/O operations) cause problems to retrieve data from the sensors. I tried to run motion detection application (motion) configured to use as little system resources as possible with no success. Rapberry overclocked to 900 Mhz performs significantly better but still only about 20% tries returns some data. Exact nature of the problem is a mystery to me as for example top indicates that still more there 80% of CPU is free.

Other question to consider is: whether the readings are correct during high humidity? My outdoor sensors tend to indicate 99% humidity pretty frequently which seems suspicious. I have compared data obtained from 3 different sensors (namely WH 2080 clone, Oregon Scientific's RMS300 and DHT-22) and some differ significantly.

#### Conversion to HTML and generating charts with dht2ht.pl

Perl script dht2ht.pl creates a HTML table and charts showing temperature/humidity readings as well as dew point, calculated with the following approximation formula: $$D_p = (237.7 \cdot \gamma(T, H) ) / (17.271 - \gamma(T, H) )$$

where: $$\gamma(T, H) = 17.271 \cdot T / (237.7 + T) + \log (H / 100.0)$$

Script outcome is available here. All scripts and other stuff discussed in this blog post are available here.

Google.docs sheet containing readings from all my 3 sensors is available here. (Note: for some important reasons Adafruit_DHT_googledocs.ex.py script started adding data from the 162th line of the spreadsheet.)

url | Sat, 08/12/2012 13:39 | tagi: , , , , , ,